Adding unique key constraints in custom tables
I don't believe there's a way to specify that a certain column in a custom table should be unique, beyond writing a custom insert event that tries to detect other items with the same field.
This would ideally be implemented using the SQL unique key constraint properties, and in the custom table Fields UI it could simply show as another "Validation Rule" dropdown option, or as a checkbox similar to "Required".
Idan Avrahami commented
+1 and +1 on multi-field unique constraints
Allowing multi-field unique constraints would be great too.