Its impossible. Color is not just an indication of skin pigmentation, but often also of social status, culture, religion, and other factors, all of which affect how you view them.
I dont want to be colorblind though. Who wants to lose their ability to enjoy colors?