If that's the case, couldn't you say that about everything we do? I hold the door for somebody behind me, so that must mean the deed was done because I want to be recognized with a "thank you", thus feeding my ego.
With this line of thinking, wouldn't the bigger test be allowing the females to wear whatever they want, thus causing the males to have even more restraint, and vice-versa?
This is where the thinking of Muslims is rightfully destroyed. It's utterly hypocritical.