Well, what's your chest size?
I'd agree, typically, unless you have 32 Ds or something.
But the reason a Men's small fits you "ladylike" is because men typically have wider frames through the shoulders than women, so the shirts are wider. Wider shirt is a parallel to depth if width isn't there as it will pull the shoulders in and push the chest out.
Also, athletic shirts tend to stretch well.