Of course it shouldn't be an issue but it is. He is the president, public image is very important even when you aren't possibly the most controversial (and prone to the media jumping on you for everything) president in American history.
It does call his judgement into question, what he was going for here, which is certainly a concern.