I think it's stupid not to celebrate. Just enjoy the goal. The type of celebration might matter of course. A celebration like Adebayor's (which was insanely funny and I'll always love him for it) is something you don't forget if you're on the receiving end. I can understand not wanting a player back after that.
