Their fans actually bother me less than the fans of most big clubs.
Basically, being very much an establishment club, they remind me too much of Rangers. I could say the same of Real Madrid and their position in Spain.
But I also can't stand clubs that make too big a show of being anti-establishment and are always adopting a 'holier than thou' approach to dealing with the establishment clubs, even though they are frequently very influential and very unpleasant clubs in their own right. I'm thinking of the likes of Celtic, Barcelona, Roma, Fiorentina for that category.