Rugby is a middle class sport in England? I had no idea.
Rugby union is. Tends to be played and followed by middle class people from central and southern England, with some exceptions. The phrase "a sport for thugs, played by gentlemen" gives some insight to union's background.
Rugby league on the other hand is very working class, has a smaller following, mostly people from northern England (Lancashire and Yorkshire) follow and play it.
In Wales union is the national sport so has no class divide. Scotland is more in line with England, however. This graphic is fairly accurate on whether union is seen as a "posh" sport or not, or whether it has other biases:
With England doing well at the Euros you see more people who would ordinarily be interested in classic "middle class" sports like rugby or tennis, or even cricket, following the football team, right on the bandwagon.