no it's not racist, even the "french" refer to themselves as "de souche", france's insistence on assimilation of immigrants but still treating them as second class citizen is a joke just like its economy, its industry, its war prowess etc...
I don't understand all the jersey obsession in the first place. Unless going on a game (probably not every game as well) or playing football myself, I'd definitely never wear one. What, you go out and wear a jersey?
I could go with one to a beach on a bike from time to time or just walking around home sometimes. Definetly not something to casualy wear to go out to town.