Better off going casual anyway.
I've had a similar experience to your one on the train when going to Hampden for Aberdeen v Celtic.
Celtic were getting mouthy and thought they were in the majority because there weren't any Aberdeen tops or scarves on show in the carriage.
They didn't realise that there were about thirty Aberdeen casuals surrounding them.