nobody of us can really tell you what's right because we dont understand the situations half as well as you do.
anyway,
if you chose:
1. to ignore him: NEVER GIVE IN IF HE KEEPS BOTHERING YOU.
2 to tell him in his face: Be cautious od his reaction and have friends with you in the day...