They shouldn't, if only those who've watched the game vote then it could be more interesting.
But it's, of course, down to any member's' conscious not to vote when he/she has not seen the match.
It's a fun enough thing to try out, but it won't be fun if it doesn't work.