I heard a story that there was a cricket match played in Germany in which Hitler was invited. After 5 days, the game ended in a draw. Hitler was furious and forbade any German to play cricket.
I'm not sure if this game actually happened or is a myth, but cricket had been getting a foothold in Germany before the Nazis, and they discouraged it to the extent where it fell off the sporting radar for decades.
In terms of ideals I don't think first class cricket with its long duration, focus on form, style and gentlemanly play really sat well with those of the Nazis. To me it is the most stereotypically English game which was then adapted and improved once it made its way to the then-colonies.
Personally I don't think you'll ever see a Test nation that wasn't once part of the Commonwealth, but for limited overs I don't see why it couldn't be more popular.