The problem in part, is environment, and nurture for that matter.
Even though you grew up with little, it sounds like your parents instilled in you a great work ethic and a drive to improve your condition, to learn, to grow, to value education, and so on...
That means you had fantastic parents. Or so goes my opinion.
Many people in poverty have parents who devalue education, who don't support their children learning, growing, expanding their horizons, so to speak... or at least just don't give a shit. It's also a societal-cultural thing in the poor classes in North America of late, to place very little emphasis on education. It's part of the reason the inner city public school system has fallen into decay and ruin in America. Sure, there are exceptions, and some people manage to overcome the odds stacked heavily against them, but that's asking a lot of a child, when the environment they are raised in is so opposed to any kind of opportunity for success.