I think in America we have this idea that you're born with certain abilities and if you aren't immediately good at something, you shouldn't pursue it. That is true to an extent, but practice and studying are important in both math and the hard sciences. In addition, kids need more exposure to lab work in grade school so that they will feel comfortable in a science lab once they reach college.
Another reason that students might decide not to pursue a career in science is lack of funding. I considered a career as a molecular biologist, but heard that competition for funding was fierce. And after getting a Ph.d. plus post-doctoral experience, you would be in your late 20's or early 30's before starting your career in science. That may discourage many people, as well.