To answer with a strict "age" and some personal experience, my stay-at-home mom worked with me on basic math and reading when I was 2. When I was in preschool, I was the only student in my class able to read. I then went on to our elementary school's "Gifted and Talented" program, and was always in advanced classes of reading and math. I skipped 1st grade because most of the basic skills they would start working on, I was already competent in.
With all of those factors in mind, my answer from personal experience is "as soon as possible!". Not everyone has the luxury to be a stay-at-home parent, but any opportunity you get, try to work with children on learning skills. It will give them a leg up, and can get them interested in different topics.