Tough question - we held back one of our children because the teachers thought he was 'immature' in some ways but as it turns out, he was quite brilliant. It could be that he is either not stimulated enough by what is being taught (or there is a problem with the teacher/student personality mix) - or he may be further ahead than what is being taught.
I would suggest trying to test out reading and writing at home and see if you get the same response - if he is worried about reading or writing in the home setting, there could be a learning 'variability' (I like that better than disability).
We also found that our daughter in kindergarten of all things had a total aversion to one particular teacher's 'style' - it was just a fluke. I tended to think my daughter was being 'immature' but then when the school actually fired her and a bunch of other parents came forward to express their problems, it turned out it wasn't my daughter at all!
As you can see, there are many reasons why a child has a hard time in school - there are many more than I mentioned. Just some things to look at in considering what is best for your boy....good luck! Parenting is a full time job!