Perhaps you could take some time off from writing (which you seem to do a lot of -- good for you!) and spend it responding to comments. Once you've caught up, you can go back to writing. When your comments start to pile up again, take some more time off to respond to them.
You could also set aside a day or two each week, or so many hours per week, to devote to your comments.
Hope this helps.