Don't know how you are doing it, but my husband picked up a simple tool (reasonable cost) , an edger of sorts, from Home Depot that sands it all away in minutes. I have used it on our baseboard caulking and it works well.
You can buy some caulk remover at Home Depot or Lowes. Find out if the caulk you want to remove is either latex or silicon. Latex caulk remover is easy -- you just squeeze it out of the tube and let it sit on the caulk a few hours and then it comes right off. I had to do this in my guest bathroom and it was not difficult.
Now silicon caulking is a real challenge to remove. I had to do this in my master bathroom. I am not sure but I think you have to use rubbing alcohol and then scrap it off. I hired a handyman to do this dirty work. If you go to one of the home improvement stores, ask the sales clerk about removing silicon caulk -- they may have something new on the market now to remove it easily.