Are you drying them thoroughly and removing the skin before battering them? If so it could be the batter recipe itself, try a simple blend of AP flour (1 & 1/2 cups), baking powder (1 tsp), salt and black pepper (1 tsp each). make sure you do the egg wash first then dip in the flour. when you put them in the fryer do it by hand or with a spider and slowly lower them into the oil. If you drop them in from any height that could knock the batter loose too.