Accept that there's a good chance you're going to get hurt, and get back on and ride. The possibility of getting hurt generally doesn't stop people from playing sports they enjoy or driving cars, and the same should go for horses. If you're going to spend time with them and think you're not going to get hurt, you're deluded. About all you can do is make sure you know your skills, don't get on a horse that's too much for you, and if need be do falling drills. I know this last one sounds strange, but it really does lessen the fear if you already have some muscle memory for landing in a roll so you don't get hurt.
For me, I've been around horses my entire life...and for my first pony I had a Welsh/Shetland cross who would flop over whenever she was done with the ride. Literally. No matter what speed, without warning, she'd just drop. Luckily I never rode her with a saddle, and got very good at paying attention to her body language and even the most minor twitches of her muscles. With every subsequent horse I've ever owned or worked with, this has been invaluable...I have been bucked off quite a many times, but it no longer comes as a complete surprise to me because I always have at least a split-second warning before it happens.
Personally, working horses from the ground worries me more than being bucked off...I've had my head stomped into the ground and lost consciousness before, and another horse that tried to kick me in the head who had deadly aim for my left temple, I moved just in time and he split the fencepost behind me instead. However, even this hasn't stopped me...I suppose all you really need is a love of horses, an understanding for what their body language means so that you can limit the possibility of injury, and the ability to learn from injuries but not let it psychologically cripple you.