I've been in this situation before. I never had the courage to tell him how I really felt through fear of ruining things, and him running. Another girl told him how she felt and he started a more serious relationship with her, and ours ended. So my advice would be to find away to tell him how you feel no matter how scary it might seem. And try to say it directly to him, not via text message or email. Bear in mind that you have a right to your feelings but it could come as a shock to him since your relationship has only been casual up to this point, so after you've told him please give him some time and space to think about what you've said so he can make up his own mind about it.
Another way to go about it would be to say something subtle to him about how you feel you're developing stronger feelings for him than what you originally had, and just see what his reaction is when you say that, at least it might get you talking about where you really stand with eachother. You lose more when you never tell the person you love how you feel, so you really need to find a way to overcome your fear of communicating your feelings to him.
Also bear in mind, your feelings could be that of lust and it could be just a crush, because if he's just coming over to your place every few weeks he's probably not giving you what you're really looking for or deserve in a relationship no matter how hot he might be it doesn't sound like he's taking you out on proper dates etc and he's potentially seeing other women (since you have an open relationship at the moment).