If you're "scared to loose him" then you're lying to yourself about not having feelings for him. The opposite of love is " indifference".
You wouldn't care about "losing him" because you know you could replace him in a heartbeat! What you're afraid of is not finding another man who will (love you) as much as this man does! In other words you think you're LUCKY to have him!
You're getting something out of the deal.
If you really thought you were this amazing woman whom men would gladly jump at the chance to be with and marry you'd be singing Beyoncé's "Irreplaceable" song.
"To the left, to the left. Everything you own in the box to the left. You must not know 'bout me. I could have another you in a minute. Matter fact he'll be here in a minute... Don't you ever for a second get to thinking you're irreplaceable."
Only someone who has no feelings left for a person would say that. :)
Odds are you are using him for something until someone "better" comes along. You will stick with him and even cheat on him.
It takes courage to walk away from a known present into an unknown future. Essentially you are robbing both him and you of time you guys could be with other people in a (mutually loving) relationship.