I suggest you look at your relationship overall before you make any decisions. How long have you been together, how you make each other feel, do you have any thoughts of the future? Think about what you want. Then ask him what he wants.
It also might not be that he doesn't want to spend time with you, but that he's busy or is getting a bad vibe from you. If you are only thinking about the negative aspects of your relationship then he's gonna pick up on that and think you don't want to be with him. You have to let him know that you want to be with him and that you enjoy being with him. But without being desperate, because you don't NEED him to be with you, you simply want him there.
I hope this helps you in some way.