My parents experienced the same sort of situation. I think it comes down to how much you value him. Is it more than your own happiness? For some wives their husbands immediate happiness is more important than their own happiness. It is a strange loop where they are only happy if their husbands are happy.
I think maybe in your case, it's not enough that he is happy for you to be happy. But of course, that's up to you to decide. If you are unhappy with your position on his priority list you should talk to him about it. If that is too uncomfortable then talk to someone else you trust to work out exactly how you feel because in the that is what is important. Take care :)