No one can tell you what to do.
After all this is YOUR life.
However I have known several people who have been in relationships or marriages where sex drives did not match.
A relationship where one person would like to have sex 4 times a week while their mate is fine with 1 time a week could be in some serious trouble.
That's 208 times VS 52 times a year!
Sexual frustration is a leading cause of infidelity.
I would say if you are in a relationship which does not allow you to be your "true authentic self" there are likely to be problems. Ultimately we all want someone who wants we want out of life. Sex is not a small thing in a marriage.
No one dreams of "forsaking all others" for someone who has little or no sexual desire for them. Sex in a marriage is a physical expression of love and intimacy. Generally speaking "sexless marriages" contain less snuggling, hand holding, hugs, wet kisses,taking showers together, and overall romantic gestures.
Ideally you want to be on the same page when it comes to stuff like that.