Splitting up if things don't work out is always better than playing pretend and cheating in the meanwhile.
That being said, I disagree with those who say "Sex isn't everything".
True, there are many more things that are important in a relationship: your shared traits, your interests, the way you handle things, the way he makes you smile, etc. Personality truly *is* important, it'd be shallow to go for a man who is very attractive but a jerk in personality. It wouldn't work out. But that doesn't mean that physical attractiveness, being good-looking, being a good cuddler and having a good sexual relationship aren't important. A person with a good personality but to whom you are not physically attracted to, can be your best friend or, like you said, a brother. For a life partner, you'd want the total package.
If that is saying that lack of one of the things I listed there is a good reason to just give up, I say no. Things don't always work out, that's why you talk, communicate, express your desires and work on those issues together.