I think that's more than possible with people who ARE NOT one's "significant other", or with a whole and healthy kind of romantic relationship; but I really don't believe, or accept, the idea that someone can truly be "in love" with someone else but not like being around him/her.
To me, it's not a matter of "in a perfect world". It's a matter of whether it's "perfect love", and while people may always have their little arguments in even "a perfect love" I don't think those with the right kind of romantic love ever question how true or whole it is at its roots.
The way I see it, there is "loving" someone. There is "caring about" someone (slightly different from being able to out-and-out call it "love". Then there is "liking someone". To me, it's one of those three things that we can feel toward someone on that "caring scale". There is also "not liking something the other person does" or "not liking some isolated thing about the other person". But, to me, when it comes to how we would label our feelings toward someone in general, those feelings would fall under one of those three, clear-cut, categories or else under the fourth, which would be "being in love with them and having a whole and healthy kind of romantic relationship" that has grown past the infatuation stage (a whole, separate thing not necessarily related to this particular question).
People so often say that none of us can ever be perfect. I don't dispute that. Too many people think, though, that no love can/will ever be perfect. THAT is a sentiment that I very much dispute, and I know I'm not alone in my thinking when it comes to that. :)