I think it CAN matter, but not necessarily to the people IN the relationship. I'm 41. My "boyfriend"-type person is 21. We have a lot in common and we get along well. Has the age thing come up? Oh, for sure. Has it been a problem? Yeah, it has. But not for US; for the people around us.
People want to give advice. People want to be experts on everything. They're not. Let them say whatever they want; it's really none of their business.
If everyone is legal and willing, then it is what it is.
We HAVE had problems, btw. But those problems would happen in ANY relationship with people of ANY age.
It's about the person inside. That's what matters.