I wouldn't complain. I probably wouldn't let the cat into my house, though. I'd probably give him some food outside. Lots of times, neighborhood cats are well loved and cared for, but their "people" don't get home in time for their need need for food. Sometimes, though, people just don't take good care of their own cats. Either way, I wouldn't complain to the people because if it just happened their cat was hungry before they got home I don't see it as a big deal to give the cat a little snack. If he's outside he's eating a lot worse than that, so eating what they don't know about isn't really an issue. If they were people who didn't take good care of their cat (or couldn't for some reason), I'd just see it as "decent" to feed the hungry cat. Of course, some people wouldn't feed it no matter what, and they'd say that the cat will stop showing up if he doesn't get food. That's true too (much of the time, anyway).
To be honest, I'd worry that if I complained about the cat to neighbors who didn't feed him for two days that they may get fed up with having trouble with neighbors and send the cat away. Then, too, if it's a Tom Cat and the neighbor cats with the food are girl cats, he may prefer to hang out with the neighbor-lady cats; and his owners may not even know where he is (or be able to do much about it other than have him neutered, provided they can afford it).
BUT, I'm someone with two neighbor-cats that I love (and who are well cared for as far as I can see). I give them occasional treats. They're my faithful buddies who meet me, or my guests, when we pull up out front. They seem hungrier on Saturday night than other nights. All in all, they're the best of all pet-worlds - the loyal friends and enthusiastic, affectionate, greeters without the vomit- and litter-box clean-up. As you can see, maybe I shouldn't have tried to come up with a good answer here. I love a good cat-friend (or two) and am happy to feed any of them. :) Even with that, though, I still don't let either my cat-buddies into the house. I suppose if it were super cold out I might do that, though.
Something else is that cats that are stressed out or have a medical condition can have fierce appetites that go way beyond what seems reasonable, so the people may not even realize that whatever they're feeding their cat isn't enough for his appetite. :/ Then, too, there's "complaining" and there's "mentioning as if it isn't all that big of a problem".... :/