Your doctor uses the word, "flu"? (or are you interpreting what your doctor says to mean "flu"?
Doctors usually use terms like "upper respiratory infection" or "intestinal virus". You probably already know that a wide variety of bacterial or viral infections can cause someone to be sick with any number of symptoms (or similar symptoms). If the infection is bacterial, doctors usually prescribe antibiotics; so it's probably safe to assume your children have had a variety of viral infections those times when you've brought them to the doctor.
If the doctor keeps using the word, "flu" I think you should ask him/her if s/he means "influenza" (which is, of course, what "flu" is short for). If this doctor says that s/he has been referring to influenza every time, then I'd think you need a new doctor because that would seem a little odd to me (unless, of course, you're only talking about a time or two, or one doctor visit for each of a few different children).
If the doctor tells you s/he has been calling every infection "flu" then I'd probably find another doctor too, because s/he is either being awfully cavalier about medical words, or else s/he is underestimating your ability to understand simple terms like "upper respiratory infection" and "intestinal virus".
If your doctor hasn't used the word "flu", and it's just your interpretation of what s/he has said (a lot of people misinterpret "infection" for "flu"), then I hope the above comments clarify that your doctor isn't calling everything "flu".
Then again, though, influenza ("flu") is usually accompanied by several symptoms, depending on the type of flu. How extreme symptoms are in any case of flu can depend on the person, his exposure, etc.
Either way, I think you ought to start by asking the doctor, "When you say 'flu' do you mean they have one form or another of influenza every time?" Ask him/her about the frequency of that diagnosis, and ask if s/he means different types of flu, or if it's unusual that the diagnosis would be that so often. Give him/her the chance to explain what s/he means and thinks, and why. Pediatricians are usually happy to explain things to parents who have questions. If this doctor isn't - then THAT, by itself, would be reason to find a new doctor (because, again, most pediatricians understand that parents have questions).