Why is he bringing the women to the house in the first place? Is he a tattoo artist, music instructor, or something like that? If not yes, I would feel disrespected! I myself do not believe in going into a male friends house if his girlfriend/wife is not present. I think that is disrepectful! Why would you need to talk to "my man" in "my house"?
And it works both ways if he is happy for you to have your male friends alone in the house with you then fine...
But I think he may just stop if he thought you would be brining men into the house when he is not there...;-)
Seriously, if you do not like it let him know, you should be the most important woman in his life and if he still chooses to do this against your objections you need a new man...One who puts you where you belong...First.