If you know they have the problem, then it should not present the problem for you. If the invited guest has a problem, it would be proper for him to tell you in advance so you can either adjust the menu, have something special set aside for him, or he might thank you for the invitation, but not wanting to cause you any additional trouble apologize for not being able to attend.
I have a friend who is allergic to garlic and a niece who is allergic to tree nuts. Even the slightest bit can cause serious distress for the person allergic to garlic and death for my niece. Being inform is the key to keeping the proverbial wrench out of your party plans.