The obvious would be to see a dentist. But I figure you know that already.
A salt water bath would help with the irritation and be somewhat soothing. Put a teaspoon of salt into a warm glass of water. Swish around. Spit out. Now, you can add some great antibacterial action by putting garlic, thyme and oregano into the water. They all have anti-biotic, antibacterial properties in them that can help fight infection.
Get some clove or clove oil. If you use the clove. crush them up and stir into a hot tea. Swish it around in your mouth and then spit out. If you find clove oil. dab a drop or two onto affected area.
If it isn't better in a few days, you should see a dentist. Mouth infections can lead to other bodily infections.