I have not heard of any issues with pregnant women and ferrets.
That being said, there are diseases like giardia that can be spread though contact with feces.
If you change the litter box, I would wear some type of protection(gloves, mask) to be on the safe side. To be even safer, maybe another person could handle litter duty. Wash your hands after contact with your ferrets.
Ferrets and humans are susceptible to the same strains of cold and flu, so if you or your ferrets are showing cold or flu symptoms I would limit contact.
Make sure your ferrets are up to date on shots.
Once the baby is born, I would not allow the ferret and baby to play. Some folks think this is adorable, but babies don't know how hard to grab a ferret and both could get hurt in the wrong situation.