The short answer for me is "Yes". However, if the right man/woman came along while your kids were still living with you then, to me, that is fate.
Hopefully this new person will be accepting of your children and not try to impose their "rule" over them. That's when problems will begin as your protective instinct will kick in and it could cause issues with your new partner.
However, if you guys are willing to work through the issues, it is possible that everything might work out fine.