I believe you are capable of answering this question for yourself. No one can tell you what to do. Every choice has positive and negative effects based on your perspective. This is your relationship and the commitment required extends far beyond just you and Daniel. There are children who need reassurance and nurturing, and they need cohesive support and grounded parenting.
Everyone will be affected by the decisions you and Daniel make for your future together and this the extended family, too. There are quite a lot of considerations you and he need to address unless you have already. You ask this question because there appears to be doubt in your mind. Be true to yourself. Know what you can and cannot live with in your life.
I am confident you will do what is best for everyone concerned. Be true to yourself. Keep in touch.