This is a very difficult question to answer, as there is no straightforward answer to it. The instant response is that you should go, as he is not setting a good example for your son and obviously you are upset about the whole situation, which in turn your son will pick up on.
However, saying all that, you should try to talk and explain your feelings to the father as he could play a vital and important role in the upbringing of your son.
If after you have talked he still plays around, then best to start afresh as a single mum. Which although difficult is not impossible. Many, many mums are on there own and they probably cope better that way than if they were with an incompetent male. Remember, if you do go, don't be afraid to seek help. There are plenty of support networks where you can turn for advice on bringing up your child whether on your own or not. You will not be the only single mum they have seen, and you may make new friends in the process, maybe even a single dad!!