You can and should take your children; however, you may want to talk first to a divorce lawyer if your reason for leaving is not abuse, or (if the reason is abuse) a women's shelter/advocacy group in order to get your story and reasons for leaving "on record".
Here's why: When women leave their marriages it is common for men to accuse (and even honestly believe) their wife is insane (for wanting to leave "wonderful them"). I don't know what the laws are in each state, but getting you picked up by the police or an ambulance is as easy as telling authorities he thinks you may hurt yourself and/or the children. Believing "all it takes is explaining" to straighten out "the misunderstanding" is incorrect. Once you get picked up you can be in a no-win situation.
Don't count on your family or friends to be your back up. They may think you're crazy for wanting to leave the guy as well. So, talk to someone in the system.
If you can't talk to someone ahead of time go to the court clerk's office and ask what papers to file for divorce. There may be an "indigence" form if you don't have the money to file. The best thing, though, is for you to get to people in the system before he does.