Yes, you should be worried. If you're not communicating, not sleeping in the same room and having other stresses in your lives, then it's time to do something about it and now.
Marriage is hard and takes a lot of work. Having been married for over 14 years myself with five kids, I know how easy it is to let your relationship slide to the back burner, but it's important that when you see this happening that you start taking steps to stop it.
You need to set aside the time on a reglar basis (daily would be great, but at least a few times a week would be good) to start talking. You don't need to only talk about the kids or bills, but also things that brought you together in the first place. Remember what is was that made you fall in love in the beginning.
Wake up everyday and think of at least one thing that you can do for your husband that will make him feel better. Love notes in his lunch or on his pillow can be a great thing to get things going.
There are great books out there to help you and counseling would be good too (though if money is a problem, I can see how this might be difficult). In any case, now you know there is a problem and so you should do something about it--the first step is to re-open communication. Not always an easy step, but one that is worth it.
Good luck to you! Now that I think about it, you've just inspired me to make another hub....