I think there are two many possible reasons/kinds of reasons that you could be feeling the way you do, and I don't think anyone on an Internet site can offer you solid suggestions (or at least not without more information than given).
This probably isn't the kind of reply you want, but I really think you and your husband should set up an appointment with a counselor together. Maybe it would only take a visit or two to get some solid input about whatever is going on/not going on. Whether this is a little kid you see only on weekends usually or is one who hasn't got his own mother in the picture and is being raised by you - both two completely different types of situations. Whether your husband is creating the situation or not, whether someone is expecting more of you than you have to give, whether its both you and your husband's child who feel "on the edge" (or just you), whether the child's mother has raised a child who is out-of-control, whether you're someone who isn't skilled at establishing a healthy respect from the child - all things that could create problems.
Again, talking to a counselor about the issues; and approaching it from the standpoint of wanting to build a better relationship (rather than "having issues") is, I think, the most sensible thing to do.