Generally speaking "worry" comes as result of observing "changes" in the dynamics of a relationship or marriage.
Especially if these are "sudden changes".
If for example your husband use to spend most of his weekends with you and then suddenly stopped doing so it's only natural to wonder if he's no longer "in love" with you or is unhappy.
Ultimately if you don't trust someone it's impossible for (you) to be "all in" regarding the relationship/marriage. There's a tendency to start pulling back and begin investigating.
If something doesn't feel right to you it's probably not right for you.
There are only two ways to experience joy and peace of mind in relationships: We either get what we want or we learn to be happy with what have. Accept them (as is) or move on.