The only way to know the answer is to ask him directly. Don't go into any confrontation but ask him in a serious way if he's really that busy or is there is something you need to know about. If he tries to dodge the answer or gets annoyed at you for asking this, you may want to investigate it further, without letting him know of course. Judging a response is not easy and unfounded suspicion may ruin a relationship. So tread carefully.