As I've gotten older, I like my birthday gifts to be more about "doing" something, instead of getting a gift. It's more memorable, if it's an event. Age 40 is a great time to set this kind of standard.
How about if you bought 4 tickets to a sporting event and invited a couple of his friends to go with him? I did this for my dad at Christmas -- he's very hard to buy for, as well. We have a local college team and I bought 4 tickets to a game, one for him, my brother, my husband and my son. They had a blast. (And there's no reason one of the tickets can't be for you, but I wanted my dad to have a boy's night.) The event could also be a trip or lessons. Since he's athletic, in what sport could he use a little coaching? You can get golf swings analyzed, for example. Another thought -- I once gave my sister-in-law a cooking lesson (which I took with her). Cooking can favor us girls, but all the classes I've taken involved eating what we cooked, so that could please a culinary guy.
Whatever you do, just consider giving your hubby a great memory. : )