I have never done it for 40 days but I have done it for many days. After a few days you get all slowed down and become amazingly aware of your own surroundings. In the natural world there are none of the people distractions. And if you cut food down to a minimum or a fast there is no food distractions. There is really only you and yourself. At that level you can begin to really put your energy and thought into prayer and listening. Now if you are a very spiritual person you can commune with your God. One part that is really cool is that the rituals all fade away. The man made strictures and structures are irrelevant.
There is no better place, space or way to really commune with God. I would think that is why Jesus did it. One could look at it and think it is some kind of denial ritual, but it is in fact the opposite. It is a totally all about me kind of deal. Me and nature and God that is. If you want to get yourself together with God it is the ticket.