Jokingly, it would be lugging around the numerous & heavy equipment! However, in my mind there is one major difficulty in being a drummer / percussionist. This is the tendency of mistakenly "speeding up" the tempo when playing drums to songs. I've noticed many drummers do this including those in long established bands. And, I believe it is most difficult when one is “in the zone”. Being in the zone is losing oneself in the music, where nothing else exists, & it is extremely pleasurable!
Another item, in my humble opinion, is reading tablature, which is the contemporary method of writing orchestration(s) for drum sets. X's Are positioned on a line in a measure showing where / when each instrument is to be struck..This is as opposed to traditional musical notations with whole, half, quarter, eighth notes etc. And so, reading the score is more-so involved as opposed to playing a single percussion instrument such as the timpani, or the snare drum. And as well, there is the physical work / nature of playing the drums. One must remember that all the appendages of the human body, both feet, arms / hands are actively used when performing on drums, so there is the physical end of it as well. One must be in fairly good shape to do an awesome “Wipe Out” on the drums!
Should you want to see an amazing drummer check out Jeff Beck's "Space Boogie" on YouTube. Jeff Beck is a progressive rock / fusion instrumentalist from the '70's. http://www.youtube.com/watch?v=WPc819ryy0g