Well, I've had jobs that were difficult for different reasons.
Roughneck (working on a rig) - very physically demanding, not to mention dangerous as hell (I was in the hospital twice in a year). And you have to learn to deal with sleep deprivation.
Chef - High heat, high stress, high speed, high emotion.
Chief of Security - Not only dealing with the petty squabbles of employees, but working in Vegas provided all sorts of interesting characters to deal with. Violence almost every day.
Glazier (window installer) - Couldn't be afraid of heights on that job. Found myself dangling from the side of a building more than once. Replacing 12/12 eighth inch panes of glass on top of a two story greenhouse was pretty scary as well.
But out of all that I think my most difficult job is being a father. Trying to teach your children right from wrong, teaching them to be intelligent and kind. And protecting them from all the scavengers of the world.