It's not easy being a manager and I think most of it is hard, at least at first. Once you get used to it more and more things seem simple. When it comes down to it, your attitude plays an important role in the level of difficulty in managing. Many things come into play, but if you stay positive and try your best you will get positive results thus making things easier for you to manage.
It also depends on what you are the manager of and how high your rank or position is. Take it from a past manager's point of view, stay positive and just do your job. You will find it's not as hard as you thought or was at first.