I think this depends on where you are at in life. I have had some tremendous memories but the main reason I live is to make new ones so I must go with the latter. Why live if you can't make new memories? You're pretty much dead anyways if that's the case.
The hardest by far would be not remembering my mother who passed away a few years back but I believe she would want me to have memories of my childrens journey through life as I get older. I'm still fairly young so God willing I will have plenty more memories to make...I just hope I get to remember those.