Probably Goldeneye and/or Perfect Dark from the N64. It wasn't so much the main game that was hard, it was unlocking the cheats. Completing a specific level with a specific requirement, in order to get that cool new cheat was often maddening.
Two levels stand out to me, one was the train level in Goldeneye. I no longer remember what the cheat was, or the requirement, but I remember my hands shaking so bad that I kept shooting the person I was supposed to save right at the end of the level. And then in Perfect Dark, there was a level on an airplane where there were just too many enemies, and not enough life, before I reached the last room. I did eventually unlock both of those chests, but I would never do it again.