A character's death always has to serve some kind of purpose to the overarching story. Maybe one character's death prevents the hero from learning a crucial fact before it's too late, or maybe their death puts the hero in a deep state of depression that he has to pull himself out of in order to solve the story's crisis. When you're thinking of killing someone off, you can't just do it because it's dramatic--it has to add something.
In Gurren Lagann, the side characters' deaths give the main characters the conviction and drive to press on for their sakes. Death adds a sense of purpose to our heroes' journey, to secure a future in which those still alive won't have to live in fear.
In Monster, the many people who've died demonstrate the sheer cold maliciousness of the villain. Death is cheap and plentiful, and it directly conflicts with the hero's unshakable optimism and desire to save everyone.
In Fate/Zero, death is a constant, brutal, bloody reminder of the sins our heroes must commit in order to fulfill their dreams. In order to win the Holy Grail War and attain the titular wish-granting relic, the main character outright claims he'll do whatever it takes to save the world ("I'll make sure the blood I shed in Fuyuki City is the last that humanity will ever shed. Even if that means staining my hands with every evil in this world, I don't care. If it will save the world, I'll do it gladly.").
Just always remember that, while a character's death makes for great drama, there also has to be purpose to it. If nothing in the story really changes as the result of that death--if the diagram of the plot doesn't even make a bump--then it's just going to feel manipulative and pointless. The death of Heel-Face-Turn Femme-Fatale has to have a lingering impact beyond just the scene of her death; it has to be something that haunts the hero or prevents him from solving the mystery or puts the villain in a more vulnerable state.