If you are asking about a newborn or a very young baby my immediate answer would be never just let them cry.
Babies aren't like adults, they don't complain just to hear their own voice. If an infant is crying there is a reason. Sometimes the reason might be "I want held" but in the grand scheme of things being held is just as important to an infant as a clean bottom or a full belly.
If you just can't take it anymore it is a safety issue for the baby, a few minutes to take a time-out is okay. In that case, make sure the baby is in a safe place and be sure to give the baby lots of comfort and reassurance when you get back.
Remember though that to a baby, his caregiver is his whole world. To develop normally, the baby needs to know all of his needs will be met, even the need for human contact.