People often think that "mum" is short for mother, but actually the word is an onomonopeia, a word that imitates the source of the word. in this case it came from the sound of trying to talk through closed lips. Go ahead, try it. Sounds like mmm, right?
One historian traces the phrase back to the mummers or mimes, who acted out various skits in silence. A form of the phrase was used in John Palsgrave's 1540 translation of the Latin text The Comedye of Acolastus, "I dare not to do so moche as put my hande to my mouthe, and saye mum, is counseyle."
But Shakespeare also used the phrase, in a slightly different form, in Henry VI, Part 2, "Seal up your lips and give no words but mum."
From there it became a common way of telling someone to stay silent.