In this instance 'mom' should be capitalized, because it is used as a specific name, not a general name. The basic rule is, if you could insert a person's name into the sentence, then capitalize it.
1. "Hi, Mom, I'm home," Pete yelled, as he came through the front door.
2. Joe's mom handed him a letter.