I believe it's personal preference or what you grew up hearing. Webster's Dictionary defines them as follows:
Handbag: a bag held in the hand or hung from a shoulder strap and used for carrying small personal articles and money
Purse: a small bag for money (2) : a receptacle (as a pocketbook) for carrying money and often other small objects
Mostly I say purse, even though it's the size of a tote bag sometimes! Occasionally I say to one of the kids "grab my pocketbook", meaning my purse.