If I asked him and he said 500 people then it would matter but since I don't know then no, it doesn't matter. Don't bother with this because no one is going to tell the truth anyway. And if they do and you don't accept the number then you'll judge them. Just focus on what you have with the person now and not their past.
Also, it might matter if they slept with a lot of people that I know or in the same social circle. Otherwise (as long as they are disease free) out of sight out of mind.