Absolutely. Sometimes that childhood passion becomes a well-loved hobby, it might be something you pass on to your children, it may serve to guide your adult career choices. Just because you loved it as a kid doesn't mean that it's immature to pursue it now as an adult.
The challenge with pursuing a childhood passion that is not connected to your work is the time it takes away from family time. The best solution I've found is to look for ways to include one or all family members in all or a portion of your hobby.