Physically fit for what? That depends on your individual needs. Yes, brisk walking promotes cardiovascular fitness.
But what if you're a world class golfer? Then you may need to add specific strength training exercises for the pectorals, obliques, deltoids, and abs.
About optimal health . . . Recent research suggests that people with large thighs live longer than people with skinny thighs. Why is that? I'll go out on a limb here, and say that it's because of more lean muscle mass in the quadriceps muscles.
If so, brisk walking -- on level ground -- is not adequate for optimal health. This particular aerobic exercise does not do very much for the quads. If optimal health is your goal, then you may need to add something to strengthen the quads.
I do the difficult version of proper-form Bulgarian squats. I am not suggesting that you should follow suit.
You may choose to do brisk walking one day, and serious bicycling the next day. You'll be getting cardio on both days. On walking days, you'll do a good job of exercising the calves, butt muscles, and hip flexors. On bicycle days, you'll do a good job of working the quadriceps muscles.