ensure you know what each family member likes or dislikes. Then, plan so that your itinerary includes activities which each person will enjoy or at least most activities they will enjoy.
If you can't ensure that everyone enjoys everything, then maybe split the family up at certain times. For example, when the ladies go shopping, the men can go golfing.
Last but not least, be flexible and be patient.