I think the races of the people in question need to form part of the equation. Black women seem to get angry when Black men date white women for a number of reasons. Black men seem to get angry when Black women date white men for a number of different reasons. Both sides have valid arguments for that anger, but I think they are talking over each other rather than to each other, so it is rare to find common grounds for understanding.
As a non-Black woman of color, I do not like to see men of color dating white women only in certain instances. It seems to me that the more famous and powerful men become, the whiter their partners appear to be. For me, this represents a trophy wife or girlfriend, an indication that one has "made it." Men of color who have money and power can attract white women because of their money and power. I think these women feel that they are being a bit rebellious and daring by dating or marrying a man of color, but would they still do so if that man was poor or had less to offer? I don't think so. I'm thinking of Ice-T and Coco here. Not sure why they came to mind.
I think Black men might feel protective of Black women for similar reasons. I think a lot of the negativity comes down to the racism that people still don't want to talk about in spite of Eric Holder's call for the US not to shy away from racial issues.
It is this racist baggage that informs people's perceptions of interracial dating. We see the color before we see the person.