I think the accent is a major factor. It's far easier for Brits to understand American TV voices than it is for American's to understand British accents. I really struggle to see an American understanding some of our British slang and some UK accents are really hard to understand.
I have an American friend who looks at me with a puzzled expression on their face anytime someone on the TV speaks in a broad Scottish accent, any type of Welsh accent and some of the London accents are really rough.
Also I think it's something to do with the sheer size of the USA where the number of potential viewers means it's financially viable to reshoot the show. Since the UK is far smaller than the USA, then it's more financially sound to just sell them the US version, rather than reshoot the whole thing.