I don't know for sure, but would guess that it's more likely a brain glitch than a physical problem. Most tongue-twisters have repeated syllables which are similar (but not all the same.) I'd conjecture that the brain has trouble sequencing these rapidly, and 'confuses itself.'
Intuitively, based upon my experience as a musician, I would expect that you could master a tongue-twister completely if you practiced it diligently. But at that point, the tongue-twister would be handled in the brain like a musical passage, or a specific 'move' in your favorite sport: it wouldn't be dealt with anymore at a conceptual or linguistic level, but rather as a learned pattern in (physical/somatic) memory.