I'm still in a post-holiday/"I-don't-feel-like-doing-anything-to-strenuous" mode, so I found this question a kind-of-fun one over which to do a little grammar "wheel-spinning". :)
I can see how it might bug someone, but I'm not sure I agree that using "key" in that phrase like that is incorrect. This seems like an interesting enough question to entertain, though. (It hasn't occurred to me to look into this particular "grammar dilemma" in the past. :) ).
One definition of "key" (as a noun) is "a means of gaining or preventing access, entrance or control". Another definition: "something that prevents a means of access".
My take on the phrase you don't like is that when people use the word that way they're using it metaphorically and "what it's a key to" is being implied. In English grammar there is a place for both metaphor and implying. I'm not sure if someone who is truly an authority on grammar rules would say that I'm correct in my non-authority assessment of this grammar dilemma, but it's why I don't see this particular phrase as a pet peeve for me. Maybe you're right; maybe I should be seeing it differently. I don't know. I don't think I use that phrase very often, myself. But, based on the reasons above, I'm not really sure it's actually incorrect grammar. :)
As far as whether it's a cliche or not goes, if something is very commonly used but is actually correct grammar that applies I'm not really sure it's truly a cliche. (Again, not sure.) Somebody could think of sentences like, "I'm tired," or even "I love you." - very commonly used and not really cliches (at least not most of the time, and at least I don't think they are when used in the right situation).