I think Greg has a pretty good assessment of the situation. In order to compete, a 3rd party is going to have to attract viable candidates, and at this point, I don't see any extremist viewpoints being attractive to enough voters to get one elected. Unfortunately, this time around, They didn't have the chance of a snowball in you-know-where. The Libertarian Party platform has almost become a middle-of-the-road credible party, but there were two problems. 1. they had good men for state and local elections, but when these two men were asked questions about pressing national and international issues, they came across as Lum and Abner. 2. they were just a whisper in a hurricane of noise and dissonant name-calling by the two major parties (And this also affected the Green Party). It was like a rabbit being run over by a diesel.
The Green Party seemed to have an intelligent person for a candidate, but only a few people knew anything about her. You don't become POTUS by hiding your light under a basket. Also, the party line was a little extreme for the average Joe American. They just don't understand when a party walks in and proposes "I'm shutting this down because it's killing you and mutating your kids into dwarf lizard monkeys."
Average Joe replies, "You can't do that, I'll lose my job!" There has to be a viable alternative presented by the party to Joe. Joe then voted for the candidate who shouted "jobs" the loudest.
Third parties, are ya listening?