My personal favorite book is the Prisoner of Azkaban because the central villain wasn't Voldemort (for the first and only time in the series), but at the same time it also kicked off the larger story that would carry the series to its conclusion. (Wormtail's escape). And my least favorite is Half Blood Prince. It just really failed to recapture the magic of the first five, and felt like filler.
Of the movies, my favorite is probably Goblet of Fire, because of its great action sequences. And my least favorite movie, ironically, is the Prisoner of Azkaban. While it was a good movie on its own, it was a terrible adaptation.