The best tip I ever got and still use, is to pat some kind of powder (talcum or something) under the eye before applying mascara. It keeps the area dry and oil free so you have much less under eye smudges. Of course you want to be sure you don't use so much that you end up with white smudges either.
The other one is when I used to wear eye liner, I bought my brushes from an art store. That way I could have the tiniest brush for a very thin line right on the eyelash line--so it more or less highlighted the eye rather than looked like a black line.