Worms love compost rich soil. My folks used to have a worm bed and what my mom did was to throw the vegetable skins, ends and pieces that she chopped off into that spot. She also threw coffee grounds, egg shells, leaves and grass clippings into the mix. Anywhere you dug your shovel into that soil there was always a bunch of worms.
I inherited my mother's love for gardening and found that mulch does the same thing. Worms love to bed down beneath the layer of mulch where the soil is dark and there is an abundance of nutrients.