It could be stuck in there with old oily varnish if the previous owner wasn't on top of the engine's maintenance. When you remove the cap that retains it, does it push out at all, or does it not budge?
As of now, I would not recommend trying to yank it out with pliers or anything. It could score the inside of the hole. Maybe try shooting some Liquid Wrench or other penetrating oil up in there to dissolve the varnish? Are you due for an oil change any time soon?