Yep, it's a pain. The upper end basically corrodes in place. Spray it with PB Blaster, let it soak a long time, and pull like you're trying to pull the car down on top of you - it should break free. Make sure the cable is disconnected at the e-brake handle, of course.
Liberally goop your new cables with antisieze at the end where it goes in to that tube, that will help if you should ever have to do it again.