I did it and it was a great decision. Ride quality improved tenfold, and honestly, it handled as well or better. You get back a lot of forgiveness in the suspension, which I think helps handling on rough roads. Poly is really only advantageous on a track where conditions are kept ideal, in my opinion. So I highly recommend a full stock bushing setup. These cars handle incredibly well with stock stuff. A lot of people swear by poly, but I think that it is because the poly is replacing 20 year old worn-out rubber parts in most cases.
I am also still "stuck" with poly sway bushings because of my ST bar set. Seriously considering ditching it for stock though since it is inevitable that the rear chassis mounts will rip off again, even with the TMS reinforcements.