Agreed.It's more than just lifting tires in rocks. Sway bars by their nature cause an opposing force across the axle which in effect causes the suspension and body to squat when it's active (called suspension jacking). This is usually what you want when the road surface is durable and flat and you have both wheels on it.
But a sway bar will work against you when one tire is no longer on that smooth, flat surface. One such case is if you run a wheel over the edge of a roadway. There's a point in a grade cross section called the hinge point, where the mostly flat slope transitions to become steeper. If the sway bar is trying to keep your body plane parallel to this newly steeper plane you reinforce the tendency for center of mass (which you doubt raised with a RTT or roof rack with MaxTrax, shovel and Hi-Lift) to push you over.
Another case is pot holes, corrugations and washboard. Having the sway bar impart a cross-axle force tends to unsettle suspension so instead of each shock independently damping oscillations you get more jarring.
It doesn't have to be rock crawling for there to be benefits to disconnecting your sway bar and letting the suspension articulate. Also increasing spring or damping rates can alleviate some of the body lean and sway issues that the sway bar was originally compensating for.
Ideally, I'd want the ability to disconnect the front bar on my Tacoma, but there isn't a simple solution for that, and unlike the 4Runner, Toyota doesn't even offer KDSS on the Tacoma. I know the compromise I'm making by leaving it installed, and for the way I use my truck, it's the right way to go.