Just a guess here - how about "driving" in an elevator? Would you brake if you didn't see the outside and feel the movement? The glasses can be dimmed on request pretty easily...

My point was that self-driving cars change the paradigm completely. What was once necessary is now no longer so... so there are plenty of new options to explore. All in all, I am guessing it won't be (much) worse than it is now. (well, apart from privacy standpoint and similar... :)