Well, I dunno if you've ever driven a performance car say around 420hp but having one extra passenger is absolutely noticable and undesirable, it really slows you down. 4 passengers in the car and it's no longer a fun drive. Not a rounding error as you can directly feel it.

I expect dragging around 120 lbs extra fuel for 100k miles does become noticable on the bank book, you'd be surprised.