I'm so sick of this trope. The point of numeracy is not in case you're suddenly caught without a calculator, although longhand arithmetic is definitely a good skill to possess.
The question reveals the problem, a victim outlook. You have no responsibility or agency, and your feelings justify it. It's not only bs, it's unhealthy and self-defeating. The outcome is predictable.