I've spent a lot of time helping to hire coworkers for a rather good remote position, and having worked with the ones that passed all of our filters and also accepted the position. It's still a bit early, but I've found that the difference between someone who actually got a proper CS degree, and someone who learned to "code" on their own is pretty stark.
The fact is that calculus, algorithms, data modelling, and the like are often very relevant to programming once you are past the entry level positions.
I don't have a CS degree, but even in entry level positions, it crops up when you least expect it. My company gathers data off of truck engines, with a data sample every .3 seconds. The data samples contain things like instantaneous "fuel rate," "vehicle speed" and other "rate" values. If the manager says to me "erroneousfunk, I want the cumulative fuel usage values over time, in a graph" I need to know that they're looking for the second derivative of the fuel rate, and how to model and summarize that data in the database appropriately for a fast processing and lookups across any combination of companies/drivers/trucks/date ranges they can think of.
The fact is that calculus, algorithms, data modelling, and the like are often very relevant to programming once you are past the entry level positions.