But people DO do the thing right. As a loose guideline, I think you can reasonably infer that anyone who is telling "you are doing Agile wrong" has first-hand experience with an agile process that is working reasonably well. So take that as a starting point.
And the reason they are telling you that: because an agile process that is working well is like night and day to all the crazy processes that preceded it. When it works, it is amazing!
If I build a manual transmission car, and you insist on driving it like an automatic, it's not going to work for you, even though it's an amazing car. There are steps and processes that you must follow in order for the thing to work for you. If you don't, it won't. Simple as that.
Put another way, if you enter the olympics for breakdancing, and then just move your body around in a floppy way, you will not win the gold medal.
When nobody manages to do the thing right, then the thing isn't good. It doesn't matter what platonic ideal you hold for it.