I find this so confusing - I’m working on a fairly large project using ESP-IDF and find it an absolute joy to use. Far better than any other microcontroller framework.
That's the main problem with it - it's a framework.
I don't want a framework that takes over an entire project and mandates the use of a given build system, configuration system, source code structure, etc. This tends to break apart when you want to have more complex build steps; eg. matrix builds, subsequent build actions on artifacts, integration with codegen like string interning, multiple target platforms which are likely not an ESP32 (like a simulation target running on the host), integration with linters/checkers, integration with test frameworks, etc.
And the technology choices ESP-IDF made are also... controversial (CMake and Kconfig! Plus a whole bunch of Python glue to actually make it work together).
Just give me libraries that I can build/link against and let me bring my own build system - or pick a build system that is actually extensible and will scale to more complex scenarios.