SSO being the single point for security is the entire point.
Compare that to every application having a login mask and querying active directory. There are so many more points of failure. If even just a single application has a vulnerability, then an adversary can just log the credentials and hope he finds one with elevated access rights.
It is vastly easier to manage and strongly secure that single point, for administrators, security professionals and users.
You can enforce a much higher security credential, as it only has to be used once. You don't have to worry that your users are re-using the same weaker credential across multiple independent services (hint: they will).
Using SSO does not mean you have no defence in depth. You should always consider defence in depth in the context of the overall system. It's not a strong reason IMHO against centralising or simplifying one type of control.
I can see why this is an unpopular opinion for IT administrators and security responders alike, but I totally agree. Centralised authentication means if an attacker can compromise the SSO layer, they've compromised everything. This is far less desirable than an attacker having to compromise each system individually.
I think most often these days the users themselves are the centralized point of entry for attackers—a focused attacker is not necessarily going after every application a company uses and trying to breach it individually, but instead trying to compromise a user who has access to all of them.
So in that scenario, centralizing auth is desirable because you don’t have that user holding dozens of weak passwords, and you can monitor access and use heuristics to lock the user out of everything automatically if, say, they access the SSO portal from an unexpected IP. You can also always set things up to require re-auth or MFA whenever someone actually signs into something through the IdP, as an extra layer on top.
Especially when it's "modern" SSO aka browser-based crap which means the same software that you use to browse untrusted sites on a daily basis is also the one in control of your entire SSO session.