That's a really good point that I feel like isn't talked about enough. Unsafe rust is a lot harder to write correctly than bog standard C, because you have to uphold the invariants to avoid undefined behavior (1). It's why there's a whole ebook about it (2).
That doesn't mean it's impossible to write correct unsafe code, it's just not as obvious as "trust me bro I know better than borrowck." You can't actually elide the invariants Rust upholds, you just have to take over from the compiler when it can't prove them.
Another critical consideration of this is that if you make a mistake in unsafe rust code it could manifest itself as a 'bug' in the 'safe' rust code due to invariants that the safe code depends on not being upheld. It is literally undefined behaviour, in the "rm -rf / and insult your mother" sense.
That doesn't mean it's impossible to write correct unsafe code, it's just not as obvious as "trust me bro I know better than borrowck." You can't actually elide the invariants Rust upholds, you just have to take over from the compiler when it can't prove them.
(1) https://doc.rust-lang.org/reference/behavior-considered-unde...
(2) https://doc.rust-lang.org/nomicon/