I feel like this needs a big asterisk. Can you ship a a non-trivial iOS or Mac app that uses SwiftUI or other first-party APIs without Xcode? Is it practical? Those are real questions, some cursory searching did not turn up a concrete answer.
It is possible and practical in lots of cases. And it's necessary to use the CLI tools directly in some situations, such as when deploying from CI servers rather than building by hand.
Is it possible for 100% of situations? I don't know, because I haven't tried 100% of the situations. And in one case I haven't figured out yet (AUv3).
What I'm interested is in the CLI tool for my own use, not necessarily for agents.