May I ask what stack you employ to meet these goals?
Many tend to reach for Gitlab CI or Github Actions but these piles of "executable yaml" never appear to be up to the task of complex deployment logic you describe in your post, not including that they don't account for multi-repo or composed artifact workflows naturally. The state of the art, if you can call it that, is Jenkins where you can drop into raw-ish groovy/java for the logic pieces when you need to. But then you run into the constant struggle of working around Jenkin's leaky abstractions and peculiarities.
You can patch together a pile of bash, python, go et al but you land in a worse place where there is no guiding structure to the automation for onboarding, enhancement, and maintenance.
I'm curious of other's experiences building complex build / deployment pipelines where up-front you have consistent entry structure to the automation but have all the escape hatches one would need to implement custom logic when required, in a type safe, potentially compiled, testable way (ie: pipelines as 'actual' code).
Of course one could write their own automation engine to avoid yaml hell and all that. However I am not seeing any pervasive solutions being presented that don't present "yet another (yaml | json | xml | cue | whatever) task dag launching containers running random scripts from wherever".
>> May I ask what stack you employ to meet these goals?
Nothing fancy :-) Ansible + Jenkins pipelines
>> these piles of "executable yaml" never appear to be up to the task of complex deployment
Agreed, there's an untapped market here - somewhere between your enterprise GitOps model like i'm alluding to (usually heavily integrated with Kubernetes) but repackaged for smaller scale use.
Many tend to reach for Gitlab CI or Github Actions but these piles of "executable yaml" never appear to be up to the task of complex deployment logic you describe in your post, not including that they don't account for multi-repo or composed artifact workflows naturally. The state of the art, if you can call it that, is Jenkins where you can drop into raw-ish groovy/java for the logic pieces when you need to. But then you run into the constant struggle of working around Jenkin's leaky abstractions and peculiarities.
You can patch together a pile of bash, python, go et al but you land in a worse place where there is no guiding structure to the automation for onboarding, enhancement, and maintenance.
I'm curious of other's experiences building complex build / deployment pipelines where up-front you have consistent entry structure to the automation but have all the escape hatches one would need to implement custom logic when required, in a type safe, potentially compiled, testable way (ie: pipelines as 'actual' code).
Of course one could write their own automation engine to avoid yaml hell and all that. However I am not seeing any pervasive solutions being presented that don't present "yet another (yaml | json | xml | cue | whatever) task dag launching containers running random scripts from wherever".