If you're trying to validate new software on known good hardware, then your goal at any scale is to insert a suite of embedded software acceptance tests into your development pipeline usually somewhere between build and release. CI infra is ideal for this, my counterparts all use Jenkins.
Robotframework is a great tool for expressing tests. For simple setups, Jenkins has an RF plugin to kick off tests.
The tricky part is managing many test resources. One jenkins server triggering robot tests on one bench PC with one embedded target is a great way to develop an initial test suite. But once your org gives assigns you more hardware, the tricky part is figuring out how to use it maximally (either optimizing for fastest time to completion or for least time idle of your dedicated test hardware.) This means the tests themselves can no longer "drive" the test resources directly, they need to lease them from a resource manager service.
Larger companies with nearly homogeneous hardware may get by with a basic priority queue. But some test setups with an L1 or L2 switch between them may support a DSL to allow tests to request resources that are physically or logically connected, and may need abstraction layers for common tasks such as "physically disconnect power" which the Resource Manager could implement as a SNMP command to a PDU or as a SCPI command to a DC supply -- the test itself should be declaring what has to happen, not how.
I've scoured the internet and plied counterparts in other industries for any example of commercially available prior art. Everyone I've talked to wrote their own. It took a week to produce something acceptable for my use case. But if the RF devs were to maintain a similar solution, I'd switch to that in a heartbeat.
Robotframework is a great tool for expressing tests. For simple setups, Jenkins has an RF plugin to kick off tests.
The tricky part is managing many test resources. One jenkins server triggering robot tests on one bench PC with one embedded target is a great way to develop an initial test suite. But once your org gives assigns you more hardware, the tricky part is figuring out how to use it maximally (either optimizing for fastest time to completion or for least time idle of your dedicated test hardware.) This means the tests themselves can no longer "drive" the test resources directly, they need to lease them from a resource manager service.
Larger companies with nearly homogeneous hardware may get by with a basic priority queue. But some test setups with an L1 or L2 switch between them may support a DSL to allow tests to request resources that are physically or logically connected, and may need abstraction layers for common tasks such as "physically disconnect power" which the Resource Manager could implement as a SNMP command to a PDU or as a SCPI command to a DC supply -- the test itself should be declaring what has to happen, not how.
I've scoured the internet and plied counterparts in other industries for any example of commercially available prior art. Everyone I've talked to wrote their own. It took a week to produce something acceptable for my use case. But if the RF devs were to maintain a similar solution, I'd switch to that in a heartbeat.