Ultimately though it is all about time efficiency, solo or small teams have little manpower and since they don't even know if their first, second, or fifth game will sell any at all it may not make strategic sense to spend months reproducing common engine features in order to save a hypothetical percentage of revenue that they may never ultimately make. You need to test the gameplay concept as cheaply as possible.
Creating a game engine from scratch is FUN, and if game development is a fun hobby then full steam ahead. If however it is a business that you plan to make money on, you're unlikely to recoup the time investment into a bespoke engine just to test the market for your games. If the game turned out to be successful you could always piecemeal replace the commercial engine with bespoke via updates (inc. using money to hire people from the success).
> You need to test the gameplay concept as cheaply as possible.
There are some edge case concepts that I have found to be completely impenetrable on commercial engines.
Multiplayer is the #1 thing in my mind. I've gotten the FPS multiplayer examples to "work" for UE and Unity, but the confidence I have in these solutions is not great. I've seen what Unity can do (BattleBit), but I don't know how many decades of game industry black magic and hackarounds it requires to force a commercial engine to behave that well.
I really think the answer is that it depends. If you aren't trying to have a "perfect" version of some gameplay aspect (i.e. multiplayer in my case), then the commercial engines will absolutely be the fastest path to validating your idea. Put differently, I think some game ideas are not possible to validate on commercial engines. But, I don't think this is very common.
It took me about 3 days to build the engine, including custom physics, gravity/jumping, collision detection including raycasting, and about 3 weeks to finalize it and start working on the game. Game was released in under a month and we already have a small community. It was definitely worth spending a little up front time to avoid a service provider, and have more control over the mechanics and look/feel of the game.
By the way, you re-use a lot of the same code when you build in-house, like you said often times your 1st title is not a hit, but you improve upon a lot of that engine code and release subsequent games. Most AAA developers do this.
Ultimately though it is all about time efficiency, solo or small teams have little manpower and since they don't even know if their first, second, or fifth game will sell any at all it may not make strategic sense to spend months reproducing common engine features in order to save a hypothetical percentage of revenue that they may never ultimately make. You need to test the gameplay concept as cheaply as possible.
Creating a game engine from scratch is FUN, and if game development is a fun hobby then full steam ahead. If however it is a business that you plan to make money on, you're unlikely to recoup the time investment into a bespoke engine just to test the market for your games. If the game turned out to be successful you could always piecemeal replace the commercial engine with bespoke via updates (inc. using money to hire people from the success).