Everyone told me "use Unity" "the biggest mistake you can make is building your own engine". How well did that advice age?
Nobody needs Unity or Unreal.
My most recent game (playshadowvane.com) is built on 100% proprietary technology, I built everything in-house down to the physics.
Always ignore mainstream game dev advice, they are just trying to sell you products. Build as much as you can in-house, it's not only better for the look/feel of the game, but no company will be able to rug pull you later.
Ultimately though it is all about time efficiency, solo or small teams have little manpower and since they don't even know if their first, second, or fifth game will sell any at all it may not make strategic sense to spend months reproducing common engine features in order to save a hypothetical percentage of revenue that they may never ultimately make. You need to test the gameplay concept as cheaply as possible.
Creating a game engine from scratch is FUN, and if game development is a fun hobby then full steam ahead. If however it is a business that you plan to make money on, you're unlikely to recoup the time investment into a bespoke engine just to test the market for your games. If the game turned out to be successful you could always piecemeal replace the commercial engine with bespoke via updates (inc. using money to hire people from the success).
> You need to test the gameplay concept as cheaply as possible.
There are some edge case concepts that I have found to be completely impenetrable on commercial engines.
Multiplayer is the #1 thing in my mind. I've gotten the FPS multiplayer examples to "work" for UE and Unity, but the confidence I have in these solutions is not great. I've seen what Unity can do (BattleBit), but I don't know how many decades of game industry black magic and hackarounds it requires to force a commercial engine to behave that well.
I really think the answer is that it depends. If you aren't trying to have a "perfect" version of some gameplay aspect (i.e. multiplayer in my case), then the commercial engines will absolutely be the fastest path to validating your idea. Put differently, I think some game ideas are not possible to validate on commercial engines. But, I don't think this is very common.
It took me about 3 days to build the engine, including custom physics, gravity/jumping, collision detection including raycasting, and about 3 weeks to finalize it and start working on the game. Game was released in under a month and we already have a small community. It was definitely worth spending a little up front time to avoid a service provider, and have more control over the mechanics and look/feel of the game.
By the way, you re-use a lot of the same code when you build in-house, like you said often times your 1st title is not a hit, but you improve upon a lot of that engine code and release subsequent games. Most AAA developers do this.
Now try to hire 20 level designers and 5 engineers who know how to work with your engine/tools.
Same applies to most software. It isn't that hard to make a new programming language or a http framework. But once you make one finding other people who know how to use it is a pain in the ass.
Level designers use tools, not really contributing at the game engine level. Long before I built Shadowvane I built a map editor in Three.js that anybody could use. If the game became profitable, in-house tooling would sophisticate and the team would be the ones shaping it. But yes if for some reason my first hire was a level designer, they could add value day 1.
Nobody needs Unity or Unreal.
My most recent game (playshadowvane.com) is built on 100% proprietary technology, I built everything in-house down to the physics.
Always ignore mainstream game dev advice, they are just trying to sell you products. Build as much as you can in-house, it's not only better for the look/feel of the game, but no company will be able to rug pull you later.