> how is it best to scope out something on the scale of capnp/kj in the context of a bigger company? Do you just give a team a year and let them run?
No, frankly, I think that would be a recipe for disaster.
The team needs to instead be tasked with a specific use case, and they need to build the platform to solve the specific problems they face while working on that use case. If you tell people to develop a platform without a specific use case, they will almost certainly build something that solves the wrong problems. Programmers (like humans in general) are mostly really bad at guessing which features are truly needed, vs. what sounds neat but really won't ever be used.
So, sadly, I think that businesses should not directly engage in such projects. But, they should be on the lookout for useful tools that their developers have built in the service of other projects, and be willing to factor those out into a separate project later.
Unfortunately, this all makes it very hard to effect revolutionary change in infrastructure. When the infrastructure isn't your main project, you probably aren't going to make risky bets on new ideas there -- you're going to mostly stick to what is known to work.
So how do we get revolutionary changes? That's tough. I suppose that does require letting a team run wild, but you have to acknowledge that 90% of the time they will fail and produce something that is worthless. If the business is big enough that they can make such bets (Google), then great. But for most tech companies I don't think it's justifiable.
No, frankly, I think that would be a recipe for disaster.
The team needs to instead be tasked with a specific use case, and they need to build the platform to solve the specific problems they face while working on that use case. If you tell people to develop a platform without a specific use case, they will almost certainly build something that solves the wrong problems. Programmers (like humans in general) are mostly really bad at guessing which features are truly needed, vs. what sounds neat but really won't ever be used.
So, sadly, I think that businesses should not directly engage in such projects. But, they should be on the lookout for useful tools that their developers have built in the service of other projects, and be willing to factor those out into a separate project later.
Unfortunately, this all makes it very hard to effect revolutionary change in infrastructure. When the infrastructure isn't your main project, you probably aren't going to make risky bets on new ideas there -- you're going to mostly stick to what is known to work.
So how do we get revolutionary changes? That's tough. I suppose that does require letting a team run wild, but you have to acknowledge that 90% of the time they will fail and produce something that is worthless. If the business is big enough that they can make such bets (Google), then great. But for most tech companies I don't think it's justifiable.