I took over a team that was struggling with a very large architectural migration that had been going for a couple years. Two years later we have largely gotten things back to a healthy state, though we have only achieved maybe 20% of the original technical ambitions, the team is an order of magnitude stronger than when it started, which in many ways is more important than the exact state of the system. The migration introduced two major new technologies being incubated by outside infra teams, a new data model meant to coalesce 500+ fields comprised of data stored from a dozen or more databases, serving hundreds of clients across dozens of teams, and exposing data on hundreds of customer facing surfaces representing both sides of a C2C marketplace.
The first thing I would say is take all advice you get with a huge grain of salt. Details matter, and the particular details that matter the most vary tremendously from project to project. That said, here's my advice:
- Be clear on the goals up front and along the way. It's already a red flag that you don't lead with the goal and say things like "I don't have many details to share as I don't know what's relevant". In the heady early days of a big project, there will be many rose-tinted ideas of problems that can be solved, and people will keep tacking them on without the burden of knowing the stumbling blocks that will inevitably come. You need to keep the goal in mind at all times so you can ruthlessly make tradeoffs every step of the way. It's even okay if the goal changes, but be explicit about it.
- Make sure you find a way to do it incrementally. If you find you have code accumulating that is not being exercised in the running system for more than a few weeks at a time, that's a huge red flag. Kent Becks Trough of Despair [1] from a few days ago is relevant to this point. You need to be very careful that your trough doesn't grow wider than you can handle. It's surprisingly easy for that to happen given the nature of software system complexity growth. The risk is even greater if you have a lot of resources at your disposal because more cooks in the kitchen means hire risk of losing cohesion.
- There's no substitute for seniority up and down the chain. One or two weak links can really derail the entire effort. And it's not just about technical strength, communication and social aspects are equally important. Every single front line engineer will likely run into issues that will be relevant outside of their scope, but will they recognize that for areas they are not focused on? When a project is too big for any one individual to understand all the details, you need a critical mass of big picture thinkers, and some lightweight ways for informal conversations to be sparked and escalated (or de-escalated) as the importance comes into focus
- If you ever ask an engineer why they're doing something and the answer is "because XXX told me to" or "because that's the plan", it's time for a quick sit-down. Engineers who don't know why they're doing something will not make good choices when the unforeseen arises (which it always does).
- Know your clients. Are they internal, external? Are there ghost or second-order clients due to leaking internal details or other encapsulation violations? Will you still support all the features they need? What actions will they need to take to support you? What rate of change can they support? You can have the perfect end-state in mind, and then get tripped up by mundane constraints on your clients that you were not fully aware of.
The first thing I would say is take all advice you get with a huge grain of salt. Details matter, and the particular details that matter the most vary tremendously from project to project. That said, here's my advice:
- Be clear on the goals up front and along the way. It's already a red flag that you don't lead with the goal and say things like "I don't have many details to share as I don't know what's relevant". In the heady early days of a big project, there will be many rose-tinted ideas of problems that can be solved, and people will keep tacking them on without the burden of knowing the stumbling blocks that will inevitably come. You need to keep the goal in mind at all times so you can ruthlessly make tradeoffs every step of the way. It's even okay if the goal changes, but be explicit about it.
- Make sure you find a way to do it incrementally. If you find you have code accumulating that is not being exercised in the running system for more than a few weeks at a time, that's a huge red flag. Kent Becks Trough of Despair [1] from a few days ago is relevant to this point. You need to be very careful that your trough doesn't grow wider than you can handle. It's surprisingly easy for that to happen given the nature of software system complexity growth. The risk is even greater if you have a lot of resources at your disposal because more cooks in the kitchen means hire risk of losing cohesion.
- There's no substitute for seniority up and down the chain. One or two weak links can really derail the entire effort. And it's not just about technical strength, communication and social aspects are equally important. Every single front line engineer will likely run into issues that will be relevant outside of their scope, but will they recognize that for areas they are not focused on? When a project is too big for any one individual to understand all the details, you need a critical mass of big picture thinkers, and some lightweight ways for informal conversations to be sparked and escalated (or de-escalated) as the importance comes into focus
- If you ever ask an engineer why they're doing something and the answer is "because XXX told me to" or "because that's the plan", it's time for a quick sit-down. Engineers who don't know why they're doing something will not make good choices when the unforeseen arises (which it always does).
- Know your clients. Are they internal, external? Are there ghost or second-order clients due to leaking internal details or other encapsulation violations? Will you still support all the features they need? What actions will they need to take to support you? What rate of change can they support? You can have the perfect end-state in mind, and then get tripped up by mundane constraints on your clients that you were not fully aware of.