> 1) Developers cannot reliably estimate software timelines. That often comes with experience and age, but many never learn that skill. It is a skill like programming itself. It comes with practice.
This is true. But this is the problem that Agile is supposed to help solve -- with the caveat that most companies don't do true Agile. I guess people are now calling it Continuous Delivery, but the idea is the same: you don't tell people when a feature will be done, only when you will be working on it. But even then, even with experienced estimators, the rule of thumb is to add 15-30% to their estimates.
> 2) Break down in communication between management (product managers) and engineers who are writing code.
Totally agree that this is very common. Unfortunately, the best way to solve this is with process, and developers hate process. But developers often don't realize that they can use the process to their advantage: a strong engineering lead can refuse to accept poorly defined requirements. That allows you to push back on product to force them to define the requirement better.
IMO this breakdown is caused by just that: developers who say "oh yeah, I can do that", then start working on the problem and realize that there are a lot of undefined edge cases. If they promised it to the client in 6 months, fine, but define what "it" is.
From the perspective of an MBA, if we have to bring in a contractor to implement it, fine. I know where I can go to hire certified rockstar contract developers at $200/hr, which I will do if the client is lucrative enough. There is no feature I can think of that would take more than 6 months of development.
Process doesn't work, it can't work, because Process isn't the problem, it's top-down management infrastructure. Process is orthogonal to that.
In a top-down management style, your so-called "Lead Developer" is just the first head on the chopping block. They're leaders of puppet regimes. If the real leaders don't like them, they install a new one. It comes in many forms: layoffs, "performance" related dismissal ("you're not getting enough done!" because of all the rejected requirements docs), even reassignment to "re-engineering initiatives" that have no real chance of seeing the light of day.
Finding a management infrastructure that actually cares about feedback is incredibly rare. I have personally never found it, in 10 years across 6 companies. No amount of process is going to change a bad manager.
Any company of a certain size must convert to a top-down management infrastructure. I've seen what happens in large companies that don't -- you get 15 teams building different versions of the same product, and internal politics decide whose version becomes the 'official' one. But invariably, the internal user community settles on the one that's actually best for their use case, so you end up having to support 3 or 4 versions of every product you create. It's an absolute support and integration nightmare.
You talk about a business like it's supposed to be a democracy. It's not. You need clear direction, and democracy is messy and opaque if your company is too big for you to know the name of everyone who works there. Now the way you describe top down management is a bad implementation of it -- executives need to work to develop their middle managers -- but every company eventually has to convert from a democratic decision making process to a chain of command.
I've been working recently with a number of household-name startups that are having this exact issue. They've hit a plateau in their growth because the culture is dead-set against top-down management, but the company is now too big for people to self-organize. Engineering initiatives that would have taken 2 months in startup mode now take 2 years because they have to go back and rebuild half their systems to make them flexible enough to support the multiple use cases that they now have. Or because they were a startup, they didn't worry about geographic redundancy because that wasn't part of the MVP. So it was put off for later, and now they're a billion dollar company operating out of a single datacenter because their databases and applications were built without multi-datacenter being a requirement. Product can't even reliably tell engineering what features to implement because nobody on the product side has enough authority to shout over the crowd.
So having an infrastructure that cares about feedback is good, but the infrastructure also needs to empower people to do something about said feedback.
I know, I really understand what you're saying. And I am not saying a democracy is a better option. But I just don't think the things you're describing are attributable to top-down management. It certainly has never prevented the problems you've mentioned in any of the places I've ever worked. I agree that clear direction is needed, but your garden-variety middle manager isn't going to provide that. What he will usually provide is a sense that the failures of the organization are the fault of the leaf-nodes.
For every SV-brand startup, there are dozens of small-time consultoware shops serving their local market. They're the kind of places that have been around for 20 years and still only have 100 employees each. They can't ever seem to grow beyond that size because the one-way communication flow from sales to development keeps them in massive technical debt.
Middle managers don't provide direction; they are communication points. As to how effective they can be, it all depends on how the org is structured. If it's a product-centric organization (i.e. P&L rolls up under product and operating costs are charged back to the product org) then it can work pretty well. If it's a functional organization (revenue rolls up through a sales org on a regional basis and is combined with costs at the corporate level) then there's really no amount of management that will make it a good place for developers to work -- under the latter model, finance is running the company and you're a pure cost center. The manager's primary focus will be cost control, not product quality. That's a bad situation to work under.
I agree with your second assessment, but it's more of an issue with the consultoware business model than anything. Most consultoware shops start up thinking they're going to get hockey-stick growth, but it turns out every implementation is super-custom and it ends up as a consulting business. Many companies fail to realize this has happened, and end up operating like a software company when they should be operating like a professional services company with hourly billing. Under a professional services model, if a customer wants a feature, you can quote them a price for developing and implementing it. Tech debt gets dealt with because it becomes a bottleneck for revenue growth when you can't charge clients to add functionality.
Very quick reply because I have to get to bed (it's 1:30 am here!). I shouldn't work so hard... :-P
Estimation (i.e. pulling numbers out of your posterier) is not necessary (or usually desirable). What you want to do break down the problems into roughly equal size pieces -- the more the better. Each piece will take a certain amount of time. You can measure this and record the average. This becomes the estimated amount of time for each piece. Obviously the estimate has an error with some kind of distribution (my sleep adled brain is unable to remember what a likely distribution for the amount of time something takes).
It doesn't really matter because the sum of all the pieces will give you an estimate for the overall completion time for all the pieces. This estimate will have a normal distribution.
If you write down a list of all the pieces in priority order, you can draw a line at one piece of functionality and determine when it is likely to be completed, along with error bars.
Of course you are unlikely to actually to do the things on your list because if you were able to correctly guess all your requirements before you started the project then you would be in fantasy land. So what you do is tell the stake holders that this is what you are going to do unless they tell you that they want to change their mind.
Of course when you start they will say, "We're not going to change our mind. We're 100% certain that this is right". To which you reply, "Excellent". Every 2 weeks or so, once you have done some work and can show it to the stake holders, you should also have a list of new ideas. You should explain these new ideas and say, "Do you want to change the list to add these new ideas". Whether they say "Yes" or "No", you should reply with, "Excellent" and carry on.
Often the stake holders will have an "emergency" and decide that they have new opportunities which change your list. To this you should say, "Excellent" and change the list.
You should release whenever your stake holders are happy to release.
In this way, you have given the best estimate you can to help the stake holders plan. Any changes to the plan will be made by the stake holders themselves at the earliest possible time. You don't get caught up trying to deliver at any particular date because the stake holders are aware that they are changing the deliverables. You may have to ask them where they want to draw the line and give them new estimates for completion every time they change something, just to reinforce this idea.
The really tricky bit is breaking the work down into small enough pieces. This is very, very difficult and requires a lot of experience. The sucess or failure of the above will very likely rest on the ability to do this, so expect to pay a lot of money for someone with a proven track record doing it. It is often worth hiring a consultant if you have nobody on staff with this experience.
This is true. But this is the problem that Agile is supposed to help solve -- with the caveat that most companies don't do true Agile. I guess people are now calling it Continuous Delivery, but the idea is the same: you don't tell people when a feature will be done, only when you will be working on it. But even then, even with experienced estimators, the rule of thumb is to add 15-30% to their estimates.
> 2) Break down in communication between management (product managers) and engineers who are writing code.
Totally agree that this is very common. Unfortunately, the best way to solve this is with process, and developers hate process. But developers often don't realize that they can use the process to their advantage: a strong engineering lead can refuse to accept poorly defined requirements. That allows you to push back on product to force them to define the requirement better.
IMO this breakdown is caused by just that: developers who say "oh yeah, I can do that", then start working on the problem and realize that there are a lot of undefined edge cases. If they promised it to the client in 6 months, fine, but define what "it" is.
From the perspective of an MBA, if we have to bring in a contractor to implement it, fine. I know where I can go to hire certified rockstar contract developers at $200/hr, which I will do if the client is lucrative enough. There is no feature I can think of that would take more than 6 months of development.