That's simply not true. I've inherited something just as bad as this. We did a full rewrite and it was quite successful and the company went on to triple the revenue.
> get some testing in place
Writing tests for something that is already not functional, will be a waste of time. How do you fix the things that the test prove are broken? It is better to spend the time figuring out what all the features are, document them and then rewrite, with tests.
The problem with people new to the company starting a rewrite from scratch is that they often are poorly informed on why things were the way they were before. If you start big, you can have bad outcomes where the new system might be objectively worse than the old one... but you are stuck trying to get the new thing out for the next 5 years because too many people sunk too much political capital into it.
As an example, I worked at an ad-tech startup that swapped it's tech team out when it had ~100 million in revenue (via acqui-hire shenanigans). The new tech team immediately committed to rewriting the code base into ruby micro-services and were struck by strange old tech decisions like "why does our tracking pixel return a purple image?". The team went so far as to stop anyone from committing to the main service for several years in a vain attempt to speed up the rewrite/architecture migration.
These refactors inevitably failed to produce a meaningful impact to revenue, as a matter of fact the company's revenue had begun to decline. The company eventually did another house cleaning on the tech team and had some minor future successes - but this whole adventure effectively cost their entire Series D round along with 3 years of product development.
You're making a silent assumption that the original team is well informed about why the things are like they are and that they know what they are doing. I think it is not always the case.
I've been to a project once where the mess in the original system was the result of the original team not knowing what they were doing and just doing permutation based programming - applying random changes until it kinda worked. The situation was very similar to that described by the OP. They even chose J2EE just because the CTO heard other companies were using it, despite not having a single engineer knowing J2EE. Overall after a year of development the original system barely even worked (it required manual intervention a few times per day to keep running!), and even an imperfect rewrite done by a student was already better after 2 weeks of coding.
So I believe the level you're starting the rewrite from is quite an important factor.
Then of course there is a whole world of a difference between "They don't know what they are doing" vs "I don't like their tech stack and want to master <insert a shiny new toy here>". There former can be recognized objectively by:
The original team may not have been the best at the task, but they still managed to deliver 100 MM in revenue. Sometimes the things they leave behind/ignore simply don’t matter to the business/useful tech.
Particular to ad tech, the lifespan of any particular software is lower than you’d expect (unless your google/Facebook). Technology that pays out big one year will become pretty meh within 3 years. In the case above I’d argue that the new tech team didn’t really understand this dynamic and so they focused on the wrong things such as rewriting functionality that didn’t matter for the future. Or making big bets on aspects of the product which were irrelevant.
To the OP, we don’t know that the lifespan of any of these php files is greater than an individual contract. If the business can be modeled as solve a contract by putting a php file on prod - rewriting may be entirely worthless as the code can be “write once, read never”.
We (my good friend and I who both have 20+ years of experience) were brought in specifically do to the rewrite. We were new to the company. We actually had to rebuild the entire IT department while we were at it as well.
> new tech team immediately committed to rewriting the code base into ruby micro-services
well... sigh.
> These refactors inevitably failed to produce a meaningful impact to revenue
It sounds like less about the refactor itself and more about the skills of the team doing the refactor. You certainly can't expect a refactor to go well if the team makes poor decisions to begin with.
> It sounds like less about the refactor itself and more about the skills of the team doing the refactor. You certainly can't expect a refactor to go well if the team makes poor decisions to begin with.
This has been my biggest struggle with rewrites where I’m currently working. We have several large, messy old codebases that everyone agrees “needs a rewrite” to (1) correct for all the early assumptions in business needs that turned out wrong, (2) deal with old PHP code that is very prone to breakage with every major new PHP version released, and (3) add much needed architectural patterns specific to our needs.
I’ve seen rewrites of portions of the project work when they involve myself and one other mid-level dev who has a grasp on solid sw engineering practices, but when the rest of the (more senior) team get involved on the bigger “full rewrite”, they end up quickly making all the same mistakes that led to the previous project being the mess that it is.
Sure, it will be using fancy new PHP 8 features, and our Laravel framework will force some level of dependency injection, but the you start seeing giant God classes being injected over here, but duplicated code copy-pasted over there, all done by “senior” devs you feel you can’t question too strongly.
To that end, an open and collaborative culture in which you start the rewrite with some agreed upon principles, group code reviews and egos kept in check, are all necessary for this to work.
You have a great experience and did a great job indeed. My only question is how does one get 20 years of such experience without horrific flashbacks of “let’s just rewrite it” decisions. Do you do rewrites/redesigns often? What’s your success rate?
I've done what I would consider as four rewrites that I can remember as large events in my life (although not fully what you'd expect). But all are good stories in my opinion.
First one was the above example. It was for the largest hardcore porn company on the planet. Myself and my good friend Jeff rebuilt an already successful business IT department from the ground up and made it even more successful. Ever heard of 'the armory in sf'?
Second was that jeff and I were hired as contractors by Rob @ Pivotal Labs (ceo) to help the CloudFoundry team rewrite itself after he had bought the team and trimmed it down to only the good people. That one was a huge mess. We spent a lot of time deep in shitty ruby code using print statements trying to figure out what 'type' of an object something was and, of course, backfilling tests. It was a fun project and both Jeff and I learned the Pivotal way, which was probably the most enlightening thing I had ever learned about how to develop software correctly from a PM perspective. If you want to improve your skills beyond just slinging code, spend some time figuring their methodology out. Much of it is documented in Pivotal Tracker help documentation and blog posts.
Third one was not really a rewrite, but the original two founders, who were not technical, had tried to hire a guy and got burned because the guy couldn't finish the job. Sadly, they had already paid the person a ton of money and got really nothing functional out of it. We (jeff and I again!) just started over. We did a MVP in 3 months (to the exact date, because we both know how to write stories using pivotal tracker and do proper estimates) and ended up doing $80m in revenue, in our first year with an initial team of 5 people.
Fourth one was three guys (who were also not technical) I kind of randomly met after I moved to Vietnam. They were deploying litecoin asic miners into a giant Vietnamese military telco (technically, they are all military). They had hired another guy to do the IT work and he was messing it all up. They invited me out to help install machines, I came out, rebuilt their networking layout and then proceeded to eventually fix their machines because the software 'firmware' that was on them was horrible. I also added monitoring with prometheus so that I could 'see' issues with all these machines. That first day on the job, they fired the other guy and made me CTO. We ended up deploying in another datacenter as well. It was a really wild experience with a ton more stories.
Life has been, um, interesting. Thanks for reading this far.
Not everything has been a success. For example, unless you're stupid rich and can afford years of losses, never start/own a night club or you might end up working for the rest of your life to pay off your debts.
The problem is that most developers are crap and self centered on working with the tech they like.
You need to work with someone who doesn't care about filling up their CV with "ruby microservices" and get stuff done.
If I went into a business to do a rewrite and decided to use $shinyNewTech because I want to build up rust experience I'd probably end up wasting years with little results.
My best wrong answer is that there were different colored pixels for different front-end versions, and the app had some radically different responses depending on the version. Maybe MENA would return white, SE Asia green, people who signed up during a sale would return blue, whatever. After a while, the other pixels were removed and only one shade of purple were used for everyone, but the code for processing them was not removed. So now, if the tracking pixel is not a precise shade of purple, some unexpected shenanigans ensue.
I worked on an app once that used two different, equally ancient libraries to a) generate thumbnails and b) create a png from a pdf. While modifying part of this process I started realizing that there were conditions where you'd get a PDF thumbnail at the end, but its output had a red tint to it.
Input looked fine and invoking each step manually worked fine as well.
Come to find out that certain PDFs contained color calibration information that, combined with how we were calling it, would treat ARGB as RGB. The input would have transparency info defined and the thumbnail generator would happily repurpose the alpha channel as the red channel instead.
I did a rewrite of a 30 year old bit of perl/php2 over the last year. Not knowing why things were the way they were was really useful for the younger team members and me to get familiar with the codebase and the business context.
Anecdotal: I asked people why they keep incorrectly using jQuery methods and produce ambiguous, difficult-to-maintain code in the year of 2022. (we still have jQuery as a dependency for legacy code.) The response was that they were not aware that native counterparts like document.querySelectorAll exist in the browser. They just copied the old jQuery code, modified them, and it worked.
I am pretty sure this kind of thing exists in any large legacy codebase.
You don't need comprehensive tests for tests to start delivering value.
Figure out the single most important flow in the application - user registration and checkout in an e-commerce app, for example.
Write an automated end-to-end test for that. You could go with full browser automation using something like Playwright, or you could use code that exercises HTTP endpoints without browser automation. Either is fine.
The value provided here immense. You now have an early warning system for if someone breaks the flow that makes the money!
You also now have the beginnings of a larger test suite. Adding tests to an existing test suite is massively easier then starting a new test suite from scratch.
You're assuming the existing flow is working perfectly and I agree with you that testing is a godsend. I constantly yell that testing is great. Heck, I even worked for Pivotal Labs that does TDD and pair development, and loved it.
Let's say you start to write tests and start to see issues crop up. Now what? How do you fix those things?
Github actions!? They don't even have source control to begin with. There are so many steps necessary to just get to that point, why bother?
If the existing code base already has extremely slow movement and people are unwilling to touch anything for fear of breaking it... you're never going to get past that. Let's say you do even fix that one thing... how do you know it isn't breaking something else?
It is a rats nest of compounding issues and all you are doing is putting a bandaid on a gushing open wound. Time to bring in a couple talented developers and start over. Define the MVP that does what they've learned their customers actually need from their 'v1' and go from there. Focus on adding features (with tests) instead of trying to repair a car that doesn't pass the smog test.
> Let's say you start to write tests and start to see issues crop up. Now what? How do you fix those things?
I assumed the tests wouldn't be for correctness, but for compatibility. If issues crop up, you reproduce the issues exactly in the rewrite until you can prove no one depends on them (Chesterton's fence and all).
The backwards-compatibility-at-all-costs approach makes sense if the product has downstream integrations that depend on the current interface. If your product is self-contained, then you're free to take the clean slate approach.
> I assumed the tests wouldn't be for correctness, but for compatibility.
You're assuming that the people coming in to write these tests can even make that distinction. How do you even know what the compatibility should be without really diving deep into the code itself? Given how screwed up the codebase already is, it could be multiple layers of things work against each other. OP mentioned multiple versions of jquery on the same page as an example.
Writing tests for something like that is really a waste of time. Better to just figure out what's correct and rewrite correct code. Then write tests for that correct code... that's what moves things forward.
> How do you even know what the compatibility should be without really diving deep into the code itself?
You can pretty much black-box the code and only deep dive when there are differences. Here's what I've done in the past for a rewrite of an over-the-network service:
1. Grab happy-path results from prod (wireshark pcap, HTTP Archive, etc), write end-to-end tests based on these to enable development-time tests that would catch the most blatant of regressions.
2. Add a lot of logging to the old system and in corresponding places in the new system. If you have sufficient volumes, you can compare statistical anomalies between the 2 systems
3. Get a production traffic from a port mirror, compare the response of your rewritten service against the old service one route at a time. Log any discrepancies and fix them before going live, this is how you catch hard-to-test compat issues
4. Optionally perform phased roll out, with option to roll back
5. Monitor roll out for an acceptable period, if successful, delete old code/route and move to the next one.
The above makes sense when backwards compatibility is absolutely necessary. however, the upsides is once you've set up the tooling and the processes, subsequent changes are faster.
All of that, while technically correct and possible, is vastly more complicated and time intensive than a rewrite of what the OPs description of the codebase is.
Yes, it absolutely is - but the trade off is a far lower risk of introducing breaking changes. Depending on the industry/market/clients - it may be the right tradeoff
In my eyes, a rewrite won't be introducing breaking changes. It would be to figure out what functionality makes money, then replicate that functionality as best as possible so that the company can continue to make money as well as build upon the product to make even more money.
We're talking about a webapp here, not rocket science.
The biggest problem isn't even the codebase in this situation.
When you keep finding bugs like that while refactoring and making things better, it will demoralise you. The productivity will stop when that happens.
It also require above average engineers to fix the mess and own it for which there is not much benefit.
Your refactoring broke things? Now it's your turn to fix it and also ship your deliverables which you were originally hired for. Get paged for things that weren't your problem.
If I was a manager and assigned this kind of refactoring work, I will attach a significant bonus otherwise I know my engineers will start thinking of switching to other places unless we pay big tech salaries.
People keep quoting Joel's post about why refactoring is better than rewrite but if your refactor is essentially a rewrite and your team is small or inexperienced - it's not clear which is better.
Parallel construction and slowly replacing things is a lot of unpaid work. Just the sheer complexity of doing it bit by bit for each piece is untenable for a 3 person team where most likely other two might not want to get into it.
> It also require above average engineers to fix the mess and own it for which there is not much benefit.
That's not true, it doesn't require above average engineers. It requires a tech lead that has the desire and backing to make a change, and engineers willing to listen and change. It doesn't require a 10x engineer to start using version control, or to tell their team to start using version control for example..
One would think, but how do you go from source control to deployment on the production server though? If they were editing files on the server directly, there could be a whole mess of symlinks and whatever else on there. Even worse, how do you even test things to see if you break anything?
Just start somewhere. These guys are making changes, actual functional changes and bug fixes in that environment meaning they already have all the problems you imagine are going to get in the way of fixing this mess. So stop fretting and just start small with one tiny thing. It doesn't really matter with what. You don't even need automated tests necessarily. It's a small simple flow that needs 10 minutes to run the same test steps manually for? Write them down and do it manually, I don't care. Just Do it.
Been there, done that. Slightly differently where they had a test server and prod server. So already better except one day I made a change and copied to prod. Yes it was manual. Just scp the files over to prod. And stuff broke. Turned out someone had fixed a bug directly in prod but never made the change on the test server.
First thing I did was to introduce version control and create a script to do make deployment automatic meaning it was just a version control update on Prod (also scripting languages here). Magically we never had an issue with bugs reappearing after that.
Pretty simple change and you can go from there.
The above code base was over 20 years old and made use of various different scripting languages and technologies including some of the business logic being in stored procedures. Zero test coverage anywhere. You just 'hide' small incremental changes to make things better in everything you do. Gotta touch this part because they want a change? Well it could break anyhow so make it better and if it breaks, it breaks and you fix it. It needs judgment though. Don't rewrite an entire module when the ask was adding a field somewhere. Make it proportional to the change you need to make and sometimes it's not going to be worth it to make something better. Just leave it.
Not sure the little hammer will fix much. And making folks use a method in new code pisses them off. "You say important I do this your way this time, even though there are 1000 examples of doing it the other way. I feel persecuted and your way is pointless, because it doesn't fix everything anyway. And its slowing me down and making me look bad."
Not rational but folks don't have to explain their feelings. You will be hated.
The little hammer definitely fixes. It does it in the same way as water cut the grand canyon. The beauty is that it works over time.
Now as for how to get the other devs on board, I agree with you that you can't just barge in and tell them everything they are doing is wrong etc. I never said to do that and I'm replying to a specific comment in the thread not the original Ask HN.
I.e. when I write about what I've done in the past, I got buyin from my boss and my colleagues on what I was going to do. But I didn't just sit there and kept doing what they had done over the past years. I changed lots of other little things too in the same manner.
So if we do want to talk about the original Ask HN and how to get the existing employees not to hate you, you can start by letting them tell you about what they think the problems are. What are their pain points. They might just not know what to do about them but actually see them as problems too. Maybe they've tried things already but failed or got shot down by others in the company. Maybe they did try to introduce version control but their non tech boss shot them down.
Of course it may not work out. Some people really are just stupid and won't listen even if you try to help them and make them part of the solution.
I'm not sure where you're pulling that from. There's no mention of startup here. Neither in the original (actually the opposite I'd say, 12 years and just a business unit).
None of what I said is a big-company process in any way. If in your book using source control is a big company process that will sink a startup then be my guest and I will just hope we never have to work together. Source control is a no-brainer that I even use just for myself, have used in teams of two and teams dozens to hundres. The amount of process around is what scale with the kind of company. Source control is useful by itself in every single size of company.
Code review, coding standards, required tests for everything, multiple stages of deployment - are not simple and can stall development. Done wrong they can sink a company.
It's easy to read the worst possible construction on what other people write here. It's never a good idea.
Btw I worked at a startup for 8 years. It was still a startup, depending on new investment to meet the monthly. In any case the described dev group was behaving in a way that used to be typical of startups. And even business units in larger organizations have runway.
Yeah, lot of worms...and if while refactoring things break. You are on the hook for scanning through that complex monster at 3 am and finding the issue and fixing it for no additional pay in most cases.
They can literally copy the whole directory from their local machine to production as a first step for all I care.
How do they test things on production? If there’s a bug how do they revert to the previous version? There are way more issues without source control than with.
Doesn't Git support symlinks? Empty directories could be trouble though. One would have to put a .GITKEEP into every directory before checkin, and a step at deployment time to remove them again.
How do 2 junior devs manage to rewrite the entire product while also meeting the ongoing goals of the business?
You're trying to spec features on a moving target.
Even if they were able to do 50% time on the rewrite you'll never actually get to feature parity.
The only viable plan, unless the company has an appetite to triple the dev headcount, is to set an expectation that features will have an increased dev time, then as you spec new features you also spec out how far you will go into the codebase refactoring what the new features touch.
But it is functional. Grandparent post is suggesting that all the currently used functionality should have tests written for it. It makes sense, as that way they can gather the requirements of a rewrite at the same time.
We don't know that it is functional... maybe the company is only making $20m and should be making $60m. Like I said, we tripled the revenue with a rewrite.
What we did was make the case that we could increase revenue by being able to add valuable features more easily/quickly. We started with a super MVP rewrite that kept the basic valuable features, launched, then spent the rest of our time adding features (with tests). Hugely successful.
The key, of course, will be to get 1-2 top notch developers in place to set things up correctly from the beginning. You're never going to be effective with a few jr's who don't have that level of experience.
> We don't know that it is functional... maybe the company is only making $20m and should be making $60m. Like I said, we tripled the revenue with a rewrite.
It's $20m functional. It's possible it could be better but unless this is the kind of huge org where 20m is nothing (doesn't sound like it) you really need the behaviors documented before you start screwing with it. It's very likely this thing has some pretty complex business logic that is absolutely critical to maintain.
> you really need the behaviors documented before you start screwing with it. It's very likely this thing has some pretty complex business logic that is absolutely critical to maintain.
Nothing I said suggested otherwise. Absolutely critical for whomever is doing a rewrite to understand everything they can about the application and the business, before writing a single line of code.
You sound frustrated that you've joined a company with an absolute stinker of a codebase, because you're confident you could deliver much better results having refactored it first. You're managing a group of people probably enormously under-productive because of the weight of the technical debt they're under. Every change takes months. It's riddled with hard-to-fix bugs. It's insecure. There are serious bus factor problems.
Many of us have been in this exact position before, multiple times. Many of us have seen somebody say "our only choice is a full rewrite" - some of us were the one making that decision. Many of us have seen that decision go disastrously wrong.
For me, the problem was my inability to do what I'm good at: write tests, write implementations that pass that test, etc. Every time I suggested doing something, somebody would have a reason why that would fail because of some unclear piece of the code. So rather than continuously getting blocked, I tried to step into my comfort zone of writing greenfield code. I built a working application that was a much nicer codebase, but it didn't match the original "spec" from customer expectations, so I spent months trying to adjust to that. I basically gave up managing the team because I was so busy writing the code. In the end, I left and the company threw away the rewritten code. They're still in business using the shitty old codebase, with the same development team working on it.
If you really want to do the rewrite, accept how massively risky and stressful it will be. The existing team will spend the whole team trying to prove you were wrong and they were right, so you need to get them to buy into that decision. You need to upskill them in order to apply the patterns you want. And you need to tease apart those bits of the codebase which are genuinely awful from those that for you are merely unfamiliar.
“I won the lottery, you can too. If you don’t buy a ticket, you’re never gonna win right…?”
There is a lot of evidence rewrites are hard to do well, and especially prone to failure.
…you might pull it off, it’s not impossible, sure. …but are you seriously saying it’s the approach everyone should take because it worked for you once?
Here my $0.02 meaningless anecdotal evidence: I’ve done a rewrite twice and it was a disaster once and went fine the second time. 50% strike rate, for me, personally, on a team of 8.
What’s your rate? How big was your team, how big was the project? What was the budget? Did you do it on time and on budget? It’s pretty easy to say, oh yeah, I rewrote some piece of crap that was a few hundred lines in my spare time.
…but the OP is dealing with a poorly documented system that’s very big, and very important and basically working fine. You’re dishing out bad advice here, because you happened to get lucky once.
Poor form.
Good advice: play it safe, use boring technology and migrate things piece by piece.
Big, high risk high reward plays are for things you do when a) things are on fire, or b) the cost of failure is very low, or c) you’re prepared to walk away and get a new job when they don’t work out.
> How do you fix the things that the test prove are broken?
Uhm. The tests don’t do any such things.
> It is better to spend the time figuring out what all the features are, document them
Yes. And the tests you should write are executable documentation showing how things are. It is like taking a plaster cast of a fossil. You don’t go “i think this is how a brachiosaurus fibula should look like” and then try to force the bones into that shape. You mould the plaster cast (your tests) to the shape of the fossil (the code running in production). Then if during excavation (the rewrite) something changes or get jostled you will know immediately that it happened, because the cast (the tests) no longer fit.
> We did a full rewrite and it was quite successful and the company went on to triple the revenue.
Which sure beats some other company coming along and "rewriting" the same or similar functionality in a competing product and killing your own revenue.
But it does come down to how big the codebase is and how long it would take for an MVP to be a realistic replacement. If there are parts that are complex but unlikely to need changing soon you can usually find ways to hide them behind some extra layer.
Is there any reason you couldn't just introduce proper processes (source control, PRs, CI/CD etc.) around the existing code though?
Kudos to you for successfully delivering in a similar situation. That said, I think your advice is a bit cavalier. The industry is littered with the carcasses of failed rewrites. The fact that you have done it in one context does not mean that this team can pull it off in another.
I'll also say there's a lot of semantics at play here. What is a "rewrite", what is a "test" vs a "document", what is "functional"? I read your main point being that one should avoid sunk-cost fallacy and find the right places to cut bait and write off unsalvageable pieces. The art of major tech debt cleanup is how big of pieces can you bite off without overwhelming the team or breaking the product.
> Writing tests for something that is already not functional, will be a waste of time.
This is not TDD; it's writing tests to confirm the features that work now. Then, when you make changes, you can get an early warning if something starts going south.
Of course a full rewrite can be successful. This is the problem when people base their entire critical thinking on blog posts. They then go on to preach it everywhere as well!
The blog posts are warnings about what not to do. People, naturally, when they don't fully understand something or can't grasp the complexity of something want to rebuild. Because writing also helps us understand that is what we are building. But its a trap, what you've rewritten will never be the same as before and there lies the footguns.
The blogs are plainly stating, "even though you feel you should rewrite, you probably shouldn't."
It is a 12 year old legacy product. What specification exists other than, "Yesterday it did X when I clicked the button, but now it does not do that anymore."
This is the point: I don’t TDD, but i am a big fan of tests. I’m this case the incorrect spec can be flagged, but all the other incorrect specs will also be there. If your Fix doesn’t break a spec, great, but if it does you can check if that spec was correct. It’s a back and forth between code and business requirements
You must have missed the part where it makes 20M revenue per year.
I gotta love hacker news, people who think the fact a backend is written in horrid PHP means it is "already not functional" while they spend their days learning something like Haskell that make them negative revenue per year.
Who knows if that 20m revenue should be 60m? They could be held back greatly by the fact that the developers are not motivated to change anything.
I also don't know Haskell and have no desire to learn it. I prefer to build products in static compiled languages where I can more easily hire developers.
That's simply not true. I've inherited something just as bad as this. We did a full rewrite and it was quite successful and the company went on to triple the revenue.
> get some testing in place
Writing tests for something that is already not functional, will be a waste of time. How do you fix the things that the test prove are broken? It is better to spend the time figuring out what all the features are, document them and then rewrite, with tests.