One of my favorite lessons "learned the hard way" by someone else. You fundamentally cannot run a business on poor data practices. It will sink you before you realize what happened.
I like Whitehurst and have had the chance to meet him a few times. If you want to understand his ethos check out "The Open Organization". It details his transition from Delta to Red Hat and the transformation of his leadership style being around a lot of smart engineers and team players ready to constructively challenge each other. Good read for anyone interested in building inclusive company culture in tech.
Honestly, I thought IBM would be feel less rudderless with him at the helm. I wanted IBM to become more 'red' and not RH to become more 'big blue'- but it appears this isn't the case after all... seems like a short tenure.
Exception to this rule was NeXT purchasing Apple for negative $400 million.
Most of the NeXT senior staff (jabs, avi, etc) Went on to take over roles at Apple. They also brought in all the NeXT technology and threw out Apples' Nextstep/Openstep->Mac OSX
Steve Jobs had been gone for 12 years, there was plenty of culture clash and turf wars. Steve just managed to prevail.
When he returned, the workforce was pampered and attached to projects that weren't going anywhere. Steve removed a number of perks, and killed some of the biggest, most internally popular projects (Newton and OpenDoc come to mind). Lot's of people left.
But Steve came back to Apple as a savvy politician. He managed to get the board of directors in his back pocket, and sell his vision of Apple to the employees. Those that stayed, stayed because they came to believe in his vision, not because he placated them.
I will say this though, I think there's a truth underneath your statement. Apple attracted people who wanted to change the world. The culture may have been different, but there was still a core motivation in Apple employees that came from Steve's original vision of computing.
Minor correction: I wouldn’t call Newton an “internally popular product”. At least, working on Newton I didn’t feel that way. And just before SJ came back, Apple had spun Newton out into a separate company because it wasn’t really fitting in. SJ spun it back in and killed it.
The decision was in the hands of a board whose sole legal goal, these days, is securing maximal financial return for shareholders (and the shareholders agreed -- there was an explicit vote). A soul is a luxury that companies don't usually get to keep past IPO.
Sole? No. No matter how many times people repeat it. Or, at least, companies can and do spend money on things that can be argued to be in service of the long-term profit and image of the company. Otherwise, how could a company do charitable giving?
That said, if someone makes an offer for a public company, the board does pretty much need to put it to a shareholder vote although they can negotiate for a higher price and ultimately make a recommendation. But it's up to the shareholders. They're the ones that own the company.
This is why I'm in favor of founder controlled shares (Founder's Fund companies tend to do this - Zuckerberg at FB).
Founder's retain total voting control of the company and can do what they think is in the best interest of the company.
The board can complain and vote against it in favor of stupid short-term decisions that harm in the company, but in the end Zuck can tell them to fuck off and buy instagram because he thinks it's the right thing to do.
While I don't completely disagree, a Benevolent (or not) Dictator For Life can have it's own set of problems. And I'd not that it's generally not considered a great governance model for large open source projects--though that's obviously a somewhat different situation.
I agree founder led companies are more like dictatorships (though feudal earldoms may be a better analogy?), this becomes less true in older companies no longer lead by a founder. I agree there’s risk (founder could lose touch or go crazy), but I think that risk is less likely than bad board decisions by non-founders. It’s also bounded a bit by the ability to retain high quality talent.
I also agree that for collaborative open source projects a different approach is often preferable (as well as governments too obviously).
> And I'd [note?] that it's generally not considered a great governance model for large open source projects
Citation needed? Offhand, it's worked great for linux, and while perl and python have had some infamous problems, they don't seem particularly specific to the BDFL model, and they seemed to work well before that.
"long-term profit" is the long-term financial returns to shareholders -- and corporate philanthropy is generally held to be justified in B-school circles only in so far as it builds up marketable goodwill (again, adding shareholder value), not for its own sake.
This is why some companies and interests are better off staying private. SpaceX is a perfect example. Musk doesn't want to be accountable to shareholders for return on investment and annual results.
At that point though, just about any reasonable actions can be justified even if they're not in the interests of immediate quarter earnings per share. And that's not what most people arguing that the board is legally required to maximize profits are talking about.
That really shouldn't have been a surprise to anybody, certainly no one who has been through a corporate merger before.
The aquiring company always makes promises they won't change the aquired company, the aquired company's management parrots that line and in the end, the aquired company always gets subsumed into the aquiring company.
As someone who spent 8 years inside Red Hat, this is exactly why anecdotal "evidence" is garbage.
Complaining that an engineering company is full of mostly white males (in an industry which is full of mostly white mails) is an example of nothing. Red Hat started programs to push LGBTQ+ people in 2014, women in leadership in 2017, neurodiversity acceptance in 2018, black leadership in 2018. You may think all of those things are late, but as driven company missions which they spend real money on (and real money on community involvement on), it's not insignificant.
A "good-old-boy's club" at a company where only 25% of engineering was even in the US is a little mindblowing. The vast majority of RH's engineers when I left were in Pune, Brno, Tel Aviv, and scattered across Europe (eastern and western), with a reasonable showing from Brazil.
Open source is, for better or worse, a place where a cult of personality matters. "This is a bad idea and we shouldn't do it" isn't personal. It's how the community interacts. It's an invitation to argue back and say "this is a good idea because XXX". This is basically how any upstream mailing list works also.
While Red Hat did have some cultural problems, they were primarily around resistance to establishing an engineering presence in China (and it's hard to fault them for that -- my team was truly geographically distributed, and adding someone in APAC would have meant we literally never could have met at the same time), and regional cultural attitudes.
I left last summer, along with a lot of other people, but your comment is apropos of nothing Red Hat was from 2013-2021.
Take that appeal to authority elsewhere, and the insults with it. As someone who was also at Red Hat for eight years (2009-2017) I'd say GP might have been off base but certainly not garbage. I heard more grossly sexist comments (or saw them on IRC) there than anywhere else. We're talking stuff that would lead to immediate mandatory diversity training the first time, or instant dismissal the second, at many companies (including IBM). "Who did she have to blow to get that job" particularly sticks in my mind. Most people didn't even bat an eye because that kind of stuff is totally allowable in the open-source communities these folks were part of. I also saw at least one person fired for refusal to work with people he considered "beneath" him socially. You might say it's good that the company took action, but it took over a year for that to happen and I don't think that would have been the case at a truly inclusive company. I suppose you can dismiss these (and plenty more I could relate) as mere anecdotes, but you haven't even provided those so that would be pretty hypocritical.
The vast majority of people I knew at Red Hat ranged from industry-average to absolutely excellent, but overall I'd say even Facebook had a better internal culture in terms of diversity etc. Painting RH as anything more than a typical tech company in those areas is a bit absurd.
For sensitive issues, it's hard to imagine this will lead to a constructive, curious conversation. It would be interesting, for example, if you and the GP tried to examine why you have such different experiences? Maybe you were in different places, working for different people, and saw different things? Maybe different job descriptions? The world is always much bigger than what we imagine.
This is not a "sensitive" issue, unless you consider GP's opinion "sensitive". I don't. I also don't think it would lead to a curious, constructive conversation. There is probably nothing OP could offer which would change what the experience of 8 years in RH engineering showed me. At best, it would lead to another long reply where I tried to explain what RH was actually like versus GP's impression.
It's essentially GP posting on the Debian/OpenBSD/LK ML and saying "wow, these guys are arrogant". Well, they have strong opinions, and they need to justify them. If you don't have thick skin, you should probably avoid upstream discussions. It doesn't mean you should cast aspersions on the developers or their communities/companies. Linus is an incredibly nice person, but you wouldn't think it if all you saw was his mails blasting Intel for Spectre/Meltdown. You'd have a very different opinion of the man than who he is in 99% of interactions, and commenting about it on HN would be likely to get a reply similar to mine.
GP only had "anecdotal" experience with Red Hat versus actually working inside the company. When I started, I was worried that it would be different "behind the curtain". It was not. The other who mentioned junior engineers speaking up if they disagreed was spot on.
In 201x (2015? 2017? I dunno), RH changed their pay structure from being bimonthly to biweekly in the US to align with the rest of the world. There were hundreds of replies to the entire company debating whether it was better to pay us once a month and manage your money better, or once a week to be more consistent, or...
That's what the company was. It was not by the time I left. Granted, when I started, RH HQ was still on a college campus. If all you ever read was upstream mailing lists, you'd get the feeling that everyone (Linux, Theo [not that he works for RH, obviously], Lennart, whomever) other than Dan Walsh was arrogant assholes. That's not representative of who they actually are.
Painting Red Hat with a broad brush from incidental interactions versus the perspective of someone who spent nearly a decade there is dramatically different. Did it have problems? Sure. But none of the ones GP mentioned. The world is much bigger than what GP imagined from his anecdotal experience.
RH was a unicorn, and it's gone. LinkedIn last summer was essentially rats fleeing a sinking ship. IBM spent $35B on a company with no actual assets (a few patents) other than smart, passionate people. They would have had to really try to screw it up. And they did. I don't honestly know if there will be another RH in the future, but I'm hoping there will be.
I agree with the rest of your comment but I just want to respond to this part:
>If you don't have thick skin, you should probably avoid upstream discussions
I have a humble request, please discontinue this attitude on your own projects and please encourage other upstreams to also discontinue it. I'm baffled as to why so many open source maintainers seem to confuse needing to defend technical decisions with "having a thick skin," they are not the same. It's perfectly possible to be opinionated and strongly scrutinize a technical decision, while also not being harsh and rude towards the person presenting it. If you believe those mailing lists are unsalvageably hostile, then in my opinion, they should just be shut down. That's not the kind of place to have official project communication.
> I also don't think it would lead to a curious, constructive conversation. There is probably nothing OP could offer which would change what the experience of 8 years in RH engineering showed me.
To me, the intentional insistence that you want to and will remain ignorant undermines the credibility of the rest of the comment. Who knows what else have you shut your mind to and thus overlooked at RH.
On the contrary, I find it very realistic. We don't have time to argue with every single individual who thinks we're wrong, no matter how much we might think _they're_ the ones in the wrong. With people like the GP, there's just no convincing them, you just have to move on and not waste your own time trying to debate with them.
> We don't have time to argue with every single individual who thinks we're wrong
Sure, but then don't enter the discussion at all. It's generally no faster to fight with the other person than to try to understand them.
> no matter how much we might think _they're_ the ones in the wrong
That assumes that's the point of the discussion. Do it to find out about the other person, help them find out about you, and to explore what they know and what you know.
> there's just no convincing them
I think that assumes, again, this adversarial fight. I'm not trying to convince them (see above). And often, people are open and curious if you approach or respond to them differently.
Generally most M&A's like this turn into a talent smasher event (think atom smasher, aka particle accelerator), and the people scatter to the four winds in hundreds/thousands of different landing zones instead of coalescing into another convergent organization. Emergent consensus centered around such a large organization takes a lot of effort and time to build because it operates through social trust pathways (that predominantly operate in the time domain), and once fissioned, the byproducts do not transitively transfer that trust, it has to be built anew.
This adds to my suspicions that the long-term IBM C-suite goal of the Red Hat acquisition is to eventually convert Red Hat to be like their captive mainframe revenue and profit stream. I would not at all be surprised to see a gradual deterioration in Red Hat quality in the coming 10 years.
Watch the general level of support engineering skills in what will eventually be a Blue-washed Red Hat L2 Support. That is your canary. What you are used to today in that tier will be gradually relegated to the tech leads for L2. Don't look for tech leads by title; look for the leads who gates decisions to escalate to L3. Highly-skilled L2 organizations grant most if not all their support engineers autonomy to escalate to L3. Profit-squeezing L2 organizations (what I suspect may be happening with Red Hat in the coming years) will gate those escalations, and staff vacancies with relatively down-skilled engineers.
Note that I also enjoy working with these relatively down-skilled engineers. No complaints about "bedside manners", and they can address common scenarios. Given enough time and exposure to more experienced staff, they would absolutely step up to highly-skilled L2 levels. The challenge is often these situations see those more experienced staff leave too soon. In my experience you need at least 7, ideally 14 years to season new staff to those skill levels, and the attrition rate is high over that time; troubleshooting well under pressure with grace is a very difficult combination of skills and characteristics to find.
Two ways I've seen to effectively deal with this on the customer end is find an alternative solution (not really feasible in this case at this time, though I'm watching several possible developments), or up-skill your staff. You'll know you've reached the right skill level when your engineers can delve into the internals of the product sufficiently that they routinely convince the down-skill L2 support engineers to escalate their lead within a day or two of engaging.
If my suspicions bear fruit (I really hope not, the vast majority of my clients hold massive people and capital investments in the Red Hat ecosystem), then this is extremely bullish for AWS. Their customer-obsession-to-a-fault culture would swiftly exploit IBM's strategy, and there would be no contest. I can see several simultaneous attacks AWS could mount to neutralize Red Hat's position if IBM's strategy at the end of the day really is what I currently suspect.
I'd really like to hear from others that have found other workarounds for this eventuality.
I don't see how RH customers can become a captive revenue stream like mainframe has; there are a few other distributions with enterprise support (like SUSE/Ubuntu) to migrate to as well as community distributions (like Debian) that can be used if you have in-house engineers.
Disclosure: I might be biased as a Debian user/contributor.
> I don't see how RH customers can become a captive revenue stream like mainframe has...
In the 10-15 year time frame: IBM can't. Simple as that.
In the 3-10 year time frame however, which not coincidentally happens to be in the range of CEO tenures, the momentum of enterprise infrastructure relying upon a specific platform delivers a kind of lock-in that absolutely can be exploited by IBM leadership.
Look at Oracle. There are still quite a few use cases where their arguably painful pricing, licensing, and account relationship management experience are worth it. For some use cases, there is no other acceptable solution; I've seen them, and Oracle is damn impressive if you operate at that rarefied strata. But for arguably in the Pareto distribution of scenarios where Oracle is used, much of it like CRUD applications, Postgres does just fine.
Yet Oracle today is as dominant as ever in many enterprises.
When it comes to what can and cannot become captive revenue streams in the software market, I've learned to accept that the market can often be irrational for far longer than I can remain solvent.
I really like Debian. It is what I use in my personal servers. I'd like nothing better than to see it supplant Red Hat. But in the enterprise, the Debian story is under-developed. Debian is very rarely supported by application vendors. The LTS duration is half that of Red Hat. The members of the big, profitable enterprise market move that slowly.
I think those are tough problems to solve by someone like Ubuntu basing off a Debian distro to solve on their own. All the other challenges of interfacing with enterprise customers, communicating in their lingo, meeting them half-way, *etc., are addressable by a company organized around doing that using Debian, but there are inevitable questions of resolving differences of opinion in the future between the community and the company. I suspect Red Hat being both the distro and the commercial entity traded solving a lot of those types of problems with a different, more commercially-acceptable set of problems.
If RedHat were to be dissolved, there aren't enough companies paying for work on Free Software to absorb all the engineers that work at RedHat. For Free Software, this is one of the big dangers of having so many people working at the same place. If RH goes down, there are going to be a lot of projects with greatly reduced maintenance and some outright abandonment.
I'm curious if our interactions could be biased. For example, we're clients of RH (RHEL & Openshift) so I'm guessing we interact with more presentable humans. Everyone I've met has been genuinely wonderful and supportive across eng, sales, product, TAM, etc. One of the few vendors I enjoy working with.
I used inclusive in this context to mean that a junior engineer is expected to speak up and offer input in an executive meeting if they have something to offer. Cutting through hierarchy, "good ideas come from everywhere". This could be idealistic but it helped me in eliciting input.
I don't know what it's like to be on the inside or close periphery to daily core operations at RH. I agree and understand completely what you are saying. "Free speech" has (recently?) been a guise to act terribly toward each other in a number of settings. Trolls gonna troll. It's important to maintain decorum in the workplace and a culture that defies that can turn toxic quickly.
Redhat are one of the few real open source companies that made a commercial success of things.
From the outside they seemed pretty open. I've got no idea about how they did with black / LGBTQ+ - but it seems a big jump (in an industry where there are not that many black female LGBTQ+ engineers) to say they were so bad.
What is the engineering company with lots of black female LGBTQ+ folks? It might be helpful to be able to look at a model that has worked rather than just trashing RedHat for not being an engineering company that gets this right (and all these companies are pushing initiatives in most of these areas).
I support getting the sampling rate and quality up. To what level? Idk.
Have you ever been in a room with a DJ playing low quality music on a decent sound system? It's awful, grating. Now at least your everyday aspiring DJ has access to higher bitrate tunes and doesn't need to blast low quality junk out of the PA system.
Overnight on the USS Yorktown (CV-10) was awesome, one of the highlights of all the scouting trips we took. It felt like we had full run of the ship, and my buddies and I explored quite a bit. My first impressions of WWII history and Battle of Midway where USS Yorktown (CV-5) sank were from the tours and videos shown that day.
Great article. Even in a large enterprise, where the prevailing organizational ideology centers around building the largest headcount you can for political sway...
I've kept my team small on purpose, and we were able to be very effective by adopting clojure. We have sway because we deliver- and that's a different type of leverage than headcount. We don't get thrown every hot potato, instead we're consistently aligned with the critical portfolios.
A small dedicated team with tools such as clojure will outdeliver, and outcompete a larger team who cannot remain agile and require significant overhead to manage.
This totally makes sense in a startup context, but how does this work in a large enterprise? For instance, who signs off on "critical portfolios" being migrated to clojure, and who maintains these pieces when your team moves onto the next project?
First, if you haven't worked in a large "non-tech" enterprise, you might be surprised to find out it's just 100 startups with a common name and financing. The bigger the company, the less technical cohesion across teams. It's a blessing and a curse. I'm technically in a "business unit" as opposed to IT, so we're more focused on tangible KPIs. When you turn around innovative results in the biz domain consistently, no one really questions your tooling.
We started a new space Data Eng/ Data Sci and then it became critical. We've been able to maintain products by building long term platforms/systems that meet current and (anticipated) future needs in clojure. Initially, product and platform/system was all the same team. We've grown a little and now individuals tend to focus more on one or the other, but we're still <15 eng and shipping daily.
We maintain everything we've built to date, and we planned for that going in. Clojure's other superpower is that it is stable AF over long periods of time, handles refactors and testing well.
As for signoff in adopting clojure... our VP is old hat in clojure/lisp. We got lucky.
Yes, this would certainly be naughty under current law, but perhaps not in a legal regime where "all speech is OK!" Note that the overt act need not be committed by the speaker, too.
There's even ambiguity about elements of this in current law. If one were to advocate for the violent overthrow of the government, and begin running training exercises to help people prepare to overthrow the government at some unspecified future date--- it is unclear whether this is protected by the First Amendment. SCOTUS mentioned -- but did not address -- this problem in Stewart v McCoy (2002):
... While the requirement that the consequence be “imminent” is justified with respect to mere advocacy, the same justification does not necessarily adhere to some speech that performs a teaching function. As our cases have long identified, the First Amendment does not prevent restrictions on speech that have “clear support in public danger.” Thomas v. Collins, 323 U.S. 516, 530 (1945). Long range planning of criminal enterprises–which may include oral advice, training exercises, and perhaps the preparation of written materials–involves speech that should not be glibly characterized as mere “advocacy” and certainly may create significant public danger. Our cases have not yet considered whether, and if so to what extent, the First Amendment protects such instructional speech. Our denial of certiorari in this case should not be taken as an endorsement of the reasoning of the Court of Appeals....
I work for a large hosp. operations company and serve as the Dir. Engineering for our clinical operations group. Hacking Healthcare is required reading for new members of my team. It serves as an excellent introduction (with a healthy amount of critique) to the dynamics in the hc technology ecosystem. Thank you for providing this perspective on the industry and its challenges with tech.
We've been successful developing using open source technology internally. In fact, I take a fairly hard stance on disallowing proprietary healthcare specific "solutions" from working their way into our stack (aside from the EHR itself, it has staying power). We're lucky in that we are positioned as somewhat of a startup within a larger org, and are able to take that approach.
To avoid some of the issues you raise, we generally are working to reduce the surface area of the EHR to become simply the transactional backend which is then mirrored to a larger ecosystem of custom apps. This has the effect of boxing in the regulated entity. We focus on data integration (by spending $$$$ on custom HL7 interfaces, unfortunately not everyone can afford) to get outside of the walled garden. This means we can use the information/data for new and interesting purposes without worrying about the EHR vendor's roadblocks/tolls. More importantly to some people, we don't disrupt the billing cycle that originates from the EHR.
Do you notice any trends where healthcare operations/providers are starting to develop internal technology that integrates with the EHR to compliment vs. replace the core transactional system?
It's Duff (David) instead of Fred but thanks. Fred is doing great too. Are you a former CHL/TXR managed or sub-owned group or facility?
Unfortunately I see the opposite trend right now, more silos, more lip service to interoperability, more tolls. I think driven by the burden of regulatory overhead. Moving forward there could be a shift to a "patient owned" record where providers and facilities feed standardized formats into a patient owned/managed "personal cloud". I hope that continues to pick up steam.
Hi Duff! Apologies for the mixup, should have seen that in your handle. No, I'm not part of CHL/TXR.
While not open source by any stretch, I see the personal EHR space being ushered along by companies following Apple's lead. Aside from complex patients, I think the generally healthy/mild-chronic person is uninterested in owning/managing their health data unfortunately. Apple is contributing useful tools to understand fundamental determinants of health including cardiovascular, sleep, fitness at massive scale. It just happens to come with your watch and phone, and their vision for health is starting to come into focus. This puts the patient in the position of generating the primary data (sensors, etc.) and sharing it with their care team on their terms (more or less). As telehealth becomes more prominent, I suspect the patient will be required to engage with their data more often as it will be the means of conveying a shared understanding vs. observations recorded in the clinical setting and stashed away in centralized EHRs.
Furthermore, if labs and other diagnostics are available directly to consumers it puts the individual in a position of ownership. I think the default position is whoever generates the data owns it, and determines how easy/hard it is to share with others. If the individual is empowered to generate information about themselves- this will start to swing toward "patient owned." I too look forward to more of this, but it will have to come with more direct to consumer and digital offerings.
One of the coolest examples I've seen of individuals taking ownership in open source med tech is openaps.org . I'm not one, but T1D's are some of the most resourceful and resilient folks around. Good on them for building a community to solve real problems together. Shout out to the #wearenotwaiting crew.
Hey - I'm trying to get into health tech (EHR integrations, HL7, FHIR etc), do you have any additional "required reading" suggestions beyond Hacking Healthcare?
One of the reasons Hacking Healthcare happened is the huge dearth of material for just this question, unfortunately I don't have anything else on that still. One absolutely timeless work not directly about technology but that is still very relevant is "The House of God". It is "fiction" but it is probably the most realistic depiction of the actual happenings at hospitals that exists and as a result something technologists can benefit from.
Our team signed a contract with Cognitect for Datomic late 2017.
This clause was in-place and stood out to me as well. I had a chance to ask their legal team about it. The clause is written in legal-ese, which always sounds overbearing.
I asked the question in the positive sense, "what if have some really nice metrics from our use cases, and want to talk about them at a conference?" They simply asked to be consulted and request written permission to share. The intent, like others have noted, is to request (legally: insist) that Cognitect have a chance to review and point out potential implementation issues (good or bad) prior to customers making performance statements about their product.
The clause can/does put a damper on 'notes from the field' reports, which often help when deciding on tech direction.I look for community based reports to reinforce perceptions of a tool (to a degree). Completely agree with OP, do your own performance testing.
One thing I will say is that it would be hard for someone who hasn't invested in learning the inner-workings of Datomic's decoupled architecture to pick apart storage speed vs. transactor speed. For example, storage speed (SQL, DEV, Dynamo, etc.) is not a concern of Datomic, but a key dependency to measurable perf. This may change in the AWS service announced today, and become more uniform on dynamo and S3 "storage resources". https://docs.datomic.com/cloud/whatis/architecture.html#stor...
Datomic is a unique product and there are many ways to make it sing (or blow up) depending on how you use it. We designed data models, streaming processes, and queries with Datomic in mind and have had success. Exactly how much success, I'm not at liberty to say just yet.
https://www.canadianbusiness.com/the-last-days-of-target-can...