I assume Crowdstrike is software you usually want to update quickly, given it is (ironically) designed to counter threats to your system.
Very easy for us to second guess today of course. But in another scenario a manager is being torn a new one because they fell victim to a ransomware attack via a zero day systems were left vulnerable to because Crowdstrike wasn’t updated in a timely manner.
Maybe, if there's a new zero-day major exploit that is spreading like wildfire. That's not the normal case. Most successful exploits and ransom attacks are using old vulnerabilites against unpatched and unprotected systems.
Mostly, if you are reasonably timely about keeping updates applied, you're fine.
> Maybe, if there's a new zero-day major exploit that is spreading like wildfire. That's not the normal case.
Sure. And Crowstrike releasing an update that bricks machines is also not the normal case. We're debating between two edges cases here, the answers aren’t simple. A zero day spreading like wildfire is not normal but if it were to happen it could be just as, if not more, destructive than what we’re seeing with Crowdstrike.
In the context of the GP where they were actively treating a heart attack, the act of restarting the computer (let alone it never come back) in of itself seems like an issue.
I believe this update didn't restart the computer, just loaded some new data into kernel. Which didn't crash anything the previous 1000 times. A successful background update could hurt performance, but probably machines where that's considered a problem just don't run a general-purpose multitasking OS?
Crowdstrike pushed a configuration change that was a malformed file, which was picked up by every computer running a the agent (millions of computers across the globe). It's not like hospitals and IT systems are manually running this update and can roll it back.
As to why they didn't catch this during tests or why they don't use perform gradual change rollouts to hosts, your guess is as good as mine. I hope we get a public postmortem for this.
Considering Crowdstrike mentioned in their blog that systems that had their 'falcon sensor' installed weren't affected [1], and the update is falcon content, I'm not sure it was a malformed file, but just software that required this sensor to be installed. Perhaps their QA only checked if the update broke systems with this sensor installed, and didn't do a regression check on windows systems without it.
It says that if a system isn’t “affected”, meaning it doesn’t reboot in a loop, then the “protection” works and nothing needs to be done. That’s because the Crowdstrike central systems, on which rely the agents running on the clients’ systems, are working well.
The “sensor” is what the clients actually install and run on their machines in order to “use Crowdstrike”.
The crash happened in a file named csagent.sys which on my machine was something like a week old.
Likely because staggered updates would harm their overall security services. I'm guessing these software offer telemetry that gets shared across their clientele, so that gets hampered if you have a thousand different software versions.
My guess is this was an auto-update pushed out by whatever central management server they use. Given CS is supposed to protect your from malware, IT may have staged and pushed the update in one go.
High-end hospital-management software is not simple stuff, to roll your own. And the (very few) specialty companies which produce such software may see no reason to support a variety of OS's.
Are you sure that argument still holds when everyone has Android/iOS phone with apps that talk to Linux servers, and some use Windows desktops and servers as well?
There isn't, and never was, a benevolent dictator choosing the OS for computers in medical settings.
Instead, it's a bunch of independent-ish, for-profit software & hardware companies. Each one trying to make it cheap & easy to develop their own product, and to maximize sales. Given the dominance of MS-DOS and Windows on cheap-ish & ubiquitous PC's, starting in the early-ish 1980's, the current situation was pretty much inevitable.
To add detail for those that don't understand, the big healthcare players barely have unix teams, and the small mom and pop groups literally have desktops sitting under the receptionist desk running the shittiest software imaginable.
The big health products are built on windows because they are built by outsourced software shops and target the majority of builds which are basically the equivalent of bob's hardware store still running windows 95 on their point of sale box.
The major players that took over this space for the big players had to migrate from this, so they still targeted "wintel" platforms because the vast majority of healthcare servers are windows.
Its basically the tech equivalent of everything evolved from the width of oxen for railway.
Because of critical mass. A significant amount of non-technically inclined people use Windows. Some use Mac. And they're intimidated by anything different.
There's a bunch of non-web proprietary software medical offices use to access patient files, result histories, prescription dispensation etc. At least here in Ontario my doctor uses an actual windows application to accomplish all that.
Then they use those apps. The point is that since they usage of the OS as such is so minimal as to be irrelevant as long as it has a launcher and an X in the top corner.
Question is: why half+ of Fortune 500 companies allowed Crowdstrike - Windows hackers - access and total control of their not-a-ms-windows business ? Obviously Crowdstrike do not do medicine or lifting cranes differentiation. "In the middle of the surgery" is not in their use case docs!
There was somewhere Mercedes pitstop image with wall of BSoD monitors :) But that is not Crowdstrike business either...
And all that via public internet and misc clouds. Banks have their own fibre lines, why hospitals can't?
Airports should disconnect from Internet too, selling tickets can be separate infra, synchronization between POSes and checkout don't need to be in real time.
There is only one sane way to prevent such events: EOD controlled by organization and this is sharply incompatible with 3rd party on-line EOD providers. But they can sell it in a box and do real time support when called.
Auditing: using Windows plus AV plus malware protection means you demonstrate compliance faster than trying to prove your particular version on Linux is secure. Hospitals have to demonstrate compliance in very short timeframes and every second counts. If you fail to achieve this, some or all of your units can be closed.
Dependency chains: many pieces of kit either only have drivers on windows or work much better on Windows. You are at the mercy of the least OS diverse piece of kit. Label printers are notorious for this as an e.g.
Staffing: Many of your staff know how to do their jobs excellently, but will struggle with tech. You need them to be able assume a look and feel, because you dont want them fighting UX differences when every second counts. Their stress level is roughly equiv. to their worst 10 seconds of their day. And staff will quit or strike over UX. Even UI colour changes due to virtualization down scaling have triggered strife.
Change Mgmt: Hospitals are conservative and rarely push the envelope. We are seeing a major shift at the moment in key areas (EMR) but this still happening slowly. No one is interested in increasing their risk just because Linux exists and has Win64 compatability. There is literally no driver for change away from windows.
> What are the hard problems? I can think of a few, but I'm probably wrong.
Billing and insurance reimbursement process change all the time and is a headache to keep up to date. E.g. the actual dentist software is paint but with mainly the bucket and some way to quickly insert teeth objects to match your mouth. I.e. almost no medical skill in the software itself helping the user.
It's not just that. A large portion of IT people who work in these industries find Windows much easier to administer. They're very resistant to switching out even if it was possible and everything the company needed was available elsewhere.
Even if they did switch, they'd then want to install all the equivalent monitoring crap. If such existed, it would likely be some custom kernel driver and it could bring a unix system to its knees when shit goes wrong too.
one major issue with zotero is the lack of android support. they are working on an android version or app or something since forever.
then is the way you store the pfds. if you want to sync between multiple computers you have to either know how to work with webdav or know how to point zotero at the location where you have your pdfs or (what they most certainly love) pay a lot of money for not so much storage space on their system. that last thing is what i don't like because i just don't trust anyone these days. you get invested in a system, build your routine around it only for them to shut it down, sell it watever and then puff you have to start over.
people keep calling zotero foss but if they were truly foss they would have a much more transparent way for people to roll their own selfhosted zotero server. instead, what they have is a dump of an old version, with next to zero documentation and a bunch of stubborn people that have managed to get something working but not quite.
I get that they are trying to make money but I am sure they could do that and be more transparent.
The other thing is the reliance on so many plugins. While zotero itself may last a while, who can say anything about the many devs of the many plugins that you end up relying on in order to make zotero bow to your routine? I like zotfile and a few others, but how long are they going to last? Also, reinstalling my system is a huge pain to get back to my routine because I have to remember all the settings for each and every plugin I install. They should come up with a way to save all these settings and restore them, and no don't do it through another plugin!
Here's a guide I found useful to set up zotero storage. In brief, it relies on zotfile to flatten the storage (keep all pdfs in one directory) and better bibtex.
I realized that it helped me to get rid of exactly the pain with fresh installs that you mention. I realized that the two plugins give me most of the functionality that I want.
the problem comes from scientometrics. you have to pump up your numbers as a researcher if you ever want to get that grant, or that promotion or that tenure position.
there was a time when there were far less journals and articles published per year. people spent a lot more time on an article and it shows. read an article today and you are left with nothing. back in the day (and by that I mean before about 2010) you had everything you needed to understand the subject and form your ideas. today is about tonnes of references (and not the useful kind either!) and inventing catchy acronyms.
but now that scientometrics is so important everyone is chasing the numbers and not the quality. you need that high impact factor up, that h-index, the influence score and the citations up. and since you have no chance to spend more time to increase the quality of your article, the next best thing is to increase the quantity you push into the grind. these predatory publications are precisely the answer to this artificially increased demand. yet people continue to be surprised.
think about it for a second: consider how many PhDs are awarded each year, they have to go somewhere and most of them want to go up. hence the increased demand. PhD courses are a huge business for universities, and this business drives the publishing industry. mix into this the fact that in academia, you have to change your research subject about every three years to keep it real for the grant masters with the big project money, you get to this disaster today.
try to build a bibliography today on a new subject and you will find tonnes of articles and then try to find the ones where you can actually understand where the science on that subject is at.
i don't know what the solution is, but counting scienctometric indices is not it.
