Apache Ozone has multiple 100+ petabyte clusters in production. The capacity is on HDDs and metadata is on SSDs. Updated docs (staging for new docs): https://kerneltime.github.io/ozone-site/
https://github.com/oneconcern/datamon
Had written this git for data tool few years back (works with GCS but can be made to work with S3)
1. No server side
2. Immutable data (via GCS policies)
3. Ability to mount data sets as filesystems
4. Integrated with k8s.
It was built to work for the needs of the startup funding it, but I would love it if it could be extended.
MIT has an amazing program for System Design and Management. Dan and I are both graduates of this program. Some of the courses I would recommend include System Architecture, System Safety, and System Dynamics. Most of the content is available on OCW.
What is taught is not software-specific, but is entirely applicable to software, outside of the world of 'throw everything on the wall and see what sticks' as long as the venture capitalist can be shown growth at all costs. I wish more software developers were mindful of complexity and architecture.
ESD.34 ESD.342 16.863J 15.871 15.783J ESD.33 to name a few
Also, 15.965 my favorite and offered by my thesis advisor Michael A. M. Davies: Based on which it is likely that OpenAI won't be walking away with the cake.
That is the share of value capture from the market that they just made real. There are legitimate scenarios where first movers do not have an advantage.
Could you elaborate a bit more on what you got out of the program? I never heard about it before but i'm intrigued. Did you find a particular course memorable?
The tag line within SDM was that it is a program for those who want to lead engineering and not leave engineering (MBA)
I think the meta framework for thinking and being able to step away from the madness of releasing a v1 product and having tools for thinking about the bigger picture.
Also, MIT.. it is a very rewarding ecosystem to be in.
I recently came across a term "Functional Medicine" https://www.ifm.org which looks at the body as a whole and tries to root cause metabolic issues which can often result in IBS cures.. most older medicinal practices always considered the body as a unified system which make sense in terms of how it is engineered but modern medicine tries to break things down and often misses the cross specialization impacts. Best of luck!
If each part of the body has its own specialised discipline, then holistic (as in "of the whole" not "based on rubbish") medicine is the exception not the norm. Example: surgery to fix sports or occupational injuries will not correct the movement patterns or muscle imbalances that caused it, and thus it is likely to recur.
I don't get it. Like how are you a knee doctor if you're not aware of what knees do or how they should work in the overall system.
On a system as incredibly simple as, say, an old carbureted motorcycle you don't just look at the spark plugs if there is poor ignition. You look at the carb, air filter, rings, valves, top dead center sensor, timing chains, the cdi/tdi, the coils, and the battery (although probably not in that order).
If the problem is the battery no sensible home mechanic (and no ethical professional mechanic) stops there, you check the voltage regulator and the stator or you're going to find yourself replacing a lot of $100 batteries for want of a $30 IC or a piece of $2 copper wire with a break in the insulation.
The exact same process applies in debugging or designing a physics experiment or making a pcb or repairing a house or therapy, or...just life. Like, you're never going to keep your room clean if your wardrobe is too small to hold all your clothes....
I can't even begin to comprehend a mindset which claims to be part of STEM (or even as professional as a tradesman) in which this concept is foreign or novel.
A knee doctor will be aware of what knees do and how they support your musculoskeletal system. The problem is institutional: if you go in with a knee injury, you'll be triaged and sent to the knee clinic for scans and assessment for a break, sprain, etc. If that problem is found, it will be treated. But it may be that your knee pain was actually caused by bad posture (it's pretty shocking the number and variety of injuries you can sustain from bad posture) and neither triage nor the knee clinic are assessing your posture. They probably could send you to somebody to do so, but the institutional will is not there because that's not how the medical system is designed.
A whole systems health model would involve essentially a debugging process for injuries and maladies in order to find root causes and treat them. It would require a totally different (transdisciplinary) approach to health though, and institutions don't really change.
as the saying goes, the lunatics are running the asylum.
In reality it's all systemic pressures everywhere and nobody, not even institutional administrators, likely have much in the way of free choice over anything that matters.
The rationalist celebrities could look more at dynamical systems, but then they'd have to admit that systematic effects exist and that daddy Thiel et al are using them to f* everyone over forever in order to pad their already bloated wallets.
True, being a libertarian does disincentivise looking at systems and systemic effects because the biggest system we live in is capitalism and the systemic analyses that have already been done on it don't paint the prettiest picture.
indeed he will, unless we learn how to manipulate him. This is why I'm confused as to why rationalists seem to have done basically no research into complexity theory (systems/cybernetics). Scott Alexander's done the Moloch pieces, and this one article on dynamical systems [1] but aside from that the closest they get is machine learning, which is a descendant.
Yeah, but any competent DB admin will at least read the logs, do a few A/B tests, blocking out different parts or using queries which should be equivalent and then say 'something is hinky with the way this query is compiling, query z seems to give a different answer if query y is written like foo even though I see no link. Probably a subtle bug in your code I haven't spotted. Maybe ask someone who knows more about the compilation step?'
Even if they 'solve' it by rewriting query y, they'll not claim that query y was the problem, and may even leave a comment or commit log saying they didn't understand why writing it like har fixed it.
Note the word competent.. Also, insurance companies will often have opinions on what tests were valid and might reject paying for tests. So if a DB admin spins up a cluster to debug his hunch of a SQL Compiler issue, that bill will be paid by the customer and not the insurance and might bankrupt the customer..
I had an apt with a cardiologist where he literally read a standard template of care and when I tried to see if he is interested diving deeper into inflammation causes, tests and ways to avoid the standard medications which only deal with symptoms he was least interested as it all dealt with metabolic issues and not the heart per se.. the system is run to see how can a patient be billed and who owns the organ. Insurance companies have swim lanes and medical practitioners are educated and trained to remain in those swim lanes.. as long as things are kept simple for prescribed cures and billing, the gravy train continues.. it not in their best interest to solve the problem. Have you ever been to a annual physical where god forbid you discuss issues that are not part of the normal annual health checkup? You will actually be asked to schedule a separate appointment as the billing for that is separate..
A lot of "functional medicine" happens outside of insurance payment coverage.
Some more interesting things to follow:
https://cholesterolcode.com/https://cynthialimd.com/faqs/
Wow! Allegations without context or material. Who would have thought someone on twitter would sling mud!
If the Indian government is going to implement the Aadhaar card.. I rather would like folks such as Viral Shah on the team to make sure the implementation is headed in the right direction.. The scale and operational challenges faced are significant. The context for that twitter thread is a leak of private data that occurred. If competent engineers are not involved, it will only get worse.
This is analogous to the US Govt implementing a plan to provide universal healthcare but random developers getting angry when a competent engineer works to make sure it is done right..
My LG washer dryer app needs to know my location... why? They won’t let me add it without giving app location access.. so does the YI camera now.. this is getting a bit out of hand.
Agree with most of these, but man do I ever miss the free and very well provisioned salad bar at the office :( I've eaten more grilled cheese and peanut butter sandwiches during WFH than any self-respecting adult ever should.
