The core issue is dimensional data modeling was introduced to address limitations on hardware (disk drives) and limited capacity.
With the advent of unlimited storage and separation of computer and storage, dimensional data modeling would only be possible if there was strong data governance in a system like SAP or a COE.
I would separate dimensional or relational modeling from data governance. You can dump all the data you have in one S3 bucket, and AWS can take care of near-real-time ingestion. So no need for staging, ODS, data warehouse, data marts, hub and spoke, all that jazz. And no data modeling required, just ingest as is and dump it there. Great.
Now what is this dump good for? It's just bunch of bytes of information which now needs to be interpreted. There's different perspectives (sales vs manufacturing vs procurement vs finance etc). There's data quality issues that need to be identified and resolved. There's PII and other compliance stuff. You have to watch out for giving permissions to sensitive information (ever dealt with payroll data? It's fun) Your data dump isn't doing any of that by itself. And I think people tend to simply stop at the data dump stage and then give access to analysts and data scientists and tell them to go do reports and outbound data feeds.
The problem with Palantir is they target gov. agencies.
Most of the time companies who have systems like Palantir, I’m thinking the SAP, Oracle, blah
Blah, have to report earnings to the street through a 10k or have to comply with regulations like Sarbanes Oxley.
They will also have in-house IT staff to monitor the logs etc.
The programs installing the Foundry system have an incentive to hide the data from prying eyes and therefore it never leaves the Palantir ecosystem. The government doesn’t hire independent consultants, auditors etc to confirm if it’s being used or not.
They simply have to demonstrate trustworthiness to a security officer and hope an IG doesn’t have an external equivalent of a Forward deployed engineer.
So while the technology is mediocre, it’s the nebulousness or the lack of audit-ability and the are the people writing checks the same people signing them.
So I sympathize with Karp talking about technology being fine it’s the apparatus surrounding it that says “just trust us” that gives pause, especially in today’s culture of conflict.
If I told you that 90% of all transactions get routed through a foreign companies software, you might pause but it’s been like that for years (SAP). The difference is there are controls in place.
American business operates like a feudal system with lords and peasants. Management are the lords and everyone else are the expendable peasants.
Occasionally, the lords are forced to acknowledge that the peasants are just as essential as they are. Without them there is nothing and no one to "lord" over.
This is the abherrent phase that Boeing has just entered. The system will be forced to either return to normal or simply cease to exist a la GE.
> Management are the lords and everyone else are the expendable peasants
Boards are the lords. Management without a board seat are the bourgeoisie; Boeing engineers, too. To the degree we have peasants in modern America, it’s in agriculture and retail, not at Boeing.
I was quite surprised how authoritarian American/Canadian business environments were and subconsciously/covertly racist without saying it outright.
I still vividly remember when I was in my 20s, an American executive from a fortune 500 company forgetting to unmute himself and accidentally unmasking himself and the attitude of those around him.
They were at the forefront of "woke" and "diversity". This is when I learned it was just another slogan like "human rights" to make the citizens feel good about themselves.
As somebody from Eastern Block, it is not much worse. Government will forcibly create monopoly businesses, they have no reason to improve their products and to top it off, you can't create competing business, because government does not allow that.
Just look what was the result after 1989 - companies who could compete with western ones, because government did not required innovation last 40 years and 5 year plans were solved by throwing more people at the problem or outright lying. Result was that products were often obsolete long before manufacturing even started, what was made were poor quality (i.e. we called it finish it at home) and much more expensive than in the rotten West.
