One of the easiest approache is torch.compile, it's the latest iteration of pytorch compiler (previous methods were : TorchScript and FX Tracing.)
You simply write
model = torch.compile(model)
"Across these 163 open-source models torch.compile works 93% of time, and the model runs 43% faster in training on an NVIDIA A100 GPU. At Float32 precision, it runs 21% faster on average and at AMP Precision it runs 51% faster on average."[1]
What google is trying to do, is to involve more people in the R&D of these kind of methods.
The near term promise is that you can use AMD, CUDA, TPUs, CPUs etc without explicit vendor support for the framework on which the model was developed.
Disclaimer: I will be very handwavey, reality is complex.
This is achieved by compiling the graph into some intermediate representation. And then implementing the right backend. For projects here, look at stableHLO, IREE, openXLA.
You can argue that Jax's jit compiler is a form of such compiler, mapping the traced operations down to XLA, which then does its own bit of magic to make it work on your backend.
It's transformations and abstractions all the way down.
Bodybuilders do this thing called "bulking and cutting." The best way to add muscle fast is to overeat. Work out lots. Sleep lots. Eat lots.
You get fat, but you also get muscular because food is never a limiting factor.
Then, they lose the extra fat with a crash diet.
Google, FB and such are such money machines that they never have to cut. They can just bulk. The others... they want some of that rapid growth potential too, but can't afford to add fat forever.
This analogy doesn’t seem to add any explanatory power.
Corporations can’t magically know the optimal number of employees to maximize profits. And even if they could this will change over time. So we should expect them to cut when they have too many and hire when they have too few.
>Google, FB and such are such money machines that they never have to cut.
Not sure what you are trying to achieve by saying something so blatantly untrue that a two-word query into any search engine can debunk it as fast as your browser loads a web page.
What makes this hard to read/follow is the grandiose moral vision... and the various levels of credulity it's met with.
If it's words from Ilya, Sam, the board... the words are all about alignment, benefiting humanity and such.
Meanwhile, all parties involved are super serious tycoons who are super serious about riding the AI wave, establishing moats, monopolies and the next AdWords, azure, etc.
These are such extreme opposite vocabularies that it's just hard to bridge. It's two conversations happening under totally different assumptions and everyone assumes at least someone is being totally disingenuous.
Meanwhile, "AI alignment" is such a charismatic topic. Meanwhile, the less futuristic but more applicable, "alignment questions" are about the alignment of msft, openai, other investors and consortium members.
If Ilya, Sam or any of them are actually worried about si alignment... They should at least give credence to the idea that we're all worried about their human alignment.
> the words are all about alignment, benefiting humanity and such.
that's why you only consider the actions taken, not the words spoken.
And in fact, i fail to believe that there are any real altruists out there. Esp. not rich ones. After all, if they're really altruistic, they would've given all their wealth away - before their supposed death (and even then, i doubt the money given to their "charitable foundations" count for real!).
Not necessarily. Money keeps you in the game. Giving it all away means you are at the bottom being not that effective. And you can donate money in your will.
One of those things that are widely true, but rarely admitted to. It seems to be very much a maturity thing. The older,larger and more governed an org is, the more likely such a pattern is.
Habits become precedents. Precedents become rules. A pattern emerges is everyone operates within rules. Staying within the ruleset, represents known safety. Even if something is dubious.. as long as it's within the rule set you are safe.
Is shaky principle tentatively applied once.. it doesn't have that kind of safety. That means it's less likely to be stretched and made absurd.
There is a logic to trimming unused budgets. not perfect, but it wouldn't surprise me if it worked well enough, often enough. If a department keeps going over budget, well.. they need more budget.. or maybe less work. Where is that budget going to come from? Departments without enough budget.
It's hard to get more legible, than last year's expenditure as the starting point for the next years budget. Nothing very notable about birthing this "principle."
I'm sure it makes sense, often. At least in the sense that it's the easiest, good enough method.
If there's a new management, using old methods is helpful. They don't know enough.. and this just gets the job done. If budgets become contentious, sticking to "principles" helps smooth things.
That is the point though.. whether it's a big-hype management method like agile.. or some unofficial budgetary principle that happened to work before these are principles. We like to be principled, especially when we don't really know what to do.
There's a literary trope of a bone casting seer. It takes a wise person to cast bones. It's an art and science. Sure, you have to know what all the bones mean. But, you also have to figure out it's a good idea to fight to fight this particular battle, build a town in that particular place... And also to understand the role bone casting plays in this particular case.
Bones must be cast, because we like external validation. It helps to bring everyone together, and calms underconfident, overwhelmed, or underunited leadership.
Knowing when to cast them, why, all the different implications.. how to define the question, how to approach the answer.. those are jobs for the seer, not the bones themselves.
Things are better when soldiers watch the stones, and chieftains watch the seer. If and when that flips, the paradigm is not at its best.
Sure, considering the platform-game paradigm different products sources cross subsidize one another a different stages of their life cycles. This is just how firms work, and is often mostly a matter of accounting.
Investors and executives.. everyone in 2023 is hyper focused on "Thiel Monopoly."
Platform, moat, aggregation theory, network effects, first mover advantages.. all those ways of thinking about it.
There's no point in being bing to Google's AdWords... So the big question is pathway to being the adWords. "Winning." That's the paradigm. This is where big returns will be.
However.. we should always remember, but the future is harder to see from the past. Post fact analysis, can often make things seem a lot simpler and more inevitable than they ever were.
It's not clear what a winner even is here. What are the bottlenecks to be controlled. What are the business models, revenue sources. What represents the "LLM Google," America online, Yahoo or a 90s dumb pipe.
FYIW I think all the big text have powerful plays available.. including keeping powder dry.
No doubt that proximity to openAI, control, influence, access to IP.. all strategic assets. That's why they're all invested an involved in the consortium.
That said assets or not strategies. It's hard to have strategies when strategic goals are unclear.
You can nominate a strategic goal from here, try to stay upstream, make exploratory investments and bets... There is no rush for the prize, unless the price is known.
Obviously, I'm assuming the prixe is not AGI and a solution to everything... That kind of abstraction is useful, but I do not think it's operative.
It's not a race currently, to see who's R&D lab turns on the first super intelligent consciousness.
Assuming I'm correct on that, we really have no idea which applications LLM capabilities companies are actually competing for.
>>Parents, meanwhile, can either work to balance the easy sources with...
It's possible to some extent. Different parents will have different levels of ability and success.
This is relevant (but difficult) advice for individual parents, but it's a minor point if we're discussing society. People (parents and kids) exist in an environment. "Just going out to play," isn't part of the world/environment/culture. The way kids/parents did thing differently in the past was by existing in a different culture.
Easy Vs hard stimulation is one paradigm, but you can't just view everything through this lens. Life is more complex. There are looping and knotting causal relationships and we can't know all of them. It's a complex.
Life is just very screen based. Ours and theirs.
They study via screens, and don't really know how to "do school" analogue. Social life is, largely through screens... and increasingly part of the media spectrum. Work will, eventually, be screened-based. So are life's administrative tasks.
Life's administrative tasks got way worse with computwrisation. Now I have to book my own flight, make sure they match with hotels, take my own reading from electric, water and gas meters, register everywhere myself and endlessly prove my indetity to hindreds of institutions.
I have an email mailbox for changes in terms and conditions that get sent to me, it is about 800 emails. If I hired a lawyer to go through them, it would probably cost $100,000
True. Arguably, work and social life have also gotten more difficult, at least for some.
Meanwhile.. it's also not overwhelmingly true that administration, and other things heavily affected by computerisation are "better," more efficient or productive.
That's easy to see in non-commercial space. It's also true in commercial space. Government. Universities. Education.
This is all tightly coupled with op's point. Digitization, computerization, networking... These really were major revolutions. They impacted and totally reformed everything. However, this has been more of a change than an improvement in many applications. Slightly better in some ways, slightly worse and others, ambiguous, mixed... complex.
The computerization of social life, entertainment.. childhood experiences. Etc.
There are long causality chains.. and we don't really understand the consequences of everything.
But in any case.. I think you're absolutely right on TCs and administrative life.
Technology made it easy to have more, more complex paperwork. That allows us to produce a lot more paperwork. A handwritten contract with two lawyers present.. is a marginal cost. Even a sign here, take-it-or-leave-it contract (EG financing, or opening an account) representing some marginal cost.
Computerization makes this whole thing more efficient. Automated emails, tick-to-agree pop-ups... Those allow humanity to create many more contracts than previously possible.
That's efficiency, of a sort. If you consider contracts themselves to be an output... The efficiency gains a greater than Henry Ford's entire lifetime.
I think we've learned something. Technological change, and it's proliferation does not necessarily aggregate in an idealic way.
I think the fear of paperclip maximizing AIs.. I think this is more of an analogy to our experience with recent tech shifts.. beyond its veracity as a danger vector for AI specifically.
This is the literal answer to "why." Also bans on various casino hacks.