That's a quite different thing, OpenAI has billions of USD/year cash flow, and when you have that there's many many potential way to achieve profitability on different time horizons. It's not a situation of chance but a situation of choice.
Anyway, how much that matters for an investor is hard to form a clear answer to - investors are after all not directly looking for profitability as such, but for valuation growth. The two are linked but not the same -- any investor in OpenAI today probably also places themselves into a game of chance, betting on OpenAI making more breakthroughs and increasing the cash flow even more -- not just becoming profitable at the same rate of cash flow. So there's still some of the same risk baked into this investment.
But with a new startup like LeCun's is going to be, it's 100% on the risk side and 0% on the optionality side. The path to profitability for a startup would be something like 1) a breakthrough is made 2) that breakthrough is utilized in a way that generates cash flow 3) the company becomes profitable (and at this point hopefully the valuation is good.)
There's a lot of things that can go wrong at every step here (aside from the obvious), including e.g. making a breakthrough that doesn't represent a defensible mote for your startup, failing to build the structure of the business necessary to generate cashflow, ... OpenAI et al already have a lot of that behind them, and while that doesn't mean that they don't face upcoming risks and challenges, the huge amount of cashflow they have available helps them overcome these issues far more easily than a startup, which will stop solving problems if you stop feeding money into it.
> That's a quite different thing, OpenAI has billions of USD/year cash flow, and when you have that there's many many potential way to achieve profitability on different time horizons. It's not a situation of chance but a situation of choice.
Talk is cheap. Until they're actually cash flow positive, I'll believe it when I see it
It's also because around 20 years ago there was a "reset" when we switched from x86 to x86_64. When AMD introduced x86_64, it made a bunch of the previously optional extension (SSE up to a certain version etc) a mandatory part of x86_64. Gentoo systems could already be optimized before on x86 using those instructions, but now (2004ish) every system using x86_64 was automatically always taking full advantage of all of these instructions*.
Since then we've slowly started accumulating optional extensions again; newer SSE versions, AVX, encryption and virtualization extensions, probably some more newfangled AI stuff I'm not on top of. So very slowly it might have started again to make sense for an approach like Gentoo to exist**.
* usual caveats apply; if the compiler can figure out that using the instruction is useful etc.
** but the same caveats as back then apply. A lot of software can't really take advantage of these new instructions, because newer instructions have been getting increasingly more use-case-specific; and applications that can greatly benefit from them will already have alternative code-pathes to take advantage of them anyway. Also a lot of the stuff happening in hardware acceleration has moved to GPUs, which have a feature discovery process independent of CPU instruction set anyway.
The llama.cpp package on Debian and Ubuntu is also rather clever in that it's built for x86-64-v1, x86-64-v2, x86-64-v3, and x86-64-v4. It benefits quite dramatically from using the newest instructions, but the library doesn't have dynamic instruction selection itself. Instead, ld.so decides which version of libggml.so to load depending on your hardware capabilities.
> llama.cpp package on Debian and Ubuntu is also rather clever … ld.so decides which version of libggml.so to load depending on your hardware capabilities
Why is this "clever"? This is pretty much how "fat" binaries are supposed to work, no? At least, such packaging is the norm for Android.
Isn’t the whole thrust of this thread that most normal algorithms see little to no speed up from things like avx, and therefore multiversioning those things that do makes more sense that compiling the whole OS for a newer set of cpu features?
In my experiments claude4 opus generated by far the best code (for my taste and purposes) but it's also a pretty expensive model. I think I used up $40 in one evening of frantic vibe-coding.
It's even worse than that, because the way they extract the causal link is just a regex, so
"vaccines > autism"
because
"Even though the article was fraudulent and was retracted, 1 in 4 parents still believe vaccines can cause autism."
I think this could be solved much better by using even a modestly powerful LLM to do the causal extraction... The website claims "an estimated extraction precision of 83% " but I doubt this is an even remotely sensible estimate.
Also when you visit most japanese websites you can see this phenomenon.
I've read an explanation once that this is because culturally, japanese people perceive a wealth of information and choice as being re-assuring and trustworthy, while most westerners feel more re-assured by seeing less content and choice presented in a more minimalist kind of way.
I actually prefer content style of Japanese websites. I get all the relevant info on one screen instead of having to scroll/click thru tens. The western style websites are very inefficient and hide info (feels scammy with lack of info).
In my view, there is a major flaw in his argument is his distinction into pure engineering and science:
> We can make a rough distinction between pure engineering and science. There is no sharp boundary, but it’s a useful first approximation. Pure engineering seeks to produce a product that may be of some use. Science seeks understanding. If the topic is human intelligence, or cognitive capacities of other organisms, science seeks understanding of these biological systems.
If you take this approach, of course it follows that we should laugh at Tom Jones.
But a more differentiated approach is to recognize that science also falls into (at least) two categories; the science that we do because it expands our capability into something that we were previously incapable of, and the one that does not. (we typically do a lot more of the former than the latter, for obvious practical reasons)
Of course it is interesting from a historical perspective to understand the seafaring exploits of Polynesians, but as soon as there was a better way of navigating (i.e. by stars or by GPS) the investigation of this matter was relegated to the second type of science, more of a historical kind of investigation. Fundamentally we investigate things in science that are interesting because we believe the understanding we can gain from it can move us forwards somehow.
Could it be interesting to understand how Hamilton was thinking when he came up with imaginary numbers? Sure. Are a lot of mathematicians today concerning themselves with studying this? No, because the frontier has been moved far beyond.*
When you take this view, it´s clear that his statement
> These considerations bring up a minor problem with the current LLM enthusiasm: its total absurdity, as in the hypothetical cases where we recognize it at once. But there are much more serious problems than absurdity.
is not warranted. Consider the following, in his own analogy:
> These considerations bring up a minor problem with the current GPS enthusiasm: its total absurdity, as in the hypothetical cases where we recognize it at ones. But there are much more serious problems than absurdity. One is that GPS systems are designed in such a way that they cannot tell us anything about navigation, planning routes or other aspects of orientation, a matter of principle, irremediable.
* I´m making a simplifying assumption here that we can´t learn anything useful for modern navigation anymore from studying Polynesians or ants; this might well be untrue, but that is also the case for learning something about language from LLMs, which according to Chomsky is apparently impossible and not even up for debate.
I came to comments to ask a question, but considering that it is two days old already, I will try to ask you in this thread.
What you think about his argument about “not being able to distinguish possible language from impossible”?
And why is it inherent in ML design?
Does he assume that there could be such an instrument/algorithm that could do that with a certainty level higher than LLM/some ml model?
I mean, certainly they can be used to make a prediction/answer to this question, but he argues that this answer has no credibility? I mean, LLM is literally a model, ie probability distribution over what is language and what is not, what gives?
Current models are probably tuned more “strictly” to follow existing languages closely, ie that will say “no-no” to some yet-unknown language, but isn’t this improvable in theory?
Or is he arguing precisely that this “exterior” is not directly correlated with “internal processes and faculties” and cannot make such predictions in principle?
Is there any strong reason to use GeoParquet instead of straight up parquet if all I'm interested in is storing and operating on lat/lons?
I'm curious if it compresses them better or something like that. I see lots of people online saying it compresses well (but mostly compared to .shp or similar) but normal parquet (.gz.parquet or .snappy.parquet) already does that really well. So it's not clear to me if I should spend time investigating it...
I mostly process normal parquet with spark and sometimes clickhouse right now.
Based on my reading of the GeoParquet spec, the main difference is that geometries are stored as WKB using Parquet's byte array type. Byte arrays can be delta-encoded. There is also some additional metadata stored like CRS and a bounding box.
When using EPSG:4326 lat/lons, I don't think GeoParquet would give you any benefits over just having separate columns (this is what I typically do, and it's plenty fast).
If you are using range requests to fetch only parts of parquet files at a time, you could potentially sort your data using a hilbert curve, which could limit the number of row groups that need to be fetched to execute a query.
Sure but all pre-made, battle-tested tree datastructures you'd use in production in all languages already come with some form of iterator that you can just for-loop over, so the original articles point is still moot.
I think figuring out the fastest version of a shader at runtime is very non-trivial, I'm not aware of any game or engine that can do this.
I think it'd be possible in principle, because most APIs (D3D, GL, Vulkan etc) expose performance counters (which may or may not be reliable depending on the vendor), and you could in principle construct a representative test scene that you replay a couple times to measure different optimizations. But a lot of games are quite dynamic, having dynamically generated scenes and also dynamically generated shaders, so the number of combinations you might have to test seems like an obstacle. Also you might have to ask the user to spend time waiting on the benchmark to finish.
You could probably just do this ahead of time with a bunch of different GPU generations from each vendor if you have the hardware, and then hard-code the most important decision. So not saying it'd be impossible, but yeah I'm not aware of any existing infrastructure for this.
The last time I did anything like this (it was for CPU linear algebra code designed to run in very heterogeneous clusters), I first came up with a parameterization that approximated how I'd expect an algorithm to perform. Then, once for each hardware combination, you sweep through the possible parameterization space. I used log-scaled quantization to make it cheap to index into an array of function pointers based on input specifics.
The important thing to note is that you can do that computation just once, like when you install the game, and it isn't that slow. Your parameterization won't be perfect, but it's not bad to create routines that are much faster than any one implementation on nearly every architecture.
Subpoenas need some probable cause to be issued. It'd be interesting to estimate what percentage of target population you'd need to subpoena complete details on in order to build a social graph of the whole population, but I feel that that volume of legal activity would create waves even if it were only around 1%. Most court systems are overburdended as is, a significant increase would be noticeable.
Of course, if you were just getting that information as a matter of course in many arrests and were willing to accumulate it slowly over time it's very possible. I'm preoccupied with scraping issues as that's my only available avenue.
Anyway, how much that matters for an investor is hard to form a clear answer to - investors are after all not directly looking for profitability as such, but for valuation growth. The two are linked but not the same -- any investor in OpenAI today probably also places themselves into a game of chance, betting on OpenAI making more breakthroughs and increasing the cash flow even more -- not just becoming profitable at the same rate of cash flow. So there's still some of the same risk baked into this investment.
But with a new startup like LeCun's is going to be, it's 100% on the risk side and 0% on the optionality side. The path to profitability for a startup would be something like 1) a breakthrough is made 2) that breakthrough is utilized in a way that generates cash flow 3) the company becomes profitable (and at this point hopefully the valuation is good.)
There's a lot of things that can go wrong at every step here (aside from the obvious), including e.g. making a breakthrough that doesn't represent a defensible mote for your startup, failing to build the structure of the business necessary to generate cashflow, ... OpenAI et al already have a lot of that behind them, and while that doesn't mean that they don't face upcoming risks and challenges, the huge amount of cashflow they have available helps them overcome these issues far more easily than a startup, which will stop solving problems if you stop feeding money into it.