Bringing up information theory and godels incompleteness theorem in regards to ai advancement is like saying there are hard physical limits on shrinking the dye in the 1980s. Yes hard axiomatic limits exist, but within those limits is a huge problem space. Even if we limit it to solving problems on a screen using existing techniques, that problem space is huge and occupying a large amount of time for the knowledge workers who make up ~25% of the labor force.
Unlike shrinking the dye though we are not near that limit, we still have a lot of improvement to make before AI can solve all the problems in that space. I think we can both agree the current LLMs aren’t able to replace all screen work. But it is making big strides in that problem space, such as going from failing basic arithmetic to solving the hardest math problems in a couple years.
Like with shrinking the dye there’s not going to be a singular breakthrough, like recursive self improvement, that’ll throw us up against those limits almost immediately. Progress is done incrementally over years with a bunch of different smaller improvements, like opaque recurrence. This does have some acceleration as the technology helps with the design of the technology, computers help chip architects design better chips, which make better computers, but a human is still required. Same with AI, every lab is probably heavily relying on coding agents to assist them with developing and deploying new techniques. The AI may not be creating these techniques, but it is accelerating the creation of these techniques.
Once again, you’re trying to analogize your way into something you haven’t studied.
What exactly is your purpose here? You still can’t even articulate the fundamental flaws LLMs have with tasks not fully within the training data. Right now, today, there is no frontier model that can operate on tasks without resorting to reward hacking once its outside the small class of problems its training data covers. The solution for this is to invest significant time from domain experts to meticulously define how to solve tasks in other domains.
Here’s a trivial example: Try getting Claude to generate coherent COBOL. Or TCL. Or even Powershell. Any language that has low representation on StackOverflow is a language that Claude can’t speak until someone teaches it how. Even the Python it generates has limited expressiveness or extensibility.
Everywhere you look, the AI is limited by the fact that it can’t generate its own new information. Navier-Stokes and statements in pure mathematics like it are the absolute best case scenario for agentic work against rigorous specification. The theorem statement itself is already a rigorous specification. It has undergone decades of auditing by the mathematical community and its rendering in Lean is a straightforward translation defined in terms of battle-tested mathematical objects from mathlib. The verifier, the Lean theorem prover, has been extensively audited and specifically designed to avoid the types of unsoundness that would make it vulnerable to reward hacks.
No other domains outside of mathematics have such rigorous specifications. Yet, somehow, you believe there’s magic pixie dust somewhere within the LLM that will help it achieve something without human interventions and that, somehow, we’re “close” to that accomplishment. Don’t quit your day job.
I have accepted that llms have hard limits, like you said with regards to information theory and godels incompleteness theorem. I don’t have enough knowledge to articulate those limits, just like I don’t know the physics that prevents you from shrinking the dye further but I accept that they exist.
So we agree there are hard limits, what I am saying is:
within those limits is a huge problem space
millions of people are putting labor in that problem space today
LLMs have not saturated that problem space yet
LLMs are making gains in that problem space, like with math and reasoning.
I’m not arguing where that “finish line” is because I know I don’t know enough about that. I do understand that 99.999% of problems aren’t passed that finish line and there’s a lot to go before LLMs reach that line.
Almost all use cases for llms don’t require creating novel math techniques, so why are you focusing on them? If “can’t create genuinely new techniques” disqualifies something from being economically transformative or intellectually significant, that standard would disqualify most human knowledge labor too.
Bringing up information theory and godels incompleteness theorem in regards to ai advancement is like saying there are hard physical limits on shrinking the dye in the 1980s. Yes hard axiomatic limits exist, but within those limits is a huge problem space. Even if we limit it to solving problems on a screen using existing techniques, that problem space is huge and occupying a large amount of time for the knowledge workers who make up ~25% of the labor force.
Unlike shrinking the dye though we are not near that limit, we still have a lot of improvement to make before AI can solve all the problems in that space. I think we can both agree the current LLMs aren’t able to replace all screen work. But it is making big strides in that problem space, such as going from failing basic arithmetic to solving the hardest math problems in a couple years.
Like with shrinking the dye there’s not going to be a singular breakthrough, like recursive self improvement, that’ll throw us up against those limits almost immediately. Progress is done incrementally over years with a bunch of different smaller improvements, like opaque recurrence. This does have some acceleration as the technology helps with the design of the technology, computers help chip architects design better chips, which make better computers, but a human is still required. Same with AI, every lab is probably heavily relying on coding agents to assist them with developing and deploying new techniques. The AI may not be creating these techniques, but it is accelerating the creation of these techniques.
Once again, you’re trying to analogize your way into something you haven’t studied.
What exactly is your purpose here? You still can’t even articulate the fundamental flaws LLMs have with tasks not fully within the training data. Right now, today, there is no frontier model that can operate on tasks without resorting to reward hacking once its outside the small class of problems its training data covers. The solution for this is to invest significant time from domain experts to meticulously define how to solve tasks in other domains.
Here’s a trivial example: Try getting Claude to generate coherent COBOL. Or TCL. Or even Powershell. Any language that has low representation on StackOverflow is a language that Claude can’t speak until someone teaches it how. Even the Python it generates has limited expressiveness or extensibility.
Everywhere you look, the AI is limited by the fact that it can’t generate its own new information. Navier-Stokes and statements in pure mathematics like it are the absolute best case scenario for agentic work against rigorous specification. The theorem statement itself is already a rigorous specification. It has undergone decades of auditing by the mathematical community and its rendering in Lean is a straightforward translation defined in terms of battle-tested mathematical objects from mathlib. The verifier, the Lean theorem prover, has been extensively audited and specifically designed to avoid the types of unsoundness that would make it vulnerable to reward hacks.
No other domains outside of mathematics have such rigorous specifications. Yet, somehow, you believe there’s magic pixie dust somewhere within the LLM that will help it achieve something without human interventions and that, somehow, we’re “close” to that accomplishment. Don’t quit your day job.
Explain to me how the analogy is wrong then.
I have accepted that llms have hard limits, like you said with regards to information theory and godels incompleteness theorem. I don’t have enough knowledge to articulate those limits, just like I don’t know the physics that prevents you from shrinking the dye further but I accept that they exist.
So we agree there are hard limits, what I am saying is:
I’m not arguing where that “finish line” is because I know I don’t know enough about that. I do understand that 99.999% of problems aren’t passed that finish line and there’s a lot to go before LLMs reach that line.
Almost all use cases for llms don’t require creating novel math techniques, so why are you focusing on them? If “can’t create genuinely new techniques” disqualifies something from being economically transformative or intellectually significant, that standard would disqualify most human knowledge labor too.