Wake me when it's going to spontaneously fix my leaky faucet because if it doesn't do it nobody else will. Until it has that capability I don't really care.
Is solving a snake like puzzle game in the least number of moves really what defines intelligence?
It's pretty close to how we measure IQ. The standard test is basically a series of puzzles.<p>I know there's a lot of people who complain that we're moving goalposts, but I think that the progress in LLMs really just shows that we don't know how to really measure intelligence in the first place, if we understand it as "human-like agency / ingenuity / adaptability". For decades, we used the Turing test as the proxy for AGI, but then early LLMs could clearly pass for a human in a casual conversation while clearly not matching human performance on most other tasks.<p>Since then, every benchmark we come up with, it turns out that an LLM can be fine-tuned to solve it while still clearly lacking <i>something</i>. They make very non-human mistakes, are easily tricked because they have a pretty tenuous grasp of reality, etc. But I think this just shows that AGI is a meaningless marketing term. We could as well be arguing if they have souls.
When I was 18, my high school girlfriend took me to the local Mensa chapter’s New Year’s party because her mother was a member and she was used to hanging out there.<p>It was a useful lesson that whatever IQ tests measure, it is completely devoid of value or interest to me.
No, but figuring out that you're <i>playing a snake-like puzzle game at all</i> in an extremely general input domain and <i>then</i> solving it in the least number of moves definitely feels like evidence of intelligence.
You forget the benchmark. The human subjects were told they were being timed. If you believe the lowest time is the primary metric you will absolutely trial and error at speed instead of meticulously plan out your moves to minimize that metric.<p>LLMs are not timed and given that it costs tens of thousands of dollars to run this test they're not optimizing for speed.<p>So you've got a deceptive test, with one metric being told to humans and not applied to LLM and a hidden metric humans aren't aware of but LLMs are as the test.<p>This is flawed from the get go. It almost seems like this was deliberately setup to be able to claim AGI and superiority of LLMs
There's currently a big market for figuring out ways to measure intelligence. With a particular interest in ways that humans can score much higher than LLMs. If you have some ideas please do share!
Why? Seems like benchmarks that closely mirror the tasks you'd want an LLM to help with would be a lot more useful than some general intelligence benchmark.
I am no where close to qualified to do that. Hell, experts can't even define what intelligence is, much less define a test for it
This is like arguing about whether a hot dog is a sandwich (of course it is) or whether the chicken or the egg was first (obviously the egg since all chickens come from eggs). Intelligence is just problem solving in the context of self-awareness. Machines don't have it and never will but they can simulate the process given inputs. You can argue whether humans and animals truly possess self-awareness and in what degree, but the definition of intelligence is as simple as the hot dog debate.
It was defined in Animal Intelligence by George John Ramones in 1882 as "intelligence is the capacity to do the right thing at the right time. It is the ability to respond to the opportunities and challenges presented by a context"
I'm also unclear as to how basic inferential logic puzzles spells out intelligence<p>I think if you summed up measures of intelligence as 'can it do basic symbolic logic in a chain with memory' then yes, you've now achieved the intelligence of an e. coli colony [0], congratulations<p>[0] <a href="https://journals.aps.org/prx/abstract/10.1103/PhysRevX.10.031017" rel="nofollow">https://journals.aps.org/prx/abstract/10.1103/PhysRevX.10.03...</a>
You should read more on the ARC prize, it actually has a pretty long history. We're on the 3rd iteration because they keep getting saturated. If you look at the score history over time on ARC AGI 1, 2 and 3 it's pretty impressive.<p><a href="https://arcprize.org/">https://arcprize.org/</a>
1.5 years ago Gemini Pro 2.5 needed 1 page of thinking for every move in tic-tac-toe.<p>Playing tic-tac-toe or snake does not imply AGI, but is required to claim AGI.
If you have never seen the game before probably.
[dead]
The instant/no reasoning performed extremely well<p><pre><code> none 35.2%, $49,791 96.7%, $23,457
</code></pre>
35.2% on the standard harness, that's above Opus 5 on high.
"For a cost comparison, during our controlled testing, human participants were paid $115 per 90-minute session, plus $5 per game completed. Participants attempted approximately nine games per session, roughly $12.78 per attempted game before bonuses.<p>Most of this fee pays for the participant’s time and willingness to take the test, rather than the energy their brain uses (a closer proxy to compare with AI). If we look at only the brain’s energy, and price it as electricity, the estimate drops to about 0.6 cents per session, or 0.067 cents per game attempted."<p>Well I dont know about all of you, but I am celebrating meat based humans...
I think raw brain energy is not a fair comparison. Humans are not willing and able to serve requests at identical competence all hours of the day. You have to invest considerable resources to get a person to even do so for part of the day.
Why are you making the assumption that a person's time is worthless? I'd argue that it is the single most valuable resource we all have.
> Astra’s progress helps clarify which AI capabilities are out of reach and which questions remain open.<p>Okay. But I don't think this entire article at all explained which AI capabilities remain out of reach. Did I miss something? Other than "oh I guess it could still get even more superhuman on ARC-AGI-3 than it is?"
Anything you can verify to be right or wrong can be done by a model. All benchmarks will be saturated
Disagree.<p>Examples:<p>- predict a coinflip: easy to verify, hard to learn<p>- earn $100: easy to verify, hard to learn<p>- increase paid subscriptions in an A/B test: easy to verify, hard to learn<p>I won't get into it, but there are many properties beyond verifiability that are needed to saturate a benchmark.
these just need more compute:<p>- earn $100: easy to verify, hard to learn<p>- increase paid subscriptions in an A/B test: easy to verify, hard to learn<p>but we both know these examples go against the spirit of my point
Yup. Only subjective taste remains.
What are these numbers? Why do they add up to a few hundred thousand dollars?
Who paid for that? With what?
what the hell is that score/cost curve lol
99.9% with the right harness? Ok, we're at AGI then.<p>Prediction:<p>We will now see the goalposts moved towards "well, a human costs less / is more efficient" - that will prevail for a few months until they come up with some other test that humans can do easily but is hard for the bots. This cycle will continue for ever and in 25 years, despite having hyper intelligent embodied robots or whatever, we'll still be arguing about if the singularity is here and if we're at AGI for the rest of my life most likely.
They explain it here: <a href="https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/" rel="nofollow">https://openai.com/index/how-two-settings-tripled-our-arc-ag...</a><p>TLDR: The official ARC harness throws away old context and reasoning. No real-world harness is this bad, the model has to re-learn the game repeatedly. OpenAI basically just added standard compaction. Their harness is still "general".
The goal moving is by design, that’s why they use something as ill defined as AGI
We were already there with Opus 5 -> <a href="https://www.linkedin.com/pulse/nvidias-harness-hit-100-arc-agi-3-kubesimplify-chfvf" rel="nofollow">https://www.linkedin.com/pulse/nvidias-harness-hit-100-arc-a...</a>
Then why is unemployment around 4%? You believe we have AGI and yet it can’t do anyone’s job?
That’s because AGI, like a lot of terms, has no meaning besides what each individual subjectively projects onto it.
I agree, like the average human isn't generally intelligent.<p>IMO, AGI is literally no different from ASI, though people think it is. Like, Imagine you have 1,000 generally intelligent humans working for you (which nobody is really) and you were to point them at your pet project. That would be amazing!
> We will now see the goalposts moved<p>It's already happening :)