Key points
- Richard Sutton asserts that raw computational power alone is insufficient for achieving true intelligence.
- Former OpenAI researcher Jerry Tworek claims transformer scaling has misled a generation of scientists.
- Sutton advocates for agents with world models that learn continuously from real-world experience.
Turing Award winner Richard Sutton published a new post arguing that raw compute alone is insufficient for intelligence, according to a report by The Decoder.reported He contends that AI science requires clever new methods and remains far from finished.Richard Sutton says we need smarter algorithms, not just bigger computers, to make AI truly intelligent.Sutton argues that the current paradigm of scaling transformers is insufficient and that continuous learning with world models is necessary for progress. This stance reinforces his long-held view that fundamental algorithmic shifts are necessary for progress in artificial general intelligence.
Evidence: reported
The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiastsaifleet-decoder
The figures
According to the source material, The source material references one specific year: 2019, when Sutton published his influential essay The Bitter Lesson.reported No other numerical data, such as model parameter counts, training costs or performance metrics, are provided in the report. The discussion centres on qualitative arguments about scaling limits rather than quantitative benchmarks.
Evidence: reported
The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiastsaifleet-decoder
Behind the figures
Sutton’s position is based on his analysis of historical trends in AI development, as described in his 2019 essay and recent statements.reported The Decoder reports that Sutton argues human expertise acts as a bottleneck because the real world is infinitely complex.reported He posits that any simulation of reality is microscopic compared to actual experience, limiting the effectiveness of synthetic data generation.
Evidence: reported
The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiastsaifleet-decoder
Evidence: reported
The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiastsaifleet-decoder
What changes, what does not
Sutton’s core argument remains consistent: general methods that scale with compute eventually outperform human-engineered approaches. However, he now emphasises that current large language models fail to scale effectively because they do not keep learning after training. The change lies in his critique of synthetic data, which he calls a big mistake, and his call for continuous learning mechanisms that preserve existing knowledge.
What builders can do with it
Developers can use Sutton’s arguments to evaluate their reliance on scaling transformer models versus investing in novel learning architectures. The material suggests builders should explore agents with world models that continuously learn from their own experience. This implies a shift away from static training on internet text toward systems that adapt in real time without catastrophic forgetting.
Why it matters
- Policy readers must consider whether current AI safety frameworks address continuous learning risks, as Sutton highlights the need for agents that learn from experience.
- The claim that synthetic data is a big mistake suggests regulatory scrutiny of data provenance may be insufficient if the underlying scaling method is flawed.
- Tworek’s warning that scientists have been reward-hacked indicates a potential systemic bias in research funding and talent allocation toward scaling rather than innovation.
- The emphasis on world models implies future AI systems may require different evaluation metrics than those used for static language models.
Topics: AI research, scaling limits, Richard Sutton, world models, synthetic data
Updated
For readers
Policy
- Sutton’s call for new learning methods may influence future AI governance frameworks focusing on continuous learning risks.
- The critique of synthetic data suggests regulators should consider the quality and origin of training data more carefully.
Researchers
- Tworek’s warning about reward-hacked brains highlights the need for diverse research approaches beyond transformer scaling.
- Sutton’s emphasis on world models provides a direction for developing agents that learn continuously from experience.
Investors
- The growing wave of startups betting on new AI approaches indicates market interest in alternatives to scaling transformers.
- Sutton’s arguments suggest that investing in novel algorithms may yield better long-term returns than simply increasing compute.
Ask the article
Local keyword search of this article; no generated answers.Did this article give you the evidence you needed?
Your choice stays on this device. No public tally.Questions readers ask
What is the bitter lesson in AI?
The bitter lesson argues that general methods scaling with compute eventually outperform sophisticated human-engineered approaches that rely on human assumptions.
Why does Sutton criticise synthetic data?
Sutton calls synthetic data a big mistake because the real world is infinitely complex and any simulation is only microscopic compared to actual experience.
What does Tworek mean by reward-hacked brains?
Tworek argues that the temporary success of transformer scaling, combined with a misreading of the bitter lesson, has misled scientists into believing scaling alone is sufficient.
Connected stories
Reading receipt
- Richard Sutton asserts that raw computational power alone is insufficient for achieving true intelligence.
- Former OpenAI researcher Jerry Tworek claims transformer scaling has misled a generation of scientists.
- Sutton advocates for agents with world models that learn continuously from real-world experience.