• home
• about
• new
• essays
• AI
• book

Frank Visser, graduated as a psychologist of culture and religion, founded IntegralWorld in 1997. He worked as production manager for various publishing houses and as service manager for various internet companies and lives in Amsterdam. Books: Ken Wilber: Thought as Passion (SUNY, 2003), and The Corona Conspiracy: Combatting Disinformation about the Coronavirus (Kindle, 2020).

Model Collapse

The AI Industry's Self-Consumption Debate

Frank Visser / ChatGPT

Introduction

The phrase model collapse has become one of the most discussed and misunderstood concepts in contemporary artificial intelligence. It refers to a potential failure mode in which AI systems increasingly train on data generated by previous AI systems, leading to a gradual degradation of quality, diversity, and reliability. Some researchers see model collapse as an existential threat to future AI development; others view it as a manageable technical challenge. The controversy reveals deeper disagreements about the nature of intelligence, learning, and the sustainability of the current AI paradigm.[1]

Model Collapse: When AI Learns from Itself

Modern large language models are trained on enormous collections of human-generated text gathered from books, articles, websites, forums, and social media. The success of systems such as OpenAI's GPT series, Google's Gemini, and Anthropic's Claude depends heavily on the richness and diversity of this human-produced corpus.

The concern arises because the internet is becoming increasingly saturated with AI-generated content. Articles, blog posts, social media comments, product reviews, and even academic summaries are now routinely produced by machines. Future AI models may therefore encounter training datasets contaminated by earlier generations of AI output.

Researchers compare this process to photocopying a photocopy repeatedly. Each generation introduces subtle distortions. Over time, information is lost, patterns become exaggerated, and diversity declines.

A widely discussed 2024 paper described this phenomenon as a statistical collapse in which rare but important features of the original data distribution gradually disappear. Models trained on synthetic data begin to reproduce only the most common patterns while neglecting outliers and nuances.

The Mathematical Basis of the Concern

The argument for model collapse is grounded in information theory and statistics.

When a model generates text, it does not perfectly reproduce the full complexity of the data on which it was trained. Instead, it approximates underlying probability distributions. The generated output therefore contains less information than the original dataset.

If future models are trained on these approximations rather than on primary human-generated material, small losses accumulate. The resulting system may become increasingly homogeneous, repetitive, and detached from reality.

Researchers describe this process as a form of distributional drift. The model begins learning from its own simplified representation of the world rather than from the world itself.

This concern is especially significant because AI-generated text often sounds authoritative regardless of its accuracy. Errors may therefore become amplified rather than corrected.

The Strong Version of the Collapse Thesis

Some commentators have interpreted model collapse as evidence that the current AI boom contains the seeds of its own destruction.

According to this perspective, large language models are dependent upon a finite reservoir of human-created content. Once that reservoir is overwhelmed by synthetic data, the quality of future systems will inevitably decline.

This argument fits a broader narrative about diminishing returns in AI. Training datasets have already absorbed much of the publicly available internet. Some estimates suggest that high-quality human-generated text may become increasingly scarce as a training resource.

In this scenario, AI development resembles an ecological system consuming its own environment. The more successful AI becomes, the more it transforms the information ecosystem upon which it depends.

The Skeptics Respond

Many AI researchers believe these fears are overstated.

First, training pipelines already employ sophisticated filtering mechanisms. Companies actively remove low-quality or duplicated content from training datasets. They are highly aware of synthetic contamination and have strong incentives to avoid it.

Second, not all synthetic data is harmful. In fact, carefully curated synthetic data is already widely used to improve AI performance.

The distinction lies between uncontrolled synthetic data and deliberately generated synthetic data. High-quality synthetic examples can expand training sets, create rare scenarios, and improve reasoning performance.

Some researchers therefore argue that model collapse is not an inevitable consequence of synthetic training but a consequence of poor synthetic training.

The Synthetic Data Revolution

Ironically, the same phenomenon feared as a source of collapse is also viewed as a solution to data scarcity.

Many leading laboratories now use AI-generated data to train newer models. This process is often called distillation or self-improvement.

A powerful model generates examples, explanations, or reasoning traces. These outputs are then filtered and used to train a smaller or newer model.

Supporters argue that human learning itself often involves learning from abstractions, summaries, textbooks, and cultural artifacts rather than from direct experience. AI systems may likewise benefit from structured synthetic knowledge.

The controversy therefore hinges on quality control rather than on synthetic data itself.

A Deeper Philosophical Question

The debate touches a surprisingly old philosophical issue: can a representation generate genuinely new knowledge?

Critics of current AI systems often argue that language models merely remix existing information. If that is true, repeated self-training could indeed become increasingly sterile.

Supporters counter that complex systems can generate novel insights through recombination, abstraction, and self-correction. Human science itself operates partly through this process. Researchers build upon previous summaries, theories, and interpretations rather than returning to raw observations at every step.

The key question becomes whether AI-generated abstractions preserve enough contact with reality.

Why the Controversy Matters

The model collapse debate exposes a central tension in AI development.

On one side lies the aspiration toward increasingly autonomous machine intelligence capable of generating its own educational material, research, and knowledge.

On the other lies the recognition that intelligence may remain dependent upon continual engagement with reality and with human experience.

For critics of AI hype, model collapse serves as a reminder that statistical systems cannot indefinitely bootstrap themselves into greater intelligence without external grounding.

For AI optimists, it represents a technical challenge akin to noise reduction, data cleaning, and quality assurance.

Conclusion: A Map Without a Territory?

The controversy surrounding model collapse is ultimately not just about datasets. It concerns the relationship between information and reality. AI systems derive their power from patterns embedded in human knowledge, culture, and experience. The fear is that an increasingly synthetic information environment may sever that connection.

Yet predictions of imminent collapse appear premature. AI laboratories are acutely aware of the problem and are actively developing methods to manage synthetic data. The more likely future is neither catastrophic degeneration nor effortless self-improvement, but an ongoing struggle to maintain contact between machine-generated abstractions and the human world from which they ultimately derive meaning.

Model collapse may therefore be less a fatal flaw than a modern version of an ancient epistemological warning: a map that becomes disconnected from the territory eventually ceases to be useful. The challenge for AI is ensuring that its maps remain anchored to reality even as they become increasingly capable of drawing themselves.

NOTES

[1] Parshin Shojaee et al., "The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity", machinelearning.apple.com, June 2025. (PDF)