The race to build more capable AI models has an uncomfortable supply problem: the internet is running out of clean, high-quality text. What happens next should concern anyone who cares about knowledge preservation — or who builds products on top of these models.

Reports have surfaced that some AI companies are acquiring and physically destroying rare, out-of-print books purely to digitise and ingest their contents into training datasets. Not scanning them carefully for a library archive. Not donating them to universities. Shredding them — because the economics of bulk digitisation at machine speed apparently do not leave room for preservation.

Why Training Data Has Become a Bottleneck

For the first generation of large language models, the web was enough. Common Crawl, Wikipedia, GitHub, and a few curated book corpora gave teams hundreds of billions of tokens to work with. But as models scale to trillions of parameters and multiple training runs, the quality ceiling of web-scraped data becomes a hard constraint.

Rare books represent something genuinely different: dense, domain-specific, pre-internet knowledge. Specialist medical references from the 1950s. Engineering manuals. Regional histories. Legal commentaries. Content that never made it online because it predates the web and nobody had a commercial reason to digitise it.

That makes these books valuable — but valuable in a way that creates a brutal incentive structure. If you can buy a box of out-of-print titles for a few dollars each at an estate sale, scan them in bulk, and feed the text to a training pipeline, the per-token cost of that data is extremely low. The fact that the physical object gets destroyed in high-speed digitisation is, to a purely cost-focused operator, irrelevant.

What This Reveals About the Data Economy

This is not simply a story about careless tech companies. It is a signal about how the AI industry is quietly restructuring who controls knowledge and in what form.

A few observations worth sitting with:

• Irreversibility is the real problem. A rare book destroyed for training data is gone. Future researchers, historians, and developers cannot re-examine the source. The provenance of what entered the model is lost.
• The beneficiary and the cost-bearer are different parties. The AI company captures the value. Society — particularly libraries, academic institutions, and future generations — absorbs the loss.
• Synthetic data is not yet a full substitute. There is significant research into generating synthetic training corpora, but for rare, domain-specific content, synthetic generation tends to amplify existing model biases rather than introduce genuinely new knowledge.
• Regulation is lagging badly. Copyright law has been the main lens through which AI training data gets scrutinised. But copyright does not protect physical artefacts from destruction. There is no legal framework specifically preventing a private buyer from shredding a book they legally purchased.

The Parallel to Software Engineering

If you have ever inherited a legacy codebase with no documentation, no version history, and no original developers available, you understand the specific pain of irreversible information loss. You can read what the code does. You cannot easily recover why it was written that way — the context, the constraints, the decisions that were made and unmade.

Destroying source documents creates the same problem for AI systems at a civilisational scale. The model may absorb the text, but the ability to audit, cross-reference, or correct that ingested knowledge disappears with the physical object.

# The gap between "data ingested" and "knowledge preserved"
training_pipeline = {
    "input":  "rare_book_scan.pdf",
    "output": "tokenised_training_corpus",
    "artefact_status": "destroyed",
    "audit_trail": None,   # <-- this is the problem
    "source_verifiable": False
}

This matters practically for anyone building RAG pipelines, fine-tuned models, or domain-specific AI features. If your model's knowledge of, say, 20th-century West African agricultural policy came from a destroyed monograph, and that monograph contained errors or a particular ideological framing, you have no recourse. You cannot go back to the source.

What Responsible Data Stewardship Looks Like

There is a better path, and some institutions are already on it. The Internet Archive, Project Gutenberg, and various national libraries have been digitising rare texts carefully — preserving physical copies, tagging metadata, and making digitised content openly accessible. Some AI labs have begun partnering with these institutions rather than running independent acquisition pipelines.

For AI companies serious about responsible development, the baseline should be:

• Digitise without destroying. The marginal cost of preservation versus destruction is small relative to model training budgets.
• Publish dataset cards with provenance. Which sources were used? Were originals preserved? Who held the copyright?
• Support archival institutions financially. If your business model depends on centuries of accumulated human knowledge, investing in the infrastructure that preserves that knowledge is not charity — it is supply chain maintenance.
• Prefer open, auditable datasets. For teams building on top of foundation models, preference providers who are transparent about training data sourcing.

Why This Matters for Your Project

If you are building a SaaS product, mobile app, or ML feature on top of any foundation model, your outputs inherit the epistemological quality — and the ethical liabilities — of that model's training data. Choosing infrastructure partners who treat data provenance seriously is not an abstract ethical stance. It is a practical risk management decision: models trained on poorly sourced, unauditable data are harder to correct, harder to defend legally, and more likely to fail in high-stakes domain-specific applications. As your product scales, those risks compound.

Source: Hacker News / HedgieMarkets — https://xcancel.com/HedgieMarkets/status/2081534588485296565