按年被提到
20262026 · 2 次2026
大模型与语言模型›检索增强与知识库·arXiv:2608.15896 · cs.GT

When Search Eats the Web: A Model of Corpus Erosion under Generative Extraction

Sylvain Peyronnet · 2026
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2026-08-18 · 最近 2026-09-04
2不同的人
2活跃月份
2讨论串
0被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Generative search engines (GSEs) answer user queries directly from crawled web content. The capture of value from the corpus without a visit returned to the source (we call this capture extraction) diverts the traffic that finances content production. In response, publishers may restrict crawler access to their websites. In this paper, we model the crawlable corpus as a common-pool resource: the crawlable commons. It is described by three quantities: volume, average quality, and lifetime. Under two types of responses of publishers we prove that extraction degrades all three at once: publishers opt out, renewal loses its funding, and content becomes more perishable. After a given erosion threshold, the corpus goes extinct. A myopic GSE can cross this threshold, a long-run oriented GSE stays below it. We extend our model to several competing engines and prove, under a concavity condition on the steady-state value of the commons, that the symmetric equilibrium extraction rate is nondecreasing in their number and converges to the threshold. Adding users who strictly prefer direct answers, the assumption most favorable to extraction, we prove that the socially optimal extraction rate lies strictly below the erosion threshold, and no higher than the single engine's sustainable optimum. Finally, we discuss seven survival mechanisms.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)
讨论串评论sypsyp · 2026-09

“Generative search engines may be eating the web they depend on. Spoiler: beyond a critical extraction rate, the crawlable web collapses. Spoiler bis: competition makes it worse. The paper adopts a game theoretical approach to proves it, then look at seven mechanisms that could keep the commons alive.”

When Search Eats the Web: A Model of Corpus Erosion Under Generative Extraction原评论 ↗
被提交为 story · 2 次