按年被提到
20252025 · 4 次2026
大模型与语言模型›提示工程与输出稳定性·arXiv:2504.06219 · cs.CL

Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs

Dongyang Fan, Vinko Sabolčec, Matin Ansaripour, Ayush Kumar Tarun, Martin Jaggi, Antoine Bosselut, Imanol Schlag · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2025-07-11 · 最近 2025-09-05
2不同的人
2活跃月份
3讨论串
2被引(OpenAlex)
arXiv 摘要(候选层不写导读)

The increasing adoption of web crawling opt-outs by copyright holders of online content raises critical questions about the impact of data compliance on large language model (LLM) performance. However, little is known about how these restrictions (and the resultant filtering of pretraining datasets) affect the capabilities of models trained using these corpora. In this work, we conceptualize this effect as the $\textit{data compliance gap}$ (DCG), which quantifies the performance difference between models trained on datasets that comply with web crawling opt-outs, and those that do not. We measure the data compliance gap in two settings: pretraining models from scratch and continual pretraining from existing compliant models (simulating a setting where copyrighted data could be integrated later in pretraining). Our experiments with 1.5B models show that, as of January 2025, compliance with web data opt-outs does not degrade general knowledge acquisition (close to 0\% DCG). However, in specialized domains such as biomedical research, excluding major publishers leads to performance declines. These findings suggest that while general-purpose LLMs can be trained to perform equally well using fully open data, performance in specialized domains may benefit from access to high-quality copyrighted sources later in training. Our study provides empirical insights into the long-debated trade-off between data compliance and downstream model performance, informing future discussions on AI training practices and policy decisions. Our website is available at https://data-compliance.github.io/.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论lllllm · 2025-07

“Yes this is an interesting question. In our arxiv paper [1] we did study this for news articles, and also removed duplicates of articles (decontamination). We did not observe an impact on the downstream accuracy of the LLM, in the case of news data. [1] https://arxiv.org/abs/2504.06219”

ETH Zurich and EPFL to release a LLM developed on public infrastructure原评论 ↗
评论lllllm · 2025-07

“this is what this paper tries to answer: https://arxiv.org/abs/2504.06219 the quality gap is surprisingly small between compliant and not”

ETH Zurich and EPFL to release a LLM developed on public infrastructure原评论 ↗
评论lllllm · 2025-09

“common crawl anyway respects the CCbot opt-out every time they do a crawl. we went a step further because back in old ages (2013 is our oldest training data) LLMs did not exist, so website owners opting out today of AI crawlers might like the option to also remove their past contents. arguments can be made either way but we tried to remain on the cautio…”

Apertus 70B: Truly Open - Swiss LLM by ETH, EPFL and CSCS原评论 ↗
被提交为 story · 1 次