按年被提到
20242024 · 4 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2402.00861 · cs.CL

Evaluating Large Language Models for Generalization and Robustness via Data Compression

Yucheng Li, Yunhao Guo, Frank Guerin, Chenghua Lin · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-05-28 · 最近 2026-03-02
2不同的人
5活跃月份
6讨论串
7被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Existing methods for evaluating large language models face challenges such as data contamination, sensitivity to prompts, and the high cost of benchmark creation. To address this, we propose a lossless data compression based evaluation approach that tests how models' predictive abilities generalize after their training cutoff. Specifically, we collect comprehensive test data spanning 83 months from 2017 to 2023 and split the data into training and testing periods according to models' training data cutoff. We measure: 1) the compression performance on the testing period as a measure of generalization on unseen data; and 2) the performance gap between the training and testing period as a measure of robustness. Our experiments test 14 representative large language models with various sizes on sources including Wikipedia, news articles, code, arXiv papers, and multi-modal data. We find that the compression rate of many models reduces significantly after their cutoff date, but models such as Mistral and Llama-2 demonstrate a good balance between performance and robustness. Results also suggest that models struggle to generalize on news and code data, but work especially well on arXiv papers. We also find the context size and tokenization implementation have a big impact of on the overall compression performance.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论bravura · 2024-05

“Predicting the future. I recently saw a paper getting LLM evaluation completely right in many respects: https://arxiv.org/abs//2402.00861 "Evaluating Large Language Models for Generalization and Robustness via Data Compression" They evaluate LLMs by seeing how well they compress text/news/science from the future, over time. In a sense, this measures ho…”

Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Advanced Results原评论 ↗
评论bravura · 2024-11

“Regarding keeping the test set private to avoid contamination, the comments about leakage are spot on. The real test set should always be the future. We should evaluate LLMs on text from beyond their knowledge cutoff date, by computing their per-byte perplexity or per-byte compression ratio. There's a deep theoretical connection between compression and l…”

FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI原评论 ↗
评论bravura · 2024-12

“"It is well established that compression is essentially prediction, which effectively links compression and langauge models (Delétang et al., 2023). The source coding theory from Shannon’s information theory (Shannon, 1948) suggests that the number of bits required by an optimal entropy encoder to compress a message... is equal to the NLL of the message …”

Ts_zip: Text Compression Using Large Language Models原评论 ↗
评论bravura · 2024-12

“antirez, it's probably identical to the approach in this paper: Li et al 2024, "Evaluating Large Language Models for Generalization and Robustness via Data Compression" (https://ar5iv.labs.arxiv.org/html//2402.00861). There's a pretty straight line from assigning probabilities (to a sequence of tokens) to arithmetic compression as an optimal compression …”

Short Message Compression Using LLMs原评论 ↗
评论bravura · 2026-01

“A rigorous approach to predicting the future of text was proposed by Li et al 2024, "Evaluating Large Language Models for Generalization and Robustness via Data Compression" (https://ar5iv.labs.arxiv.org/html//2402.00861) and I think that work should get more recognition. They measure compression (perplexity) on future Wikipedia, news articles, code, arX…”

TimeCapsuleLLM: LLM trained only on data from 1800-1875原评论 ↗
评论Otterly99 · 2026-03

“There is this paper that proposed data compression as a way to judge the ability of a LLM to "understand" things correctly, training on older texts and trying to predict more recent articles: https://ar5iv.labs.arxiv.org/html//2402.00861”