“Predicting the future. I recently saw a paper getting LLM evaluation completely right in many respects: https://arxiv.org/abs//2402.00861 "Evaluating Large Language Models for Generalization and Robustness via Data Compression" They evaluate LLMs by seeing how well they compress text/news/science from the future, over time. In a sense, this measures ho…”
Evaluating Large Language Models for Generalization and Robustness via Data Compression
Existing methods for evaluating large language models face challenges such as data contamination, sensitivity to prompts, and the high cost of benchmark creation. To address this, we propose a lossless data compression based evaluation approach that tests how models' predictive abilities generalize after their training cutoff. Specifically, we collect comprehensive test data spanning 83 months from 2017 to 2023 and split the data into training and testing periods according to models' training data cutoff. We measure: 1) the compression performance on the testing period as a measure of generalization on unseen data; and 2) the performance gap between the training and testing period as a measure of robustness. Our experiments test 14 representative large language models with various sizes on sources including Wikipedia, news articles, code, arXiv papers, and multi-modal data. We find that the compression rate of many models reduces significantly after their cutoff date, but models such as Mistral and Llama-2 demonstrate a good balance between performance and robustness. Results also suggest that models struggle to generalize on news and code data, but work especially well on arXiv papers. We also find the context size and tokenization implementation have a big impact of on the overall compression performance.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Regarding keeping the test set private to avoid contamination, the comments about leakage are spot on. The real test set should always be the future. We should evaluate LLMs on text from beyond their knowledge cutoff date, by computing their per-byte perplexity or per-byte compression ratio. There's a deep theoretical connection between compression and l…”
“"It is well established that compression is essentially prediction, which effectively links compression and langauge models (Delétang et al., 2023). The source coding theory from Shannon’s information theory (Shannon, 1948) suggests that the number of bits required by an optimal entropy encoder to compress a message... is equal to the NLL of the message …”
“antirez, it's probably identical to the approach in this paper: Li et al 2024, "Evaluating Large Language Models for Generalization and Robustness via Data Compression" (https://ar5iv.labs.arxiv.org/html//2402.00861). There's a pretty straight line from assigning probabilities (to a sequence of tokens) to arithmetic compression as an optimal compression …”
“A rigorous approach to predicting the future of text was proposed by Li et al 2024, "Evaluating Large Language Models for Generalization and Robustness via Data Compression" (https://ar5iv.labs.arxiv.org/html//2402.00861) and I think that work should get more recognition. They measure compression (perplexity) on future Wikipedia, news articles, code, arX…”
“There is this paper that proposed data compression as a way to judge the ability of a LLM to "understand" things correctly, training on older texts and trying to predict more recent articles: https://ar5iv.labs.arxiv.org/html//2402.00861”