按年被提到
20242024 · 3 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2405.16684 · cs.CL

gzip Predicts Data-dependent Scaling Laws

Rohan Pandey · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-05-28 · 最近 2024-06-12
2不同的人
2活跃月份
3讨论串
4被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Past work has established scaling laws that predict the performance of a neural language model (LM) as a function of its parameter count and the number of tokens it's trained on, enabling optimal allocation of a fixed compute budget. Are these scaling laws agnostic to training data as some prior work suggests? We generate training datasets of varying complexities by modulating the syntactic properties of a PCFG, finding that 1) scaling laws are sensitive to differences in data complexity and that 2) gzip, a compression algorithm, is an effective predictor of how data complexity impacts scaling properties. We propose a new data-dependent scaling law for LM's that accounts for the training data's gzip-compressibility; its compute-optimal frontier increases in dataset size preference (over parameter count preference) as training data becomes harder to compress.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论KhoomeiK · 2024-05

“Hey HN! Chinchilla (DeepMind 2022) tells us that when we scale up our language model training, we should scale the parameters and data equally. Over the last several months I've been hacking on a research project to determine if the optimal compute allocation (scaling law) for training an LLM is sensitive to training data complexity. I found that as data…”

Show HN: Chinchilla Scaling Laws Are Not Universal原评论 ↗
被提交为 story · 2 次