“Worth checking out the explanations of novel jokes on page 38 of the same paper.”
PaLM: Scaling Language Modeling with Pathways
5400 亿参数稠密 Transformer,少样本能力随规模跃升
作者用 Pathways 系统在 6144 块 TPU v4 上训练了 5400 亿参数的稠密 Transformer 语言模型 PaLM,在数百个语言理解与生成基准上取得少样本学习的最优结果。在一批多步推理任务上它超过了微调的最优水平,在 BIG-bench 上超过人类平均水平,许多任务随模型规模出现不连续的跃升;论文还分析了偏见与毒性、不同规模下的训练数据记忆,并讨论伦理问题。
评论者用它说明规模带来的惊人能力,常引用其中解释笑话、回答文化梗的例子,以及训练 token 数、与 LaMDA 的对比等细节。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“their performance continues to surprise us as they scale[2].”
“The PaLM paper[1] says it made use of 700B tokens.”
“And then looking at the PaLM paper PaLM on page 5 we have:”
“Check section 7 of the PaLM paper for example.”
“Note: 540B parameters -> Google Research's PaLM.”
看全部 13 条 →
“examples on Page 36 of the paper: PaLM and tell me if it's just”
“I believe we're already at the footsteps of AI understanding cultural references”
“Work such as PaLM[2] show that language models evolve the ability to "reason" over the internal data representation.”
“PaLM itself is probably quite a bit better than LamDa in several tasks, according to a chart and a couple of tables here: PaLM”
“PaLM itself is probably quite a bit better than LamDa in several tasks, according to a chart and a couple of tables here: PaLM”
“You can look at any training run comparing multiple parameter sizes on the dataset.”
“Btw, PALM2 has far fewer parameters than V1.”
- Palm: Scaling Language Modeling with Pathways [pdf] ↗lawrenceyan · 2022-04-09 · 2 分
- Palm: Scaling Language Modeling with Pathways ↗ArtWomb · 2022-04-22 · 2 分
- Palm: Scaling Language Modeling with Pathways ↗bookofjoe · 2022-06-21 · 1 分