“There's a neat argument against these models doing interpolation: the manifold of the data is so sparse that it's infinitesimally unlikely for a good predictor to be doing interpolation between existing points on the manifold.”
Learning in High Dimension Always Amounts to Extrapolation
高维数据中几乎不存在插值,学习本质上是外推
插值指样本落在数据集的凸包内或其边界上,外推则是落在凸包之外。作者从经验和理论两方面论证:在任何高维(大于 100 维)数据集上,插值几乎必然不会发生,这反驳了「先进算法有效是因为能正确插值训练数据」以及「各类任务和数据集中普遍发生插值」这两种误解,并对现有插值/外推定义能否作为泛化性能指标提出挑战。
评论者用它反驳「LLM 只是超级插值」「统计鹦鹉」之类的说法,强调在高维空间里几乎一切都是外推。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“In high dimensional spaces basically everything is extrapolation including in pixel space and embedding space.”
“Definition 1. Interpolation occurs for a sample x whenever this sample belongs to the convex hull of a set of samples X,”
“These models are pretty much always extrapolating [0]”
“That's not at all what they do. This myth of "supercharged interpolation" really needs to die. Learning in High Dimension Always Amounts to Extrapolation”
“There's this idea of "supercharged interpolation", the idea that they just take out certain texts and switch out words or whatever, that is not true.”
看全部 16 条 →
“This is not at all what it is doing. "Supercharged Interpolation" is false and makes no sense.”
“"Supercharged Interpolation" is not something that actually exists.”
“"Supercharged Interpolation" is not a real thing.”
“Learning in high dimensions always results in extrapolation.”
“We know that the curse of dimensionality means LLMs are using extrapolation as an example.”
“If you like ML, this is also related with the results of this paper [0], where they show that learning in high dimensions amounts to extrapolation, as opposed to interpolation.”
“The "statistical parrot" parrots have been demonstrably wrong for years (see e.g. LeCun et al[1]).”
“In these Yann Lecun has argued that interpolation is meaningless in the high dimensional spaces these "curves" are embedded in.”
“depending on how much you care about the semantics of these terms Learning in High Dimension Always Amounts to Extrapolation”
“Learning in High Dimension Always Amounts to Extrapolation”
- Learning in High Dimension Always Amounts to Extrapolation ↗og_kalu · 2023-10-28 · 1 分
- Learning in High Dimension Always Amounts to Extrapolation ↗tosh · 2023-11-07 · 1 分