按年被提到
20232023 · 3 次2026
大模型与语言模型›词向量与预训练模型·arXiv:2303.09540 · cs.LG

SemDeDup: Data-efficient learning at web-scale through semantic deduplication

Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, Ari S. Morcos · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-03-17 · 最近 2025-10-18
2不同的人
6活跃月份
6讨论串
74被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Progress in machine learning has been driven in large part by massive increases in data. However, large web-scale datasets such as LAION are largely uncurated beyond searches for exact duplicates, potentially leaving much redundancy. Here, we introduce SemDeDup, a method which leverages embeddings from pre-trained models to identify and remove semantic duplicates: data pairs which are semantically similar, but not exactly identical. Removing semantic duplicates preserves performance and speeds up learning. Analyzing a subset of LAION, we show that SemDeDup can remove 50% of the data with minimal performance loss, effectively halving training time. Moreover, performance increases out of distribution. Also, analyzing language models trained on C4, a partially curated dataset, we show that SemDeDup improves over prior approaches while providing efficiency gains. SemDeDup provides an example of how simple ways of leveraging quality embeddings can be used to make models learn faster with less data.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论godelski · 2023-05

“> I don't think you took more than a passing glance, if any at those papers. I'll assume in good faith but let's try to keep this in mind both ways. > What you describe is impossible with these 3. Definitely possible. I did not write my comment as a paper but I did provide plenty of evidence. I specifically ask that you pay close attention to my HumanE…”

Governance of Superintelligence原评论 ↗
评论godelski · 2023-07

“For this case, yes. In the general sense, no. That's what I'm saying. And to be clear, this is actually a common problem, not an uncommon one. Here's a bit more why. In general, can you tell me how I can identify duplicates in my dataset? Ken's methods only work under certain assumptions. The Filipino test only works because there is an exact match. It w…”

“Gzip beats BERT?” Part 2: dataset issues, improved speed, and results原评论 ↗
评论godelski · 2024-02

“Does anyone know how much spoilage are in these datasets? Common crawl has a lot of websites in it, including Reddit and Stack*. I'm certain there are lots of questions in those datasets and we want to differentiate recall from problem solving (often confused). I have a deep distrust when using large datasets like this given a common one with 60 authors a…”

DeepSeekMath 7B achieved 51.7% on MATH benchmark原评论 ↗
评论godelski · 2024-07

“I think you're close enough that the differences probably aren't too important. But if you want a bit more nuance, then read on. For disclosure, I'm in the second camp here. But I'll also say that I have a lot of very strong evidence to support this position, and that I do this from the perspective of a researcher. There's a few big problems when making …”

Reasoning in Large Language Models: A Geometric Perspective原评论 ↗
评论godelski · 2025-10

“Maybe given our different niches we interact with different people? But I'm uncertain because I believe what I'm saying is highly visible. I forgot, which NeurIPS(?) conference were so many wearing "Scale is all you need" shirts? > My first impression there is that the toolbox has essentially run dry and we need something more to advance the field This i…”

Andrej Karpathy – It will take a decade to work through the issues with agents原评论 ↗
被提交为 story · 1 次