按年被提到
20222022 · 1 次2026
视觉、语音与图像生成›卷积网络与图像识别·arXiv:2205.04596 · cs.CV

When does dough become a bagel? Analyzing the remaining mistakes on ImageNet

Vijay Vasudevan, Benjamin Caine, Raphael Gontijo-Lopes, Sara Fridovich-Keil, Rebecca Roelofs · 2022
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2022-05-17 · 最近 2024-10-23
2不同的人
2活跃月份
2讨论串
14被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Image classification accuracy on the ImageNet dataset has been a barometer for progress in computer vision over the last decade. Several recent papers have questioned the degree to which the benchmark remains useful to the community, yet innovations continue to contribute gains to performance, with today's largest models achieving 90%+ top-1 accuracy. To help contextualize progress on ImageNet and provide a more meaningful evaluation for today's state-of-the-art models, we manually review and categorize every remaining mistake that a few top models make in order to provide insight into the long-tail of errors on one of the most benchmarked datasets in computer vision. We focus on the multi-label subset evaluation of ImageNet, where today's best models achieve upwards of 97% top-1 accuracy. Our analysis reveals that nearly half of the supposed mistakes are not mistakes at all, and we uncover new valid multi-labels, demonstrating that, without careful review, we are significantly underestimating the performance of these models. On the other hand, we also find that today's best models still make a significant number of mistakes (40%) that are obviously wrong to human reviewers. To calibrate future progress on ImageNet, we provide an updated multi-label evaluation set, and we curate ImageNet-Major: a 68-example "major error" slice of the obvious mistakes made by today's top models -- a slice where models should achieve near perfection, but today are far from doing so.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论Datenstrom · 2022-05

“Link to the actual paper [1] for anyone who wants to skip the blog like me. Its a good read covering some problems I have been working on for a while. [1]: https://arxiv.org/pdf/2205.04596.pdf”

Assessing the Historical Accuracy of ImageNet原评论 ↗
评论gwern · 2024-10

“> Is there much in the way of datasets of things that are recognizable but triggering incorrect something-or-other (what is it exactly? Semantic sensations?) There are some datasets which collect 'hard' examples. One I love for the title alone is "When does dough become a bagel? Analyzing the remaining mistakes on ImageNet" https://arxiv.org/abs/2205.045…”

Seeing faces in things: A model and dataset for pareidolia原评论 ↗