按年被提到
20192022 · 2 次2026
机器学习与深度学习›贝叶斯方法与 AutoML·arXiv:1911.00068 · stat.ML

Confident Learning: Estimating Uncertainty in Dataset Labels

Curtis G. Northcutt, Lu Jiang, Isaac L. Chuang · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2019-11-04 · 最近 2022-09-08
2不同的人
4活跃月份
4讨论串
845被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Learning exists in the context of data, yet notions of confidence typically focus on model predictions, not label quality. Confident learning (CL) is an alternative approach which focuses instead on label quality by characterizing and identifying label errors in datasets, based on the principles of pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence. Whereas numerous studies have developed these principles independently, here, we combine them, building on the assumption of a class-conditional noise process to directly estimate the joint distribution between noisy (given) labels and uncorrupted (unknown) labels. This results in a generalized CL which is provably consistent and experimentally performant. We present sufficient conditions where CL exactly finds label errors, and show CL performance exceeding seven recent competitive approaches for learning with noisy labels on the CIFAR dataset. Uniquely, the CL framework is not coupled to a specific data modality or model (e.g., we use CL to find several label errors in the presumed error-free MNIST dataset and improve sentiment classification on text data in Amazon Reviews). We also employ CL on ImageNet to quantify ontological class overlap (e.g., estimating 645 "missile" images are mislabeled as their parent class "projectile"), and moderately increase model accuracy (e.g., for ResNet) by cleaning data prior to training. These results are replicable using the open-source cleanlab release.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论cgn · 2019-11

“Hi, Hackers. I'm excited to share confident learning for characterizing, finding, and learning with label errors in datasets. To promote and standardize future research in learning with noisy labels and weak supervision, I've also open-sourced the cleanlab Python package: https://pypi.org/project/cleanlab/ Post: https://l7.curtisnorthcutt.com/confident-l…”

Announcing Confident Learning: Finding and Learning with Dataset Label Errors原评论 ↗
评论cgn · 2021-04

“I am an author on the paper, and just saw this thread. Thanks for all of your comments. Note this work focuses on errors in test sets, not training labels. Test sets are foundation of benchmarking for machine learning. We study test sets (versus training sets), because training sets are already studied in-depth in confident learning (https://arxiv.org/pdf…”

Error-riddled data sets are warping our sense of how good AI is原评论 ↗
评论calebchiam · 2022-08

“Cleanlab (https://github.com/cleanlab/cleanlab) is a family of algorithms for automatically finding issues in datasets. It might seem surprising that it’s possible to automatically identify label errors and out-of-distribution data; Cleanlab does this using the algorithms published in https://arxiv.org/abs/1911.00068. Cleanlab’s algorithms, while clever,…”

Show HN: Cleanlab Vizzy – automatically find label errors and bad data原评论 ↗
评论calebchiam · 2022-09

“I built Vizzy as a hackathon project, with the goal of explaining how relatively simple (but clever) algorithms can be a powerful tool to automatically find issues in datasets, including label errors and out-of-distribution data. Vizzy uses a JavaScript port of (a part of) https://github.com/cleanlab/cleanlab, which implements the algorithms described in…”

Show HN: How simple (but clever) algorithms can find label issues in datasets原评论 ↗