按年被提到
20232023 · 6 次2026
大模型与语言模型›对话式大模型·arXiv:2303.15056 · cs.CL

ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

Fabrizio Gilardi, Meysam Alizadeh, Maël Kubli · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-03-28 · 最近 2023-09-13
2不同的人
4活跃月份
6讨论串
1,323被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Many NLP applications require manual data annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd-workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using a sample of 2,382 tweets, we demonstrate that ChatGPT outperforms crowd-workers for several annotation tasks, including relevance, stance, topics, and frames detection. Specifically, the zero-shot accuracy of ChatGPT exceeds that of crowd-workers for four out of five tasks, while ChatGPT's intercoder agreement exceeds that of both crowd-workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003 -- about twenty times cheaper than MTurk. These results show the potential of large language models to drastically increase the efficiency of text classification.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论og_kalu · 2023-06

“There's the paper that shows ChatGPT outperforming MTurk Crowdworkers and another one showing GPT-4 going to to toe with experts (and blowing past crowdworkers) who set the benchmarks they were evaluating. https://arxiv.org/abs/2303.15056 https://www.artisana.ai/articles/gpt-4-outperforms-elite-crowdworkers-saving-researchers-usd500-000-and-20 If quali…”

33-46% of workers on MTurk used LLMs in a text production task原评论 ↗
评论og_kalu · 2023-06

“>should you just cut out the middle-model and be using ChatGPT as your classifier? Oh you certainly could. See here: GPT-3.5 outperforming elite crowdworkers on MTurk for Text annotation https://arxiv.org/abs/2303.15056 GPT-4 going toe to toe with expertrs (and significantly outperforming crowdworkers) on NLP tasks https://www.artisana.ai/articles/gpt…”

The idea maze for AI startups (2015)原评论 ↗
评论og_kalu · 2023-07

“Or just use the tools directly since they no longer underperform human actors. GPT-4 going toe to toe with the experts who set the benchmarks, way overperforming the crowdworkers https://www.artisana.ai/articles/gpt-4-outperforms-elite-crowdworkers-saving-researchers-usd500-000-and-20 GPT-3.5 outperforming crowdworkers https://arxiv.org/abs/2303.15056 …”

Mechanical Turk workers are using AI to automate being human原评论 ↗
评论og_kalu · 2023-07

“Current State of the art (GPT-4) is mostly on par with experts and much better than crowdworkers. Might be overkill though. https://www.artisana.ai/articles/gpt-4-outperforms-elite-crowdworkers-saving-researchers-usd500-000-and-20 3.5 (what is used here) is better than crowd workers https://arxiv.org/abs/2303.15056”

What we learned from using GPT for 500k+ classifications原评论 ↗
评论og_kalu · 2023-09

“LLMs are better at that though. Sure you may not require them but it certainly wouldn't be for a lack of accuracy. https://www.artisana.ai/articles/gpt-4-outperforms-elite-crowdworkers-saving-researchers-usd500-000-and-20 https://arxiv.org/abs/2303.15056”

Against LLM Maximalism原评论 ↗
被提交为 story · 1 次