“There's the paper that shows ChatGPT outperforming MTurk Crowdworkers and another one showing GPT-4 going to to toe with experts (and blowing past crowdworkers) who set the benchmarks they were evaluating. https://arxiv.org/abs/2303.15056 https://www.artisana.ai/articles/gpt-4-outperforms-elite-crowdworkers-saving-researchers-usd500-000-and-20 If quali…”
ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Many NLP applications require manual data annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd-workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using a sample of 2,382 tweets, we demonstrate that ChatGPT outperforms crowd-workers for several annotation tasks, including relevance, stance, topics, and frames detection. Specifically, the zero-shot accuracy of ChatGPT exceeds that of crowd-workers for four out of five tasks, while ChatGPT's intercoder agreement exceeds that of both crowd-workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003 -- about twenty times cheaper than MTurk. These results show the potential of large language models to drastically increase the efficiency of text classification.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“>should you just cut out the middle-model and be using ChatGPT as your classifier? Oh you certainly could. See here: GPT-3.5 outperforming elite crowdworkers on MTurk for Text annotation https://arxiv.org/abs/2303.15056 GPT-4 going toe to toe with expertrs (and significantly outperforming crowdworkers) on NLP tasks https://www.artisana.ai/articles/gpt…”
“Or just use the tools directly since they no longer underperform human actors. GPT-4 going toe to toe with the experts who set the benchmarks, way overperforming the crowdworkers https://www.artisana.ai/articles/gpt-4-outperforms-elite-crowdworkers-saving-researchers-usd500-000-and-20 GPT-3.5 outperforming crowdworkers https://arxiv.org/abs/2303.15056 …”
“Current State of the art (GPT-4) is mostly on par with experts and much better than crowdworkers. Might be overkill though. https://www.artisana.ai/articles/gpt-4-outperforms-elite-crowdworkers-saving-researchers-usd500-000-and-20 3.5 (what is used here) is better than crowd workers https://arxiv.org/abs/2303.15056”
“LLMs are better at that though. Sure you may not require them but it certainly wouldn't be for a lack of accuracy. https://www.artisana.ai/articles/gpt-4-outperforms-elite-crowdworkers-saving-researchers-usd500-000-and-20 https://arxiv.org/abs/2303.15056”
- ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks ↗georgehill · 2023-03-28 · 3 分