按年被提到
20212025 · 2 次2026
大模型与语言模型›大模型评测与真实性·arXiv:2110.08193 · cs.CL

BBQ: A Hand-Built Bias Benchmark for Question Answering

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, Samuel R. Bowman · 2021
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2025-01-29 · 最近 2025-02-04
2不同的人
2活跃月份
2讨论串
262被引(OpenAlex)
arXiv 摘要(候选层不写导读)

It is well documented that NLP models learn social biases, but little work has been done on how these biases manifest in model outputs for applied tasks like question answering (QA). We introduce the Bias Benchmark for QA (BBQ), a dataset of question sets constructed by the authors that highlight attested social biases against people belonging to protected classes along nine social dimensions relevant for U.S. English-speaking contexts. Our task evaluates model responses at two levels: (i) given an under-informative context, we test how strongly responses reflect social biases, and (ii) given an adequately informative context, we test whether the model's biases override a correct answer choice. We find that models often rely on stereotypes when the context is under-informative, meaning the model's outputs consistently reproduce harmful biases in this setting. Though models are more accurate when the context provides an informative answer, they still rely on stereotypes and average up to 3.4 percentage points higher accuracy when the correct answer aligns with a social bias than when it conflicts, with this difference widening to over 5 points on examples targeting gender for most models tested.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论JumpCrisscross · 2025-01

“They measure bias using "the Bias Benchmark for QA (BBQ), a dataset of question sets...that highlight attested social biases against people belonging to protected classes along nine social dimensions relevant for U.S. English-speaking contexts. Our task evaluates model responses at two levels: (i) given an under-informative context, we test how strongly r…”

DeepSeek's Hidden Bias: How We Cut It by 76% Without Performance Loss原评论 ↗
评论westurner · 2025-02

“That would disincentive this type of research, for example: "DeepSeek's Hidden Bias: How We Cut It by 76% Without Performance Loss" (2025) https://news.ycombinator.com/item?id=42868271 https://news.ycombinator.com/item?id=42891042 TIL about BBQ: Bias Benchmark for QA "BBQ: A Hand-Built Bias Benchmark for Question Answering" (2021) https://arxiv.org/ab…”

US bill proposes jail time for people who download DeepSeek原评论 ↗