按年被提到
20252025 · 2 次2026
大模型与语言模型›大模型评测与真实性·arXiv:2511.13029 · cs.CL

AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models

Declan Jackson, William Keating, George Cameron, Micah Hill-Smith · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2025-11-18 · 最近 2026-04-08
3不同的人
2活跃月份
3讨论串
3被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Existing language model evaluations primarily measure general capabilities, yet reliable use of these models across a range of domains demands factual accuracy and recognition of knowledge gaps. We introduce AA-Omniscience, a benchmark designed to measure both factual recall and knowledge calibration across 6,000 questions. Questions are derived from authoritative academic and industry sources, and cover 42 economically relevant topics within six different domains. The evaluation measures a model's Omniscience Index, a bounded metric (-100 to 100) measuring factual recall that jointly penalizes hallucinations and rewards abstention when uncertain, with 0 equating to a model that answers questions correctly as much as it does incorrectly. Among evaluated models, Claude 4.1 Opus attains the highest score (4.8), making it one of only three models to score above zero. These results reveal persistent factuality and calibration weaknesses across frontier models. Performance also varies by domain, with the models from three different research labs leading across the six domains. This performance variability suggests models should be chosen according to the demands of the use case rather than general performance for tasks where knowledge is important.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论simianwords · 2026-04

“They all use the tool search, no? Please correct me if I'm wrong. My criteria was using ChatGPT which explicitly allows it. https://arxiv.org/html/2511.13029v1 if you don't believe me. BTW this was your original point >Anyway, it's trivial to get pretty much any model to make things up. Don't we all know this? That's why I was surprised by your positi…”

The Future of Everything Is Lies, I Guess原评论 ↗
被提交为 story · 2 次