“They all use the tool search, no? Please correct me if I'm wrong. My criteria was using ChatGPT which explicitly allows it. https://arxiv.org/html/2511.13029v1 if you don't believe me. BTW this was your original point >Anyway, it's trivial to get pretty much any model to make things up. Don't we all know this? That's why I was surprised by your positi…”
AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models
Existing language model evaluations primarily measure general capabilities, yet reliable use of these models across a range of domains demands factual accuracy and recognition of knowledge gaps. We introduce AA-Omniscience, a benchmark designed to measure both factual recall and knowledge calibration across 6,000 questions. Questions are derived from authoritative academic and industry sources, and cover 42 economically relevant topics within six different domains. The evaluation measures a model's Omniscience Index, a bounded metric (-100 to 100) measuring factual recall that jointly penalizes hallucinations and rewards abstention when uncertain, with 0 equating to a model that answers questions correctly as much as it does incorrectly. Among evaluated models, Claude 4.1 Opus attains the highest score (4.8), making it one of only three models to score above zero. These results reveal persistent factuality and calibration weaknesses across frontier models. Performance also varies by domain, with the models from three different research labs leading across the six domains. This performance variability suggests models should be chosen according to the demands of the use case rather than general performance for tasks where knowledge is important.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及- AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Language Models ↗declanjackson · 2025-11-18 · 6 分
- AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in LLMs ↗gmays · 2025-11-19 · 2 分