按年被提到
20242024 · 3 次2026
大模型与语言模型›大模型推理与思维链·arXiv:2405.19616 · cs.AI

Easy Problems That LLMs Get Wrong

Sean Williams, James Huckle · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2024-05-31 · 最近 2024-09-06
3不同的人
2活跃月份
2讨论串
2被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We introduce a comprehensive Linguistic Benchmark designed to evaluate the limitations of Large Language Models (LLMs) in domains such as logical reasoning, spatial intelligence, and linguistic understanding, among others. Through a series of straightforward questions, it uncovers the significant limitations of well-regarded models to perform tasks that humans manage with ease. It also highlights the potential of prompt engineering to mitigate some errors and underscores the necessity for better training methodologies. Our findings stress the importance of grounding LLMs with human reasoning and common sense, emphasising the need for human-in-the-loop for enterprise applications. We hope this work paves the way for future research to enhance the usefulness and reliability of new models.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论smusamashah · 2024-09

“We need results from these harder/different benchmarks which give pretty bad scores to current top LLMs. https://www.wolfram.com/llm-benchmarking-project/ https://help.kagi.com/kagi/ai/llm-benchmark.html Edit: There are few other benchmarks that give pretty low scores (<20%) to top LLMs. Can't find them atm. There was a benchmark with common sense eas…”

Reflection 70B, the top open-source model原评论 ↗
评论thethirdone · 2024-09

“The sample answers for the horse race question are crazy. [0] Pretty much all the LLM really want to split 6 horses into two groups of three. Only LLAMA 3 makes the justification that only 2 horses can be raced at a time, but then gets its modified question wrong by racing three horses. I personally would consider an answer that presumes some restriction…”

Reflection 70B, the top open-source model原评论 ↗
被提交为 story · 1 次