按年被提到
20232023 · 1 次2026
大模型与语言模型›对话式大模型·arXiv:2311.18805 · cs.CL

Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Text

Qi Cao, Takeshi Kojima, Yutaka Matsuo, Yusuke Iwasawa · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-12-03 · 最近 2024-07-06
2不同的人
2活跃月份
2讨论串
10被引(OpenAlex)
arXiv 摘要(候选层不写导读)

While Large Language Models (LLMs) have achieved remarkable performance in many tasks, much about their inner workings remains unclear. In this study, we present novel experimental insights into the resilience of LLMs, particularly GPT-4, when subjected to extensive character-level permutations. To investigate this, we first propose the Scrambled Bench, a suite designed to measure the capacity of LLMs to handle scrambled input, in terms of both recovering scrambled sentences and answering questions given scrambled context. The experimental results indicate that most powerful LLMs demonstrate the capability akin to typoglycemia, a phenomenon where humans can understand the meaning of words even when the letters within those words are scrambled, as long as the first and last letters remain in place. More surprisingly, we found that only GPT-4 nearly flawlessly processes inputs with unnatural errors, even under the extreme condition, a task that poses significant challenges for other LLMs and often even for humans. Specifically, GPT-4 can almost perfectly reconstruct the original sentences from scrambled ones, decreasing the edit distance by 95%, even when all letters within each word are entirely scrambled. It is counter-intuitive that LLMs can exhibit such resilience despite severe disruption to input tokenization caused by scrambled text.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论Lerc · 2024-07

“I don't think this is an accurate representation of LLM understanding. The significance of answering questions given in Base64 is that it can give an answer without writing out the human readable form first. That means it is understanding language in a non-language tokenised form. This does not extend to being able to manipulate text and words well, such…”

Tokens are a big reason today's generative AI falls short原评论 ↗
被提交为 story · 1 次