按年被提到
20232023 · 3 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2306.07174 · cs.CL

Augmenting Language Models with Long-Term Memory

Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2023-06-13 · 最近 2023-11-30
3不同的人
2活跃月份
3讨论串
105被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Existing large language models (LLMs) can only afford fix-sized inputs due to the input length limit, preventing them from utilizing rich long-context information from past inputs. To address this, we propose a framework, Language Models Augmented with Long-Term Memory (LongMem), which enables LLMs to memorize long history. We design a novel decoupled network architecture with the original backbone LLM frozen as a memory encoder and an adaptive residual side-network as a memory retriever and reader. Such a decoupled memory design can easily cache and update long-term past contexts for memory retrieval without suffering from memory staleness. Enhanced with memory-augmented adaptation training, LongMem can thus memorize long past context and use long-term memory for language modeling. The proposed memory retrieval module can handle unlimited-length context in its memory bank to benefit various downstream tasks. Typically, LongMem can enlarge the long-form memory to 65k tokens and thus cache many-shot extra demonstration examples as long-form memory for in-context learning. Experiments show that our method outperforms strong long-context models on ChapterBreak, a challenging long-context modeling benchmark, and achieves remarkable improvements on memory-augmented in-context learning over LLMs. The results demonstrate that the proposed method is effective in helping language models to memorize and utilize long-form contents. Our code is open-sourced at https://aka.ms/LongMem.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论josephg · 2023-06

“> For example, is it better to have 100x longer prompts, or 100x bigger models? Ultimately we need both. There's no point having a superintelligent code assistant which doesn't have enough working memory to understand what your program is doing. And there's no point having 100x longer prompts if the system isn't smart enough to contribute code changes. …”

The Secret Sauce behind 100K context window in LLMs: all tricks in one place原评论 ↗
评论filterfiber · 2023-11

“Yeah that's a fairly well studied one. Most of these techniques are rather "lossy" compared to extending the context window. The most likely "real solution" is going to be using various tricks and finetuning on higher context lengths to just extend the context window. Here's a bunch of other related methods, Summarizing context - https://arxiv.org/abs/2…”

Is anyone using self hosted LLM day to day and training it like a new employee原评论 ↗
被提交为 story · 1 次