按年被提到
20192019 · 1 次2026
大模型与语言模型›检索增强与知识库·arXiv:1909.07606 · cs.CL

K-BERT: Enabling Language Representation with Knowledge Graph

Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, Ping Wang · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2019-12-16 · 最近 2020-02-10
2不同的人
2活跃月份
2讨论串
850被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Pre-trained language representation models, such as BERT, capture a general language representation from large-scale corpora, but lack domain-specific knowledge. When reading a domain text, experts make inferences with relevant knowledge. For machines to achieve this capability, we propose a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge. However, too much knowledge incorporation may divert the sentence from its correct meaning, which is called knowledge noise (KN) issue. To overcome KN, K-BERT introduces soft-position and visible matrix to limit the impact of knowledge. K-BERT can easily inject domain knowledge into the models by equipped with a KG without pre-training by-self because it is capable of loading model parameters from the pre-trained BERT. Our investigation reveals promising results in twelve NLP tasks. Especially in domain-specific tasks (including finance, law, and medicine), K-BERT significantly outperforms BERT, which demonstrates that K-BERT is an excellent choice for solving the knowledge-driven problems that require experts.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论phreeza · 2019-12

“Maybe something like K-BERT, the idea is simple enough, and can leverage pretrained transformer models. https://arxiv.org/abs/1909.07606”

Play AI Dungeon 2. Become a Dragon. Eat the Moon原评论 ↗
评论ctoth · 2020-02

“Some papers which you might find interesting: Lin, B. Y., Chen, X., Chen, J., & Ren, X. (2019). KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning. 2822–2832. https://doi.org/10.18653/v1/d19-1282 Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., & Wang, P. (2019). K-BERT: Enabling Language Representation with Knowledge Graph. Retrie…”

Turing-NLG: A 17B-parameter language model原评论 ↗