引用tmzt · 2025-01
“A full speech-to-speech model [1] working on an M1 Mac would be incredible.”
借它设想完整的语音到语音模型能在 M1 Mac 上运行
首个实时全双工语音对话大模型,延迟约200毫秒
传统语音对话由语音活动检测、识别、文本对话、语音合成等独立组件串联,延迟高、丢失情绪等非语言信息,也无法处理重叠和打断。Moshi 把对话看作语音到语音的生成:以文本语言模型为骨干,生成神经音频编解码器残差量化器的语音 token,并把自己和用户的语音建模为并行流,从而无需显式说话轮次。其“Inner Monologue”方法先预测与音频对齐的文本 token 作为前缀,显著提升语言质量,也可用于流式识别和合成;模型理论延迟 160 毫秒,实际约 200 毫秒。
HN 评论者把它当作全双工实时语音对话的开创者来引用,希望能在本地设备上跑端到端语音模型,也有人指向论文第 4.2 节了解训练音频如何转写。
“A full speech-to-speech model [1] working on an M1 Mac would be incredible.”
“See Section 4.2 in the Moshi paper: Moshi”
“which aims to address this problem by allowing the model to listen and generate audio at the same time.”
“I think the full duplex mode that Moshi pioneered is probably where the puck is going to end up: Moshi”
“and of course Moshi.”