“The linked paper is a much more interesting read: https://arxiv.org/pdf/2608.16157 I think there are many other significant inference improvements which an be built out that are being ignored because the API between client and inference stack would be tricky to nail down. This is not a criticism of the research but instead the presentation but the repo …”
FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及- FreeToken: Efficient Edge-Native Moe Serving with Bandwidth-Adaptive Execution ↗throwaway2027 · 2026-08-21 · 1 分
- FreeToken: Efficient Edge-Native Moe Serving with Bandwidth-Adaptive Execution ↗theanonymousone · 2026-08-22 · 3 分
- FreeToken: Efficient Edge-Native Moe Serving with Bandwidth-Adaptive Execution ↗Anon84 · 2026-08-24 · 1 分
- FreeToken: Efficient Edge-Native Moe Serving with Bandwidth-Adaptive Execution ↗gmays · 2026-08-30 · 2 分
- FreeToken: Datacenter-class intelligence on the hardware you own ↗lioeters · 2026-11-02 · 3 分