按年被提到
20242026 · 2 次2026
大模型与语言模型›提示工程与输出稳定性·arXiv:2410.20247 · cs.LG

Model Equality Testing: Which Model Is This API Serving?

Irena Gao, Percy Liang, Carlos Guestrin · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2026-04-28 · 最近 2026-10-26
2不同的人
2活跃月份
2讨论串
4被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Users often interact with large language models through black-box inference APIs, both for closed- and open-weight models (e.g., Llama models are popularly accessed via Amazon Bedrock and Azure AI Studio). In order to cut costs or add functionality, API providers may quantize, watermark, or finetune the underlying model, changing the output distribution -- possibly without notifying users. We formalize detecting such distortions as Model Equality Testing, a two-sample testing problem, where the user collects samples from the API and a reference distribution and conducts a statistical test to see if the two distributions are the same. We find that tests based on the Maximum Mean Discrepancy between distributions are powerful for this task: a test built on a simple string kernel achieves a median of 77.4% power against a range of distortions, using an average of just 10 samples per prompt. We then apply this test to commercial inference APIs from Summer 2024 for four Llama models, finding that 11 out of 31 endpoints serve different distributions than reference weights released by Meta.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论spindump8930 · 2026-04

“Remember that models on different inference platforms might not necessarily give exactly the same results, adding another axis of non-determinism to development. Things like quantization, custom model serving silicon, batching, or other inference optimizations might mean a model from the original provider performs differently from the hosted one:/ This …”

OpenAI models coming to Amazon Bedrock: Interview with OpenAI and AWS CEOs原评论 ↗
被提交为 story · 1 次