Qwen released BabelArena, a multilingual agent benchmark designed to test tool-using models across 23 languages, 13 domains, and four benchmark families.
Unlike evaluations that translate only the task, BabelArena localizes prompts, tasks, tool descriptions, and environment content. Qwen’s release notes say no evaluated model leads every benchmark family. Qwen also reports that all five tested models perform worse in low-resource languages, and that those runs can use up to roughly twice the input tokens of English.
For builders, the useful part is full-workflow regression testing: a model that handles an English prompt may still fail when tools and surrounding context change language. The BabelArena dataset is public, although the current standard subset excludes ClawEval, which is distributed separately.