Evaluating how well AI models perform in languages other than English has long been a messy endeavor. Most benchmarks are simply translated from English using basic tools, which often leads to "semantic drift"—where the core meaning of a question shifts—or "context loss," where cultural or linguistic nuances vanish. These errors result in misleading scores that do not reflect an AI's true capabilities in local languages.

To solve this, researchers at INSAIT developed a new automated framework that uses "test-time compute scaling." Instead of a single-shot translation, the system generates multiple candidate translations and uses a ranking method called T-RANK to pick the best one. This approach effectively uses more processing power during the translation phase to ensure the final output preserves the original task's structure and difficulty.

The team applied this pipeline to eight Eastern and Southern European languages, including Ukrainian, Bulgarian, and Greek. By using a model-based evaluation approach—where a highly capable AI is used to grade the quality of others—they found their translations were preferred four to one over previous versions. This framework is now open-sourced, providing a blueprint for more reliable and reproducible global AI development.

This work is crucial because it bridges the gap between English-centric AI and the rest of the world. By refining how we measure performance in diverse languages, developers can more accurately adjust models for specific regions, ensuring that AI benefits are not limited by linguistic barriers or poor evaluation data.

Evaluating how well AI models perform in languages other than English has long been a messy endeavor. Most benchmarks are simply translated from English using basic tools, which often leads to "semantic drift"—where the core meaning of a question shifts—or "context loss," where cultural or linguistic nuances vanish. These errors result in misleading scores that do not reflect an AI's true capabilities in local languages.

To solve this, researchers at INSAIT developed a new automated framework that uses "test-time compute scaling." Instead of a single-shot translation, the system generates multiple candidate translations and uses a ranking method called T-RANK to pick the best one. This approach effectively uses more processing power during the translation phase to ensure the final output preserves the original task's structure and difficulty.

The team applied this pipeline to eight Eastern and Southern European languages, including Ukrainian, Bulgarian, and Greek. By using a model-based evaluation approach—where a highly capable AI is used to grade the quality of others—they found their translations were preferred four to one over previous versions. This framework is now open-sourced, providing a blueprint for more reliable and reproducible global AI development.

This work is crucial because it bridges the gap between English-centric AI and the rest of the world. By refining how we measure performance in diverse languages, developers can more accurately adjust models for specific regions, ensuring that AI benefits are not limited by linguistic barriers or poor evaluation data.