Evaluating generative AI models often feels like hitting a moving target. Developers usually rely on static rubrics—a fixed set of rules like "is it polite?"—to grade outputs. However, a creative story needs different benchmarks than a complex Python script or a legal brief.

To solve this, Amazon SageMaker AI has introduced a rubric-based judge powered by its Amazon Nova foundation models. Instead of a one-size-fits-all approach, this LLM-as-a-Judge analyzes the user’s specific prompt and builds a custom checklist in real-time. For instance, when summarizing medical records, the system automatically prioritizes medical accuracy and empathetic tone without requiring manual human intervention.

The judge provides a deep-dive analysis through structured YAML outputs. It assigns importance weights to different criteria and provides a "weighted score" to indicate its confidence in a specific preference. This level of transparency helps engineering teams pinpoint exactly where a model is failing, such as detecting if a model is becoming more accurate but losing its conversational clarity.

By implementing reconciled agreement—where the judge evaluates responses in multiple orders to ensure consistency—Amazon aims to provide a more trustworthy alternative to human-led evaluation. This tool streamlines the development lifecycle for any SFT model, allowing for faster checkpoint selection and automated quality control for large-scale training datasets.

Evaluating generative AI models often feels like hitting a moving target. Developers usually rely on static rubrics—a fixed set of rules like "is it polite?"—to grade outputs. However, a creative story needs different benchmarks than a complex Python script or a legal brief.

To solve this, Amazon SageMaker AI has introduced a rubric-based judge powered by its Amazon Nova foundation models. Instead of a one-size-fits-all approach, this LLM-as-a-Judge analyzes the user’s specific prompt and builds a custom checklist in real-time. For instance, when summarizing medical records, the system automatically prioritizes medical accuracy and empathetic tone without requiring manual human intervention.

The judge provides a deep-dive analysis through structured YAML outputs. It assigns importance weights to different criteria and provides a "weighted score" to indicate its confidence in a specific preference. This level of transparency helps engineering teams pinpoint exactly where a model is failing, such as detecting if a model is becoming more accurate but losing its conversational clarity.

By implementing reconciled agreement—where the judge evaluates responses in multiple orders to ensure consistency—Amazon aims to provide a more trustworthy alternative to human-led evaluation. This tool streamlines the development lifecycle for any SFT model, allowing for faster checkpoint selection and automated quality control for large-scale training datasets.