Evaluating AI agents often relies on single-turn tests, which judge a single question and answer in isolation. However, real-world users engage in multi-turn conversations where one response shapes the next, creating a chain of context that static tests fail to capture. AWS has addressed this gap by introducing ActorSimulator within the Strands Evaluation SDK, a tool designed to simulate realistic, goal-oriented users programmatically rather than relying on fixed scripts.

Unlike simple prompts, ActorSimulator uses Large Language Models to create dynamic "actors" with specific personas, such as a budget-conscious traveler or a technical expert. These actors maintain consistency in their communication style and persistence in pursuing a defined goal, such as resolving a complex booking issue. This approach ensures the agent is tested against the unpredictable twists of human dialogue, including follow-up questions, requests for clarification, and abrupt changes in direction.

The system tracks whether goals are successfully met and provides structured reasoning for every simulated user action, offering transparency into why a conversation succeeded or failed. By integrating with OpenTelemetry, developers can capture detailed traces of tool calls and model behavior across the entire conversation. This automated approach allows engineering teams to scale testing without the massive overhead of manual human evaluation, effectively identifying where agents lose track of complex user needs over time.

Evaluating AI agents often relies on single-turn tests, which judge a single question and answer in isolation. However, real-world users engage in multi-turn conversations where one response shapes the next, creating a chain of context that static tests fail to capture. AWS has addressed this gap by introducing ActorSimulator within the Strands Evaluation SDK, a tool designed to simulate realistic, goal-oriented users programmatically rather than relying on fixed scripts.

Unlike simple prompts, ActorSimulator uses Large Language Models to create dynamic "actors" with specific personas, such as a budget-conscious traveler or a technical expert. These actors maintain consistency in their communication style and persistence in pursuing a defined goal, such as resolving a complex booking issue. This approach ensures the agent is tested against the unpredictable twists of human dialogue, including follow-up questions, requests for clarification, and abrupt changes in direction.

The system tracks whether goals are successfully met and provides structured reasoning for every simulated user action, offering transparency into why a conversation succeeded or failed. By integrating with OpenTelemetry, developers can capture detailed traces of tool calls and model behavior across the entire conversation. This automated approach allows engineering teams to scale testing without the massive overhead of manual human evaluation, effectively identifying where agents lose track of complex user needs over time.