Platform Engineer with AWS AgentCore Evaluation

About the Position

We are looking for a Platform Engineer with experience in CI/CD automation, cloud observability, and AI platform operations to help build and maintain quality evaluation pipelines for agent based systems. In this role, you will integrate automated evaluation frameworks into deployment workflows, develop monitoring solutions, and support continuous quality validation across production environments.

About the Project

The project focuses on establishing a scalable evaluation and monitoring platform for AI agents and intelligent applications. The solution provides automated deployment gates, online quality assessment, operational visibility, and regression detection capabilities to help maintain high standards of reliability and performance.

About the Team

You will work closely with platform engineers, DevOps specialists, AI engineers, and cloud architects. The team collaborates on automation, observability, deployment processes, and operational excellence, ensuring that AI systems meet quality requirements throughout the software delivery lifecycle.

Responsibilities

• Integrate AWS AgentCore Evaluation API into CI/CD pipelines to support automated deployment quality gates
• Configure and maintain on demand evaluation workflows as part of build and release processes
• Develop and support GitHub Actions workflows for automated evaluation execution and deployment validation
• Configure and manage online evaluation modes with appropriate sampling strategies
• Build and maintain CloudWatch dashboards to track evaluation metrics for agents and evaluators over time
• Design and configure CloudWatch alarms to identify quality degradation and operational issues
• Collaborate with AI and platform engineering teams to define evaluation thresholds and quality standards
• Monitor evaluation results and contribute to regression detection and release validation processes
• Support observability, reporting, and operational monitoring initiatives across AI platforms
• Create and maintain technical documentation related to evaluation workflows, monitoring, and deployment automation

Requirements

• 3+ years of experience in platform engineering, DevOps engineering, or a related field
• Experience integrating quality validation processes into CI/CD pipelines
• Hands on experience designing and maintaining deployment gates for automated release workflows
• Experience building and supporting AWS CloudWatch dashboards and monitoring solutions
• Experience configuring CloudWatch alarms and operational alerting strategies
• Knowledge of cloud infrastructure, automation, and deployment best practices
• Experience working with monitoring, observability, and platform operations tools
• Strong analytical and troubleshooting skills
• Good written and verbal communication skills

Nice to Have

• Experience with AWS AgentCore Evaluation online mode configuration and sampling strategies
• Experience using evaluations to compare runtime versions and identify regressions
• Knowledge of CloudWatch Embedded Metric Format (EMF) for structured metric collection and reporting
• Experience with AI platform monitoring and evaluation frameworks
• Exposure to automated quality assurance processes for machine learning or agent based systems
• Experience supporting enterprise scale cloud platforms and deployment pipelines

Technologies

AWS AgentCore Evaluation API, AWS AgentCore Runtime, AWS CloudWatch, CloudWatch Embedded Metric Format (EMF), GitHub Actions, CI/CD Pipelines, Python, AWS Services, Monitoring and Observability Tools, Quality Evaluation Frameworks