Platform Engineer with AWS AgentCore Evaluation
About the Position
We are looking for a Platform Engineer with experience in CI/CD automation, cloud observability, and AI platform operations to help build and maintain quality evaluation pipelines for agent based systems. In this role, you will integrate automated evaluation frameworks into deployment workflows, develop monitoring solutions, and support continuous quality validation across production environments.
About the Project
The project focuses on establishing a scalable evaluation and monitoring platform for AI agents and intelligent applications. The solution provides automated deployment gates, online quality assessment, operational visibility, and regression detection capabilities to help maintain high standards of reliability and performance.
About the Team
You will work closely with platform engineers, DevOps specialists, AI engineers, and cloud architects. The team collaborates on automation, observability, deployment processes, and operational excellence, ensuring that AI systems meet quality requirements throughout the software delivery lifecycle.
Responsibilities
- Design and implement semantic discovery capabilities using vector embeddings, OpenSearch, or pgvector
- Develop and maintain Python based APIs for secure credential retrieval and secrets access
- Integrate and manage HashiCorp Vault and AWS Secrets Manager within platform services
- Build and optimize semantic indexing pipelines and search experiences
- Design and maintain semantic domain taxonomies to improve information discovery
- Collaborate with platform, security, and application teams to ensure secure access patterns
- Implement monitoring, logging, and operational best practices for platform services
- Support MCP server capabilities and integrations across enterprise systems
- Contribute to architecture discussions, technical design reviews, and platform roadmap initiatives
- Ensure compliance with security, governance, and credential management standards
Requirements
- 3+ years of experience in platform engineering or backend engineering
- Hands on experience with secrets management solutions such as HashiCorp Vault or AWS Secrets Manager
- Experience implementing vector search, semantic search, or embedding based search solutions
- Semantic discovery implementation using vector embeddings, OpenSearch, or pgvector
- HashiCorp Vault integration for credential and secrets management
- AWS Secrets Manager integration as an alternative secrets management solution
- Development of Python APIs for credential retrieval and access management
- MCP server capabilities and semantic domain taxonomy design
Nice to Have
- Experience with AWS Bedrock embedding models for semantic indexing
- Experience with OpenSearch Service or pgvector on Amazon RDS
- Knowledge of OAuth token lifecycle management
Technologies
AWS AgentCore Evaluation API, AWS AgentCore Runtime, AWS CloudWatch, CloudWatch Embedded Metric Format (EMF), GitHub Actions, CI/CD Pipelines, Python, AWS Services, Monitoring and Observability Tools, Quality Evaluation Frameworks
