Note: The job is a remote job and is open to candidates in USA. OVA.Work is seeking an AI Monitoring Engineer to design, implement, and manage monitoring solutions for AI and machine learning systems. The ideal candidate will ensure the reliability and performance of AI services by developing observability frameworks and collaborating with various engineering teams.
Responsibilities
- Design and implement monitoring frameworks for AI applications, machine learning models, and AI infrastructure
- Monitor model performance, prediction quality, latency, throughput, and availability
- Detect and analyze data drift, model drift, concept drift, and inference anomalies
- Build dashboards and alerts for AI system health using observability tools
- Develop Service Level Indicators (SLIs), Service Level Objectives (SLOs), and operational metrics for AI services
- Implement logging, distributed tracing, and telemetry for AI applications
- Configure automated alerting and incident response workflows
- Monitor GPU, CPU, memory, storage, and network utilization for AI workloads
- Analyze AI system performance trends and recommend optimization strategies
- Collaborate with AI Engineers and Data Scientists to improve model reliability and operational performance
- Support root cause analysis (RCA) for production incidents and implement preventive measures
- Maintain monitoring documentation, operational runbooks, and reporting dashboards
- Ensure compliance with monitoring, security, governance, and operational standards
Skills
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field
- 3+ years of experience in Monitoring Engineering, Site Reliability Engineering (SRE), DevOps, Platform Engineering, MLOps, or Cloud Operations
- Experience monitoring production AI or machine learning systems
- Strong programming skills in Python and scripting with Bash
- Experience with Linux administration
- Hands-on experience with Docker and Kubernetes
- Experience with AWS, Microsoft Azure, or Google Cloud Platform
- Strong understanding of distributed systems, cloud infrastructure, and networking
- Experience with observability tools such as Prometheus, Grafana, OpenTelemetry, ELK Stack, Datadog, Splunk, or New Relic
- Experience with SQL and time-series databases for operational reporting
- Experience with machine learning monitoring platforms such as Evidently AI, Arize AI, Fiddler AI, WhyLabs, or Azure AI monitoring capabilities
- Experience monitoring Large Language Models (LLMs) and Generative AI applications
- Knowledge of AI evaluation metrics, prompt monitoring, hallucination detection, and Retrieval-Augmented Generation (RAG) observability
- Experience with MLOps platforms such as MLflow, Kubeflow, SageMaker, Vertex AI, or Azure Machine Learning
- Familiarity with vector databases and model serving platforms
- Experience monitoring GPU-enabled infrastructure and AI inference systems
- Knowledge of Responsible AI, fairness monitoring, and AI governance
Company Overview