Note: The job is a remote job and is open to candidates in USA. Mount Sinai Health System is one of the largest academic medical systems in the New York metro area, and they are seeking a Machine Learning Engineer I to join their SinAI Assurance Lab. The role involves designing, building, and deploying large language model applications while ensuring compliance and performance standards are met across AI systems.
Responsibilities
- Designing, building, and deploying large language model (LLM) applications including retrieval-augmented generation (RAG) systems, agentic platforms, and clinical chatbots
- Designing, maintaining, and optimizing data infrastructure and model validation pipelines that ensure all AI systems are rigorously validated for compliance, performance, and patient safety
- Collaborating with AI product teams, clinical and technical stakeholders, DevOps engineers, and the AI Governance Committee to engineer scalable data flows that support model validation, real-time monitoring
- Building and maintaining robust ETL pipelines for structured and unstructured clinical data from EHR, imaging, and text sources
- Designing systems to automate data preparation, lineage tracking, and reproducibility for AI model inputs and outputs
- Developing data infrastructure for benchmarking and stress-testing models in clinical simulation environments
- Collaborating with DevOps and cloud teams to ensure deployment pipelines meet compliance and performance standards
- Setting up and monitoring model tracking infrastructure for evaluation metrics and drift detection
- Assisting in the development of standards and procedures affecting data management, design and maintenance
- Documenting all standards and procedures
- Engineering and maintaining pipelines that support pre-deployment model validation and post-deployment monitoring
- Collaborating with Data Scientists and Clinical Product Owners to validate data integrity, reproducibility, and fairness in AI workflows
- Ensuring compliance with HIPAA, ethical guidelines, and institutional governance policies on sensitive health data use
- Building dashboards and tools that provide observability across the ML lifecycle: data, models, outcomes
- Designing, building, and deploying LLM-powered applications including clinical chatbots, copilots, and decision-support tools for end-users across MSHS
- Developing retrieval-augmented generation (RAG) pipelines that integrate vector databases with clinical knowledge sources, EHR data, and institutional documents
- Building agentic platforms and multi-agent workflows using frameworks such as LangChain, LlamaIndex, LangGraph, CrewAI, or equivalent
- Operationalizing LLM deployment, including inference optimization, latency and cost tuning, model serving, and integration of safety guardrails
- Implementing prompt engineering, prompt versioning, and structured prompt-evaluation workflows across model providers and versions
- Fine-tuning and adapting foundation models to clinical and operational use cases where appropriate
- Building LLM evaluation harnesses covering accuracy, hallucination, safety, bias, sycophancy, and clinical appropriateness, with red-teaming and stress-testing of deployed systems
- Effectively communicating technical findings related to model and data integrity to governance teams, clinical stakeholders, and leadership
- Maintaining clear and well-organized documentation of data workflows, platform architecture, and validation processes
- Helping write internal reports on data infrastructure resilience, validation system status, and operational risk
- Staying informed on industry best practices in data engineering and healthcare-focused machine learning
- Possessing an extremely flexible attitude and willingness to work with multiple types of technologies and languages
- Continuous interest in updating skill sets and knowledge of trends in the Big Data Technology space
- Working closely with cross-functional teams including data scientists, healthcare providers, and IT professionals to understand data requirements, develop solutions, and support data-driven decision-making
Skills
- Bachelor's degree in Computer Science, Statistics, Mathematics, or related field
- Knowledge of at least one programming language among Scala, Python, Java, C, or C++
- Knowledge of big data technologies (e.g., Hadoop, Spark)
- Knowledge of Software Development Lifecycle
- Self-motivated with a demonstrated ability to work independently, and to exercise independent judgment in developing complex techniques or programs in a dynamic environment
- Act as the major contributor in the development and operationalization of four different applications
- Play a key technical role in maintaining deployed products
- Understanding of machine learning algorithms (Supervised, Unsupervised ML algorithms)
- Familiarity with SQL or other database languages
- Master's degree in a quantitative discipline (e.g., Statistics, Operations Research, Bioinformatics, Economics, Computational Biology, Computer Science, Information Technology, Mathematics, Physics) or equivalent practical experience
- 2+ years of experience in data engineering, software engineering, or machine learning
- Proficient in Python and SQL
- Proficiency in at least one cloud computing platforms (e.g., AWS, Azure, GCP)
- Intermediate knowledge of Machine Learning
- Familiarity with ML lifecycle management tools (e.g., MLflow, Kubeflow, Airflow)
- Experience on deployment and operationalization of ML Systems
- Experience with monitoring tools for AI model tracking
- Understanding of DevOps principles, CI/CD pipelines, and containerization (e.g., Docker, Kubernetes)
- Experience with version control systems (e.g., Git) Knowledge of big data technologies (e.g., Hadoop, Spark)
- Hands-on experience building and deploying LLM-based applications in production (chatbots, copilots, summarization, Q&A, or decision-support tools)
- Experience designing and implementing retrieval-augmented generation (RAG) architectures, including chunking strategies, embedding models, and vector databases (e.g., Pinecone, Weaviate, FAISS, pgvector, Milvus)
- Experience with agentic frameworks and orchestration libraries (e.g., LangChain, LlamaIndex, LangGraph, CrewAI, AutoGen, Semantic Kernel) including tool/function calling and multi-agent workflows
- Experience building conversational AI / chatbot systems, including dialog state management, memory, and integration with enterprise systems
- Familiarity with foundation model APIs and SDKs (e.g., OpenAI, Anthropic, Google, Azure OpenAI, AWS Bedrock) and open-weight model families (e.g., Llama, Mistral, Qwen, Gemma)
- Working knowledge of prompt engineering, prompt evaluation, and LLM observability/evaluation tooling (e.g., LangSmith, Langfuse, Arize, Ragas, DeepEval)
- Familiarity with fine-tuning and model adaptation techniques (e.g., supervised fine-tuning, LoRA/QLoRA, PEFT, instruction tuning, RLHF/DPO) and serving stacks (e.g., vLLM, TGI, Triton)
- Awareness of LLM safety, guardrails, and evaluation practices (hallucination, bias, sycophancy, jailbreak resistance) — experience with healthcare-specific evaluation is a plus
- Strong problem-solving skills and ability to work in cross-functional teams
Company Overview
Company H1B Sponsorship