Senior AI Evaluation Engineer

About Us:
LTM is a global technology consulting and digital solutions company that enables enterprises across industries to reimagine business models, accelerate innovation, and maximize growth by harnessing digital technologies. As a digital transformation partner to more than 700+ clients, LTM brings extensive domain and technology expertise to help drive superior competitive differentiation, customer experiences, and business outcomes in a converging world. Powered by nearly 90,000 talented and entrepreneurial professionals across more than 30 countries, LTM a Larsen & Toubro Group company combines the industry-acclaimed strengths of erstwhile Larsen and Toubro Infotech and Mindtree in solving the most complex business challenges and delivering transformation at scale. For more information, please visit

Title: Senior AI Evaluation Engineer
Location: Remote

This role does not offer Visa Sponsorship. Applicants must have Independent Authorization to work in the United States.
Key Responsibilities·

  • Design and operate end-to-end evaluation frameworks for agentic AI systems (offline regression, online monitoring, A/B testing)
  • Define quality rubrics, scoring architectures, and evaluation standards across multi-agent workflows
  • Build LLM-as-judge and Agent-as-judge pipelines assessing trajectory, tool-call accuracy, groundedness, and safety
  • Own and deliver evaluation infrastructure end-to-end; mentor teams and uplevel customers on evals best practices

Must Have·

  • GenAI Experience (Level 400) with Advanced Python · Experience building production grade systems· Hands-on experience with agent evaluation: offline/on-demand regression testing and online/continuous live-traffic scoring
  • LLM-as-judge proficiency: rubric design, scoring at session, response, and tool-call granularity
  • Core eval metrics: task completion, tool-call accuracy, trajectory quality, groundedness, safety
  • Agentic framework experience (Strands, LangGraph, CrewAI or similar)
  • AWS Bedrock (Models, Guardrails) + CloudWatch; Lambda, S3, IAM fundamentals

Should Have·

  • Bedrock AgentCore Evaluations (on-demand SDK runner, online monitoring, A/B testing)
  • Agent-as-a-Judge framework; HITL evaluation pipelines
  • Benchmarking frameworks (tau-bench, SWE-bench, GAIA) and custom dataset construction
  • RAG quality evaluation (faithfulness, context precision, recall)
  • CI/CD integration for eval regression; observability tooling (tracing, structured logging)
  • Legacy technology familiarity (Java/Spring, Sybase, DB2, or COBOL) as source context for agentic modernization or integration pattern design
  • Financial services or regulated-industry experience (compliance-aware delivery: trade surveillance, data controls, audit requirements)
  • Large-scale portfolio delivery experience — reusable patterns or components applied across 50+ applications, including wave planning and throughput measurement

Benefits/perks listed below may vary depending on the nature of your employment with LTM (LTIM):
Benefits and Perks:

  • Medical Plan Covering Medical, Dental, Vision
  • Term and Long-Term Disability Coverage
  • Plan with Company match.
  • Insurance
  • Time, Sick Leave, Paid Holidays
  • Paternity and Maternity Leave

The range displayed on each job posting reflects the minimum and maximum salary target for the position across all US locations. Within the range, individual pay is determined by work location and job level and additional factors including job-related skills, experience, and relevant education or training. Depending on the position offered, other forms of compensation may be provided as part of overall compensation like an annual performance-based bonus, sales incentive pay and other forms of bonus or variable compensation.
Disclaimer: The compensation and benefits information provided herein is accurate as of the date of this posting.
LTM is an equal opportunity employer that is committed to diversity in the workplace. Our employment decisions are made without regard to race, color, creed, religion, sex (including pregnancy, childbirth or related medical conditions), gender identity or expression, national origin, ancestry, age, family-care status, veteran status, marital status, civil union status, domestic partnership status, military service, handicap or disability or history of handicap or disability, genetic information, atypical hereditary cellular or blood trait, union affiliation, affectional or sexual orientation or preference, or any other characteristic protected by applicable federal, state, or local law, except where such considerations are bona fide occupational qualifications permitted by law.
Safe return to office:
In order to comply with LTM s company COVID-19 vaccine mandate, candidates must be able to provide proof of full vaccination against COVID-19 before or by the date of hire. Alternatively, one may submit a request for reasonable accommodation from LTM s COVID-19 vaccination mandate for approval, in accordance with applicable state and federal law, by the date of hire. Any request is subject to review through LTM s applicable processes.

Back to blog

Other Jobs To Apply

Online Amazon Work From Home Part Time Jobs (No Experience/Entry Level)

Video Editor - Internship

Delta Airlines Remote Customer Support - Work From Your Own Home

GMC Appointment Setter/Cold Caller - Permanent work from home, morning shift

Remote Virtual Assistant - Admin Support (No Experience Needed)

bolthires KDP Book Publishing VA/Virtual Assistants - No Experience Necessary - We Will Train You

amazon flex $15+/ Hour (Sign on Bonus)!

Remote Part-Time Night Customer Support Specialist - Flexible Night Shift Roles

Chat Moderator Job - No Prior Experience Needed, Earn $25-$35/Hour

Work From Home Product Tester – $25-$45/hr – No Experience Needed

Hourly pay Office Admin Assistant

Administrative Assistant - Data Entry (Remote)

[PART_TIME Remote] Target Remote Careers From Home $26/Hour

Senior Network Engineer, India Remote, Singapore Remote

Construction -Rebar entry level

Part-Time Virtual Assistant (2–3 hours per week)

Hiring Now: Remote Chat Agent (No Experience Necessary)

Representative II, Claims Processing - Scottsdale, Arizona (Onsite)

Amazon Customer Service Representative (Remote)

Remote College reputed company Intern – Part-Time Software reputed company & IT Solutions at reputed company

Student Loan Sales Intern (Flexible, Part-Time) | CampusReel

Hallmark Territory and Installation Supervisor (part-time) Beverly Hills, FL 34465

Medical Billing Clerk (Work From Home) - No Exp.

Part Time Work At Home Amex Virtual Assistant a...

Experienced Part-Time Remote Interview Specialist – Talent Acquisition and Educational Evaluator for brighthush

Remote Global Travel Coordinator

Online Hotel Booker No Experience Needed

amazon flex $15+/ Hour (Sign on Bonus)!

Licensed Pharmacy Technician – Data Entry Specialist – Remote reputed company Position at reputed company

Flexible Work From Home Jobs: Immediate Start, No Experience Needed

Experienced Part-Time Data Entry Specialist – Remote Administrative Support and Database Management at brighthush

**Experienced Remote Chat Agent – No Experience Needed – Work Remotely Worldwide**

E‑Commerce Account Growth Intern

reputed company: Online Jobs for 17 Year Old With No Experience

Real-Time Corporate Travel Concierge (Remote)

Senior Cloud Engineer

Live Customer Service - Evening Remote Part Time Jobs with Flexible Hours

Urgent Hiring for Remote Data Entry Jobs - Apply Today

Part Time Tutor Jobs in Sitiawan and Across Malaysia

Remote Social Media Customer Support Specialist – Engaging Audiences, Enhancing Brand Experience, and Delivering Exceptional Service for reputed company (reputed company / Part‑Time, Work‑From‑Home)

Assistant Team Leader - Premium Travel & Concierge

Remote Data Entry Associate – Student‑Friendly, No‑Experience Required, Flexible Home‑Based Schedule, Earn While You reputed company

Immediate Hiring: Online Content Typist and Transcription Specialist for Students to Earn Money Weekly from Home

Internship (We pay for each project closure) JobID JC0015821

Dynamic Part-Time Remote Customer Email Support Specialist – Flexible Schedule for College Students, Championing Women in Customer Service Excellence at brighthush

Remote Specialist - Leaves of Absence (LOA) & Workers Comp Programs (Remote Contractor)

Dominos Content Moderator Jobs (Remote, Part Time) $35/Hour

Professional Resident (Medicaid) (Remote)

SDET / QA 100% Remote

Hiring Now - Work from Home - No Experience

Common Interview Questions And Answers

1. HOW DO YOU PLAN YOUR DAY?

This is what this question poses: When do you focus and start working seriously? What are the hours you work optimally? Are you a night owl? A morning bird? Remote teams can be made up of people working on different shifts and around the world, so you won't necessarily be stuck in the 9-5 schedule if it's not for you...

2. HOW DO YOU USE THE DIFFERENT COMMUNICATION TOOLS IN DIFFERENT SITUATIONS?

When you're working on a remote team, there's no way to chat in the hallway between meetings or catch up on the latest project during an office carpool. Therefore, virtual communication will be absolutely essential to get your work done...

3. WHAT IS "WORKING REMOTE" REALLY FOR YOU?

Many people want to work remotely because of the flexibility it allows. You can work anywhere and at any time of the day...

4. WHAT DO YOU NEED IN YOUR PHYSICAL WORKSPACE TO SUCCEED IN YOUR WORK?

With this question, companies are looking to see what equipment they may need to provide you with and to verify how aware you are of what remote working could mean for you physically and logistically...

5. HOW DO YOU PROCESS INFORMATION?

Several years ago, I was working in a team to plan a big event. My supervisor made us all work as a team before the big day. One of our activities has been to find out how each of us processes information...

6. HOW DO YOU MANAGE THE CALENDAR AND THE PROGRAM? WHICH APPLICATIONS / SYSTEM DO YOU USE?

Or you may receive even more specific questions, such as: What's on your calendar? Do you plan blocks of time to do certain types of work? Do you have an open calendar that everyone can see?...

7. HOW DO YOU ORGANIZE FILES, LINKS, AND TABS ON YOUR COMPUTER?

Just like your schedule, how you track files and other information is very important. After all, everything is digital!...

8. HOW TO PRIORITIZE WORK?

The day I watched Marie Forleo's film separating the important from the urgent, my life changed. Not all remote jobs start fast, but most of them are...

9. HOW DO YOU PREPARE FOR A MEETING AND PREPARE A MEETING? WHAT DO YOU SEE HAPPENING DURING THE MEETING?

Just as communication is essential when working remotely, so is organization. Because you won't have those opportunities in the elevator or a casual conversation in the lunchroom, you should take advantage of the little time you have in a video or phone conference...

10. HOW DO YOU USE TECHNOLOGY ON A DAILY BASIS, IN YOUR WORK AND FOR YOUR PLEASURE?

This is a great question because it shows your comfort level with technology, which is very important for a remote worker because you will be working with technology over time...