Get the app Chrome extension Freelance ideasGigsSign inFind my gigs

ProfilePush / H1B jobs / AI Evaluation Specialist

AI Evaluation Specialist

System One (TechUSA) · Atlanta, GA · contract · posted 14 days ago

This posting says US citizenship is required. System One (TechUSA) sponsors H1B for other roles, so look at its other jobs below.
$155,000 a yearpay
Atlanta, GAlocation
contractjob type

Apply on System One (TechUSA)'s siteCheck my AI Match score

About the job

Job Title: AI Evaluation Specialist

US Citizenship or Green Card required per government contract

Clearance: Must be able to obtain Public Trust

Salary: $155K

Job Description:

In this position, you’ll evaluate enterprise AI tools and capabilities to determine their effectiveness, safety, usability, and readiness for business adoption. You’ll partner with product managers, data scientists, AI engineers, software engineers, security and privacy teams, and business stakeholders to define evaluation criteria and conduct structured assessments of generative AI products and platforms, including tools such as OpenAI and life sciences models like Rosalind. You will design repeatable evaluation frameworks, test real-world use cases, analyze model and product behavior, and translate findings into clear recommendations. You will be responsible for comparing AI solutions across quality, accuracy, reliability, safety, privacy and security.

Key Job Responsibilities

Plan and execute hands-on evaluations of AI tools such as OpenAI and life sciences models like Rosalind using representative business scenarios, workflows, prompts, datasets, and user personas.

Assess AI solutions for response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall task effectiveness.

Design and maintain standardized evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools, models, agents, and AI-enabled applications.

Evaluate safety, privacy, security, data handling, governance, and responsible AI considerations in collaboration with security, privacy, legal, and governance stakeholders.

Conduct comparative evaluations and proof-of-concept assessments across multiple AI products, models, configurations, and vendors, including analysis of functionality, integration requirements, performance, and cost.

Evaluate AI agents and tool-enabled workflows, including retrieval-augmented generation (RAG), enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol (MCP)-enabled capabilities where applicable.

Stay current with advancements in generative AI, model evaluation, AI safety, responsible AI, and emerging enterprise AI products and incorporate relevant methods into the evaluation program.

Engage in knowledge sharing and contribute to building a strong, collaborative team environment.

Job Requirements:

5+ years of experience in model evaluation, model quality assurance with hands-on experience evaluating AI-enabled products or services

Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria

Experience with hands-on generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes

Experience assessing AI systems across dimensions such as accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost

Ability to demonstrate familiarity with common generative AI architectures and capabilities, including large language models, retrieval-augmented generation (RAG), AI agents, APIs, enterprise search, and tool or connector integrations

Bachelor's degree

Must be a U.S. Citizen or Lawful Permanent Resident (Green Card Holder)

Job Desirable Skills:

Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models like Rosalind or comparable generative AI solutions

Experience with automated LLM evaluation frameworks, model-as-a-judge approaches, human-in-the-loop evaluation, red teaming, or adversarial testing

Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows

Experience developing dashboards or reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results

Ability to work with datasets and evaluation results using tools such as Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics, identify patterns, and summarize findings

Possession of an industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI

Master’s degree preferred

#M1

#LI-VH1

Ref: #851-Rockville-S1

More System One (TechUSA) jobs this week

IT Support Technician IISystem One (TechUSA) · Washington, DC · contract · yesterday
View
Sales RepresentativeSystem One (TechUSA) · Pittsfield, MA · full-time · yesterday
View
Planning Project CoordinatorSystem One (TechUSA) · Merrifield, VA · contract · yesterday
View
Microsoft Enterprise Architect – Intune SMESystem One (TechUSA) · Washington, DC · contract · yesterday
View
Lead Sales Operations AnalystSystem One (TechUSA) · San Jose, CA · contract · yesterday
View
Sales ManagerSystem One (TechUSA) · Pittsfield, MA · full-time · yesterday
View

Questions

Does System One (TechUSA) sponsor H1B for this job?

This posting says US citizenship is required. System One (TechUSA) sponsors H1B for other roles, so look at its other jobs below.

How do I apply for this AI Evaluation Specialist job?

Apply on System One (TechUSA)'s own careers page with the Apply button. To see how well your resume fits first, ProfilePush's AI Match scores it 0 to 100 for free.

How well do you fit this job?

Upload your resume and AI scores this job, and every sponsor job like it, 0 to 100. Free to start.

Get my match score