Clera logo

Clera

Data Scientist, Agent Evaluations & Quality

Work from home

Remote role where the employee must remain based in a particular country.

United States only

From the employer's advert: On-site in Palo Alto, California, United States.

Employer listed it 8 weeks ago · Added 8 weeks ago

Listed for 5 days before it closed.

This job has closed

Clera has taken this listing down, so it is no longer accepting applications. The details below are kept for reference only.

Browse open roles

Salary

Not stated

Location

United States only

Timezone

Not stated

Contract

Full-time

Experience

Mid

Category

Data

The employer did not publish a salary, and we have too few comparable roles to estimate one.

What the advert says

Quoted word for word from the employer's advert. Where it says nothing, we say so.

Working hours
Not mentioned in the advert
Where you can work from
On-site in Palo Alto, California, United States.
Contract
Not mentioned in the advert
Experience asked for
Not mentioned in the advert
Pay
Not mentioned in the advert

Remote flexibility

Work from home

This is a remote role, but the employee must be based in United States. It is work from home rather than work from anywhere.

What the employer says

  • Source listing states candidate location: "Palo Alto, Remote"

What Nomaders makes of it

  • Residency required in United States
  • Payroll and tax are likely handled in that country only

The quotes above are the employer's own words; the reading is ours. Always check the original listing and employment terms before working from another country.

Before you apply

Questions worth asking: Nomaders interpretation

  • No pay is stated. Ask for the salary range before investing time in interviews.
  • The advert does not say whether this is employment or a contractor arrangement. Ask, as it affects tax and benefits.

Produced automatically from this advert's text; not reviewed by a person. The employer's own page is the final word.

About the role

About the Role

This role sits at the intersection of applied data science and AI product quality for a small, fast-moving AI productivity startup building autonomous agents that handle email, calendar, browser, and business software tasks. You will own the measurement of agent quality end-to-end: turning ambiguous product behavior into rigorous, actionable evaluation systems that directly guide engineering and product decisions.

What You'll Do

Architect and maintain automated evaluation pipelines that measure agent quality across capabilities and product surfaces.

Translate agent capabilities into explicit success criteria, including pass, partial-pass, and failure definitions for complex multi-step tasks.

Build representative gold datasets and regression suites covering common workflows, edge cases, ambiguous requests, and adversarial scenarios.

Define and track metrics such as task success, tool-selection accuracy, instruction adherence, factual consistency, latency, cost, and reliability.

Design deterministic and model-based graders, calibrate LLM-as-a-judge systems, and measure grader agreement, false positives, and false negatives.

Analyze traces, tool calls, model outputs, and production outcomes to identify root causes and build a useful failure taxonomy.

Compare models, prompts, tools, and capability implementations using rigorous offline experiments and production evidence.

Build dashboards and release-quality signals that make evaluation results understandable and actionable for engineering, product, and leadership.

Partner with capability engineers to recommend improvements and verify that fixes raise quality without unacceptable regressions in cost, latency, or reliability.

What We're Looking For

5+ years in data science, machine learning, or analytics roles, with a focus on evaluation systems, metrics frameworks, or quality measurement for production systems.

Demonstrated experience designing and implementing evaluation frameworks, grading systems, and success criteria for ML or AI systems in production.

Strong Python and SQL proficiency with the ability to build automated data pipelines and production-quality analysis code at scale.

Solid statistical and experimental design knowledge: sampling, variance, uncertainty quantification, bias detection, confounding variables, and significance testing for non-deterministic systems.

Experience with ground-truth data development: labeling guideline design, annotation quality control, ambiguity resolution, and dataset maintenance.

Working knowledge of LLM behavior, tool use, retrieval systems, multi-step execution, and practical failure modes of language model systems.

Ability to connect quantitative patterns to individual system traces and identify failure origins across model, prompt, context, tools, data, and application logic.

Experience communicating evaluation results, methodology, uncertainty, and trade-offs to both technical and non-technical stakeholders.

Comfort operating with high ownership in ambiguous, fast-moving environments, independently turning open-ended quality questions into evaluation systems.

Experience with LLM-as-a-judge systems, agentic or multi-step task evaluation, or benchmarking platforms for AI systems is a strong plus.

Location

Requirements

The employer hasn't listed requirements separately, they're described in the role summary above and on the original listing.

Benefits

No benefits package published with this listing. Ask about it at first interview.

Applying

This listing has closed, so there is no application link. The details above are kept for reference only.

Found 2mo ago. Last checked 9 Oct. Always confirm the details on the original posting, salary and location can change after publication.

Listing sourced from Company boards.

Find roles that are still open

This role is closed, applications are no longer accepted.