Nous Research
More roles at Nous ResearchMachine Learning Engineer, Evals
Remote work allowed only within certain countries or regions.
North America
Employer listed it 3 months ago · Added 2 months ago
Been open since 3 months ago. Long-running listings are sometimes left up after the role is filled.
Salary
Not stated
Location
North America
Timezone
US East
Contract
Full-time
Experience
Mid
Category
Data
The employer did not publish a salary, and we have too few comparable roles to estimate one.
What the advert says
Quoted word for word from the employer's advert. Where it says nothing, we say so.
- Working hours
- Not mentioned in the advert
- Where you can work from
- Location eligibility not confirmed
- Contract
3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience from coursework, an internship, a side project, open source work, or a job
- Experience asked for
- Not mentioned in the advert
- Pay
- Not mentioned in the advert
Remote flexibility
Region Restricted
Remote work is allowed, but only for candidates based in North America.
What the employer says
- Source listing states candidate location: "Americas (US time zones), Remote"
What Nomaders makes of it
- Applications outside the listed area are usually rejected
- Timezone overlap with the listed area is often expected
The quotes above are the employer's own words; the reading is ours. Always check the original listing and employment terms before working from another country.
Before you apply
Experience: employer states
Written as a requirement
- “Experience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.), with real opinions on what it does well and where it falls short”
Written as a nice-to-have
- “Hands-on experience with LLMs: prompting, few-shot design, and ideally fine-tuning or RAG; regular use of coding agents”
Questions worth asking: Nomaders interpretation
- The advert does not say where you can be based. Ask which countries they can legally employ or contract you from.
- No pay is stated. Ask for the salary range before investing time in interviews.
Produced automatically from this advert's text; not reviewed by a person. The employer's own page is the final word.
About the role
The Role
You'll work across the lab on agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together. This is a high-growth, high-ownership role on a small team, and you'll ship evaluation infrastructure that researchers depend on from day one.
Responsibilities
Run the full eval pipeline end to end and reproduce known results during onboarding, pairing with a senior engineer on your first task
Build a judge calibration protocol: sample human-labeled decisions, measure agreement (κ, per-class P/R), identify drift zones, and document it so anyone can re-run it
Extend an existing benchmark (GAIA, τ-Bench, SWE-bench slice, etc.) with new tasks targeting known capability gaps, including the prompt, environment, rubric, automated grader, and QA
Run failure analysis on model outputs: categorize failure modes, quantify prevalence, and write up findings with recommendations for training data, judge prompts, or benchmark changes
Own a recurring eval workflow (weekly regression suite, judge drift dashboard, red-team evaluation for a new capability) and ship tooling researchers actually use
Qualifications
3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience from coursework, an internship, a side project, open source work, or a job
Experience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.), with real opinions on what it does well and where it falls short
Hands-on experience with LLMs: prompting, few-shot design, and ideally fine-tuning or RAG; regular use of coding agents
Solid Python. You write clean, tested, version-controlled code that a colleague could run without you babysitting it
Comfort with Git, CI/CD basics, Docker, and the Linux command line (SSH, tmux, debugging a remote job)
Understanding of basic eval statistics: why accuracy misleads on imbalanced judges, what Cohen's κ measures, how to think about confidence intervals on a metric
At least 3 of the following: you can explain why LLM-as-judge needs calibration; you've done failure analysis and can tell model bugs apart from prompt, grader, or retrieval issues; you know at least two agent benchmarks (GAIA, AgentBench, τ-Bench, MINT, SWE-bench, WebShop, ALFWorld) and a limitation of each; you've designed or extended an eval dataset with happy paths, edge cases, and adversarial examples; you've thought about non-determinism in eval, how you sample, how many runs, how you report variance
You communicate clearly to both researchers and engineers, in the right language for each
You're comfortable with ambiguity, can turn a half-formed request into a plan, and know when to ask for help
Preferred
RLVR / RLHF pipeline experience
Training data curation experience
Distributed eval orchestration experience
Benchmark design from scratch
Red teaming and adversarial eval experience
Requirements
- ·3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience from coursework, an internship, a side project, open source work, or a job
- ·Experience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.), with real opinions on what it does well and where it falls short
- ·Hands-on experience with LLMs: prompting, few-shot design, and ideally fine-tuning or RAG; regular use of coding agents
- ·Solid Python. You write clean, tested, version-controlled code that a colleague could run without you babysitting it
- ·Comfort with Git, CI/CD basics, Docker, and the Linux command line (SSH, tmux, debugging a remote job)
Benefits
No benefits package published with this listing. Ask about it at first interview.
How to apply
- 1Check the flexibility label above, region restricted, matches where you plan to live and work.
- 2Tailor your CV to the role at Nous Research, mentioning your remote working experience and working hours (US East).
- 3Apply directly on the employer's careers page using the button below. Nomaders never handles your application.
Found 2mo ago. Last checked today. Always confirm the details on the original posting, salary and location can change after publication.
Listing sourced from Company boards.
Similar roles
Other open data roles with comparable remote rules.
Browse all open rolesPay not stated · You'll be taken to the employer's careers page.