Robots and Pencils
More roles at Robots and PencilsStaff Applied Scientist
Remote role where the employee must remain based in a particular country.
Canada only
From the listing details: Source listing states candidate location: "CA Remote, Canada - Remote"
Employer listed it 6 days ago · Added 10 days ago
First listed 10 days ago and still open.
Salary
$176,612 to $243,680
Location
Canada only
Timezone
Not stated
Contract
Full-time
Experience
Lead
Category
Data
Stated by the employer in the job description
What the advert says
Quoted word for word from the employer's advert. Where it says nothing, we say so.
- Working hours
- Not mentioned in the advert
- Where you can work from
- Location eligibility not confirmed
- Contract
- Not mentioned in the advert
- Experience asked for
- Not mentioned in the advert
- Pay
Our salary range is $176,612 - $243,680 CAD
Remote flexibility
Work from home
This is a remote role, but the employee must be based in Canada. It is work from home rather than work from anywhere.
What the employer says
- Source listing states candidate location: "CA Remote, Canada - Remote"
What Nomaders makes of it
- Residency required in Canada
- Payroll and tax are likely handled in that country only
The quotes above are the employer's own words; the reading is ours. Always check the original listing and employment terms before working from another country.
Before you apply
Questions worth asking: Nomaders interpretation
- The advert does not say where you can be based. Ask which countries they can legally employ or contract you from.
- The advert does not say whether this is employment or a contractor arrangement. Ask, as it affects tax and benefits.
Produced automatically from this advert's text; not reviewed by a person. The employer's own page is the final word.
About the role
We're looking for a Staff Applied Scientist to lead the design and build of an evaluation framework for an enterprise planner-based agent. You will work directly with the client's EA engineering team to benchmark their agent across planner models, executor LLMs and inference optimization levers. Every proposed change, whether a model upgrade, a harness change or a routing strategy, becomes a controlled, statistically sound experiment with a measurable dollar outcome.
This is a research-heavy, customer-facing role with a significant build component. You will own both the experimental method and the tooling that makes it repeatable: a scenario-driven user simulator, isolated experiment execution, statistical comparison, and automated insights in a locally hosted dashboard the client can adopt as their own. The work runs in three overlapping streams (evaluation pipeline, grading and statistics, dashboard and experiment campaign) and ends in a recommendation the client can act on.
Why This Role Matters
At Robots & Pencils, we design AI systems for a human world. Our name says it all. Robots and pencils means engineering paired with creativity, because every agent we ship has to work for real people in real workflows. That balance is baked into how we operate.
Agents are only as good as the evidence behind them. Today, most teams pick a model or switch on a cost lever on instinct. Here, you replace instinct with data: you build the system that tells the client which planner model, which executor and which levers to roll out, how much quality each change costs or saves, and what it is worth in dollars at production volume. You also show how well the planner carries over from data analytics to EA tasks. This is a research role with real build accountability, and your work becomes the client's own decision-making tool. It demands people who take ownership, lead with craft, and care deeply about what they put their name on.
What You'll Do
Evaluation Pipeline & Simulation
Design the end-to-end evaluation workflow for a planner-executor agent: scenarios, user simulation, configuration, execution, grading, comparison and insights
Define the scenario schema, grading spec and validators with the client's team, who own scenario creation, so their data plugs directly into the framework (each scenario has a goal, persona, starting state and expected end state)
Build an LLM-based user simulator with a hidden goal and a stop rule that holds realistic multi-turn conversations, and check its realism against individual production logs
Build isolated, parallel experiment execution: config files, one run per config and scenario, and a harness adapter that routes all model traffic through the client's existing LLM gateway
Package optimization levers as plug-ins and implement a shortlist of 3 to 4 for v1, such as cascade-on-failure, cache-aware model switching and prompt caching
Tag every model call at the gateway and send traces to a trace store, so tokens, cache usage, latency and cost are attributed to each request
Grading, Statistics & Insights
Grade each run against the expected end state as the primary signal, with optional LLM judges for answer and plan quality
Compute the core metrics: verified success, cost per success, p95 latency, and failures traced to pipeline stage
Compare experiments on the same scenarios with paired methods: confidence intervals, win/loss/tie per scenario, and a non-inferiority test that a cost lever does not drop quality past an agreed margin. Say clearly when a result is inconclusive
Convert differences into projected dollar savings at production volume using real deployment costs, not just list token prices, and plot the cost-versus-quality frontier
Build an automated insight engine where a statistics layer computes every number and an LLM only writes them up, so each ranked finding links back to its evidence
Dashboard, Experiments & Readout
Build the locally hosted dashboard (Launch, Live, Results, Compare, Insights and Traces views), API-first with one-command local deployment so it can be integrated into the client's platform later
Run the staged campaign: production baseline, model screen, lever screen, then confirm the best combination
Analyze generalization by EA scenario category, optionally alongside a small data-analytics anchor set
Deliver a final recommendation with confidence intervals and dollar impact, plus the runbook and integration guide that support go/no-go decisions
Requirements
- ·Experience with LLM and agent evaluation methods, such as programmatic state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring and user simulation
- ·Experience building or evaluating planner-executor or other multi-step, tool-using agent architectures in production, not just prototypes
- ·A strong background in experimental design and statistical inference, including paired designs, non-inferiority testing and power analysis, and the judgment to call a result inconclusive
- ·Experience with multi-objective optimization or cost-aware ML, including Pareto frontier analysis and converting results into projected dollar savings
- ·Experience with observability and tracing for LLM systems, including attributing tokens, cache usage, latency and cost per request
Benefits
No benefits package published with this listing. Ask about it at first interview.
How to apply
- 1Check the flexibility label above, work from home, matches where you plan to live and work.
- 2Tailor your CV to the role at Robots and Pencils, mentioning your remote working experience.
- 3Apply directly on the employer's careers page using the button below. Nomaders never handles your application.
Found 11d ago. Last checked 17 Nov. Always confirm the details on the original posting, salary and location can change after publication.
Listing sourced from Company boards.
Similar roles
Other open data roles with comparable remote rules.
$176,612 to $243,680 · You'll be taken to the employer's careers page.