Fireworks AI logo

Fireworks AI

More roles at Fireworks AI

Member of Technical Staff, Evals Platform

Hybrid

Part remote, part office, you need to live within commuting distance of a named location.

Hybrid · San Mateo

Employer listed it 13 months ago · Added 8 weeks ago

Been open since 13 months ago. Long-running listings are sometimes left up after the role is filled.

Salary

$210k to $320k per year

Location

Hybrid · San Mateo

Timezone

Not stated

Contract

Full-time

Experience

Lead

Category

Software

Published by the employer

What the advert says

Quoted word for word from the employer's advert. Where it says nothing, we say so.

Working hours
Not mentioned in the advert
Where you can work from
Location eligibility not confirmed
Contract
Not mentioned in the advert
Experience asked for
Not mentioned in the advert
Pay
Not mentioned in the advert

Remote flexibility

Hybrid

This role is only partly remote, the employer expects time in the office around San Mateo, Hybrid, so you need to live within commuting distance.

What the employer says

  • Source listing states candidate location: "San Mateo, Hybrid"
  • Listing mentions "Hybrid"

What Nomaders makes of it

  • Not suitable if you plan to move between countries

The quotes above are the employer's own words; the reading is ours. Always check the original listing and employment terms before working from another country.

Before you apply

Questions worth asking: Nomaders interpretation

  • This role involves office time, but the advert does not say how often. Ask how many days a week are expected on site.
  • The advert does not say whether this is employment or a contractor arrangement. Ask, as it affects tax and benefits.

Produced automatically from this advert's text; not reviewed by a person. The employer's own page is the final word.

About the role

About Us:

Fireworks is the platform for specialized intelligence, enabling companies to build, train, and serve AI models tailored to their own data, workflows, and products. Founded by the team behind PyTorch and backed by AMD, Atreides, Benchmark Capital, Index Ventures, Lightspeed, NVIDIA, Sequoia Capital, and TCV, Fireworks powers production AI with hundreds of state-of-the-art open models across text, image, embedding, audio, and multimodal workloads. Today, Fireworks is a Series D company valued at $17.5 billion, bringing together an ambitious, collaborative team that's building the future of enterprise AI.

We are seeking a Member of Technical Staff, Evals & Post-Training Product to help define how developers improve models on Fireworks. This role sits at a unique intersection of scalable system design, deep data science, and model quality.

You will build the infrastructure and workflows that connect evaluation and post-training. Our evaluation setup has grown past its original scope, and we need someone who can take it to the next stage, improving programmatic access and scale. You will work across backend systems, sandbox infrastructure, and user-facing surfaces to make it easier to author evals, understand results, and iterate quickly.

Key Responsibilities

Scale Eval Infrastructure: Take ownership of our internal eval setup and evolve it for the future. You will design systems to eliminate single-host coordination bottlenecks, resolve log-syncing latency, and build seamless programmatic access.

Benchmark Obsession & Reproduction: Act as a "metrics obsessive." Track state-of-the-art (SOTA) benchmarks, read the latest research papers, dig deep into data discrepancies, and insist on rigorously reproducing published results internally.

Pioneer New Benchmarks: Design and build entirely new benchmarks to measure model performance on complex, emerging, or domain-specific use cases.

Own Fine-Tuning Product Experiences: Build and improve user-facing workflows for post-training, including fine-tuning experiences across SFT, RFT, and related model-improvement capabilities.

Work Closely With Users: Partner with customers and internal stakeholders to understand evaluation and fine-tuning needs, triage issues, and convert bespoke workflows into productized, reusable solutions.

Minimum Requirements

Experience: 1–7 years of software engineering or data science experience (We are hiring at multiple levels for this role).

Strong System Design Skills: You know how to architect scalable, programmatic systems and transition legacy setups into robust infrastructure.

Sandbox Infrastructure: Hands-on experience building or working with sandbox environments and sandbox infrastructure for secure code execution and testing.

Analytical & Data Science Mindset: You possess a deep understanding of LLM evaluations, how to design them, and how to use the results to guide model improvement. You are meticulous about data and metrics.

Understanding of the GenAI Lifecycle: You understand the end-to-end workflow—from prompting a base model to curating a dataset, fine-tuning, and productionizing agents.

Preferred Qualifications

Experience: 3+ years of software engineering or applied data science experience.

Frameworks & Orchestration: Experience working with the Harbor framework or similar container registry and orchestration tools.

Public Writing & Analysis: A strong interest in discovering where different models excel and fall short, with a desire to write up and publish these insights publicly (e.g., technical blogs, whitepapers).

Inference & Hardware Knowledge: Interest in the hardware side of AI—understanding GPU constraints, inference optimization techniques, and how they relate to model performance.

Startup DNA: Experience in fast-paced environments where you own features end-to-end.

Why Fireworks?

Solve Hard Problems: Tackle challenges at the forefront of AI infrastructure, from low-latency inference to scalable model serving.

Requirements

  • ·Experience: 1–7 years of software engineering or data science experience (We are hiring at multiple levels for this role).
  • ·Strong System Design Skills: You know how to architect scalable, programmatic systems and transition legacy setups into robust infrastructure.
  • ·Sandbox Infrastructure: Hands-on experience building or working with sandbox environments and sandbox infrastructure for secure code execution and testing.
  • ·Analytical & Data Science Mindset: You possess a deep understanding of LLM evaluations, how to design them, and how to use the results to guide model improvement. You are meticulous about data and metrics.
  • ·Understanding of the GenAI Lifecycle: You understand the end-to-end workflow—from prompting a base model to curating a dataset, fine-tuning, and productionizing agents.

Benefits

No benefits package published with this listing. Ask about it at first interview.

How to apply

  1. 1Check the flexibility label above, hybrid, matches where you plan to live and work.
  2. 2Tailor your CV to the role at Fireworks AI, mentioning your remote working experience.
  3. 3Apply directly on the employer's careers page using the button below. Nomaders never handles your application.

Found 2mo ago. Last checked today. Always confirm the details on the original posting, salary and location can change after publication.

Listing sourced from Company boards.

$210k to $320k per year · You'll be taken to the employer's careers page.