Sentry
Senior Software Engineer, AI Evals
Part remote, part office, you need to live within commuting distance of a named location.
Hybrid · San Francisco, California
Employer listed it 10 months ago · Added 2 months ago
Listed for 257 days before it closed.
This job has closed
Sentry has taken this listing down, so it is no longer accepting applications. The details below are kept for reference only.
Browse open rolesSalary
$155k to $400k per year
Location
Hybrid · San Francisco, California
Timezone
Not stated
Contract
Full-time
Experience
Senior
Category
Software
Published by the employer
What the advert says
Quoted word for word from the employer's advert. Where it says nothing, we say so.
- Working hours
- Not mentioned in the advert
- Where you can work from
- Location eligibility not confirmed
- Contract
- Not mentioned in the advert
- Experience asked for
Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
- Pay
The base salary range (or hourly wage range, if applicable) that Sentry reasonably expects to pay for this position is $155,000 to $400,000 USD .
Remote flexibility
Hybrid
This role is only partly remote, the employer expects time in the office around San Francisco, California, Hybrid, so you need to live within commuting distance.
What the employer says
- Source listing states candidate location: "San Francisco, California, Hybrid"
- Listing mentions "Hybrid"
What Nomaders makes of it
- Not suitable if you plan to move between countries
The quotes above are the employer's own words; the reading is ours. Always check the original listing and employment terms before working from another country.
Before you apply
Experience: employer states
Written as a nice-to-have
- “Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)”
Questions worth asking: Nomaders interpretation
- This role involves office time, but the advert does not say how often. Ask how many days a week are expected on site.
- The advert does not say whether this is employment or a contractor arrangement. Ask, as it affects tax and benefits.
Produced automatically from this advert's text; not reviewed by a person. The employer's own page is the final word.
About the role
About Sentry
Software runs the world and the pace is faster than ever. Sentry helps developers fix errors and performance issues before users notice, so teams can spend less time firefighting and more time building.
Trusted by 200,000+ organizations, Sentry is today’s application monitoring standard and our team is building its AI-native future.
About the role
As a Senior Software Engineer on Sentry’s AI/ML team, you’ll be responsible for building the evaluation infrastructure that measures the accuracy, reliability, and real-world performance of our AI systems. This role is critical to ensuring that our debugging agents and AI-powered features behave correctly, safely, and predictably as they scale. You’ll design datasets, benchmarks, and test harnesses that turn ambiguous AI behavior into measurable signals, helping the team ship AI with confidence.
In this role you will
Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring
You’ll love this job if you
Care deeply about correctness, rigor, and measurement in AI systems
Enjoy turning fuzzy product goals and model behavior into concrete tests and metrics
Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team
Thrive in cross-functional environments and enjoy influencing model design through better evaluation
Qualifications
Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
Comfort writing production-quality code (we use Python and TypeScript)
Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)
Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer tools
The base salary range (or hourly wage range, if applicable) that Sentry reasonably expects to pay for this position is $155,000 to $400,000 USD . A successful candidate’s actual base salary (or hourly wage) amount will be determined by a variety of relevant factors including, without limitation, the candidate’s work location, education, work and other relevant experience, skills, and job-related knowledge. A successful candidate will be eligible to participate in Sentry’s employee benefit plans/programs applicable to the candidate’s position (including incentive compensation, equity grants, paid time off, and group health insurance coverage). See Sentry Benefits for more details about the Company’s benefit plans/programs.
Requirements
- ·Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
- ·Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
- ·Comfort writing production-quality code (we use Python and TypeScript)
- ·Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
- ·Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)
Benefits
No benefits package published with this listing. Ask about it at first interview.
Applying
This listing has closed, so there is no application link. The details above are kept for reference only.
Found 2mo ago. Last checked 17 Oct. Always confirm the details on the original posting, salary and location can change after publication.
Listing sourced from Company boards.
Similar open jobs
This role is closed, applications are no longer accepted.