OpenAI
More roles at OpenAITraining Performance Engineer
Part remote, part office, you need to live within commuting distance of a named location.
Hybrid · San Francisco
Employer listed it 13 months ago · Added 2 months ago
Been open since 13 months ago. Long-running listings are sometimes left up after the role is filled.
Salary
$295k to $500k per year
Location
Hybrid · San Francisco
Work style
Async
Contract
Full-time
Experience
Mid
Category
Software
Published by the employer
What the advert says
Quoted word for word from the employer's advert. Where it says nothing, we say so.
- Working hours
Our work focuses on three pillars: high-performance, asynchronous, zero-copy tensor and optimizer-state-aware data movement; performant, high-uptime, fault-tolerant training frameworks (training loop, state management, resilient…
- Where you can work from
This role is based in San Francisco, CA.
- Contract
- Not mentioned in the advert
- Experience asked for
- Not mentioned in the advert
- Pay
- Not mentioned in the advert
Remote flexibility
Hybrid
This role is only partly remote, the employer expects time in the office around San Francisco, Hybrid, so you need to live within commuting distance.
What the employer says
- Source listing states candidate location: "San Francisco, Hybrid"
- Listing mentions "Hybrid"
What Nomaders makes of it
- Not suitable if you plan to move between countries
The quotes above are the employer's own words; the reading is ours. Always check the original listing and employment terms before working from another country.
Before you apply
Experience: employer states
Written as a requirement
- “AI is an extremely powerful tool that must be created with safety and human needs at its core, and to achieve our mission, we must encompass and value the many different perspectives, voices, and experiences that form the full spectrum of humanity.”
Questions worth asking: Nomaders interpretation
- This role involves office time, but the advert does not say how often. Ask how many days a week are expected on site.
- The advert does not say whether this is employment or a contractor arrangement. Ask, as it affects tax and benefits.
Produced automatically from this advert's text; not reviewed by a person. The employer's own page is the final word.
About the role
About the Team Training Runtime designs the core distributed machine-learning training runtime that powers everything from early research experiments to frontier-scale model runs. With a dual mandate to accelerate researchers and enable frontier scale, we’re building a unified, modular runtime that meets researchers where they are and moves with them up the scaling curve.
Our work focuses on three pillars: high-performance, asynchronous, zero-copy tensor and optimizer-state-aware data movement; performant, high-uptime, fault-tolerant training frameworks (training loop, state management, resilient checkpointing, deterministic orchestration, and observability); and distributed process management for long-lived, job-specific and user-provided processes.
We integrate proven large-scale capabilities into a composable, developer-facing runtime so teams can iterate quickly and run reliably at any scale, partnering closely with model-stack, research, and platform teams. Success for us is measured by raising both training throughput (how fast models train) and researcher throughput (how fast ideas become experiments and products).
About the Role As a Training Performance Engineer, you’ll drive efficiency improvements across our distributed training stack. You’ll analyze large-scale training runs, identify utilization gaps, and design optimizations that push the boundaries of throughput and uptime. This role blends deep systems understanding with practical performance engineering — analyzing GPU kernel performance, collective communication throughput, investigating I/O bottlenecks, and sharding our models so we can train them at massive scale.
You’ll help ensure that our clusters are running at peak performance, enabling OpenAI to train larger, more capable models with the same compute budget.
This role is based in San Francisco, CA. We use a hybrid work model of three days in the office per week and offer relocation assistance to new employees.
In this role, you will:
Profile end-to-end training runs to identify performance bottlenecks across compute, communication, and storage.
Optimize GPU utilization and throughput for large-scale distributed model training.
Collaborate with runtime and systems engineers to improve kernel efficiency, scheduling, and collective communication performance.
Implement model graph transforms to improve end to end throughput.
Build tooling to monitor and visualize MFU, throughput, and uptime across clusters.
Partner with researchers to ensure new model architectures scale efficiently during pre-training.
Contribute to infrastructure decisions that improve reliability and efficiency of large training jobs.
You might thrive in this role if you:
Love optimizing performance and digging into systems to understand how every layer interacts.
Have strong programming skills in Python and C++ (Rust or CUDA a plus).
Have experience running distributed training jobs on multi-GPU systems or HPC clusters.
Enjoy debugging complex distributed systems and measuring efficiency rigorously.
Have exposure to frameworks like PyTorch, JAX, or TensorFlow and an understanding of how large-scale training loops are built.
Are comfortable collaborating across teams and translating raw profiling data into practical engineering improvements.
Nice to have:
Familiarity with NCCL, MPI, or UCX communication libraries.
Experience with large-scale data loading and checkpointing systems.
Requirements
- ·Have experience running distributed training jobs on multi-GPU systems or HPC clusters.
- ·Enjoy debugging complex distributed systems and measuring efficiency rigorously.
- ·Have exposure to frameworks like PyTorch, JAX, or TensorFlow and an understanding of how large-scale training loops are built.
- ·Are comfortable collaborating across teams and translating raw profiling data into practical engineering improvements.
- ·Familiarity with NCCL, MPI, or UCX communication libraries.
Benefits
No benefits package published with this listing. Ask about it at first interview.
How to apply
- 1Check the flexibility label above, hybrid, matches where you plan to live and work.
- 2Tailor your CV to the role at OpenAI, mentioning your remote working experience and working hours (Async).
- 3Apply directly on the employer's careers page using the button below. Nomaders never handles your application.
Found 2mo ago. Last checked today. Always confirm the details on the original posting, salary and location can change after publication.
Listing sourced from Company boards.
Similar roles
Other open software roles with comparable remote rules.
$295k to $500k per year · You'll be taken to the employer's careers page.