OpenAI logo

OpenAI

More roles at OpenAI

Training Performance Engineer

Hybrid

Part remote, part office, you need to live within commuting distance of a named location.

Hybrid · San Francisco

Employer listed it 13 months ago · Added 2 months ago

Been open since 13 months ago. Long-running listings are sometimes left up after the role is filled.

Salary

$295k to $500k per year

Location

Hybrid · San Francisco

Work style

Async

Contract

Full-time

Experience

Mid

Category

Software

Published by the employer

What the advert says

Quoted word for word from the employer's advert. Where it says nothing, we say so.

Working hours
Our work focuses on three pillars: high-performance, asynchronous, zero-copy tensor and optimizer-state-aware data movement; performant, high-uptime, fault-tolerant training frameworks (training loop, state management, resilient…
Where you can work from
This role is based in San Francisco, CA.
Contract
Not mentioned in the advert
Experience asked for
Not mentioned in the advert
Pay
Not mentioned in the advert

Remote flexibility

Hybrid

This role is only partly remote, the employer expects time in the office around San Francisco, Hybrid, so you need to live within commuting distance.

What the employer says

  • Source listing states candidate location: "San Francisco, Hybrid"
  • Listing mentions "Hybrid"

What Nomaders makes of it

  • Not suitable if you plan to move between countries

The quotes above are the employer's own words; the reading is ours. Always check the original listing and employment terms before working from another country.

Before you apply

Experience: employer states

Written as a requirement

  • “AI is an extremely powerful tool that must be created with safety and human needs at its core, and to achieve our mission, we must encompass and value the many different perspectives, voices, and experiences that form the full spectrum of humanity.”

Questions worth asking: Nomaders interpretation

  • This role involves office time, but the advert does not say how often. Ask how many days a week are expected on site.
  • The advert does not say whether this is employment or a contractor arrangement. Ask, as it affects tax and benefits.

Produced automatically from this advert's text; not reviewed by a person. The employer's own page is the final word.

About the role

About the Team Training Runtime designs the core distributed machine-learning training runtime that powers everything from early research experiments to frontier-scale model runs. With a dual mandate to accelerate researchers and enable frontier scale, we’re building a unified, modular runtime that meets researchers where they are and moves with them up the scaling curve.

Our work focuses on three pillars: high-performance, asynchronous, zero-copy tensor and optimizer-state-aware data movement; performant, high-uptime, fault-tolerant training frameworks (training loop, state management, resilient checkpointing, deterministic orchestration, and observability); and distributed process management for long-lived, job-specific and user-provided processes.

We integrate proven large-scale capabilities into a composable, developer-facing runtime so teams can iterate quickly and run reliably at any scale, partnering closely with model-stack, research, and platform teams. Success for us is measured by raising both training throughput (how fast models train) and researcher throughput (how fast ideas become experiments and products).

About the Role As a Training Performance Engineer, you’ll drive efficiency improvements across our distributed training stack. You’ll analyze large-scale training runs, identify utilization gaps, and design optimizations that push the boundaries of throughput and uptime. This role blends deep systems understanding with practical performance engineering — analyzing GPU kernel performance, collective communication throughput, investigating I/O bottlenecks, and sharding our models so we can train them at massive scale.

You’ll help ensure that our clusters are running at peak performance, enabling OpenAI to train larger, more capable models with the same compute budget.

This role is based in San Francisco, CA. We use a hybrid work model of three days in the office per week and offer relocation assistance to new employees.

In this role, you will:

Profile end-to-end training runs to identify performance bottlenecks across compute, communication, and storage.

Optimize GPU utilization and throughput for large-scale distributed model training.

Collaborate with runtime and systems engineers to improve kernel efficiency, scheduling, and collective communication performance.

Implement model graph transforms to improve end to end throughput.

Build tooling to monitor and visualize MFU, throughput, and uptime across clusters.

Partner with researchers to ensure new model architectures scale efficiently during pre-training.

Contribute to infrastructure decisions that improve reliability and efficiency of large training jobs.

You might thrive in this role if you:

Love optimizing performance and digging into systems to understand how every layer interacts.

Have strong programming skills in Python and C++ (Rust or CUDA a plus).

Have experience running distributed training jobs on multi-GPU systems or HPC clusters.

Enjoy debugging complex distributed systems and measuring efficiency rigorously.

Have exposure to frameworks like PyTorch, JAX, or TensorFlow and an understanding of how large-scale training loops are built.

Are comfortable collaborating across teams and translating raw profiling data into practical engineering improvements.

Nice to have:

Familiarity with NCCL, MPI, or UCX communication libraries.

Experience with large-scale data loading and checkpointing systems.

Requirements

  • ·Have experience running distributed training jobs on multi-GPU systems or HPC clusters.
  • ·Enjoy debugging complex distributed systems and measuring efficiency rigorously.
  • ·Have exposure to frameworks like PyTorch, JAX, or TensorFlow and an understanding of how large-scale training loops are built.
  • ·Are comfortable collaborating across teams and translating raw profiling data into practical engineering improvements.
  • ·Familiarity with NCCL, MPI, or UCX communication libraries.

Benefits

No benefits package published with this listing. Ask about it at first interview.

How to apply

  1. 1Check the flexibility label above, hybrid, matches where you plan to live and work.
  2. 2Tailor your CV to the role at OpenAI, mentioning your remote working experience and working hours (Async).
  3. 3Apply directly on the employer's careers page using the button below. Nomaders never handles your application.

Found 2mo ago. Last checked today. Always confirm the details on the original posting, salary and location can change after publication.

Listing sourced from Company boards.

Similar roles

Other open software roles with comparable remote rules.

Browse all open roles

$295k to $500k per year · You'll be taken to the employer's careers page.