Delta Exchange logo

Delta Exchange

More roles at Delta Exchange

AI Evaluation Engineer

Work from home

Remote role where the employee must remain based in a particular country.

India only

From the employer's advert: Delta is bigger than all the Indian crypto exchanges combined.

Employer listed it 6 weeks ago ยท Added 8 weeks ago

Been open since 8 weeks ago. Long-running listings are sometimes left up after the role is filled.

Salary

Not stated

Location

India only

Timezone

APAC

Contract

Full-time

Experience

Mid

Category

Software

The employer did not publish a salary, and we have too few comparable roles to estimate one.

What the advert says

Quoted word for word from the employer's advert. Where it says nothing, we say so.

Working hours
Not mentioned in the advert
Where you can work from
Location eligibility not confirmed
Contract
Convert findings into actionable engineering evidence and into permanent regression coverage.
Experience asked for
Not mentioned in the advert
Pay
With a daily traded volume of ~$ 10 billion , and increasing.

Remote flexibility

Work from home

This is a remote role, but the employee must be based in India. It is work from home rather than work from anywhere.

What the employer says

  • Source listing states candidate location: "India, Remote"

What Nomaders makes of it

  • Residency required in India
  • Payroll and tax are likely handled in that country only

The quotes above are the employer's own words; the reading is ours. Always check the original listing and employment terms before working from another country.

Before you apply

Experience: employer states

Written as a requirement

  • โ€œAt least 1 year owning quality for AI products in a lead or primary-owner capacity.โ€

Questions worth asking: Nomaders interpretation

  • The advert does not say where you can be based. Ask which countries they can legally employ or contract you from.

Produced automatically from this advert's text; not reviewed by a person. The employer's own page is the final word.

About the role

About the Company

At Delta, we are reimagining and rebuilding the financial system. Join our team to make a positive impact on the future of finance.

๐ŸŽฏ Mission Driven: Re-imagine and rebuild the future of finance.

๐Ÿ’ก Most innovative cryptocurrency derivatives exchange. With a daily traded volume of ~$ 10 billion , and increasing. Delta is bigger than all the Indian crypto exchanges combined.

๐Ÿ“ˆ Offer the widest range of derivative products and have been serving traders all over the globe since 2018 and growing fast.

๐Ÿ’ช๐Ÿป The founding team is comprised of IIT and ISB graduates. Business co-founders have previously worked with Citibank, UBS and GIC; and our tech co-founder is a serial entrepreneur who previously co-founded TinyOwl and Housing.com .

๐Ÿ’ฐ Funded by top crypto funds (Sino Global Capital, CoinFund, Gumi Cryptos) and crypto projects (Aave and Kyber Network).

AI Evaluation Engineer (SDET - AI)

๐ŸŽฏ Role Overview

Delta operates three AI products used by live traders: a customer support chatbot, an API Copilot that generates and executes trading scripts, and an MCP server. This role owns the quality of those products and is responsible for measuring it objectively.

Testing probabilistic systems differs fundamentally from testing deterministic ones. The same input produces different output on every run, an incorrect answer can be entirely fluent, and a prompt change in one flow can degrade another without any visible signal. The core of this role is establishing what correct means for these systems, building the datasets and scoring that measure it, and operating the release gate that prevents a regression from reaching customers.

This is an emerging discipline with no established playbook. We are defining the methodology as we build it, and the role carries a high degree of autonomy and ownership.

๐Ÿ”ฌ What Sets This Role Apart

Most QA roles that mention AI mean using AI to do testing faster: generating test cases from a PRD, healing flaky selectors, exploring an app with an agent. Those are useful and we do them.

This role is the other thing. The system under test is itself an AI, and the hard problem is deciding whether its output is correct when the same question produces a different answer every time and a wrong answer reads as convincingly as a right one. That means building evaluation datasets, defining what correct looks like, scoring against it, and defending a number that decides whether a release ships.

If you have spent time on the second problem, this role is built for you.

๐Ÿ›  Key Responsibilities

Build and maintain golden datasets for our AI products, trace mined from production or generated, and verified against a documented source of truth.

Design layered scoring for AI outputs: deterministic rules where behaviour can be asserted, and model-graded evaluation where it cannot.

Manage evaluation runs end to end: schedule and execute them against release candidates, compare results against prior baselines, and triage failures into product defects, incorrect expectations, and platform issues.

Own and operate the release gate that determines whether a prompt or model change is approved for production.

Identify failure modes ahead of customers, including hallucination, incorrect tool selection, loss of context across conversation turns, and PII exposure.

Investigate production traces to distinguish retrieval failures from generation failures from tool failures.

Convert findings into actionable engineering evidence and into permanent regression coverage.

Requirements

  • ยท๐Ÿ›  Key Responsibilities
  • ยทBuild and maintain golden datasets for our AI products, trace mined from production or generated, and verified against a documented source of truth.
  • ยทDesign layered scoring for AI outputs: deterministic rules where behaviour can be asserted, and model-graded evaluation where it cannot.
  • ยทManage evaluation runs end to end: schedule and execute them against release candidates, compare results against prior baselines, and triage failures into product defects, incorrect expectations, and platform issues.
  • ยทOwn and operate the release gate that determines whether a prompt or model change is approved for production.

Benefits

No benefits package published with this listing. Ask about it at first interview.

How to apply

  1. 1Check the flexibility label above, work from home, matches where you plan to live and work.
  2. 2Tailor your CV to the role at Delta Exchange, mentioning your remote working experience and working hours (APAC).
  3. 3Apply directly on the employer's careers page using the button below. Nomaders never handles your application.

Found 2mo ago. Last checked 17 Nov. Always confirm the details on the original posting, salary and location can change after publication.

Listing sourced from Company boards.

Pay not stated ยท You'll be taken to the employer's careers page.