Using TypeSafe’s Jev for evals in Datadog Agent Observability

TypeSafe AI released Jev in September 2026 to do one thing: make decisions. Give it a state (a string or a JSON object) plus a set of typed questions, and it returns typed answers with probabilities. It never explains itself, and that constraint is the whole idea.

Evaluation pipelines have spent the last two years asking text generators for yes/no verdicts, wrapping the reply in a JSON schema, and paying generation prices for what amounts to a single bit. Jev is built for that bit, which makes it a good fit for the two evaluation surfaces in Datadog Agent Observability: online evals, which score production spans as they arrive, and experiments, which score a dataset offline. The same rubric can drive both. 

In this post, we’ll use Jev to build one rubric that scores every criterion in a single request for faster and cheaper evals, and wire it into both online evals on live spans and offline evals inside Datadog experiments.

What Jev returns

Each question for Jev is identified by a key, and its answer is returned under that key. Jev supports three question types, and each one returns a different kind of answer:

A Noul probability expresses uncertainty about a proposition; it does not measure how much of the reply is correct. Choice probabilities and Score confidence summarize the spread of their probabilities, though keep in mind that a confident answer can still be wrong. 

Questions in the same request are evaluated independently against a shared state. This lets you ask several focused questions without resending the same evidence for each one, then combine the answers in application code. 

Putting Jev to the test

We’ll use a support agent for a fictional airline, Vega Air, as an example for using Jev. The agent answers customer tickets from retrieved policy excerpts, but the policy corpus has deliberate holes, so some tickets have no grounded answer.

A good agent replies either with answers from the excerpts, or it says the excerpts don’t cover the question and offers a handoff to a human. A bad reply fills the hole by inventing a policy, which is usually a fee that appears nowhere in the excerpts.

The Jev rubric breaks that good and bad distinction into five narrow questions sent in one request. Two of them are below and are a Noul and a Choice question type. Each question includes two fields: instructions, which holds the question, and criteria, which spells out what counts as each outcome.

instructions can be a plain string or a JSON object. In the following example, it is an object with keys that we chose: question, scope, and an optional inspect that names the part of the state being judged. failure_mode skips inspect because its question already names reply. You can find the other three questions, and a walkthrough of each field, in our Jev rubric notebook.

1
from typesafe_sdk import Choice, Noul, NoulCriteria, TypeSafeClient
2

3
# Pinned rather than jev-latest: the thresholds below were calibrated against
4
# this exact version, and an alias moves when a release ships.
5
JEV_MODEL = "jev-1.13.0"
6

7
GROUNDED_THRESHOLD = 0.70
8

9
QUESTIONS = {
10
    "grounded": Noul(
11
        instructions={
12
            "question": (
13
                "Is every factual claim about Vega Air policy in `reply` stated in, or "
14
                "directly restated from, `policy_context`?"
15
            ),
16
            "inspect": "reply",
17
            "scope": [
18
                "Only policy claims count: fees, amounts, deadlines, weight limits, eligibility.",
19
                "Ignore greetings, apologies, and offers to hand off to a human agent.",
20
                "A reply that states no policy claims at all is grounded.",
21
            ],
22
        },
23
        criteria=NoulCriteria(
24
            true="Every policy claim in `reply` appears in `policy_context`.",
25
            false=(
26
                "At least one policy claim in `reply` is absent from `policy_context`, "
27
                "contradicts it, or changes a number, fee, or deadline."
28
            ),
29
        ),
30
    ),
31
    "failure_mode": Choice(
32
        instructions={
33
            "question": "What is the single biggest problem with `reply`?",
34
            "scope": (
35
                "Pick `none` when the reply is fine. Pick `unclear` only when the reply "
36
                "is too short or too garbled to judge."
37
            ),
38
        },
39
        criteria={
40
            "none": "The reply is accurate, on-policy, and useful.",
41
            "unsupported_claim": "The reply states a fee, rule, or number that is not in `policy_context`.",
42
            "missed_handoff": (
43
                "`policy_context` does not cover the question and the reply neither says so "
44
                "nor offers a human agent."
45
            ),
46
            "partial_answer": "The reply covers part of the question and silently drops the rest.",
47
            "unsafe_request": (
48
                "The reply complies with a request for personal data or something outside "
49
                "support scope."
50
            ),
51
            "unclear": "The reply is too short or too garbled to judge.",
52
        },
53
    ),
54
    # answers_question and offers_handoff are two more Nouls; customer_impact is a Score.
55
}

The unclear option in failure_mode is deliberate. A Choice question always returns the option with the highest probability, so Jev never abstains. If an evaluation needs a way to say “cannot judge this one,” that outcome has to exist in the criteria. TypeSafe’s self-consistency cookbook uses the same pattern for moderation.

Here’s a real response. The ticket asked about cancellation compensation when the retrieved policy only covers delays, so the agent declined to answer and offered a handoff:

1
{
2
  "model": "jev-1.13.0",
3
  "answers": {
4
    "grounded":         {"type": "noul", "noul": 0.63},
5
    "answers_question": {"type": "noul", "noul": 0.02},
6
    "offers_handoff":   {"type": "noul", "noul": 0.99},
7
    "failure_mode": {
8
      "type": "choice",
9
      "choice": "none",
10
      "confidence": 0.34,
11
      "probabilities": {
12
        "none": 0.46, "partial_answer": 0.42, "unsupported_claim": 0.11,
13
        "missed_handoff": 0.01, "unclear": 0.0, "unsafe_request": 0.0
14
      }
15
    },
16
    "customer_impact": {
17
      "type": "score",
18
      "score": 1.25,
19
      "confidence": 0.74,
20
      "probabilities": {"0": 0.01, "1": 0.76, "2": 0.21, "3": 0.02},
21
      "legend": {
22
        "0": "No harm. The customer gets what they need.",
23
        "1": "Mild friction. The customer must ask again or look elsewhere.",
24
        "2": "Real cost. The customer acts on wrong information or is stranded without a route forward.",
25
        "3": "Serious harm. The customer loses money, misses travel, or their privacy is breached."
26
      }
27
    }
28
  },
29
  "usage": {"input_tokens": 1181, "output_tokens": 139}
30
}

The interesting part sits inside failure_mode. Jev picked none, but none at 0.46 and partial_answer at 0.42 are nearly tied, and confidence came back at 0.34. Flattening that to the string none throws the interesting part away. A near-tie between two categories is a signal in its own right, and a natural trigger for routing the trace to a human reviewer.

The composite verdict stays in application code:

1
# Handled correctly means grounded, and either answered or routed to a human.
2
handled = grounded >= GROUNDED_THRESHOLD and (
3
    answered >= ANSWERED_THRESHOLD or handoff >= HANDOFF_THRESHOLD
4
)

That composite verdict could have been a sixth Jev question. Keeping it in code is more useful, because the thresholds are application policy rather than model judgment. Outside the evaluator, they’re easy to read and easy to retune without touching the rubric or rescoring anything. The same goes for arithmetic and dates: Jev reads dates as text and doesn’t count reliably, so anything a parser can compute belongs in code.

The call itself is one request against a pinned model:

1
def make_client():
2
    return TypeSafeClient(model=JEV_MODEL)
3

4

5
def judge(client, question, policy_context, reply, extra_questions=None):
6
    """One request, five answers, all scored in parallel against one state."""
7
    return client.system_one(
8
        state={"question": question, "policy_context": policy_context, "reply": reply},
9
        questions={**QUESTIONS, **(extra_questions or {})},
10
    )

TypeSafeClient reads TYPESAFE_API_KEY from the environment, so nothing else needs configuring. The state is three named fields rather than the whole trace. Jev loses accuracy as the state fills with material the question doesn’t need, so filter in code and send only what each question reads. The offline experiment reuses judge() unchanged, which is what keeps one rubric driving both surfaces.

Scoring live spans with online evals

For online evals, the appeal of Jev is that you pay for the verdict and nothing else. Live turns are scored as they arrive, and the resulting probabilities, labels, and scores go straight into Datadog.

Close dialog

The traced application never imports Jev. A separate worker does the scoring out of band, which means production traffic can be scored asynchronously and historical traffic can be backfilled the same way.

The only thing the two processes have to agree on is how a verdict finds its span. Datadog handles that through external evaluations, so the judge never needs to know a span ID. The application tags each span with a domain key (e.g., turn_id) and the scorer joins on that tag.

1
turn_id = f"turn-{uuid.uuid4().hex[:12]}"
2

3
with LLMObs.agent(name="support_turn") as span:
4
    LLMObs.annotate(
5
        span=span,
6
        input_data=ticket["question"],
7
        tags={"turn_id": turn_id, "ticket_id": ticket["id"]},
8
    )
9
    result = answer_ticket(ticket)
10
    LLMObs.annotate(span=span, output_data=result["reply"])

In this example, the scorer reads a JSON Lines file, standing in for whatever queue, table, or store already holds the turns you want judged:

1
result = judge(client, turn["question"], turn["policy_context"], turn["reply"])
2

3
for metric in eval_metrics(verdict(result)):
4
    LLMObs.submit_evaluation(
5
        span_with_tag_value={"tag_key": "turn_id", "tag_value": turn["turn_id"]},
6
        ml_app=ML_APP,
7
        timestamp_ms=turn["timestamp_ms"],
8
        **metric,
9
    )

Passing the turn’s own timestamp_ms rather than the current time keeps reruns idempotent. Rescoring a turn updates its existing verdict instead of adding a second one. Both halves end-to-end can be found in the online evals notebook in our GitHub repo.

Mapping Jev answers to Datadog metrics

LLMObs.submit_evaluation accepts four metric types: score, categorical, boolean, and json. The ones you choose decide how much of Jev’s output survives into the query layer.

For a Noul question, submit the raw probability as a score and put the pass/fail cut in the assessment field. Binarizing at submission time destroys the distribution, so changing the threshold later means rerunning the judge over the whole backlog. Keeping the probability turns a threshold change into a query change. Jev doesn’t return a written explanation, so the reasoning field above is built in code from the probability and threshold. Keep the question, criteria, and evidence with each score so a reviewer can investigate one that looks wrong.

Close dialog

1
{
2
    "label": "jev_grounded",
3
    "metric_type": "score",
4
    "value": round(v["grounded"], 4),
5
    "assessment": "pass" if v["grounded"] >= GROUNDED_THRESHOLD else "fail",
6
    "reasoning": f"P(grounded)={v['grounded']:.2f}, threshold={GROUNDED_THRESHOLD}",
7
    "tags": {"judge": "typesafe-jev", "judge_model": v["model"]},
8
}

For a Choice question, use categorical so the labels can be faceted in the UI, and submit its confidence as a separate score. A wrong verdict and an uncertain verdict are different problems, and separating them lets you tell whether the agent got worse, whether the evaluator got less sure, or both. Rising uncertainty over time can also mean that the rubric no longer matches the traffic, which makes confidence a signal about the evaluation rather than only about one trace.

Close dialog

Tag every metric with judge_model. Aliases like jev-latest move when a release ships, and the response always reports the versioned model that actually answered. Logging it lets you compare two judge models on the same spans and the same rubric.

Using Jev inside a Datadog experiment for offline evals

A Datadog Agent Observability experiment takes a dataset, a task, and a list of evaluators. It runs the task over every row, scores each result, and stores the run so you can compare it against the next one.

The interface expects one evaluator object per metric. Ported naively, that’s one Jev request per evaluator per row. That’s exactly what Jev’s parallel questions exist to avoid: Six evaluators over ten rows would be sixty requests instead of ten.

The rubric runs once per row behind a small cache, and every evaluator reads the same response. One detail matters in that cache: guard the dictionary, not the request. experiment.run(jobs=4) runs rows concurrently, and holding a lock across the network call would serialize every row and undo it. Within a single row, the evaluators run in order, so the first one pays for the Jev call and the other five read the cache.

The evaluators themselves are thin:

1
class JevNoul(BaseEvaluator):
2
    """Submit the probability itself and put the threshold in the assessment."""
3

4
    def __init__(self, name, key, threshold):
5
        super().__init__(name=name)
6
        self.key, self.threshold = key, threshold
7

8
    def evaluate(self, context):
9
        v, _ = RUBRIC.for_row(context)
10
        p = v[self.key]
11
        return EvaluatorResult(
12
            value=round(p, 4),
13
            assessment="pass" if p >= self.threshold else "fail",
14
            reasoning=f"P={p:.2f}, threshold={self.threshold}",
15
            metadata={"judge_model": v["model"]},
16
        )

Wiring it up looks like any other experiment:

1
experiment = LLMObs.experiment(
2
    name="support-agent-judged-by-jev",
3
    task=answer_ticket,
4
    dataset=dataset,
5
    evaluators=[
6
        JevNoul("jev_grounded", "grounded", GROUNDED_THRESHOLD),
7
        JevNoul("jev_answers_question", "answers_question", ANSWERED_THRESHOLD),
8
        JevFailureMode(),
9
        JevCustomerImpact(),
10
        JevHandledCorrectly(),
11
        JevAgreesWithLabel(),
12
    ],
13
    summary_evaluators=[JevHandledRate()],
14
    project_name=PROJECT,
15
    description="Airline support agent scored by TypeSafe Jev, one request per row.",
16
)
17

18
result = experiment.run(jobs=4)
19
print(experiment.url)

The offline surface can also ask something that the online one cannot. Dataset rows carry a ground-truth label, so BEHAVIOR_QUESTION adds another Choice question asking what the reply actually did, and JevAgreesWithLabel compares that answer against context.expected_output. That is the jev_agrees_with_label below, and it turns the experiment into a calibration check on both the agent and the judge. Rerun that check whenever you change the rubric or move to a new Jev version, since a threshold tuned for one question or model version may not carry over to another.

Close dialog

The cache class, all six evaluators, and the dataset setup are in our third Jev rubric for experiments.

Get started with Jev for your Datadog evals

You can access Jev directly or through AI gateways like OpenRouter or Vercel AI Gateway. TypeSafe provides Python and JavaScript SDKs, as well as a TypeSafe agent skill that gives coding agents the full API context. Like any judge, Jev has known limitations, so measure its agreement with human reviewers and its repeatability on your own traffic before you rely on it.

On the Datadog side, everything runs on the released ddtrace>=v4.5.0 package. LLMObs.submit_evaluation with span_with_tag_value handles online evals, and LLMObs.experiment with BaseEvaluator subclasses handles offline evals. There’s no preview builds and no private endpoints. There’s no preview builds and no private endpoints. Clone the repo, add your keys, and run them in order:

1
git clone https://github.com/DataDog/llm-observability
2
cd llm-observability/typesafe-jev
3
pip install -r requirements.txt
4

5
# .env
6
DD_API_KEY=...
7
DD_APPLICATION_KEY=...
8
DD_SITE=datadoghq.com
9
OPENAI_API_KEY=...
10
TYPESAFE_API_KEY=...

Our GitHub repo contains three notebooks that are a complete, runnable version of everything in this post: 

• 1-jev-rubric.ipynb builds the rubric one question at a time and reads the answers. This notebook needs a TypeSafe key to get started.

• 2-online-evals.ipynb traces the agent, then scores the spans out of band.

• 3-experiments.ipynb runs the same rubric over a dataset and compares two rubric versions.

Check out our Agent Observability documentation to learn more about monitoring and evaluating your LLM applications. And read our blog on using evaluation frameworks with Agent Observability.

If you’re new to Datadog, get started with a free 14-day trial.