The Probabilistic Contract: What LLMs Actually Guarantee

An LLM API call returns one sample from a probability distribution over possible responses, not a fixed answer. This lesson covers what that means for a production system: which controls you have over that distribution on current models, how a small error rate becomes a daily count of failures at volume, where silent truncation actually happens, and what the API contract does and doesn't guarantee.

What the API actually returns

When you call an LLM API, the model does not look up an answer. It samples from a probability distribution over the next token, then the next, then the next, until it produces a termination token or hits your max_tokens limit. The output you receive is one draw from that distribution — not the output, a output.

This distinction matters because every engineering instinct you have from deterministic software is wrong here. A function call with the same arguments returns the same value. An LLM API call with the same arguments returns a value drawn from a probability distribution over possible completions. Most of the time the high-probability region of that distribution is narrow enough that outputs look consistent. When it isn't, you have a production reliability problem with no obvious cause.

What you can and can't control

For years the standard answer to "how do I make an LLM consistent?" was temperature=0. Temperature rescales the probability distribution before sampling; at 0, the model always takes its single most likely next token.

That control is disappearing. Current frontier models (Claude Opus 4.7 and later, the Claude 5 family) reject non-default sampling parameters such as temperature and top_p with an error, and many reasoning models from other providers restrict them too. These models think before they answer, and you steer them with different controls:

• Effort — how much reasoning the model does before answering. Lower effort is faster and cheaper; higher effort helps on hard tasks.
• Structured outputs — constrain the shape of the response to a schema (Lesson 2).
• The prompt itself — instructions, examples and the context you provide.

None of these removes the randomness. Two identical requests can still return different outputs, and that was true even when temperature=0 was available: batching and parallel floating-point arithmetic on the provider's side make even "greedy" decoding drift, and provider model updates change the distribution itself. Treat run-to-run variation as a property of the system you're building on, not a setting you forgot to turn off.

The lesson from the temperature=0 era still holds: consistency is not correctness. A model that commits strongly to its most probable answer is more consistently wrong when that answer is wrong. You get reliability from validation and evals (the rest of this course), not from a sampling setting.

The call-volume failure rate problem

Probabilistic systems interact with call volume in a way that catches engineers off-guard. Consider a classification pipeline with a 1% error rate — a number that sounds acceptably small. At 1,000 calls per day, that is 10 wrong classifications every day, every day the pipeline runs. At 10,000 calls per day, it is 100.

The relevant question is not "what is my error rate?" but "how many wrong outputs per day is my application producing, and what happens to each of them?" If wrong outputs are discarded, retried, or reviewed before taking effect, the error rate is manageable. If wrong outputs flow directly into downstream systems — updated database records, sent emails, triggered actions — each wrong output is an incident.

def estimate_daily_failures(calls_per_day: int, error_rate: float) -> dict:
    """
    Estimate daily failure volume from a probabilistic pipeline.
    error_rate should be measured on your eval set, not assumed.
    """
    return {
        "calls_per_day": calls_per_day,
        "error_rate": error_rate,
        "failures_per_day": calls_per_day * error_rate,
        "failures_per_week": calls_per_day * error_rate * 7,
        "failures_per_month": calls_per_day * error_rate * 30,
    }

# 1% error rate, 5,000 calls/day
print(estimate_daily_failures(5_000, 0.01))
# {'calls_per_day': 5000, 'error_rate': 0.01, 'failures_per_day': 50.0,
#  'failures_per_week': 350.0, 'failures_per_month': 1500.0}

The practical implication: your application architecture must account for failure volume, not just error rate. Validation, retry logic, and graceful degradation are not optional polish — they are the mechanisms that translate a probabilistic system into something with acceptable production behaviour.

Silent truncation: where it actually happens

The context window is a bounded working space. A common belief is that the API quietly drops whatever doesn't fit. It doesn't: send more input than the model's context window allows and the API rejects the request with an error. That failure is loud.

The silent truncation happens in your stack, before the request is sent:

• a framework that trims chat history to fit a budget, dropping the earliest turns, which often hold the instructions
• code that slices a document to a character limit "to be safe"
• an embedding model or tokeniser with a max_length, which cuts long passages without complaint
• a retrieval step that keeps the top-k chunks and discards the one that held the answer

There is one more, on the way out: if the response reaches max_tokens, generation stops mid-answer. The call succeeds, and the partial text may still parse. Only stop_reason tells you, and on current models thinking counts toward max_tokens too.

import anthropic

client = anthropic.Anthropic()
MODEL = "claude-opus-5-5"
SYSTEM = "You are a data extraction assistant..."

# Decide what to do with an over-long document *before* sending it,
# instead of letting some layer of your stack truncate it silently.
limit = client.models.retrieve(MODEL).max_input_tokens
n_tokens = client.messages.count_tokens(
    model=MODEL,
    system=SYSTEM,
    messages=[{"role": "user", "content": document}],
).input_tokens
if n_tokens > limit - 8_000:  # leave room for the response
    raise ValueError(f"Document is {n_tokens} tokens; split it or route it for review")

response = client.messages.create(
    model=MODEL,
    max_tokens=8_000,
    system=SYSTEM,
    messages=[{"role": "user", "content": document}],
)

# Check how the response ended before you read it.
if response.stop_reason != "end_turn":
    raise RuntimeError(f"Incomplete response: stop_reason={response.stop_reason}")

Counting tokens first costs a little latency and turns "the model quietly saw half the document" into an explicit decision: split it, summarise it, or send it for review. Checking stop_reason does the same for the output.

The operational contract

The API contract provides:

• A response within the stated latency SLA (usually p95 or p99)
• Responses drawn from the model's probability distribution over completions
• Rate limiting at the stated tier (requests per minute, tokens per minute)
• Eventual model updates that may change output distributions without notice

The API contract does not provide:

• Identical outputs across model versions
• Identical outputs for identical requests, at any setting
• Warnings when your own code or framework truncated the input before sending it
• Alerts when model updates change your application's behaviour

Understanding this contract is not pessimism — it is the baseline required to build reliable systems. The rest of this course is about building the validation, eval, and observability infrastructure that makes a probabilistic API behave reliably enough to run in production.

---

Summary: LLM APIs return samples from a probability distribution, not deterministic lookups. On current models you steer that distribution with effort, structured outputs and the prompt; sampling settings like temperature are often rejected, and no setting makes outputs identical across runs or model updates. Error rates interact with call volume: a 1% error rate at 5,000 calls/day produces 50 wrong outputs daily. Over-long input makes the API return an error; silent truncation happens in your own stack (history trimming, slicing, tokeniser limits, top-k) and at max_tokens, so count tokens before the call and check stop_reason after it. The API contract guarantees SLA-bounded latency and rate-limited access; it does not guarantee identical outputs or change notifications.

Next: Treating prompts as typed function signatures — and why underconstrained prompts are the equivalent of Any return types.