The Probabilistic Contract: What LLMs Actually Guarantee

What the API actually returns

When you call an LLM API, the model does not look up an answer. It samples from a probability distribution over the next token, then the next, then the next, until it produces a termination token or hits your max_tokens limit. The output you receive is one draw from that distribution — not the output, a output.

This distinction matters because every engineering instinct you have from deterministic software is wrong here. A function call with the same arguments returns the same value. An LLM API call with the same arguments returns a value drawn from a probability distribution over possible completions. Most of the time the high-probability region of that distribution is narrow enough that outputs look consistent. When it isn't, you have a production reliability problem with no obvious cause.

Temperature and what it actually controls

Temperature is a scaling parameter applied to the logit distribution before sampling. At temperature=1.0, the distribution is sampled as-is. At temperature=0.0, the model always selects the highest-probability token — argmax sampling.

Two things are commonly misunderstood about temperature=0:

It is reproducible, not deterministic. If the model weights, infrastructure, and parallelism configuration are identical, temperature=0 produces the same output. In practice, none of those are guaranteed constant. Provider model updates (even minor ones) change the weights. Parallel inference across GPUs introduces floating-point non-determinism. The output you get at temperature=0 today may not be the output you get from the same prompt in three months.

Lower temperature does not mean higher quality. It means higher confidence — the model commits more strongly to the most probable completion. If the most probable completion is wrong, lower temperature just makes it more consistently wrong.

For extraction and classification tasks, temperature=0 or low temperature is appropriate — you want consistent outputs against a schema. For generation tasks, higher temperature produces more varied and often more useful outputs. The choice is task-dependent, not a quality dial.

The call-volume failure rate problem

Probabilistic systems interact with call volume in a way that catches engineers off-guard. Consider a classification pipeline with a 3% error rate — a number that sounds acceptably small. At 1,000 calls per day, that is 30 wrong classifications every day, every day the pipeline runs. At 10,000 calls per day, it is 300.

The relevant question is not "what is my error rate?" but "how many wrong outputs per day is my application producing, and what happens to each of them?" If wrong outputs are discarded, retried, or reviewed before taking effect, the error rate is manageable. If wrong outputs flow directly into downstream systems — updated database records, sent emails, triggered actions — each wrong output is an incident.

def estimate_daily_failures(calls_per_day: int, error_rate: float) -> dict:
    """
    Estimate daily failure volume from a probabilistic pipeline.
    error_rate should be measured on your eval set, not assumed.
    """
    return {
        "calls_per_day": calls_per_day,
        "error_rate": error_rate,
        "failures_per_day": calls_per_day * error_rate,
        "failures_per_week": calls_per_day * error_rate * 7,
        "failures_per_month": calls_per_day * error_rate * 30,
    }

# 1% error rate, 5,000 calls/day
print(estimate_daily_failures(5_000, 0.01))
# {'calls_per_day': 5000, 'error_rate': 0.01, 'failures_per_day': 50.0,
#  'failures_per_week': 350.0, 'failures_per_month': 1500.0}

The practical implication: your application architecture must account for failure volume, not just error rate. Validation, retry logic, and graceful degradation are not optional polish — they are the mechanisms that translate a probabilistic system into something with acceptable production behaviour.

Context window overflow: the silent truncation

The context window is not unlimited memory — it is a bounded working space. When your accumulated prompt exceeds the model's context limit, something gets dropped. What gets dropped depends on the client library and the truncation strategy configured (or not configured) in your code.

The default behaviour in most LLM client libraries is to truncate silently. No exception is raised. No warning is logged. The API call succeeds, and the model generates a response — but it generated that response without seeing the context that was truncated. If that context contained the instructions, the schema, or the data the model was supposed to process, the output will be wrong in ways that are difficult to diagnose.

import anthropic

client = anthropic.Anthropic()

# This will silently truncate if your messages exceed the context limit.
# No exception. The response will be based on whatever fits.
response = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=1024,
    system="You are a data extraction assistant...",
    messages=[{"role": "user", "content": very_long_document}],  # may be truncated
)

# You need to check token counts BEFORE sending if context overflow is a risk.
token_count = client.messages.count_tokens(
    model="claude-opus-4-6",
    system="You are a data extraction assistant...",
    messages=[{"role": "user", "content": very_long_document}],
)
print(f"Input tokens: {token_count.input_tokens}")
# Check against model limit before calling

Checking token counts before the call costs a small amount of latency but eliminates the silent truncation failure mode. For pipelines processing variable-length inputs, this check is mandatory.

The operational contract

The API contract provides:

• A response within the stated latency SLA (usually p95 or p99)
• Responses drawn from the model's probability distribution over completions
• Rate limiting at the stated tier (requests per minute, tokens per minute)
• Eventual model updates that may change output distributions without notice

The API contract does not provide:

• Identical outputs across model versions
• Guaranteed identical outputs at temperature=0
• Warnings when context is truncated
• Alerts when model updates change your application's behaviour

Understanding this contract is not pessimism — it is the baseline required to build reliable systems. The rest of this course is about building the validation, eval, and observability infrastructure that makes a probabilistic API behave reliably enough to run in production.

---

Summary: LLM APIs return samples from a probability distribution, not deterministic lookups. Temperature=0 is reproducible under identical conditions, not guaranteed identical across model updates. Error rates interact with call volume: a 1% error rate at 5,000 calls/day produces 50 wrong outputs daily. Context overflow is silent in most client libraries — token counting before the call eliminates the failure mode. The API contract guarantees SLA-bounded latency and rate-limited access; it does not guarantee identical outputs or change notifications.

Next: Treating prompts as typed function signatures — and why underconstrained prompts are the equivalent of Any return types.