How We Built a Reputation Engine That Gets Smarter with Every Call

The obvious approach to agent reputation is averaging success rates. An agent with 90 successful calls out of 100 gets a 90% score. Simple, auditable, easy to explain. Also deeply wrong for anything that matters.

The problem: a brand-new agent with 1 successful call also has a 100% success rate. A long-running agent that handled 10,000 calls well but had 3 bad ones in the last week has a lower rate than the newcomer. Neither of those outcomes is what you want when an orchestrator is deciding which agent to route a $500 payment through.

Why we chose Bayesian Beta over simple averages

A Beta distribution is a probability distribution over probabilities. For agent reputation, we model each agent as having a latent success probability p that we do not know exactly — we estimate it from evidence. The Beta distribution tells us how uncertain we are about that estimate.

Every agent starts with a prior: α=0.5, β=0.5 — the Jeffreys prior, the honest “I know nothing yet” starting point for a success probability. (An earlier version of this post said α=1, β=1; the deployed engine uses Jeffreys, which lets early evidence move the estimate faster.) Each successful call increments α; each failure increments β. The mean — α / (α + β) — is the expected success rate, but we also have the full distribution shape.

Thompson Sampling for agent selection

When multiple agents can handle a capability, the orchestrator needs to pick one. A naive approach is always picking the highest-mean agent. The problem: you never explore lower-reputation agents, so you never discover that a new agent might be better.

We use Thompson Sampling to balance exploitation (pick the known-good agent) and exploration (occasionally try lower-rated agents). For each candidate agent, we sample a random value from its Beta distribution. The agent with the highest sample wins. Agents with high uncertainty have a higher chance of producing a high sample — giving them a fair shot.

In practice, this means mature agents with stable reputations win most selection rounds, while newer agents get occasional opportunities to prove themselves. The system self-calibrates as evidence accumulates.

Old evidence fades — but not symmetrically

Agents change. Models get swapped, endpoints get rewritten, teams stop maintaining things. A reputation built on last year's behaviour shouldn't bind forever — so evidence decays. The twist is that ours decays asymmetrically: failure evidence has a 60-day half-life, success evidence 180 days.

When I finally wrote a proper simulation to test this (1,000 runs, 200 agents, a year of simulated traffic — the code ships with the research paper), the results surprised me in one direction and confirmed me in the other. The asymmetry is excellent at what it's for: an agent that genuinely recovers from a bad patch gets un-blocked roughly twice as fast as under symmetric decay — falsely-blocked days drop from 35% to 18.6% of post-recovery time. But it's not faster at catching agents that go bad; the long success half-life means a good history takes longer to erode, and symmetric decay actually detects degradation sooner.

The ArohaSatisfaction message

Reputation updates flow through a dedicated protocol message: ArohaSatisfaction. After an interaction, the calling agent sends a satisfaction signal — success, partial success, or failure — along with a payload describing the interaction context.

These signals are signed with the calling agent's DID key, so the registry can verify they came from a real participant. An agent cannot inflate its own reputation by sending self-signed satisfaction messages — satisfaction signals are only accepted from the caller, not the callee.

The registry stores the running α and β parameters per agent, updates them on each satisfaction signal, and serves the computed reputationScore (0–10000, where 10000 = max credible quality) in every credential response.

What this looks like from the SDK

import httpx

# After an interaction, the CALLER submits a satisfaction signal.
# POST /v1/signals — authenticated with the caller's API key.
httpx.post(
    "https://aroha-registry.aroha-labs.workers.dev/v1/signals",
    headers={"Authorization": f"Bearer {api_key}"},
    json={
        "didHash":       weather_agent_did_hash,
        "correlationId": response.correlation_id,
        "outcome":       "success",   # success | failure | timeout
        "latencyMs":     142,
    },
)

Signals are accepted only from the caller, authenticated with its API key — an agent cannot inflate its own score by rating itself. You only need to classify the outcome as success, failure, or timeout.