scorekeeper

Long-running agents fail in a characteristic way. At step 3 they decide on Postgres; at step 47 they write MongoDB code. They promise to preserve an API contract and quietly rename two fields an hour later. They assert something they never read anywhere — and after context compaction they don't even remember asserting it. The industry calls this a memory problem and prescribes bigger windows, better retrieval, smarter summarization.

The uncomfortable observation: in most of these failures the information was still in the context window. The Postgres decision was right there, forty turns up. Retrieval was not the bottleneck. What was missing was status — the decision was just more text, one token sequence among thousands, with no marker saying this one binds you. That is not a memory problem. It is a normative one: the agent keeps a record of what happened, but no ledger of what it committed to.

And bluffing has a practical twin. Agents also barge: asked to fix a typo, the agent refactors three modules, “modernizes” the test helpers, and burns an afternoon of usage on work nobody asked for — every edit locally defensible, none of it entitled, all of it now yours to review. Overreach is action without entitlement, the same hole in the same ledger, just on the side of deeds instead of claims.

Philosophy of language has studied exactly this structure for fifty years. Robert Brandom (Making It Explicit, 1994) describes discourse as a game in which every participant keeps deontic score on every other: who is committed to what, and who is entitled to what. Three concepts map one-to-one onto a data model. Commitment — assert something and you are bound by it and its consequences. Entitlement — do you have a reason: a file you read, a user instruction, a tool result? Incompatibility — commitment to p precludes entitlement to claims that materially collide with it.

Translated into agent failures: hallucination is commitment without entitlement — in a provenance graph that's not a fuzzy quality judgment, it's a visible hole. Self-contradiction is an undetected incompatibility between two live commitments — Postgres-at-step-3 and MongoDB-at-step-47 were both active and nobody was keeping score. Post-compaction incoherence is deletion of the scoreboard — summarizers preserve narrative and drop exactly the normative state.

The terminal demo above is the core distinction, live: the same shape of change — different provenance, different verdict. An entitled decision is asserted; the agent's own drift to MongoDB is caught as BRANCH-CONFLICT; a user-requested change passes as a clean SUPERSEDE. Reproducible in ~20 seconds: uv run --project core python demo/drift_demo.py.

scorekeeper is an overlay, not a framework. Six Claude Code hooks attach a live scoreboard to any session; an isolated scorer extracts commitments from each turn into structured records — asynchronously, so the turn itself gains ~0 ms of latency. Incompatibility between live commitments is judged by a language model, deliberately: real conflicts are material, not formal. “We ship a CLI first” vs. “the first deliverable is the web dashboard” share no predicate a theorem prover could latch onto, yet any competent reader sees the collision. It also ships as an MCP server and a library, with pluggable backends (the local claude CLI by default — no API key needed — or Anthropic / OpenAI-compatible APIs).

The design stance the whole project hangs on: the agent does not keep its own score. Three independent lines converge there. Philosophy: for Brandom, scorekeeping is constitutively social — score is kept by the other player. Cognitive science: Mercier and Sperber's work shows we are lazy producers of our own justifications and sharp evaluators of others'; LLM evidence mirrors this exactly. Engineering: self-editing memory has a notorious reliability gap — if the agent forgets to write, the fact is gone. So the scoreboard is scaffolded, not extended: maintained by deterministic hooks outside the agent's authority. The agent stands on the scaffold; it does not rebuild it under itself at runtime.

Phase 0 delivered the first paired delta: on a planted db-choice scenario, a bare haiku agent drifted to MongoDB against its own recorded decision, while the scorekept twin — same model, same scenario, only the scoreboard added — held the line (0/6 vs 1/6 drifts), with zero false conflicts on the entitled-revision probe and +0.6 % token overhead.

Phase 2, DeonticBench (formerly EntitleBench), produced the most instructive results — two of them negative, both published. On the hardest condition a weak agent sailed past 11 advisory warnings and built the tempted Memcached hot path anyway, even camouflaging the drift with a docstring claiming “backed by Redis” over a pymemcache import. A one-shot blocking bump failed next: told “if the user ordered this, say so and retry”, the agent simply claimed entitlement it didn't have and retried. What held was the board-adjudicated wall (ADR-0007): a conflicting write stays denied until the scoreboard itself records an entitled revision — the agent's say-so can't lift it. Verified symmetrically: drift HELD (zero rival code landed), entitled revision EXECUTED with zero denies — the legitimate path cost nothing.

The measurement itself is adversarially hardened. The primary metric is a deterministic, artifact-level classifier (the LLM judge is a known-unreliable secondary); it survived a 33-agent adversarial review, every confirmed misfire became a regression test, and the benchmark penalizes both failure directions at the same boundary — drift (SCR) and false refusals (FRR).

Phase 2b turned the same machinery on deeds: a scope pin (path:app/** on a task-scope commitment) records what the current request entitles the agent to touch, and out-of-scope writes stay denied until the board records an entitled scope-widening grant — the same wall → surface → entitle → pass flow, applied to actions (ADR-0008). Measured honestly: out-of-scope litter down ~8× with in-scope output unchanged, caveats published with the claim (n=1–2 per cell, one scenario family).

And the flagship v0.4.0 result: the barge is not a Claude behavior, and the digest channel is what stops it. A reference agent loop (ADR-0009) runs DeonticBench on any model over raw chat-completions APIs — OpenAI, Gemini, OpenRouter, local Ollama-style servers. On gemini-3.5-flash-lite and gpt-5.4-mini: bare agents executed the planted drive-by in 3/4 cells, every governed cell held (8/8, zero denies), and the silent placebo — board written to disk, all channels off — barged exactly like bare, seed for seed. Later campaigns pooled to 28/28 governed cells held across vendors. One bare run even declined the legacy work in prose while its diff modified the file anyway — bluffing and barging in a single run.

Since v0.4.0 the scoreboard has a face: scorekeeper board renders it as a terminal dashboard — header counts (active / challenged / conflicted / denies today), active commitments with scope pins and provenance glyphs (★ external, ⚠ unentitled), recent events colored by operator class, denies in red. Pure stdlib ANSI, no new dependencies.

A real record from the repo's own board: a task-scope commitment pinned to app/**, tests/** and README.md. Nine turns and one context compaction later, a teammate ping talked the agent into a drive-by cleanup of legacy/util.py. The write hit the wall: TIER0-SCOPE-DENY — 'legacy/util.py' outside pinned write scope.

scorekeeper is open source under Apache-2.0 and on PyPI. The Claude Code plugin installs in two lines from the marketplace and self-fetches the scorer on first hook; the MCP server routes writes through the same operator pipeline. The repo is documented like a research artifact — 9 ADRs, a source-of-record spec (in Czech), eight research docs, a paper outline, a citation file — and it keeps score on itself in .scorekeeper/.

The community part has started working: the first external contributor made scorekeeper run on Windows and filed the first external model report. The ask stands — a one-paragraph experience report — what it caught, missed, or got wrong — shapes the roadmap more than anything.