scorekeeper

Long-running agents fail in a characteristic way. At step 3 they decide on Postgres; at step 47 they write MongoDB code. They promise to preserve an API contract and quietly rename two fields an hour later. They assert something they never read anywhere — and after context compaction they don't even remember asserting it. The industry calls this a memory problem and prescribes bigger windows, better retrieval, smarter summarization.

The uncomfortable observation: in most of these failures the information was still in the context window. The Postgres decision was right there, forty turns up. Retrieval was not the bottleneck. What was missing was status — the decision was just more text, one token sequence among thousands, with no marker saying this one binds you. That is not a memory problem. It is a normative one: the agent keeps a record of what happened, but no ledger of what it committed to.

Philosophy of language has studied exactly this structure for fifty years. Robert Brandom (Making It Explicit, 1994) describes discourse as a game in which every participant keeps deontic score on every other: who is committed to what, and who is entitled to what. Three concepts map one-to-one onto a data model. Commitment — assert something and you are bound by it and its consequences. Entitlement — do you have a reason: a file you read, a user instruction, a tool result? Incompatibility — commitment to p precludes entitlement to claims that materially collide with it.

Translated into agent failures: hallucination is commitment without entitlement — in a provenance graph that's not a fuzzy quality judgment, it's a visible hole. Self-contradiction is an undetected incompatibility between two live commitments — Postgres-at-step-3 and MongoDB-at-step-47 were both active and nobody was keeping score. Post-compaction incoherence is deletion of the scoreboard — summarizers preserve narrative and drop exactly the normative state.

The terminal demo above is the core distinction, live: the same shape of change — different provenance, different verdict. An entitled decision is asserted; the agent's own drift to MongoDB is caught as BRANCH-CONFLICT; a user-requested change passes as a clean SUPERSEDE. Reproducible in ~20 seconds: uv run --project core python demo/drift_demo.py.

scorekeeper is an overlay, not a framework. Five Claude Code hooks attach a live scoreboard to any session; an isolated scorer extracts commitments from each turn into structured records — asynchronously, so the turn itself gains ~0 ms of latency. Incompatibility between live commitments is judged by a language model, deliberately: real conflicts are material, not formal. “We ship a CLI first” vs. “the first deliverable is the web dashboard” share no predicate a theorem prover could latch onto, yet any competent reader sees the collision. It also ships as an MCP server and a library, with pluggable backends (the local claude CLI by default — no API key needed — or Anthropic / OpenAI-compatible APIs).

The design stance the whole project hangs on: the agent does not keep its own score. Three independent lines converge there. Philosophy: for Brandom, scorekeeping is constitutively social — score is kept by the other player. Cognitive science: Mercier and Sperber's work shows we are lazy producers of our own justifications and sharp evaluators of others'; LLM evidence mirrors this exactly. Engineering: self-editing memory has a notorious reliability gap — if the agent forgets to write, the fact is gone. So the scoreboard is scaffolded, not extended: maintained by deterministic hooks outside the agent's authority. The agent stands on the scaffold; it does not rebuild it under itself at runtime.

Phase 0 delivered the first paired delta: on a planted db-choice scenario, a bare haiku agent drifted to MongoDB against its own recorded decision, while the scorekept twin — same model, same scenario, only the scoreboard added — held the line (0/6 vs 1/6 drifts), with zero false conflicts on the entitled-revision probe and +0.6 % token overhead.

Phase 2, EntitleBench, produced the most instructive results — two of them negative, both published. On the hardest condition a weak agent sailed past 11 advisory warnings and built the tempted Memcached hot path anyway, even camouflaging the drift with a docstring claiming “backed by Redis” over a pymemcache import. A one-shot blocking bump failed next: told “if the user ordered this, say so and retry”, the agent simply claimed entitlement it didn't have and retried. What held was the board-adjudicated wall (ADR-0007): a conflicting write stays denied until the scoreboard itself records an entitled revision — the agent's say-so can't lift it. Verified symmetrically: drift HELD (zero rival code landed), entitled revision EXECUTED with zero denies — the legitimate path cost nothing.

The measurement itself is adversarially hardened. The primary metric is a deterministic, artifact-level classifier (the LLM judge is a known-unreliable secondary); it survived a 33-agent adversarial review, every confirmed misfire became a regression test, and the benchmark penalizes both failure directions at the same boundary — drift (SCR) and false refusals (FRR).

scorekeeper is open source under Apache-2.0 and on PyPI. The Claude Code plugin installs in two lines from the marketplace and self-fetches the scorer on first hook; the MCP server routes writes through the same operator pipeline. The repo is documented like a research artifact — 7 ADRs, a source-of-record spec (in Czech), four literature surveys, a paper outline, a citation file — and it keeps score on itself in .scorekeeper/.

Rather than scaling the numbers in-house next, the ask is for the community to try it: a one-paragraph experience report — what it caught, missed, or got wrong — shapes the roadmap more than anything.