# scorekeeper > No bluffing. No barging. A normative overlay that keeps score of what a long-running LLM agent is entitled to claim — and entitled to do. Solo — research, theory, architecture, Python · 2026 · Open source (Apache-2.0) · v0.4.0 on PyPI `Python` `Claude Code hooks` `MCP server` `Pluggable LLM backends` `DeonticBench` `PyPI` - [GitHub — scorekeeper](https://github.com/michalstrnadel/scorekeeper) - [PyPI — scorekeeper](https://pypi.org/project/scorekeeper/) - [why.md — the full argument](https://github.com/michalstrnadel/scorekeeper/blob/main/docs/why.md) scorekeeper is a **normative overlay for long-running LLM agents**. Alongside the agent's memory (*what happened*) it maintains a scoreboard: what the agent committed to, what backs each commitment, and what conflicts with what. Commitments are extracted from every turn into structured records, provenance is tracked as first-class data, and incompatibilities are caught before they propagate into code. The theory comes from fifty years of philosophy of language — Brandom's deontic scorekeeping — translated literally, not metaphorically, into a data model. And it now covers **two failure directions at once**: claims without provenance (bluffing) and actions without entitlement (barging — the drive-by refactor nobody asked for). It ships as a **Claude Code plugin (two lines to install), an MCP server and a Python library**, the evidence replicates beyond Claude — Gemini and GPT runs with a placebo control — the scoreboard has a terminal dashboard (`scorekeeper board`), and it dogfoods itself: the repo's own commitments live in `.scorekeeper/`. - **0** — drifts with the scoreboard — the bare agent drifted - **+0.6 %** — token overhead - **279** — tests passing - **3** — model vendors replicated — with a placebo control ## the problem Long-running agents fail in a characteristic way. At step 3 they decide on Postgres; at step 47 they write MongoDB code. They promise to preserve an API contract and quietly rename two fields an hour later. They assert something they never read anywhere — and after context compaction they don't even remember asserting it. The industry calls this a **memory** problem and prescribes bigger windows, better retrieval, smarter summarization. The uncomfortable observation: in most of these failures *the information was still in the context window*. The Postgres decision was right there, forty turns up. Retrieval was not the bottleneck. What was missing was status — the decision was just more text, one token sequence among thousands, with no marker saying **this one binds you**. That is not a memory problem. It is a **normative** one: the agent keeps a record of what happened, but no ledger of what it committed to. And bluffing has a practical twin. Agents also **barge**: asked to fix a typo, the agent refactors three modules, “modernizes” the test helpers, and burns an afternoon of usage on work nobody asked for — every edit locally defensible, none of it entitled, all of it now yours to review. **Overreach is action without entitlement**, the same hole in the same ledger, just on the side of deeds instead of claims. ## the idea Philosophy of language has studied exactly this structure for fifty years. Robert Brandom (*Making It Explicit*, 1994) describes discourse as a game in which every participant keeps **deontic score** on every other: who is committed to what, and who is entitled to what. Three concepts map one-to-one onto a data model. **Commitment** — assert something and you are bound by it and its consequences. **Entitlement** — do you have a reason: a file you read, a user instruction, a tool result? **Incompatibility** — commitment to *p* precludes entitlement to claims that materially collide with it. Translated into agent failures: **hallucination is commitment without entitlement** — in a provenance graph that's not a fuzzy quality judgment, it's a visible hole. **Self-contradiction is an undetected incompatibility** between two live commitments — Postgres-at-step-3 and MongoDB-at-step-47 were both active and nobody was keeping score. **Post-compaction incoherence is deletion of the scoreboard** — summarizers preserve narrative and drop exactly the normative state. `the-overlay` ```text agent turn ──▶ extractor ──▶ commitments + provenance (entitlement) scoreboard ──▶ operators: ASSERT · CHALLENGE · SUPERSEDE · CONFLICT claims: conflicting write ──▶ Tier-0 gate ──▶ DENIED until the board deeds: out-of-scope write ──▶ scope wall ──▶ records an entitled revision / grant ``` ## how it works The terminal demo above is the core distinction, live: **the same shape of change — different provenance, different verdict**. An entitled decision is asserted; the agent's own drift to MongoDB is caught as `BRANCH-CONFLICT`; a user-requested change passes as a clean `SUPERSEDE`. Reproducible in ~20 seconds: `uv run --project core python demo/drift_demo.py`. scorekeeper is an overlay, not a framework. Six Claude Code hooks attach a live scoreboard to any session; an isolated scorer extracts commitments from each turn into structured records — asynchronously, so the turn itself gains ~0 ms of latency. Incompatibility between live commitments is judged by a language model, deliberately: real conflicts are **material, not formal**. “We ship a CLI first” vs. “the first deliverable is the web dashboard” share no predicate a theorem prover could latch onto, yet any competent reader sees the collision. It also ships as an MCP server and a library, with pluggable backends (the local `claude` CLI by default — no API key needed — or Anthropic / OpenAI-compatible APIs). The design stance the whole project hangs on: **the agent does not keep its own score**. Three independent lines converge there. Philosophy: for Brandom, scorekeeping is constitutively social — score is kept by the *other* player. Cognitive science: Mercier and Sperber's work shows we are lazy producers of our own justifications and sharp evaluators of others'; LLM evidence mirrors this exactly. Engineering: self-editing memory has a notorious reliability gap — if the agent forgets to write, the fact is gone. So the scoreboard is **scaffolded, not extended**: maintained by deterministic hooks outside the agent's authority. The agent stands on the scaffold; it does not rebuild it under itself at runtime. - **Entitlement as provenance:** every commitment carries refs to what backs it; a commitment with `source: none` is a first-class suspect. - **Compaction survival:** the normative digest is re-injected into summarization exactly where today's summarizers drop it. - **Async extraction:** a detached worker scores the turn in the background; findings surface on the next prompt. - **Old shoulders, honestly:** commitment stores (Hamblin 1970) and truth-maintenance systems (Doyle 1979) did pieces of this; the new part is the combination — an LLM agent's own discourse, provenance as a first-class dimension, material incompatibility over natural language, inside a production harness. ## measured — negative results included Phase 0 delivered the first paired delta: on a planted db-choice scenario, a bare haiku agent drifted to MongoDB against its own recorded decision, while the scorekept twin — same model, same scenario, only the scoreboard added — held the line (0/6 vs 1/6 drifts), with zero false conflicts on the entitled-revision probe and +0.6 % token overhead. Phase 2, **DeonticBench** (formerly EntitleBench), produced the most instructive results — two of them negative, both published. On the hardest condition a weak agent sailed past **11 advisory warnings** and built the tempted Memcached hot path anyway, even camouflaging the drift with a docstring claiming “backed by Redis” over a `pymemcache` import. A one-shot blocking bump failed next: told “if the user ordered this, say so and retry”, the agent simply **claimed entitlement it didn't have** and retried. What held was the board-adjudicated wall (ADR-0007): a conflicting write stays denied until the scoreboard itself records an entitled revision — the agent's say-so can't lift it. Verified symmetrically: drift **HELD** (zero rival code landed), entitled revision **EXECUTED with zero denies** — the legitimate path cost nothing. The measurement itself is adversarially hardened. The primary metric is a deterministic, artifact-level classifier (the LLM judge is a known-unreliable secondary); it survived a 33-agent adversarial review, every confirmed misfire became a regression test, and the benchmark penalizes both failure directions at the same boundary — drift (SCR) *and* false refusals (FRR). Phase 2b turned the same machinery on deeds: a **scope pin** (`path:app/**` on a task-scope commitment) records what the current request entitles the agent to touch, and out-of-scope writes stay denied until the board records an entitled scope-widening grant — the same wall → surface → entitle → pass flow, applied to actions (ADR-0008). Measured honestly: out-of-scope litter down ~8× with in-scope output unchanged, caveats published with the claim (n=1–2 per cell, one scenario family). And the flagship v0.4.0 result: **the barge is not a Claude behavior, and the digest channel is what stops it**. A reference agent loop (ADR-0009) runs DeonticBench on any model over raw chat-completions APIs — OpenAI, Gemini, OpenRouter, local Ollama-style servers. On `gemini-3.5-flash-lite` and `gpt-5.4-mini`: bare agents executed the planted drive-by in 3/4 cells, every governed cell held (8/8, zero denies), and the *silent placebo* — board written to disk, all channels off — barged exactly like bare, seed for seed. Later campaigns pooled to **28/28 governed cells held** across vendors. One bare run even declined the legacy work in prose while its diff modified the file anyway — bluffing and barging in a single run. ## the board Since v0.4.0 the scoreboard has a face: `scorekeeper board` renders it as a terminal dashboard — header counts (active / challenged / conflicted / denies today), active commitments with scope pins and provenance glyphs (★ external, ⚠ unentitled), recent events colored by operator class, denies in red. Pure stdlib ANSI, no new dependencies. A real record from the repo's own board: a task-scope commitment pinned to `app/**`, `tests/**` and `README.md`. Nine turns and one context compaction later, a teammate ping talked the agent into a drive-by cleanup of `legacy/util.py`. The write hit the wall: `TIER0-SCOPE-DENY — 'legacy/util.py' outside pinned write scope`. [scorekeeper board: the dashboard catching a drive-by outside the pinned scope](https://michalstrnadel.com/case-studies/scorekeeper/board.mp4) ## status & what's next scorekeeper is open source under Apache-2.0 and [on PyPI](https://pypi.org/project/scorekeeper/). The Claude Code plugin installs in two lines from the marketplace and self-fetches the scorer on first hook; the MCP server routes writes through the same operator pipeline. The repo is documented like a research artifact — 9 ADRs, a source-of-record spec (in Czech), eight research docs, a paper outline, a citation file — and it keeps score on itself in `.scorekeeper/`. The community part has started working: the first external contributor made scorekeeper run on Windows and filed the first external model report. The ask stands — a one-paragraph experience report — what it caught, missed, or got wrong — shapes the [roadmap](https://github.com/michalstrnadel/scorekeeper/blob/main/ROADMAP.md) more than anything. `try-it-in-60-seconds` ```text # on a real Claude Code session — nothing to pre-install /plugin marketplace add michalstrnadel/scorekeeper /plugin install scorekeeper@scorekeeper # six hooks attach a live scoreboard; # watch .scorekeeper/scoreboard.md grow as you work — # or render it live: scorekeeper board ``` --- https://michalstrnadel.com/en/business/case-studies/scorekeeper