The Frome Hoard survived as evidence because one detectorist stopped digging. An agent output captured as a row in an evaluation table has lost its third witness the same way.
On 11 April 2010, in a field near Frome in Somerset, a metal detectorist named Dave Crisp found the rim of a large pot with coins in it. He did the thing that makes the rest of the story possible: he stopped digging and filled in the hole. Twelve days later a team led by the archaeologist Alan Graham, with Katie Hinds and Anna Booth of the Portable Antiquities Scheme, excavated the pot over three days, removing the coins in layers so that they could tell whether the earliest lay at the bottom and the latest at the top. There were 52,503 of them, about 160 kilograms of Roman bronze and silver struck between AD 253 and 305, in a pot 45 centimetres across. The pot's walls were too thin to have carried that weight, so the archaeologists concluded it had been buried empty and the coins poured in afterwards, then sealed and covered. Sam Moorhead of the British Museum pointed out that it could not have been recovered again without breaking it, and suggested the hoard might be a communal offering to the gods rather than anyone's savings. The Treasure Valuation Committee put it at £320,250 that October.
Now run the counterfactual. Suppose Crisp had spent the weekend pulling coins out by the handful, as people did for centuries and some still do. Every one of the 52,503 coins would still exist. Each would still weigh what it weighs. Each would still carry its emperor's portrait and its legend, including the 766 struck for Carausius, the usurper who ran Britain for seven years. A catalogue of the hoard would be almost identical. What would be gone is the pot buried first, the coins poured in on one occasion, the layering, and with them the argument about what the thing was for. That testimony does not survive collection, and no later study recovers it. It is lost in the field, on the day, by the act of picking the coins up.
Hold that loss in mind. It is the whole essay.
A coin can be read three ways at once. As an economic instrument it has a weight, a fineness and a denomination, and those numbers track debasement, state finance and how much a soldier was paid. As a piece of state communication it has a portrait, a legend and a reverse type, and those carry the emperor's claims about victory, divinity and dynasty to everyone who handled it. As an archaeological artifact it has a find spot, a context and companions, and those say who lost or buried it, when, with what, and sometimes why.
Reading all three together, as one object with three testimonies, sounds like the obvious method of a mature discipline. It is not what the discipline did for most of its history, and it is worth being exact about whose argument it is. In 2011 Fleur Kemmers and Nanouschka Myrberg published "Rethinking numismatics. The archaeology of coins" in Archaeological Dialogues, arguing that coins combine "text, image and existence as material objects," and that reading them that way gives access not only to the "big history" of economies and states but to "small histories," cultural values and "the agency of humans and objects." They also noted that numismatics, central to how nineteenth-century archaeology built its methods, had since drifted into an isolated discipline of its own. In December 2023 Stefan Krmnicek of Tübingen published "Contextual numismatics: a post-processual approach illustrated by application to Roman coins" in Heritage Science, presenting the approach, in the abstract's words, "as the most recent and innovative research direction in the field of numismatics," with "a historical overview of the gradual development of an archaeological-oriented approach to numismatic material." Post-processual is a term of art from the archaeological theory of the 1980s; it means asking what objects meant to the people who used them, rather than only what can be measured about them.
So the three-witness reading is a reform position, argued from the archaeological side, against the field's default. It is not what numismatics has always held, and it is not attributable to the standard textbooks. Christopher Howgego's Ancient History from Coins, published by Routledge in 1995, is the book most people are handed first, and it is a superb demonstration of how much the first two witnesses can yield: its sections run through minting, coinage and imperialism, the political uses of coin types, circulation, and inflation and monetary crisis, from the invention of coinage around 600 BC to AD 400. It is an exemplar of the tradition the reformers are reacting to, not the source of their critique, and an essay that pinned the critique on it would be inventing a citation.
The default was the museum tray. Nineteenth- and early twentieth-century numismatics, as the reform literature describes it, studied coins apart from their archaeological context, by typology, series, iconography and economic function. Joris Aarts, writing in Archaeological Dialogues in 2005, opened with the observation that "until now, the Roman economy has been discussed primarily in economic terms." A coin on a tray can be weighed, photographed, compared with every other specimen of its type across every collection in Europe, and ordered into a catalogue that another scholar can check. That is real knowledge, and enormous amounts of it were produced this way. The great type corpora that every later study depends on are tray work.
There is also an honest defence of the tray that the reformers have to answer. Find context survives for only a small fraction of the ancient coins that exist. Most were dug up, traded, melted, collected and re-collected long before anyone thought to write down where they came from. For the bulk of the surviving record, catalogue work on trayed specimens is the only work possible. Both positions are defensible. What the reform made indefensible is treating the tray as the whole method, and treating a coin's third testimony as an optional extra rather than as evidence that a collector's shovel destroys.
Now lay an agent's output on the tray.
| coin, read three ways | agent output, read three ways |
|---|---|
| economic instrument: weight, fineness, denomination | the deliverable: is the answer right |
| state communication: portrait, legend, type | the behavioural signal: what the agent took the request to mean, what it planned, what it decided to ignore |
| archaeological artifact: find spot, deposition, companions | the system artifact: the context-window state, the tool history, the retrieval results, the prompt the model actually saw against the prompt you sent |
This taxonomy is ours. No study validates it, and it is offered as a way of seeing, not a finding. But every developer who has run an agent in production will recognise the third row, because the third row is where the surprises live. The prompt you wrote is not always the prompt the model received: a template variable rendered as its own name, a system message truncated by a context limit, a retrieval step that returned yesterday's document, a tool that errored and whose error the agent quietly worked around. None of that is visible in the answer. The answer is a coin on a tray.
Most evaluation practice scores the first row. It scores it well, with rubrics and graders and confidence intervals, and it produces comparable numbers the way catalogues produced comparable types. Then two kinds of failure arrive from the rows it did not read.
The first is the right answer for the wrong reason. The deliverable passes. The behavioural signal, had anyone read it, shows the agent inferred something other than what was asked and landed on the correct output by a route that will not survive the next case. Channel one says pass; channel two says the pass was luck; the eval records the pass.
The second is the wrong answer to a question nobody asked. The system artifact shows the model saw a different prompt from the one you sent, and channels one and two are being read as though it had not. The grader marks the answer wrong, the team tunes the prompt they wrote rather than the prompt the model saw, and the fix lands on the wrong object. I am not going to attach a rate to either failure; no published incidence data exists, and inventing a number here would be its own kind of tray work.
Both failures share a property with the Frome counterfactual. They are invisible to output-only scoring by construction, not by carelessness. A grader looking only at the deliverable cannot see them, in the same way a cataloguer looking only at a coin cannot see the pot.
Here is the structural point, and I will flag it as an argument by analogy, since that is what it is.
In numismatics the third witness is the one that collection destroys. A coin lifted from the ground without its find spot recorded has lost its depositional testimony for good. The weight and the portrait travel with the coin into any tray in the world. The context stays in the field, and if nobody wrote it down at the moment of lifting, it is gone. The damage happens at collection time, silently, and the resulting collection looks tidier for it: rows of clean specimens, each with a weight and a type, none with a hole beside it.
An agent output captured as a row in an evaluation table has lost its third witness the same way. The context-window state at the moment of generation, the sequence of tool calls and their raw returns, the rendered prompt: none of these can be reconstructed from the answer, any more than a find spot can be reconstructed from a coin. If the harness did not save them when it captured the output, they are not late; they are gone. And the eval table looks tidy precisely because the loss leaves no mark. Prompt, response, score. Three clean columns, and the one that would explain the failures is the one that was never a column.
Our own earlier piece on numismatics and training data was about a different loss: the provenance claim, the auction-catalogue sentence that says "from an old European collection" when it means that nobody knows. That is about how a field talks when the context is already missing. This essay is about the moment before, when the context is still in the ground and someone is deciding whether to write it down.
The contextual numismatists' working rule can be stated in three clauses. Read all three witnesses. Name which one you are privileging in this study, because some study always privileges one, and the honest move is to say so rather than to pretend the others corroborate it. And refuse to extract any single channel from an object without reconnecting it to the other two, since a weight without a find spot and a portrait without a weight each tell a smaller truth than the coin does.
The rule travels to agents without modification. Read the answer, the inference and the artifact. Say which one this evaluation is scoring; a benchmark that scores deliverables is fine, as long as it says it is a benchmark of deliverables and nobody mistakes it for a study of the agent. And never report a channel-one number without the channel-three capture that would let someone go back and ask why. A score with no trace behind it is a coin with no find spot: it is evidence, and it is severed from the thing that would tell you what it means.
There is one refinement the agent side can add, because it has an advantage the numismatists never had. A coin's context can only be recorded by the person standing in the field, once, by hand. An agent's context can be recorded by the harness, every time, at no marginal cost, if someone decided at build time that it should be. Crisp's decision to fill in the hole was a personal discipline that a scheme spent years cultivating in detectorists; the Portable Antiquities Scheme, run from the British Museum since 1997, has recorded more than 1.4 million objects on that basis. The agent equivalent of that discipline is a logging default, set once. Save the rendered prompt, not the template. Save the tool returns, not the summary of them. Save the context state at generation, or at least a hash of it. Then score whatever you like, because the other two witnesses will still be there when the score turns out to need explaining.
I would like to end by saying that evaluation practice is about to change, and I do not think the reference class supports it.
Aarts was arguing against a purely economic reading of Roman coins in 2005. Kemmers and Myrberg were calling for numismatics to be brought back into archaeology in 2011. A conference in Trieste in April 2022 marked thirty years of coin-find studies as a recognised subfield. And in December 2023 the contextual approach was still being introduced, by one of its leading practitioners, as "the most recent and innovative research direction in the field." That is two decades of a winning argument that has not finished winning, in a field with a fraction of the money and the turnover of ours, against a tray tradition that was tractable, comparable, publishable and produced beautiful books.
If that is the reference class, output-only evaluation is not going to be abandoned because someone wrote an essay, including this one. What moved numismatics, to the extent it has moved, was less the argument than the infrastructure and the habit: a scheme that made recording the find spot normal, a finder who filled in the hole. The portable lesson is the same size. Decide, at build time, that the third witness gets written down. Everything else in this piece follows from whether that one column exists.
Sourcing notes: the Frome Hoard details (dates, counts, weight, the layered excavation, the pot-buried-first inference, the valuation, Moorhead's suggestion) are from the Wikipedia account of the hoard, which cites the British Museum and Portable Antiquities Scheme reports; Krmnicek 2023 is quoted only from its abstract, verified verbatim through the Semantic Scholar record, because the full text sits behind a publisher authentication wall from here, so the paper's body is paraphrased and not quoted; Kemmers and Myrberg 2011 and Aarts 2005 are quoted from their Cambridge Core abstracts; Howgego 1995 is described from the Classical Review notice and the publisher's description of its sections, and, as the research for this piece verified against the full text, contains no "three witnesses" framework; the three-channel taxonomy for agent outputs is this essay's own framing, and no incidence figure is given for either failure mode because none is published.
A score with no trace behind it is a coin with no find spot.
The third witness only exists if something wrote it down at the moment of capture, and the essay's point is that for an agent this is a build-time decision rather than a personal discipline. Chain of Consciousness is that column: a tamper-evident record of the rendered prompt, the tool returns and the decision, written when it happens rather than reconstructed from the answer afterward.
pip install chain-of-consciousness · npm install chain-of-consciousness
Or the whole stack, provenance and ratings and verification together: pip install agent-trust-stack / npm install agent-trust-stack.