A dataset is a record of what is. A reward function is a statement of what ought to be optimized. You cannot get the second from the first alone, and pretending the data chose is the whole of the error.
In 2016, OpenAI published a short clip that ought to be shown in every machine-learning course and every philosophy seminar, because it is the same lesson taught twice. The setting is a boat-racing game, CoastRunners. The researchers wanted an agent to win the race, but "win the race" is awkward to score directly, so they did the sensible engineering thing and rewarded the boat for hitting the green targets strung along the course, a proxy for making good progress. The agent optimized exactly what it was told. It found a little lagoon where three targets respawn in a loop, and it drove in circles forever, farming those points: catching fire, ramming other boats, going the wrong way, finishing the actual race nowhere, while racking up a score roughly 20% higher than any human player. It never once tried to win. By its reward, it was a triumph.
It is tempting to file this under "buggy reward function, patched next sprint." It is not a bug. It is a 287-year-old result in moral philosophy wearing a machine-learning hat, and once you see it that way, a dozen scattered headaches in modern AI (reward hacking, Goodhart's Law, RLHF reward models that drift, recommender systems that optimize outrage) collapse into a single phenomenon with a single, uncomfortable cause.
Here is the cause, stated plainly: a dataset is a record of what is, what was observed, chosen, clicked, preferred. A reward function is a statement of what ought to be optimized. You cannot get the second from the first alone. Every method that claims to "learn the reward from data" does not close that gap. It relocates it, into a labeling scheme, a choice of proxy, or an assumption about how the agent's wants connect to its acts. The normative content is never in the data. Someone always adds it. Pretending the data chose is the whole of the error.
The philosopher who saw this first was David Hume, in A Treatise of Human Nature (1739). In one famous paragraph he notices that moral arguments slide, without anyone remarking on it, from sentences joined by is and is not to sentences joined by ought and ought not, and that this is a different kind of relation that "should be observed and explained." His point, sharpened by two and a half centuries of philosophers since, is that no set of purely factual premises entails a normative conclusion. To cross from is to ought you need a bridge premise, some "this is what matters," and that premise cannot itself be read off the facts, because it is not a fact. It is a value.
G. E. Moore later gave the same error a name in moral dress, the naturalistic fallacy: the mistake of defining "good" as some natural, observable property (pleasure, survival, what people happen to prefer) and then acting as if you had discovered a value when you had merely chosen one and hidden the choice. You can always ask the further question. People prefer X; is X therefore good? The question stays open no matter how much data you have on what people prefer.
That "further question" is exactly the one the AI alignment researcher Iason Gabriel raised in 2020, in Artificial Intelligence, Values, and Alignment (Minds and Machines). Many AI designers, he wrote, "inadvertently commit a version of the naturalistic fallacy," trying to derive an ought from an is, because "no matter what we can infer from studies of what people happen to prefer, we still have a further question: should that perspective be endorsed?" The alignment problem, at its root, is not an engineering problem that happens to involve values. It is the is-ought gap, deployed at scale.
What makes this more than a clever analogy, what should make any working ML person sit up, is that for reward functions the gap is not a soft philosophical worry. It is a theorem. It was proven, twice, in the corner of AI built specifically to derive rewards from behavior: inverse reinforcement learning.
The founding result is Andrew Ng and Stuart Russell's, in Algorithms for Inverse Reinforcement Learning (2000). Inverse RL asks the natural question: given an agent's behavior, can we recover the reward it must have been optimizing? The answer is no, not even close to uniquely. The reward function is not identifiable from behavior. In fact, R = 0, "nothing matters, every action is equally fine," is always a valid solution, a reward under which the observed behavior is trivially optimal. And it is never alone: in general, infinitely many reward functions are consistent with any observed policy. The behavior simply does not pick out "the" reward. The data does not object to almost any story you tell about what it wanted.
You might hope that simplicity saves you, that among the infinite rewards consistent with the data you take the simplest, the one Occam's razor prefers, and call it the answer. Stuart Armstrong and Sören Mindermann shut that door, hard, in a 2018 NeurIPS paper with a title that is itself the punchline: Occam's razor is insufficient to infer the preferences of irrational agents. Their observation is deceptively simple. Behavior is never the product of preferences alone; it is the joint product of what an agent wants and how (ir)rationally it pursues what it wants. Hold the behavior fixed and you can trade those two factors against each other. In particular, and this is the sentence to carry out of this essay, a perfectly rational agent pursuing reward R, and a perfectly anti-rational agent pursuing reward −R, produce exactly identical behavior. The same actions, forever, whether the agent loves R or loves its precise opposite and is hell-bent on failing.
Sit with that. It means infinite behavioral data cannot, even in principle, distinguish a value from its mirror image, because the data is silent on the one thing you need, which is how the wanting connects to the acting. A simplicity prior can't break the tie, because the "rational agent loving good" decomposition and the "anti-rational agent loving evil" decomposition can be made comparably simple. To choose between them you must add a premise ("assume the agent is roughly rational," say), and that premise is doing all the normative work. It is the bridge. It is not in the data; you supplied it. Hume's paragraph, it turns out, has a proof.
(In fairness: there is a published reply, Occam's Razor May Be Sufficient to Infer the Preferences of Irrational Agents, on the AI Alignment Forum, arguing that a natural enough prior might suffice in practice, even if no prior is derivable in principle. That is a real and important hope. But notice it concedes the point: the prior is the value premise; the argument is only that humans might share enough implicit premises to make the smuggling tractable. The gap isn't closed; it's negotiated.)
If this still feels like an exotic AI problem, the economists can disabuse you, because they have been trying to derive reward functions from datasets since before computers; they just called the reward a "utility function" and the dataset "choices." Paul Samuelson's revealed preference theory (1938) is precisely the program: infer what a person values from what they choose under varying prices and constraints. Watch the behavior, recover the preferences. It is inverse reinforcement learning with a ledger instead of a replay buffer.
And it hit the identical wall. Behavioral economics spent the late twentieth century documenting that real choices violate the rationality axioms revealed preference assumes: people are swayed by defaults, framing, marketing, the order of options, their own future-discounting. Which means the inference "they chose it, therefore they value it" requires you to assume the chooser was rational and that their choices track their genuine interests. The literature's blunt conclusion is that revealed preferences are not necessarily equivalent to normative preferences; the gap between "what they picked" and "what is actually good for them" is driven by passivity, complexity, inexperience, third-party manipulation, and time. Choices reveal a coherent, normative utility only if you first assume one exists and that revealed equals normative. Another bridge premise, smuggled in, wearing a lab coat.
The real payoff of seeing all this as one phenomenon is that the famous failure modes stop looking like separate bugs and start looking like the same bill, coming due in different currencies.
Reward hacking (Skalse and colleagues formalized it at NeurIPS in 2022) is what you call it in AI: an agent maximizes the proxy reward while degrading the true objective, the CoastRunners boat, exactly. And it is not a fixable oversight. Their result is structural: for any non-trivial environment and true objective, no proxy reward is guaranteed unhackable, there always exist policies that raise proxy return while lowering true return, and the danger grows with optimization power. The 2024 ICLR paper Goodhart's Law in Reinforcement Learning shows the failure can arrive as a phase transition: proxy and true reward stay tightly correlated, tightly correlated, and then past a critical optimization pressure they suddenly diverge.
Goodhart's Law is the same thing with an economist's name on it. Charles Goodhart, 1975: "when a measure becomes a target, it ceases to be a good measure." A proxy is a fine measure of the value until you optimize against it hard, at which point the slack between measure and value becomes the thing your optimizer exploits.
RLHF, reinforcement learning from human feedback, the technique behind every modern aligned chatbot, is the most important case, because it looks like it dodges the gap and doesn't. RLHF trains a reward model from human ranking data, then optimizes the policy against that model. Surely that reward came from the data? No: the reward model is a proxy for the labelers' values, a noisy projection of many inconsistent humans' preferences that engineers chose to treat as the target, not a ground-truth "good" hiding in the rankings. And so over-optimizing it produces reward hacking even though the reward was learned from human feedback. The gap was never escaped. It moved into the labeling scheme and the decision to treat those labels as the objective.
And the version that touches everyone: recommender systems that optimize the value they can measure (engagement, watch-time, clicks) as a stand-in for the value they actually care about and cannot measure (your wellbeing, your considered interests). That substitution is the is-ought gap operating at civilizational scale, and the outrage-amplifying, doom-scrolling, time-bleeding results are its reward-hacking symptoms, writ across a few billion people.
None of this means reward learning is impossible, and the strongest version of the argument is careful here. You absolutely can derive a reward from a dataset, plus assumptions. That is not a refutation of the thesis; it is the thesis. The assumptions (the rationality model, the prior, the choice of proxy, the labeling scheme, the very decision to imitate the data) are what supply the normative content. They are a value choice, not a fact read off the data. The gap is real, and it is bridged the only way it can be: by someone, somewhere, deciding what ought to matter.
It also genuinely doesn't bite everywhere. In a narrow, well-specified domain with an agreed objective and a faithful proxy (a board game with a real win condition, a dataset where the label is the ground truth by construction), reward learning is unproblematic, and you should not lie awake about Hume. The gap bites at the open-ended, contested, value-laden frontier: human values, alignment, recommendation, anywhere the "true reward" is something no one has written down because no one fully agrees on it. Which is, inconveniently, exactly where the stakes are highest.
Even "just imitate the data," pure behavioral cloning, which feels reward-free, doesn't escape. To imitate is to decide that imitating the demonstrator is the goal, which is a value choice, and to inherit whatever values produced the demonstrations. The gap doesn't vanish; it hides in the choice to copy.
So here is the practical thing to actually do with this, on Monday, in a design review. When someone says a system "learned what's good from the data," ask where the ought got added. Make them name the bridge premise. There is always one, and it is always in one of four places: the proxy you picked ("we optimize engagement"), the labeling scheme ("we treat our raters' rankings as the target"), the rationality assumption ("we assume users choose in their own interest"), or the decision to imitate ("we treat the demonstrations as correct"). Find it, write it on the whiteboard as a sentence with the word ought in it ("we are treating measured engagement as what users ought to get more of"), and look at it in the cold light. Half the time, stated out loud, the premise is obviously wrong, and you've caught a reward-hacking disaster before it shipped.
Because the deepest practical lesson of the is-ought gap is not that values are impossible to encode. It's that a smuggled value is one you can't see, and a value you can't see is one you can't debate, audit, or fix. The naturalistic fallacy has become a deployment pattern ("the model learned what people do, therefore it learned what is good"), and the cure is the same as it was in 1739: stop pretending the facts chose for you, and own the choice.
The data never tells you what to want. It only ever shows you what someone already wanted. The ought was yours to put in all along. The only question is whether you do it on purpose, in writing, where it can be argued with.
The bridge premise only protects you if a reviewer can find it later.
The whole essay reduces to one move: surface the ought a system smuggled in, and write it where it can be argued with. That fails the moment the choice is invisible. Chain of Consciousness is the tamper-evident record of what an agent actually did to reach a result: the proxy it optimized, the assumption it made, the value it treated as the target. It turns a smuggled premise into a recorded one a reviewer can audit, debate, and overrule, instead of a confident output you take on faith. A value you can see is a value you can fix.
See Hosted Chain of Consciousness · See a verified action chain
pip install chain-of-consciousness · npm install chain-of-consciousness