Critical Pitfall in Reward Learning from Human Feedback

On how dynamics misconceptions impact human feedback

Read Paper View Poster Download Presentation Experience the Study Games Code and Data

Writing a reward function in reinforcement learning is notoriously difficult. You may know what behaviour you want, but once that idea becomes a scalar objective to maximize, an agent can maximize it in ways you did not intend. Ask for cups of coffee, and a poorly specified reward might make stealing coffee look like a good strategy.

Reward learning from human feedback offers a better route. Instead of hand-writing the reward, the agent produces behaviour or plans, and a human provides feedback that the agent uses to update its reward model (Kaufmann et al., 2024). That feedback can take many forms, including expert demonstrations (Ng & Russel, 2000), pairwise preferences (Christiano et al., 2017), or corrective interventions (Bajcsy et al., 2017).

But there is another variable: the human’s understanding of how the environment works. Since, the agent’s only signal from the human is the feedback signal, if the feedback provider misunderstands the environment dynamics, their feedback can reflect that misunderstanding. If a human does not know that the ground is slippery, but pushes the agent toward the quickest route, the learner may infer that risky slipping behaviour is desired. If a human does not know that stealing coffee is possible, but repeatedly encourages quick coffee delivery, a reward learner may incorrectly treat stealing as part of the reward it must maximize for.

In (Shaheen et al., 2026), we empirically test whether this effect exists. We explore if human feedback in reward learning reflects not only latent reward, or it also involves dynamics understanding.

Why This Matters

Most reward-learning methods assume that the learner and the feedback provider share the same dynamics model. Under that assumption, feedback is treated as pure evidence about the human’s reward. Our concern is that this assumption can be false.

Traditional reward learning view — The standard view: feedback is interpreted as evidence about the human's latent reward under the learner's perfect understanding of environment dynamics.

Mismatched dynamics view — Our view: feedback is a confounded signal shaped by both reward and the human's dynamics understanding.

Research Question

The question we studied: Does a perturbed understanding of domain dynamics impact human feedback?

Answering it required an experiment that could separate reward from belief. We needed to initialize participants with one understanding of the domain dynamics, perturb that understanding, and then measure feedback without letting participants update their beliefs by watching the plan execute.

We tested three hypotheses:

H1: Internal dynamics understanding mediates the relationship between instruction framing and feedback.
H2: This mediation is robust to feedback modality and remains consistent across three feedback types.
H3: This mediation is robust to visual priming and persists across two visual contexts.

The Study Game

We designed a randomized controlled trial with 211 participants. To keep the task understandable, we used a set of simple planning games around Elfie the elf, with participants helping Elfie plan a route to a cookie without falling into holes. The world was a gridworld with holes, with the domain dynamics differences being what participants believed about the ground near the cliffs.

Experiment Flow

The study flow can roughly be divided into these steps:

Prime: expose participants to contrastive visuals, icy and grassy, then randomize them to one visual context.
Manipulate: use instruction framing to induce either a sense that Elfie was in danger of slipping or that Elfie was safe.
Measure: collect feedback while preventing participants from observing plan execution.

Prime

We first exposed participants to two visuals and then randomized them to either an icy, slippery visual context or a grassy, stable visual context.

Visual contexts — Participants first formed an initial model from visual context: grassy/stable or icy/slippery.

Manipulate

We analyzed the experiment through a causal mediation framework. We test whether the instruction framing changed feedback through the participant’s dynamics understanding, rather than through a direct change in what participants wanted Elfie to do. Then we used text instructions to update their belief about whether cliff-adjacent tiles were safe or dangerous. Only the participant’s dynamics understanding was manipulated.The reward stayed consistent: reach the cookie and avoid holes.

Instruction manipulation — Text instructions then framed the same cliff-adjacent paths as safe or dangerous.

Measurement and Feedback Types

Participants gave feedback in three different ways. This was important because a result limited to one interface might only be about that interface. Before the feedback tasks, participants were told that the games were about planning only. They would not watch Elfie execute the plan. This mattered because observing execution could have corrected their dynamics understanding during the measurement phase.

Correction game — Corrective intervention: participants edited Elfie's planned path.

Comparison game — Pairwise preference: participants chose between two candidate plans.

Off intervention game — Off-button-style intervention: participants placed obstacles to stop and redirect Elfie.

Results

The main result is: perturbed dynamics understanding impacted feedback.

First, the priming worked: participants formed different initial beliefs from the visual contexts. Second, the instruction manipulation worked: the safety framing shifted participants’ understanding of whether Elfie was at risk of slipping. Finally, the shifted dynamics understanding predicted shifted feedback behaviour. This effect persisted across both visual contexts and all three feedback types.

Study overview and results — Overview of the paper's causal story: instructions changed dynamics understanding, and changed understanding shifted feedback.

Takeaways

If you do RLHF or reward learning from human feedback, consider the feedback provider’s dynamics understanding.
If the human and learner have mismatched environment models, reward learning may critically fail.
Changing the feedback interface is not enough if the same misunderstanding impacts all feedback types.
This study gives empirical support for RLHF methods that separate what people want from what people believe will happen.

Acknowledgements

This research was supported in part by NSF grant 2047186 and the 2025 ASU Graduate Student Government JumpStart Grant. The study was approved by the Arizona State University Institutional Review Board.

References

2026

IJCAI
Empirical Evidence and Analysis of a Critical Pitfall in Reward Learning from Human Feedback

Taha Shaheen , Stephen G. West , and Yu Zhang

In Proceedings of the 35th International Joint Conference on Artificial Intelligence (IJCAI-ECAI 2026), Aug 2026

Abs Bib Website Read

Reward learning via human feedback is a crucial capability for beneficial AI. Current methods are built on decision-making theories that assume a matched dynamics model between the learning agent and the feedback provider. However, humans often form imperfect internal dynamics models, and their feedback reflects these misconceptions. While this relationship has long been hypothesised, its manifestation in sequential decision-making remains largely an assumption. Our work provides the first comprehensive empirical investigation of this relationship through a randomized controlled trial (N=211). We followed a two-stage design where we first initialized the participants’ understanding of the dynamics in a grid-world navigation domain and then manipulated it using text-based instructions. Causal mediation analysis revealed that humans’ internal models play a mediating role in feedback behaviour. We show that this relationship is invariant across visual contexts and is robust to three common feedback types: pairwise preferences, trajectory corrections, and off-switch interventions. These findings confirm a critical limitation of current reward learning methods and establish the missing psychological foundation for approaches that incorporate dynamics understanding.
@inproceedings{shaheen2026criticalpitfall, title = {Empirical Evidence and Analysis of a Critical Pitfall in Reward Learning from Human Feedback}, author = {Shaheen, Taha and West, Stephen G. and Zhang, Yu}, booktitle = {Proceedings of the 35th International Joint Conference on Artificial Intelligence (IJCAI-ECAI 2026)}, year = {2026}, month = aug, url = {https://ijcai-preprints.s3.us-west-1.amazonaws.com/2026/2656.pdf}, }

2024

A Survey of Reinforcement Learning from Human Feedback

Timo Kaufmann , Paul Weng , Viktor Bengs , and 1 more author

Transactions on Machine Learning Research, Jun 2024

Abs

Reinforcement learning from human feedback (RLHF) is a variant of reinforcement learning (RL) that learns from human feedback instead of relying on an engineered reward function. Building on prior work on the related setting of preference-based reinforcement learning (PbRL), it stands at the intersection of artificial intelligence and human-computer interaction. This positioning provides a promising approach to enhance the performance and adaptability of intelligent systems while also improving the alignment of their objectives with human values. The success in training large language models (LLMs) has impressively demonstrated this potential in recent years, where RLHF has played a decisive role in directing the model’s capabilities towards human objectives. This article provides an overview of the fundamentals of RLHF, exploring how RL agents interact with human feedback. While recent focus has been on RLHF for LLMs, our survey covers the technique across multiple domains. We provide our most comprehensive coverage in control and robotics, where many fundamental techniques originate, alongside a dedicated LLM section. We examine the core principles that underpin RLHF, how algorithms and human feedback work together, and discuss the main research trends in the field. Our goal is to give researchers and practitioners a clear understanding of this rapidly growing field.

2017

Deep Reinforcement Learning from Human Preferences

Paul F Christiano , Jan Leike , Tom B Brown , and 3 more authors

In , Jun 2017

Abs

For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than 1% of our agent’s interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any which have been previously learned from human feedback.
Learning Robot Objectives from Physical Human Interaction

Andrea Bajcsy , Dylan P. Losey , Marcia K. O’Malley , and 1 more author

In Proceedings of the 1st Annual Conference on Robot Learning, Oct 2017

Abs

When humans and robots work in close proximity, physical interaction is inevitable. Traditionally, robots treat physical interaction as a disturbance, and resume their original behavior after the interaction ends. In contrast, we argue that physical human interaction is informative: it is useful information about how the robot should be doing its task. We formalize learning from such interactions as a dynamical system in which the task objective has parameters that are part of the hidden state, and physical human interactions are observations about these parameters. We derive an online approximation of the robot’s optimal policy in this system, and test it in a user study. The results suggest that learning from physical interaction leads to better robot task performance with less human effort.

2000

Algorithms for Inverse Reinforcement Learning

Andrew Y. Ng , and Stuart Russel

In International Conference on Machine Learning, Jun 2000