publications

2026

1. IJCAI

   Empirical Evidence and Analysis of a Critical Pitfall in Reward Learning from Human Feedback

   Taha Shaheen ,  Stephen G. West ,  and  Yu Zhang

   In Proceedings of the 35th International Joint Conference on Artificial Intelligence (IJCAI-ECAI 2026), Aug 2026

   Abs Bib Website Read

   Reward learning via human feedback is a crucial capability for beneficial AI. Current methods are built on decision-making theories that assume a matched dynamics model between the learning agent and the feedback provider. However, humans often form imperfect internal dynamics models, and their feedback reflects these misconceptions. While this relationship has long been hypothesised, its manifestation in sequential decision-making remains largely an assumption. Our work provides the first comprehensive empirical investigation of this relationship through a randomized controlled trial (N=211). We followed a two-stage design where we first initialized the participants’ understanding of the dynamics in a grid-world navigation domain and then manipulated it using text-based instructions. Causal mediation analysis revealed that humans’ internal models play a mediating role in feedback behaviour. We show that this relationship is invariant across visual contexts and is robust to three common feedback types: pairwise preferences, trajectory corrections, and off-switch interventions. These findings confirm a critical limitation of current reward learning methods and establish the missing psychological foundation for approaches that incorporate dynamics understanding.

   @inproceedings{shaheen2026criticalpitfall,
     title = {Empirical Evidence and Analysis of a Critical Pitfall in Reward Learning from Human Feedback},
     author = {Shaheen, Taha and West, Stephen G. and Zhang, Yu},
     booktitle = {Proceedings of the 35th International Joint Conference on Artificial Intelligence (IJCAI-ECAI 2026)},
     year = {2026},
     month = aug,
     url = {https://ijcai-preprints.s3.us-west-1.amazonaws.com/2026/2656.pdf},
   }

2025

1. arXiv

   Active Shadowing (ASD): Manipulating Visual Perception of Robotics Behaviors via Implicit Communication

   Andrew Boateng ,  Prakhar Bhartiya ,  Taha Shaheen , and 1 more author

   arXiv preprint arXiv:2407.01468, Aug 2025

   Abs Bib Read

   Explicit communication is often valued for its directness in presenting information but requires attention during exchange, resulting in cognitive interruptions. On the other hand, implicit communication contributes to tacit and smooth interaction, making it more suitable for teaming, but requires inference for interpretation. This paper studies a novel type of implicit visual communication (IVC) using shadows via visual projection with augmented reality, referred to as active shadowing (ASD). Prior IVC methods, such as legible motion, are often used to influence the perception of robot behavior to make it more understandable. They often require changing the physical robot behavior, resulting in suboptimality. In our work, we investigate how ASD can be used to achieve similar effects without losing optimality. Our evaluations with user studies demonstrates that ASD can effectively creates ”illusions” that maintain optimal physical behavior without compromising its understandability. We also show that ASD can be more informative than other explicit communication methods, and examine the conditions under which ASD becomes less effective.

   @article{boateng2024active,
     title = {Active Shadowing (ASD): Manipulating Visual Perception of Robotics Behaviors via Implicit Communication},
     author = {Boateng, Andrew and Bhartiya, Prakhar and Shaheen, Taha and Zhang, Yu},
     journal = {arXiv preprint arXiv:2407.01468},
     year = {2025},
     url = {https://arxiv.org/abs/2407.01468},
   }

2024

1. A Survey of Reinforcement Learning from Human Feedback

   Timo Kaufmann ,  Paul Weng ,  Viktor Bengs , and 1 more author

   Transactions on Machine Learning Research, Jun 2024

   Abs

   Reinforcement learning from human feedback (RLHF) is a variant of reinforcement learning (RL) that learns from human feedback instead of relying on an engineered reward function. Building on prior work on the related setting of preference-based reinforcement learning (PbRL), it stands at the intersection of artificial intelligence and human-computer interaction. This positioning provides a promising approach to enhance the performance and adaptability of intelligent systems while also improving the alignment of their objectives with human values. The success in training large language models (LLMs) has impressively demonstrated this potential in recent years, where RLHF has played a decisive role in directing the model’s capabilities towards human objectives. This article provides an overview of the fundamentals of RLHF, exploring how RL agents interact with human feedback. While recent focus has been on RLHF for LLMs, our survey covers the technique across multiple domains. We provide our most comprehensive coverage in control and robotics, where many fundamental techniques originate, alongside a dedicated LLM section. We examine the core principles that underpin RLHF, how algorithms and human feedback work together, and discuss the main research trends in the field. Our goal is to give researchers and practitioners a clear understanding of this rapidly growing field.

2. ACM THRI

   Investigation of Low-Moral Actions by Malicious Anonymous Operators of Avatar Robots

   Taha Shaheen ,  Dražen Brščić ,  and  Takayuki Kanda

   ACM Transactions on Human-Robot Interaction, Sep 2024

   Abs Bib Website Read

   Avatar robots allow a teleoperator to interact with the people and environment of a remote place. Malicious operators can use this technology to perpetrate malicious or low-moral actions. In this study, we used hazard identification workshops to identify low-moral actions that are possible through the locomotor movement, cameras, and microphones of an avatar robot. We conducted three workshops, each with four potential future users of avatars, to brainstorm possible low-moral actions. As avatars are not yet widespread, we gave participants experience with this technology by having them control both a simulated avatar and a real avatar as a malicious anonymous operator in a variety of situations. They also experienced sharing space with an avatar controlled by a malicious anonymous operator. We categorized the ideas generated from the workshops using affinity diagram analysis and identified four major categories: violate privacy and security, inhibit, annoy, and destroy or hurt. We also identified subcategories for each. In the second half of this study, we discuss all low-moral action subcategories in terms of their detection, mitigation, and prevention by studying literature from autonomous, social, teleoperated, and telepresence robots as well as other fields where relevant.

   @article{shaheen2024lowmoralactions,
     author = {Shaheen, Taha and Br\v{s}\v{c}i\'{c}, Dra\v{z}en and Kanda, Takayuki},
     title = {Investigation of Low-Moral Actions by Malicious Anonymous Operators of Avatar Robots},
     year = {2024},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3696466},
     doi = {10.1145/3696466},
     journal = {ACM Transactions on Human-Robot Interaction},
     month = sep,
     keywords = {avatar robots, low-moral actions, hazard identification, malicious users, ethics},
   }

2021

1. Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch

   Luca Viano ,  Yu-Ting Huang ,  Parameswaran Kamalaruban , and 2 more authors

   In Advances in Neural Information Processing Systems, Sep 2021

2020

1. What Is It You Really Want of Me? Generalized Reward Learning with Biased Beliefs about Domain Dynamics

   Ze Gong ,  and  Yu Zhang

   In , Apr 2020

   Abs

   Reward learning as a method for inferring human intent and preferences has been studied extensively. Prior approaches make an implicit assumption that the human maintains a correct belief about the robot’s domain dynamics. However, this may not always hold since the human’s belief may be biased, which can ultimately lead to a misguided estimation of the human’s intent and preferences, which is often derived from human feedback on the robot’s behaviors. In this paper, we remove this restrictive assumption by considering that the human may have an inaccurate understanding of the robot. We propose a method called Generalized Reward Learning with biased beliefs about domain dynamics (GeReL) to infer both the reward function and human’s belief about the robot in a Bayesian setting based on human ratings. Due to the complex forms of the posteriors, we formulate it as a variational inference problem to infer the posteriors of the parameters that govern the reward function and human’s belief about the robot simultaneously. We evaluate our method in a simulated domain and with a user study where the user has a bias based on the robot’s appearances. The results show that our method can recover the true human preferences while subject to such biased beliefs, in contrast to prior approaches that could have misinterpreted them completely.

2018

1. Occam’s Razor Is Insufficient to Infer the Preferences of Irrational Agents

   Stuart Armstrong ,  and  Sören Mindermann

   Advances in Neural Information Processing Systems, Apr 2018

   Abs

   Inverse reinforcement learning (IRL) attempts to infer human rewards or preferences from observed behavior. Since human planning systematically deviates from rationality, several approaches have been tried to account for specific human shortcomings. However, the general problem of inferring the reward function of an agent of unknown rationality has received little attention. Unlike the well-known ambiguity problems in IRL, this one is practically relevant but cannot be resolved by observing the agent’s policy in enough environments. This paper shows (1) that a No Free Lunch result implies it is impossible to uniquely decompose a policy into a planning algorithm and reward function, and (2) that even with a reasonable simplicity prior/Occam’s razor on the set of decompositions, we cannot distinguish between the true decomposition and others that lead to high regret. To address this, we need simple ’normative’ assumptions, which cannot be deduced exclusively from observations.

2017

1. A Survey of Preference-Based Reinforcement Learning Methods

   Christian Wirth ,  Riad Akrour ,  Gerhard Neumann , and 1 more author

   Journal of Machine Learning Research, Apr 2017

   Abs

   Reinforcement learning (RL) techniques optimize the accumulated long-term reward of a suitably chosen reward function. However, designing such a reward function often requires a lot of task-specific prior knowledge. The designer needs to consider different objectives that do not only influence the learned behavior but also the learning progress. To alleviate these issues, preference-based reinforcement learning algorithms (PbRL) have been proposed that can directly learn from an expert’s preferences instead of a hand-designed numeric reward. PbRL has gained traction in recent years due to its ability to resolve the reward shaping problem, its ability to learn from non numeric rewards and the possibility to reduce the dependence on expert knowledge. We provide a unified framework for PbRL that describes the task formally and points out the different design principles that affect the evaluation task for the human as well as the computational complexity. The design principles include the type of feedback that is assumed, the representation that is learned to capture the preferences, the optimization problem that has to be solved as well as how the exploration/exploitation problem is tackled. Furthermore, we point out shortcomings of current algorithms, propose open research questions and briefly survey practical tasks that have been solved using PbRL.

2. Deep Reinforcement Learning from Human Preferences

   Paul F Christiano ,  Jan Leike ,  Tom B Brown , and 3 more authors

   In , Apr 2017

   Abs

   For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than 1% of our agent’s interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any which have been previously learned from human feedback.

3. The off-switch game

   Dylan Hadfield-Menell ,  Anca Dragan ,  Pieter Abbeel , and 1 more author

   In Proceedings of the 26th International Joint Conference on Artificial Intelligence, Apr 2017

   Abs

   It is clear that one of the primary tools we can use to mitigate the potential risk from a misbehaving AI system is the ability to turn the system off. As the capabilities of AI systems improve, it is important to ensure that such systems do not adopt subgoals that prevent a human from switching them off. This is a challenge because many formulations of rational agents create strong incentives for self-preservation. This is not caused by a built-in instinct, but because a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead. Our goal is to study the incentives an agent has to allow itself to be switched off. We analyze a simple game between a human H and a robot R, where H can press R’s off switch but R can disable the off switch. A traditional agent takes its reward function for granted: we show that such agents have an incentive to disable the off switch, except in the special case where H is perfectly rational. Our key insight is that for R to want to preserve its off switch, it needs to be uncertain about the utility associated with the outcome, and to treat H’s actions as important observations about that utility. (R also has no incentive to switch itself off in this setting.) We conclude that giving machines an appropriate level of uncertainty about their objectives leads to safer designs, and we argue that this setting is a useful generalization of the classical AI paradigm of rational agents.

4. Learning Robot Objectives from Physical Human Interaction

   Andrea Bajcsy ,  Dylan P. Losey ,  Marcia K. O’Malley , and 1 more author

   In Proceedings of the 1st Annual Conference on Robot Learning, Oct 2017

   Abs

   When humans and robots work in close proximity, physical interaction is inevitable. Traditionally, robots treat physical interaction as a disturbance, and resume their original behavior after the interaction ends. In contrast, we argue that physical human interaction is informative: it is useful information about how the robot should be doing its task. We formalize learning from such interactions as a dynamical system in which the task objective has parameters that are part of the hidden state, and physical human interactions are observations about these parameters. We derive an online approximation of the robot’s optimal policy in this system, and test it in a user study. The results suggest that learning from physical interaction leads to better robot task performance with less human effort.

2000

1. Algorithms for Inverse Reinforcement Learning

   Andrew Y. Ng ,  and  Stuart Russel

   In International Conference on Machine Learning, Jun 2000