Reward and Constraint Learning: Foundations for Human-AI Alignment
A core challenge in deploying highly capable AI systems is the value alignment problem: how can we ensure that complex agents pursue objectives that are beneficial, safe, and aligned with human intent, especially when that intent is difficult to articulate precisely? Traditional Reinforcement Learning (RL) assumes a perfect, hand-coded reward function encoding the intent, an improper assumption for real-world tasks like self-driving or human-AI collaboration. However, miss-specification of rewards has also been observed in “simpler” tasks, like video-games. The consequences of such miss-specification can be severe, e.g., unsafe driving behavior in self-driving cars or inappropriate responses in large language model-based chatbots. While there is no final solution for the aforementioned problems, there is a general theoretical foundation and substantial recent advances in the field of learning reward and constraint functions and their usage for human-AI alignment.
Course Goals
This course aims to introduce the basics for engaging with this field, providing a solid foundation and providing selected examples of current research efforts. To this end, the course will cover the following topics:
- Inverse Reinforcement Learning (IRL): Inferring the underlying reward function from expert demonstrations.
- Reward Learning and Reward Modeling (RM): Learning robust reward models directly from human feedback and preferences (e.g., in the context of Reinforcement from Human Feedback).
- Constraint Learning: Inferring implicit safety, preference, or feasibility constraints that must be satisfied independent of the associated reward.
By the end of this course, students will understand the basic modern technical toolkit required to bridge the gap between human values and reward and constraint functions for reinforcement learning and hence the basis for human-AI alignment.
Tentative Outline
The course will largely be a series of lectures introducing the basis for human-AI alignment including the formal setup, the discussion of central algorithms, a very few mathematical derivations, and discussions of recent, high-impact papers from venues like NeurIPS, ICML, and AAAI.
- [First session] Recap of RL Basics; IRL Fundamentals and Maximum Entropy Methods; Basic RL problem and its formalization; apprenticeship learning and the IRL problem (feature expectations and feature matching); Maximum Entropy IRL (MaxEnt IRL) and its probabilistic interpretation; Challenges in complicated state spaces, Generative Adversarial Imitation Learning (GAIL)
- [Second session] Deep Reward Modeling from Human Feedback: Limitations of classical IRL; Learning from comparisons (pairwise preferences); The Reinforcement Learning from Human Feedback (RLHF) approach; Deep Neural Network architectures for reward modeling; Active querying strategies for reducing human label burden.
- [Third session] Learning and Integrating Constraints: Formalizing safety and preference constraints in Constrained MDPs (CMDPs); Inferring hard and soft constraints from human demonstrations and special feedback, e.g., “stop” signals; Convex Constraint Learning (CoCoRL); Safe policy optimization using learned constraints (e.g., using Lagrange multipliers).
- [Fourth Session] Human-AI Alignment and Advanced Topics: Cooperative Inverse RL (CIRL) and its role in value alignment; Learner-Aware Teaching (adapting demonstrations based on learner biases) Connections to Explainable AI (XAI) and interpretability; Case studies in robotics and autonomous systems; Ethical implications of learned reward functions.
Prerequisites
This is course designed for students and researchers with a solid background in Machine Learning:
- Core Machine Learning: Familiarity with standard supervised and unsupervised learning techniques.
- Reinforcement Learning: Students are ideally already familiar with the fundamentals of RL, although a quick recap is done at the beginning of the course. Student’s knowledge should include in particular (1) Markov Decision Processes (MDPs) and the Bellman equations, and (2) basic RL algorithms (e.g., Q-learning, Policy Gradients).
Students not familiar with Reinforcement Learning could prepare for the course by appropriate reading, e.g., the relevant parts of the standard RL book (e.g., Sutton and Barto, Reinforcement Learning: An Introduction, Chapters 1-6).