Yunhao (Robin) Tang

I am a research scientist interested in reinforcement learning.

Currently, I am a core contributor to the Llama post-training at Meta. Previously, I spent a few years at DeepMind London, where I was a core contributor to the Gemini post-training and studied deep reinforcement learning.

Before that, I was a two-time intern at DeepMind Paris hosted by Remi Munos. I obtained my PhD at Columbia University in New York City.

Email  /  Google Scholar

• [7/2024] Great to be part of the project led by my excellent collaborators to benchmark scalable-oversight protocols. Check it out here.
• [5/2024] We have a new paper that hypothesis-tests the importance of on-policy sampling in language model alignment, check it out here!
• [5/2024] Four papers are accepted at ICML 2024. Thank you for the heavy lifting to my coauthors.
• [3/2024] Gemini 1.5 is announced. Check out the tech report here.
• [12/2023] Gemini is launched. Great to be a core contributor to Gemini, the most powerful multi-modal large language model developed by Google DeepMind. Check out the tech report here.

My recent work focuses on the understanding and developments of deep reinforcement learning algorithms and systems, spanning the following non-exhaustive list of topics

• Learning from human feedback
• Representation learning
• Distributional reinforcement learning
• Search and credit assignment
• Off-policy reinforcement learning
• Stochastic gradient estimation

See here for the full list of publications.

Is online RL really necessary for AI alignment, or do offline algorithms suffice? The answer seems to be yes according to our careful ablations.

We have benchmarked important existing scalable-oversight protocols in a comprehensive suite of QA tasks, opening the path for further future investigation.

When human feedback is pointwise rather than pairwise, we propose direct reward optimization (DRO) as the alignment algorithm.

Online preference optimization as an alignment technique turns out to be intimately related to Nash equilibrium, besides being a competitive algorithm for RLHF.

GPO unifies alignment algorithms such as DPO, IPO and SLiC as special cases. The insight, interestingly, is based on classic literature on convex losses for binary classification. At the end of the day, all algorithmic variants have similar performance-regularization trade-off though their natural strengths of regularization differ.

One of the most powerful multi-modal large language models thus far in the world.

In aligning large language models, we search for Nash Equilibrium naturally defined via the pairwise human feedback. This approach is more general purpose, imposes fewer assumptions on reward modeling, and performs better than canonical RLHF.

We introduce another addition to the family of off-policy distributional RL algorithms, importantly, without the need for importance sampling.

We shed light on what distributional equivalence of successor representations look like, and the algorithmic applications arising from the insights.

We propose VA-learning as a more sample efficient alternative to Q-learning. The sample efficiency stems from the value sharing between different actions. Intriguingly, VA-learning closely relates to dueling architecture.

We design an off-policy actor-critic algorithm based on multi-step policy improvement and policy evaluation. This algorithm improves state-of-the-art IMPALA baseline.

We provide a characterization on how TD-learning learns representations, relating random reward based TD-learning with spectral decomposition of the transition matrix.

Efficient credit assignment should account for external factors outside of agent's control, or more informally, the level of luck. We formalize such intuitions into quantile credit assignment.

We show that in certain cases quantile TD outperforms TD in mean value prediction. This hints at a general potential of distributional RL to outperform mean-based RL at its own game.

Self-predictive learning is a popular representation learning algorithm in RL, which learns a latent representation by predicting (bootstrapping) its own future latents. Intuitively, the algorithm should not work as it can collapse to trivial solutions. We identify algorithmic components to prevent the collapse and show that self-preditive learning is related to gradient-based spectral decomposition of the transition dynamics.

We provide a first proof of the convergence of quantile TD-learning, a distributional RL algorithm that drives multiple recent empirical breakthroughs.

We identify a few intriguing and fundamental differences between value-based TD-learning and distributional TD-learning.

We find that self-prediction loss is a surprisingly useful signal for exploration in extremely challenging deep RL domains. Our method: BYOL-explore, partially cracks a wide range of extremely hard exploration problems much more efficiently than prior methods.

We find that certain deliberate bias in gradient estimators could significantly reduce variance for meta RL.

How to estimate high-order derivatives of value functions? We propose a unifying framework with off-policy evaluation. Direct differentiations of off-policy estimates produce estimates to high-order derivatives of value functions, and instantiate many prior methods as special cases.

Uncorrected multi-step updates such as n-step Q-learning are ubiquitous in modern deep RL practices. We revisit Peng's Q($\lambda$), a classic uncorrected multi-step variant. Our analysis sheds light on why uncorrected updates should work in practice. The empirical result also suggests significant gains on benchmark tasks.

We propose Hindsight Expectation Maximization (hEM), an EM algorithm for goal-conditioned RL problem which combines supervised learning through the M-step and hindsight goal sampling through the E-step. We also make an intimate connection between hindsight replay and importance sampling for rare event simulations.

Why is self-imitation learning efficient? We shed light on its connections to n-step Q-learning and show that part of its gains might be attributed to trade-offs in RL operators. We also propose a n-step extension of self-imitation learning which incorporates the strengths of both n-step updates and lower-bound learning.

We establish an interpretation of MCTS as policy optimization. This interpretation leads to algorithmic variants which naturally improve over MCTS-based baselines such as AlphaZero and MuZero.

We estabilish the intimate connections between trust region policy search and off-policy evaluation. The new algorithm TayPO generalizes policy optimization objectives to high-order extentions which leads to gains on large-scale distributed agents.

We formulate cutting plane algorithms as a sequential decision making problem for generic integer pgroamming. The cutting plane agent learned via RL improves over human-designed heuristics and benfits downstream applications such as branch-and-cut.