Reinforcement Learning 2022
Learning from mixed-quality demonstrations
Reformulating robot imitation learning as a sequence of returns, states, and actions.
Project description
From the original project page. Results and time references describe that version of the work.
Reformulation of Decision Transformer for Robonomic Benchmark
Reinforcement learning typically involves an agent interacting with an environment to achieve a maximum reward. Our project disregards the traditional approach of estimating policies and simplifies Reinforcement Learning to a sequence modeling problem that can effectively be solved by the Trans- former architecture. Our project extends the capabilities of the initial Decision Transformer (DT) [4] to learn from mixed- quality input data. Our modified Decision Transformer quantifies the benefit of return-conditioned imitation learning on mixed- quality data by leveraging the robomimic datasets. We show that our Decision Transformer significantly outperforms standard behavioral cloning on mixed-quality data for the Lift and Can tasks. Overall, our Decision Transformer and semi-sparse reward function provide a new way to tackle the challenges of imitation learning with mixed-quality data. Our project below is an earlier version of the paper that I plan on publishing in the upcoming future.
The question
Can an agent learn useful behavior from demonstrations of mixed quality? This project explores return-conditioned imitation learning using robomimic datasets.
The approach
Each input token concatenates the state, return-to-go and previous action. A two-layer network and learned position embeddings feed a causal transformer. Its output parameterizes an action distribution, including a five-mode Gaussian mixture policy.
The finding
The project reports improvements over behavioral cloning on the Lift and Can tasks with mixed-quality data. It also explores a semi-sparse reward function. See the report for the experimental setup.
Research status
The linked report is the project version shared on the original website. The visualization here illustrates sequence conditioning; it does not run a trained policy.