Abstract
Natural language provides an intuitive and flexible way for humans to communicate with robots. Grounding language commands to structured task specifications enables autonomous robots to understand a broad range of natural language and solve long-horizon tasks with safety guarantees. Linear temporal logic (LTL) provides unambiguous semantics for language grounding, and its compositionality can induce skill transfer.
In this talk, I will first introduce two language grounding systems. 1) Lang2LTL is a modular system that uses large language models (LLMs) to ground navigation commands with diverse temporal patterns to LTL task specifications in novel environments without retraining. 2) Improved upon its predecessor, Lang2LTL-2 uses LLMs and a pre-trained vision-language model to ground spatiotemporal navigation commands. By translating natural language to LTL, both systems can detect infeasible task specifications and abort execution when necessary. Next, I will introduce a zero-shot transfer algorithm, LTL-Transfer, that leverages the compositionality of LTL to reuse skills learned during training to solve novel test tasks without violating any safety constraints. By integrating LTL-Transfer with Lang2LTL-2, we can develop a robotic system from natural language to verifiable robot behavior that is generalizable to unseen environments and tasks.
Relevant Papers
Grounding Complex Natural Language Commands for Temporal Tasks in Unseen Environments
LTL-Transfer: Skill Transfer for Temporal Task Specification
A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddings
Biography
Jason Xinyu Liu is a Ph.D. candidate at Brown University, advised by Prof. Stefanie Tellex. His research lies in the intersection of robotics, natural language processing, and formal methods. He is working towards developing autonomous robots that assist people. His work has appeared at CoRL, ICRA, IROS, IJCAI, and AAAI Symposiums. Jason earned his Bachelor's degree in Electrical Engineering and Computer Sciences from UC Berkeley. His research is generously funded by the NSF Graduate Research Fellowship Program and the Jack Kent Cooke Foundation Graduate Scholarship.