Pete Florence: Dense Visual Representations, NeRFs, and LLMs for Robotics

In episode 54 of The Gradient Podcast, Andrey Kurenkov (andreykurenkov.com) speaks with Pete Florence (peteflorence.com).
Note: this was recorded 2 months ago. Andrey should be getting back to putting out some episodes next year.
Pete Florence is a Research Scientist at Google Research on theRobotics at Google (research.google/teams/brain/robotics) team insideBrain Team (ai.google/research/teams/brain) inGoogle Research (research.google). His research focuses on topics in robotics, computer vision, and natural language -- including 3D learning, self-supervised learning, and policy learning in robotics. Before Google, he finished his PhD in Computer Science at MIT withRuss Tedrake (groups.csail.mit.edu/locomotion/russt.html).
Subscribe to The Gradient Podcast: Apple Podcasts (podcasts.apple.com/us/podcast/the-gradient-pod…) | Spotify (open.spotify.com/show/6onNcSqsP6hEEqmZ6TU2g8) | Pocket Casts (pca.st/itunes/1569777340) | RSS (api.substack.com/feed/podcast/265424/s/1354/pr…)Follow The Gradient on Twitter (twitter.com/gradientpub)
Outline:
* (00:00:00) Intro
* (00:01:16) Start in AI
* (00:04:15) PhD Work with Quadcopters
* (00:08:40) Dense Visual Representations
* (00:22:00) NeRFs for Robotics
* (00:39:00) Language Models for Robotics
* (00:57:00) Talking to Robots in Real Time
* (01:07:00) Limitations
* (01:14:00) Outro
Papers discussed:
* Aggressive quadrotor flight through cluttered environments using mixed integer programming (groups.csail.mit.edu/robotics-center/public_pa…)
* Integrated perception and control at high speed: Evaluating collision avoidance maneuvers without maps (groups.csail.mit.edu/robotics-center/public_pa…)
* High-speed autonomous obstacle avoidance with pushbroom stereo (groups.csail.mit.edu/robotics-center/public_pa…)
* Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation. (arxiv.org/pdf/1806.08756.pdf) (Best Paper Award, CoRL 2018)
* Self-Supervised Correspondence in Visuomotor Policy Learning (arxiv.org/abs/1909.06933) (Best Paper Award, RA-L 2020 )
* iNeRF: Inverting Neural Radiance Fields for Pose Estimation. (arxiv.org/pdf/2012.05877.pdf)
* NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields. (arxiv.org/abs/2203.01913)
* Reinforcement Learning with Neural Radiance Fields (arxiv.org/abs/2206.01634)
* Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language. (arxiv.org/pdf/2204.00598)
* Inner Monologue: Embodied Reasoning through Planning with Language Models (arxiv.org/abs/2207.05608)
* Code as Policies: Language Model Programs for Embodied Control (code-as-policies.github.io)

Get full access to The Gradient at thegradientpub.substack.com/subscribe?utm_medi…