The Gradient: Perspectives on AI

Hugo Larochelle: Deep Learning as Science

Donnerstag, 6. Juli 2023 · 1 h 48

MP3

Shownotes

In episode 80 of The Gradient Podcast, Daniel Bashir (twitter.com/spaniel_bashir) speaks to Professor Hugo Larochelle (twitter.com/hugo_larochelle?s=20).
Professor Larochelle leads the Montreal Google DeepMind team and is adjunct professor at Université de Montréal and a Canada CIFAR Chair. His research focuses on the study and development of deep learning algorithms.
Have suggestions for future podcast guests (or other feedback)? Let us know here (forms.gle/YNKGko9LCGNp1rhU7) or reach us at editor@thegradient.pub
Subscribe to The Gradient Podcast: Apple Podcasts (podcasts.apple.com/us/podcast/the-gradient-pod…) | Spotify (open.spotify.com/show/6onNcSqsP6hEEqmZ6TU2g8) | Pocket Casts (pca.st/itunes/1569777340) | RSS (api.substack.com/feed/podcast/265424/s/1354/pr…)Follow The Gradient on Twitter (twitter.com/gradientpub)
Outline:
* () Intro
* () Prof. Larochelle’s background, working in Bengio’s lab
* () Prof. Larochelle’s work and connectionism
* () 2004-2009, work with Bengio
* () Nonlocal Estimation of Manifold Structure, manifolds and deep learning
* () Manifold learning in vision and language
* () Relationship to Denoising Autoencoders and greedy layer-wise pretraining
* () From input copying to learning about local distribution structure
* () Zero-Data Learning of New Tasks
* () The phrase “extend machine learning towards AI” and terminology
* () Prescient hints of prompt engineering
* () Daniel goes on totally unnecessary tangent
* () Methods for training deep networks (strategies and robust interdependent codes)
* () Motivations for layer-wise pretraining
* () Robust Interdependent Codes and interactions between neurons in a single network layer
* () 2009-2011, postdoc in Geoff Hinton’s lab
* () Reflections on the AlexNet moment
* () Frustration with methods for evaluating unsupervised methods, NADE
* () How researchers thought about representation learning, toying with objectives instead of architectures
* () The Restricted Boltzmann Forest
* () Imposing structure for tractable learning of distributions
* () 2011-2016 at U Sherbooke (and Twitter)
* () How Prof. Larochelle approached research problems
* () How Domain Adversarial Networks came about
* () Can we still learn from Restricted Boltzmann Machines?
* () The ~ Infinite ~ Restricted Boltzmann Machine
* () The need for researchers doing different sorts of work
* () 2017-present, at MILA (and Google)
* () Modulating Early Visual Processing by Language, neuroscientific inspiration
* () Representation learning and generalization, what is a good representation (Meta-Dataset, Universal representation transformer layer, universal template, Head2Toe)
* () Meta-Dataset motivation
* () Shifting focus to the problem—good practices for “recycling deep learning”
* () Head2Toe intuitions
* () What are “universal representations” and manifold perspective on datasets, what is the right pretraining dataset
* () Prof. Larochelle’s takeaways from Fortuitous Forgetting in Connectionist Networks (arxiv.org/abs/2202.00155) (led by Hattie Zhou (thegradientpub.substack.com/p/hattie-zhou-lott…))
* () Obligatory commentary on The Present Moment and current directions in ML
* () The creation and motivations of the TMLR journal
* () Prof. Larochelle’s takeaways about doing good science, building research groups, and nurturing a research environment
* () Prof. Larochelle’s advice for aspiring researchers today
* () Outro
Links:
* Professor Larochelle’s homepage (mila.quebec/en/person/hugo-larochelle) and Twitter (twitter.com/hugo_larochelle?s=20)
* Transactions on Machine Learning Research (jmlr.org/tmlr)
* Papers
* 2004-2009
* Nonlocal Estimation of Manifold Structure (hal.science/hal-01575345/file/Non_Local_Estima…)
* Classification using Discriminative Restricted Boltzmann Machines (swoh.web.engr.illinois.edu/courses/IE598/hando…)
* Zero-data learning of new tasks (cs.utoronto.ca/~larocheh/publications/aaai2008…)
* Exploring Strategies for Training Deep Neural Networks (jmlr.org/papers/volume10/larochelle09a/laroche…)
* Deep Learning using Robust Interdependent Codes (proceedings.mlr.press/v5/larochelle09a/laroche…)
* 2009-2011
* Stacked Denoising Autoencoders (jmlr.org/papers/volume11/vincent10a/vincent10a…)
* Tractable multivariate binary density estimation and the restricted Boltzmann forest (cs.utoronto.ca/~larocheh/publications/NECO-10-…)
* The Neural Autoregressive Distribution Estimator (proceedings.mlr.press/v15/larochelle11a/laroch…)
* Learning Attentional Policies for Tracking and Recognition in Video with Deep Networks (d1wqtxts1xzle7.cloudfront.net/42205942/Learnin…)
* 2011-2016
* Practical Bayesian Optimization of Machine Learning Algorithms (proceedings.neurips.cc/paper/2012/file/0531165…)
* Learning Algorithms for the Classification Restricted Boltzmann Machine (jmlr.org/papers/volume13/larochelle12a/laroche…)
* A neural autoregressive topic model (proceedings.neurips.cc/paper/2012/file/b495ce6…)
* Domain-Adversarial Training of Neural Networks (arxiv.org/abs/1505.07818)
* NADE (jmlr.org/papers/volume17/16-272/16-272.pdf)
* An Infinite Restricted Boltzmann Machine (arxiv.org/pdf/1502.02476.pdf)
* 2017-present
* Modulating early visual processing by language (proceedings.neurips.cc/paper/2017/file/6fab6e3…)
* Meta-Dataset (arxiv.org/pdf/1903.03096.pdf)
* A Universal Representation Transformer Layer for Few-Shot Image Classification (arxiv.org/pdf/2006.11702.pdf)
* Learning a universal template for few-shot dataset generalization (proceedings.mlr.press/v139/triantafillou21a/tr…)
* Impact of aliasing on generalization in deep convolutional networks (openaccess.thecvf.com/content/ICCV2021/papers/…)
* Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning (arxiv.org/abs/2201.03529)
* Fortuitous Forgetting in Connectionist Networks (arxiv.org/abs/2202.00155)

Get full access to The Gradient at thegradientpub.substack.com/subscribe?utm_medi…

0:00 / 0:00