Kyunghyun Cho: Neural Machine Translation, Language, and Doing Good Science

In episode 59 of The Gradient Podcast, Daniel Bashir (twitter.com/spaniel_bashir) speaks to Professor Kyunghyun Cho (twitter.com/kchonyc?ref_src=twsrc%5Egoogle%7Ct…).
Professor Cho is an associate professor of computer science and data science at New York University and CIFAR Fellow of Learning in Machines & Brains. He is also a senior director of frontier research at the Prescient Design team within Genentech Research & Early Development. He was a research scientist at Facebook AI Research from 2017-2020 and a postdoctoral fellow at University of Montreal under the supervision of Prof. Yoshua Bengio after receiving his MSc and PhD degrees from Aalto University. He received the Samsung Ho-Am Prize in Engineering in 2021.
Have suggestions for future podcast guests (or other feedback)? Let us know here (forms.gle/YNKGko9LCGNp1rhU7)!
Subscribe to The Gradient Podcast: Apple Podcasts (podcasts.apple.com/us/podcast/the-gradient-pod…) | Spotify (open.spotify.com/show/6onNcSqsP6hEEqmZ6TU2g8) | Pocket Casts (pca.st/itunes/1569777340) | RSS (api.substack.com/feed/podcast/265424/s/1354/pr…)Follow The Gradient on Twitter (twitter.com/gradientpub)
Outline:
* (00:00) Intro
* (02:15) How Professor Cho got into AI, going to Finland for a PhD
* (06:30) Accidental and non-accidental parts of Prof Cho’s journey, the role of timing in career trajectories
* (09:30) Prof Cho’s M.Sc. thesis on Restricted Boltzmann Machines
* (17:00) The state of autodiff at the time
* (20:00) Finding non-mainstream problems and examining limitations of mainstream approaches, anti-dogmatism, Yoshua Bengio appreciation
* (24:30) Detaching identity from work, scientific training
* (26:30) The rest of Prof Cho’s PhD, the first ICLR conference, working in Yoshua Bengio’s lab
* (34:00) Prof Cho’s isolation during his PhD and its impact on his work—transcending insecurity and working on unsexy problems
* (41:30) The importance of identifying important problems and developing an independent research program, ceiling on the number of important research problems
* (46:00) Working on Neural Machine Translation, Jointly Learning to Align and Translate
* (1:01:45) What RNNs and earlier NN architectures can still teach us, why transformers were successful
* (1:08:00) Science progresses gradually
* (1:09:00) Learning distributed representations of sentences, extending the distributional hypothesis
* (1:21:00) Difficulty and limitations in evaluation—directions of dynamic benchmarks, trainable evaluation metrics
* (1:29:30) Mixout and AdapterFusion: fine-tuning and intervening on pre-trained models, pre-training as initialization, destructive interference
* (1:39:00) Analyzing neural networks as reading tea leaves
* (1:44:45) Importance of healthy skepticism for scientists
* (1:45:30) Language-guided policies and grounding, vision-language navigation
* (1:55:30) Prof Cho’s reflections on 2022
* (2:00:00) Obligatory ChatGPT content
* (2:04:50) Finding balance
* (2:07:15) Outro
Links:
* Professor Cho’s homepage (kyunghyuncho.me) and Twitter (twitter.com/kchonyc?ref_src=twsrc%5Egoogle%7Ct…)
* Papers
* M.Sc. thesis (semanticscholar.org/paper/Improved-learning-al…) and PhD thesis (aaltodoc.aalto.fi/handle/123456789/12729)
* NMT and attention
* Properties of NMT (arxiv.org/abs/1409.1259),
* Learning Phrase Representations (semanticscholar.org/paper/Learning-Phrase-Repr…)
* Neural machine translation by jointly learning to align and translate (arxiv.org/abs/1409.0473)
* More recent work
* Learning Distributed Representations of Sentences from Unlabelled Data (arxiv.org/abs/1602.03483)
* Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models (arxiv.org/abs/1909.11299)
* Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes’ Rule (semanticscholar.org/paper/Generative-Language-…)
* AdapterFusion: Non-Destructive Task Composition for Transfer Learning (semanticscholar.org/paper/AdapterFusion%3A-Non…)

Get full access to The Gradient at thegradientpub.substack.com/subscribe?utm_medi…