Why Nvidia builds open models with Bryan Catanzaro

One of the big stories of 2025 for me was how Nvidia massively stepped up their open model program — more releases, higher quality models, joining a small handful of companies releasing datasets, etc. In this interview, I sat down with one of the 3 VP’s leading the effort of 500+ technical staff, Bryan Catanzaro, to discuss:
* Their very impressive Nemotron 3 Nano model released in Dec. 2025, and the bigger Super and Ultra variants coming soon,
* Why Nvidia’s business clearly benefits from them building open models,
* How the Nemotron team culture was crafted in pursuit of better models,
* Megatron-LM and the current state of open-source training software,
* Career reflections and paths into AI research,
* And other topics.
The biggest takeaway I had from this interview is how Nvidia understands their unique roll as a company that and both build and directly capture the value they get from building open language models, giving them a uniquely sustainable advantage.
Bryan has a beautiful analogy for open models this early in AI’s development, and how they are a process of creating “potential energy” for AI’s future applications.
I hope you enjoy it!
Guest: Bryan Catanzaro, VP Applied Deep Learning Research (ADLR), NVIDIA. X:@ctnzr (x.com/ctnzr), LinkedIn (linkedin.com/in/bryancatanzaro), Google Scholar (scholar.google.com/citations?user=UZ6kI2AAAAAJ).
Listen on Apple Podcasts (podcasts.apple.com/us/podcast/interconnects-au…), Spotify (open.spotify.com/show/6XNzfJULeVxR7SneeesDUs), YouTube (youtube.com/@interconnects), and where ever you get your podcasts (interconnects.ai/podcast). For other Interconnects interviews, go here (interconnects.ai/t/interviews).
Nemotron Model Timeline
2019–2022 — Foundational Work
* Megatron-LM (arxiv.org/abs/1909.08053) (model parallelism framework that has become very popular again recently; alternatives:github.com/deepspeedai/DeepSpeed, PyTorch FSDP).github.com/NVIDIA/NeMo
* NeMo Framework (github.com/NVIDIA/NeMo) (NVIDIA’s end-to-end LLM stack: training recipes, data pipelines, evaluation, deployment).
Nov 2023 — Nemotron-3 8B: Enterprise-ready NeMo models. Models:huggingface.co/nvidia/nemotron-3-8b-base-4k,ch… (huggingface.co/nvidia/nemotron-3-8b-chat-4k-sft),chat-rlhf (huggingface.co/nvidia/nemotron-3-8b-chat-4k-rlhf), huggingface.co/collections/nvidia/nemotron-3-8… (developer.nvidia.com/blog/nvidia-ai-foundation…).
Feb 2024 — Nemotron-4 15B: Multilingual LLM trained to 8T tokens. Paper (arxiv.org/abs/2402.16819).
Jun 2024 — Nemotron-4 340B: Major open release detailing their synthetic data pipeline. Paper (arxiv.org/abs/2406.11704),blogs.nvidia.com/blog/nemotron-4-synthetic-dat…. Models:huggingface.co/nvidia/Nemotron-4-340B-Instruct… (catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/…).
Jul–Sep 2024 — Minitron / Nemotron-Mini: First of their pruned models, pruned from 15B.huggingface.co/nvidia/Minitron-4B-Base (base model), huggingface.co/nvidia/Nemotron-Mini-4B-Instruc… (arxiv.org/abs/2407.14679),code (github.com/NVlabs/Minitron).
Oct 2024 — Llama-3.1-Nemotron-70B: Strong post-training on Llama 3.1 70B.Model (huggingface.co/nvidia/Llama-3.1-Nemotron-70B-I…),huggingface.co/collections/nvidia/llama-31-nem…. Key dataset — huggingface.co/datasets/nvidia/HelpSteer2,paper (arxiv.org/abs/2410.01257).
Mar–Jun 2025 — Nemotron-H: First hybrid Mamba-Transformer models for inference efficiency.Paper (arxiv.org/abs/2504.03624),research page (research.nvidia.com/labs/adlr/nemotronh),developer.nvidia.com/blog/nemotron-h-reasoning…. Models:huggingface.co/nvidia/Nemotron-H-8B-Base-8K,ht… (huggingface.co/nvidia/Nemotron-H-4B-Instruct-1…).
May 2025 — Llama-Nemotron: Efficient reasoning models built ontop of Llama (still!). Paper (arxiv.org/abs/2505.00949).
Sep 2025 — Nemotron Nano 2: 9B hybrid for reasoning, continuing to improve in performance. 12B base on 20T tokens (FP8 training) pruned to 9B for post-training.research.nvidia.com/labs/adlr/files/NVIDIA-Nem…, V2 collection (huggingface.co/collections/nvidia/nvidia-nemot…).
Nov 2025 — Nemotron Nano V2 VL: 12B VLM. Report (research.nvidia.com/labs/adlr/files/NVIDIA-Nem…).
Dec 2025 — Nemotron 3: Nano/Super/Ultra family, hybrid MoE, up to 1M context. Super/Ultra H1 2026. Nano: 25T tokens, 31.6B total / ~3.2B active, releases recipes + code + datasets.nvidianews.nvidia.com/news/nvidia-debuts-nemot… Paper (arxiv.org/abs/2512.20856),Technical Report (arxiv.org/abs/2512.20848). Models:Nano-30B-BF16 (huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-3…),huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-3….
Nemotron’s Recent Datasets
NVIDIA began releasing substantially more data in 2025, including pretraining datasets — making them one of few organizations releasing high-quality pretraining data at scale (which comes with non-negligible legal risk).
Pretraining Data
Collection (huggingface.co/collections/nvidia/nemotron-pre…) —huggingface.co/datasets/nvidia/Nemotron-CC-v2,…. Math paper:arXiv:2508.15096 (arxiv.org/abs/2508.15096).
Post-Training Data
Core post-training dumps (SFT/RL blends):
* Llama Nemotron Post-Training v1.1 (huggingface.co/datasets/nvidia/Llama-Nemotron-…) (Apr 2025)
* Nemotron Post-Training v1 (huggingface.co/datasets/nvidia/Nemotron-Post-T…) (Jul 2025)
* Nemotron Post-Training v2 (huggingface.co/datasets/nvidia/Nemotron-Post-T…) (Aug 2025)
2025 reasoning/code SFT corpora:
* huggingface.co/datasets/nvidia/OpenMathReasoning (Apr 2025)
* huggingface.co/datasets/nvidia/OpenCodeReasoning (Apr 2025),huggingface.co/datasets/nvidia/OpenCodeReasoni… (May 2025)
* huggingface.co/datasets/nvidia/AceReason-1.1-SFT (Jun 2025)
* huggingface.co/datasets/nvidia/Nemotron-Math-H… (Jun 2025),huggingface.co/datasets/nvidia/Nemotron-PrismM… (Apr 2025)
NeMo Gym RLVR datasets:Collection (huggingface.co/collections/nvidia/nemo-gym)
Nemotron v3 post-training (Dec 2025):Collection (huggingface.co/collections/nvidia/nemotron-pos…)
HelpSteer (human feedback/preference):
* huggingface.co/datasets/nvidia/HelpSteer (Nov 2023)
* huggingface.co/datasets/nvidia/HelpSteer2 (Jun 2024)
* huggingface.co/datasets/nvidia/HelpSteer3 (Mar 2025)
And others, not linked here.
Chapters
* 00:00:00 Intro & Why NVIDIA Releases Open Models
* 00:05:17 Nemotron’s two jobs: systems R&D + ecosystem support
* 00:15:23 Releasing datasets, not just models
* 00:22:25 Organizing 500+ people with “invitation, not control”
* 0:37:29 Scaling Nemotron & The Evolution of Megatron
* 00:48:26 Career Reflections: From SVMs to DLSS
* 00:54:12 Lessons from the Baidu Silicon Valley AI Lab
* 00:57:25 Building an Applied Research Lab with Jensen Huang
* 01:00:44 Advice for …