Hot Takes from NeurIPS 2023!
Instant reactions from the Thirty-seventh Conference on Neural Information Processing Systems in New Orleans, LA.
Last week I had the pleasure to attend NeurIPS for the first time! Like last year’s conference it was set in New Orleans, LA U.S.A., at the Ernest N. Morial Convention Center. Somewhere north of 6,000 papers and 15,000 AI nerds researchers and professionals in artificial intelligence. It was an absolutely overwhelming experience.
I’m planning a follow-up guide for n00bs/noobs/nubs on how to get the most out of a major ML conference. For now, here are some of the highlights, hot takes, and themes I observed at this mega-conference.1
Themes
In future posts I’ll dive into many of the topics below, but here are some of the major research areas from this year’s conference (in non-wordcloud form) that I had the chance to explore. I had the privilege of speaking with many authors in these areas during the poster sessions, which I found the most valuable part of this AI-fest.
Efficient training of deep learning models. It’s not clear to me what is the most important thing to work on right now—making the current suite of models much more efficient; trainable on consumer-grade hardware even. Or is the single most important thing to push ahead on capabilities of these models and work on the plumbing as needed. I’m glad the research community is doing both. I cannot shake the suspicion that making models much cheaper to train and deploy is actually necessary to making them more capable.
The QLoRA oral talk and poster session were extremely, predictably, popular:
All the things Diffusion.
What ARE language models? There were quite a few papers with titles that seem to once-and-for-all declare the identity of AI models—Language Models are Weak Learners, Foundation Model is Efficient Multimodal Multitask Model Selector, etc.2
Graphs, GNNs.
Language, Vision… Vision and Language.
Time series! As someone with a lot of interest and background in time series analysis I was heartened and frankly surprised to see this area well-represented at NeurIPS. For now, forecasting of large-scale, high-dimensional, non-stationary time series that arise from complex, dynamic, and even in some cases adversarial systems has not been shredded by the foundation model buzzsaw.
Causal structure learning from observational data. This is very much related to the time series challenges I noted above.
Classic ML. Plenty of folks are still working on tabular data learning, k-NN algorithms, boosting trees, etc. Good for them.
Federated learning, which I know next to nothing about.
Benchmarks and Datasets. My original take was boredom with the abundance of benchmarks and evaluation datasets presented. At some point however I had a π rads change. Hot take #1: We need way more benchmark datasets in AI, across many more types of tasks…LIKE THIS ONE! As one presenter said, now you don’t have an excuse not to work on AI for climate research.
Theory. I like AI theory papers; they have actual math; I like math.
I can’t wait to do a deep dive into Reconsidering Overfitting in the Age of Overparameterized Models. This talk really helped me understand why the bias-variance “trade-off” seems to not be much of a trade-off.
Some of things I didn’t have the time/inclination to explore:
RL/Bandits
Optimization techniques.
Bayesian optimization techniques.
Semi-supervised and unsupervised learning methods
Science x AI (AI for physics, AI for Biology, Materials, etc.)
Causal inference in the experimentation setting. Hot take #2: If you can run interventions you have it easy and I am bored.
Bias, fairness, privacy etc. Although I did get a chance to stop by the poster for Evaluating the Moral Beliefs Encoded in LLMs, a fascinating paper that’s made waves over the latter half of this year. Another fascinating paper bringing the paradigm of experimental moral philosophy to AI in this way is Moral Foundations of Large Language Models. Hot take #3: Trolley problems are cool again!3
Hot take #Ω: There was surprisingly little research on robotics but I predict next year, as foundation models become embodied, there will be an explosion of papers embodying language and vision models into robots. See RT-2: Vision-Language-Action Models from Google DeepMind.4
Highlights
An excellent panel spanning lots of topics around AI featuring the Nerds’ Historian, Walter Isaacson.5
The Test of Time Award for Word2Vec (2013). Jeff and Greg gave a great acceptance speech, I cannot imagine why Ilya didn’t attend…
New Orleans:
Dictated, but not read.
Can’t we just ask them how they identify?
If you would flip the switch and murder that one guy then… No, you can’t sit with us.
Is this “the first of a new generation of God's children…the face of the shape of things to come”?
Scuttlebutt is that he’ll do a bio of Demis Hassabis in the not-too-distant whenever 🤩