There’s no data like less data?
The current paradigm appears sufficient to automate a large portion of digital knowledge work, with an economic impact measured in trillions. Not bad! But researchers like to go from zero to one, and are asking themselves what’s missing in the current paradigm. One answer is data efficiency: paraphrasing Yann Lecun, “it takes a human 10 hours to learn to drive, while Telsa FSD has to watch millions of hours of video.” So here’s my thoughts on approaches to sample complexity reduction.
Multi-task Learning
The basic idea: “you aren’t learning just one task, you are learning many tasks that share aspects, so while you need a lot of data in aggregate, the per-task amount of data that you need can be small.” The sharing can be realized via world models, representations, priors, objectives, or constraints. Whether or not to share aspects between tasks can potentially be learned from data.
The most promising directions in my opinion are:
World Model: If you are solving many tasks in the same environment, the environment structure (dynamics + observation model) is reusable across tasks. In some cases you can hand specify a world model (e.g., self-driving uses video game engines and curated scenarios; and there are high-fidelity simulations of Tokamaks), but bitter lesson says you want to learn the world model. Given a world model, you need an algorithm that can exploit it for different goals or objectives (i.e., planning, reasoning, or control). Extreme bitter lesson says you shouldn’t specify how this exploitation works either, but somehow learn it from data.
Meta-Learning: Learn a strategy over tasks that itself learns quickly on each individual task. The most bitter-lesson-pilled version of this is “learning a learning algorithm”, which has arguably already happened in language model pretraining (i.e., ICL arising because language modeling is actually a multi-task problem which demands a dynamic regret algorithm to achieve optimal perplexity). However ICL is implemented in a static computation graph, and the hope is test-time-training (TTT) is equivalent to a richer computation graph that could be even more sample efficient (analogous to the explosion in LLM capability around dynamic test-time compute).
Active Data Acquisition
The basic idea: if you let the model choose the data to sample, you can get away with using less data. This umbrella term includes topics like active learning and efficient exploration.
Information gain: Classic active learning with classifiers was disagreement based: if all historically good classifiers agree on the label, you don’t need to see it. This is a special case of Bayesian experimental design, where a “good experiment” is one that causes a big change in the posterior over hypotheses (“information gain”).
Error-driven: Information gain assumes a well-specified hypothesis class, with multiple competing hypotheses that are just as good on historical data but disagree on novel predictions. However sometimes the best hypotheses historically are failing to predict a reliable observation, e.g., the ultraviolet catastrophe. This leads to error-driven, surprise driven, or learning-progress driven data selection strategies. It’s tricky because one does not want to focus on trying to explain irreducible noise. But contextual bandit exploration can be efficiently implemented via a learning progress strategy, ensuring either good task performance or large learning progress via a constructive bound on the DEC.
To bitter lesson pill this, one would have to set up a series of problems where the algorithm has to choose data to acquire and then gets rewarded for good data selection. That last part (“what is the reward for data acquisition”) is the tricky part, but if tasks can be synthesized at scale, downstream task performance for a budget of experimentation could be the reward for data acquisition.
Unsupervised and self-supervised learning
The basic idea: “although my learning strategy is data inefficient, I have an unlimited amount of data, so it doesn’t matter”. This is true, clearly a good idea, and the basis for the current explosion in AI. It did radically reduce the sample complexity of subsequent downstream tasks, e.g., pre-train on predicting all of GitHub, then RL train for agentic coding performance. But when people complain about sample efficiency, they are partially complaining about this process. Plausibly 1) scaling pre-training reliably leads to better downstream general task performance, but 2) it’s increasingly difficult to scale pre-training; however 3) if a more sample efficient learning algorithm (TTT?) were pre-trained on the same data, it would be the equivalent of the old paradigm pre-training on a much larger amount of data, and therefore would create models with better downstream general task performance.
Architecture
This one has definitely worked in the past: I saw a workshop talk from the creator of Neurogammon, who noted in retrospect that the symmetry built into the weights of the network were worth about a factor of 20 in data reduction (equivalent performance without the weight tying did eventually occur on more modern computers).
On the one hand, architecture choices feel anti-bitter-lesson. On the other hand, no free lunch theorems indicate we will not progress on problems in general, and some prior over problems is necessary. The debate continues today, e.g., slotformer.
I view architecture choices as necessary but always worthy of skepticism.
Synthetic Data
This spans the above categories. World models can generate synthetic data (e.g., rollouts for counterfactual action sequences); self-supervised learning algorithms can generate synthetic data (e.g., mutating images in JEPA in order to create an equivalence class for the learning algorithm); meta-learning can generate synthetic data (e.g., early breakthroughs on ARC-AGI combined test-time-training with test-time data synthesis); and unsupervised learning nowadays leverages synthetic data (e.g., pre-training using data generated from the previous generation of models).
The idea of using (synthetic) data to encode constraints is quite old and still good. One nice thing about encoding priors as data: you can control the relative strength of your prior by how much synthetic vs. real data you use, and eventually let the real data dominate if enough of it accumulates.
Leave a Reply