Meta-Learning Beyond MAML

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Learning to learn

Meta-learning asks a different question than ordinary supervised learning. Instead of 'how do I learn this one task well,' it asks 'how do I learn a strategy that lets me adapt quickly to a new, related task given only a handful of examples.' The canonical setting is few-shot learning: a model is meta-trained across a distribution of many small tasks, so that at test time, given a brand-new task with just a few labeled examples, it can adapt rapidly. MAML (Model-Agnostic Meta-Learning) became the reference algorithm for this — it meta-learns an initialization such that a few steps of ordinary gradient descent, on any new task drawn from the same distribution, lands at a good solution.

Where MAML strains: cost and second-order gradients

MAML's elegance comes at a real cost. Its outer-loop update requires differentiating through the inner-loop gradient steps themselves — a second-order gradient computation — which is memory- and compute-intensive, especially as the number of inner steps or the model size grows. This motivated a wave of methods that keep MAML's goal (fast adaptation from few examples) while sidestepping its most expensive machinery.

MAML and its successors

Method	Core idea	Avoids second-order gradients?	Tradeoff
MAML	Meta-learn an initialization; adapt via a few full gradient steps	No	Most general, most expensive
First-Order MAML (FOMAML)	Drop second-order terms, use first-order approximation	Yes	Cheaper, slightly less precise adaptation
Reptile	Repeatedly move initialization toward post-adaptation weights, no explicit gradient-through-gradient	Yes	Simple to implement, comparable performance to FOMAML
Prototypical Networks	Learn an embedding space; classify by distance to class prototypes	N/A — not gradient-based adaptation	No inner-loop optimization at all; very fast inference
Meta-learned optimizers (e.g. learned LSTM update rules)	Replace SGD with a learned update function	Varies	Can adapt in fewer steps, but the meta-optimizer itself is complex to train

Metric-based meta-learning: skip the inner loop entirely

A structurally different family — metric-based or 'non-parametric' meta-learning — avoids gradient-based adaptation altogether. Prototypical Networks learn an embedding function such that, for any new few-shot task, you simply average the embeddings of each class's support examples to get a 'prototype,' then classify new points by nearest prototype (typically under Euclidean or cosine distance). There's no inner-loop gradient descent at test time at all — adaptation is just a few vector averages, making inference extremely cheap. Matching Networks and Relation Networks are close relatives, differing mainly in how the distance or similarity function between query and support examples is computed.

prototypical_network.py

python

import torch
import torch.nn.functional as F

def prototypical_loss(embed_fn, support_x, support_y, query_x, query_y, n_classes):
    support_emb = embed_fn(support_x)   # [n_support, d]
    query_emb = embed_fn(query_x)       # [n_query, d]

    # Prototype = mean embedding per class (no gradient-based adaptation at all)
    prototypes = torch.stack([
        support_emb[support_y == c].mean(dim=0) for c in range(n_classes)
    ])

    # Classify queries by negative distance to each prototype
    dists = torch.cdist(query_emb, prototypes)   # [n_query, n_classes]
    log_probs = F.log_softmax(-dists, dim=1)
    return F.nll_loss(log_probs, query_y)

Large language models blurred the line with in-context learning

Few-shot prompting of large language models — showing a handful of examples in the prompt and getting good task performance without any weight updates — is, functionally, a form of meta-learning that emerged as a byproduct of large-scale pretraining rather than an explicit meta-learning objective. It has displaced classical meta-learning methods for many NLP few-shot tasks, though classical meta-learning remains important in domains like robotics, few-shot vision, and settings where in-context examples don't fit in a context window or where weight adaptation is specifically required.

Meta-learning methods are sensitive to task distribution mismatch

Every meta-learning method assumes test-time tasks are drawn from roughly the same distribution as meta-training tasks. When a genuinely novel task falls outside that distribution, fast adaptation can fail badly — the learned initialization or embedding space simply wasn't built to generalize there. This out-of-distribution brittleness is one of the main practical limitations holding classical meta-learning back from broader deployment.

What's next?

Meta-learning and the broader few-shot adaptation problem connect directly to the practical question every team eventually faces when deploying a capable base model: given limited labeled data for a new task, is it better to adapt via a handful of in-context examples, or to actually update the model's weights through fine-tuning.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →