When a measure becomes a target, it ceases to be a good measure.
Marilyn Strathern’s formulation of Goodhart’s law
While reading the literature on generative models of T cell receptors, with a background in robot learning and inverse reinforcement learning, I expected the move to require a new set of methods. It required instead a new vocabulary for a problem I had already met several times. What follows is a short high level summary of what transfers, what does not, and which terminology makes this literature difficult to read from the outside.
These notes come out of a collaboration with Institut Curie, and more precisely out of a series of enriching coffees, or teas, with Philippe Martin, whom I first met in Montreal thanks to Josh Waterfall and then again in Paris, where the machine learning and immunology talk alternates reliably with apartment renovations, which parquet to choose and how to restore it, and the rest of the thirty-something catalogue.
The immunology these notes assume, together with the three research lines they refer to and a reading path, is set out separately in T Cell Receptors in a Nutshell.
What actually transfers
My first instinct was that sequential decision making would be the bridge. Treating a patient is a sequence of decisions: which therapy to give, when to switch, how to respond as the tumour adapts. That is a real sequential problem, and it is modelled as one. It is not, however, the problem in front of me. Designing a receptor is a single choice scored once, and most work in therapeutic design does not run a policy at all; it trains a generative model on observed data and steers it at sampling time.
My second instinct was that reward design is the thing that transfers, since in both domains the objective has to be learned rather than written down. That instinct is premature rather than wrong.
Reward hacking is what happens when something is optimised hard against a learned scorer. The scorer has blind spots, and a search that pushes far enough will find them and exploit them in place of solving the task, so the size of the effect follows the force of the search. Christiano et al. (2017) give the clearest demonstration: a robot hand trained to pick up a ball from human judgements of short video clips, rather than from a hand-written reward, learned to position itself between the camera and the ball, where it only appeared to be grasping, and the raters approved of what they saw. Brian Christian recounts the episode in The Alignment Problem. The learned scorer could not tell grasping from the appearance of grasping, and a search with enough force behind it found that out. Steering a generator at sampling time does not push hard. It tilts the output somewhat in the direction the classifier approves of, without going looking for the classifier’s mistakes. A binding classifier trained on a few thousand examples certainly has blind spots, and a mild tilt is unlikely to land on them.
This changes once the generator is trained against the classifier rather than merely steered by it, which is what applying preference learning or reinforcement would amount to here. The generator is then searching explicitly for whatever the classifier scores highest, and the classifier’s mistakes become the target rather than an accident. Reward design becomes the right thing to worry about at that point, and not before.
What the literature reports failing, meanwhile, is measurement. The failure modes on record are inflated results traced back to the way the data was divided, classifiers that learned how the negative examples were constructed rather than any biology, and models that collapse on peptides they were not trained on. None of these are failures of objective specification.
What transfers first is narrower and less glamorous than reward design. It consists of knowing that a split over rows leaks when those rows are clonally related, that a classifier can score well by detecting which pairs were assembled at random, that a new model means little until it has been compared against one that knows strictly less, and that an ablation which ought to fail, such as masking the peptide and confirming that performance drops, is more informative than another which ought to succeed. Suspicion directed at one’s own metrics is the durable instinct, and where it was acquired does not much matter.
A translation table
This is the part I consult most often. The third column matters more than the first two: the question is never whether two words correspond, but how far the correspondence can be pushed before it fails.
| Term you know | What it corresponds to here | How far it transfers |
|---|---|---|
| grouped cross-validation | splitting by donor, peptide or receptor | Transfers. The strongest one here, and the point at which the field documents itself failing. Each grouping answers a different question, and a split over rows answers none of them. |
| classifier guidance | guided or steered generation | Transfers. Almost unchanged, and probably the most useful thing to bring from the diffusion and language modelling literature. |
| behaviour cloning | training a generator on repertoire data | Transfers. This is the honest name for what most TCR generative models do. |
| distribution shift | unseen peptide, unseen HLA | Transfers. And it is the central failure reported in this literature: models that look strong collapse on held-out peptides. |
| offline RL | design without an experimental loop | Transfers, and worse. Offline RL at least has logged rewards. Here the true reward is often unavailable at any price. |
| safety constraint | autoimmunity risk | Transfers, with the stakes inverted. Not a regulariser. It is the mechanism by which patients are harmed. |
| policy | the generative model itself | Partly. There is no environment and no state transition. A bandit over sequences (each candidate is proposed, scored on its own, and nothing carries over to the next one) is a closer object than a Markov decision process. |
| trajectory | a course of treatment over time | Partly. Treatment really is a sequence of decisions, so the analogy holds there. It does not hold for the receptor: generating a sequence left to right is the chain rule, not a replay of how the molecule was built, which joins D to J first and then V to DJ. |
| expert demonstrations | observed repertoires | Partly. The repertoire was selected by survival rather than by competence at the task you care about, so the expert was optimising something else. |
| reward model | a binding or immunogenicity classifier | Partly. Trained on small, biased data covering a narrow slice of peptide space, so much weaker than a reward model fitted for reinforcement learning from human feedback. |
| exploration | library diversity | Partly. Novelty is cheap and functional novelty is not, and distance from the training data is a poor proxy for the second. |
| simulation to reality gap | in-silico structure to in-vitro assay | Partly. Closer than it first looks, since optimising against a structure predictor means optimising against an imperfect model of reality. What does not carry over are the remedies: domain randomisation and system identification both assume cheap access to the real system. |
| reward hacking | cross-reactivity, off-target binding | Not yet. Cross-reactivity comes from the proteome, i.e., the whole set of proteins a body makes, containing peptides that resemble one another, rather than from an optimiser exploiting a learned reward. It becomes the same thing once a generator is trained against a classifier. |
What does not carry over
- The simulators are not ground truth. In-silico oracles do exist, including structure prediction, docking and molecular dynamics, and projects lean on them. What is missing is a cheap oracle you can trust, because the real answer comes from an experiment with a cost and a calendar. A method that assumes you can query the true reward a million times has to be redesigned, not ported. That expense is also why reward hacking is not yet this field’s characteristic failure: no loop runs tight enough to produce it.
- Data scale is the binding constraint, not architecture. The lesson repeated throughout this literature is that model capacity rarely limits performance, whereas access to large, well-annotated and properly split data does. The ImmSET authors report performance scaling consistently with data volume across several data types, which is the same lesson stated as a curve.
- The objective is plural. Bind the target, avoid everything else, remain stable, be manufacturable, and do not provoke autoimmunity. This is not a single scalar with a Lagrange multiplier attached.
- Biology offers conserved structure to exploit. B cells and antibodies arise from the same lineage through a comparable recombination mechanism, so methods frequently transfer across receptor types. Furthermore, considerably more data exists on the antibody side, which makes it a reasonable place to validate a method before bringing it back.
What I would say to myself in the first week
The vocabulary of immunology can be picked up quickly, the judgement cannot, and the distance between the two is where someone arriving from outside does damage. What a methods background supplies is not a better architecture but a trained reflex for asking what a number was actually measured on. Hence it is worth more here than another model class. If any of this is useful, take a look at the primary sources!
Written while starting out, so the biology above should be treated as a working summary. Corrections are welcome.
1 Comment