Sakana AI introduced PC-ALM, a way to train neural networks without backpropagation. Backpropagation sends error signals backward through a whole network in a fixed order. PC-ALM instead lets each layer talk only to its immediate neighbours. Each layer runs a small feedback controller that accumulates its own prediction error. Sakana AI trained residual networks up to 1000 layers on MNIST this way. Accuracy stayed within about two points of backpropagation across depths. The method also beat standard predictive coding on Fashion-MNIST, CIFAR-10 and Tiny ImageNet. The researchers say credit signals spread faster and more evenly than before. They note one loss: the settled activations no longer differ from the forward pass. The stated motive is understanding how brains assign credit, with possible use on energy-efficient neuromorphic chips.
What changed
Local predictive coding lost credit signal in deep, narrow networks and topped out near 128 layers.
What it unlocks
Training very deep networks using only neighbouring-layer communication, with code published.
- 1000 layers trained
- within ~2 points of backprop
- width 32 residual MLP
- T = 2L inference steps
What you need to act on it
- research setting
- small benchmark tasks and residual MLPs
Sources