Pangram Labs released code, models, and a dataset for its paper on AI token value. The paper studies scaling laws for wild AI-generated web text. It uses 96M English web documents labeled human, AI, or mixed. The work trains an 800-model grid on controlled text mixtures. It evaluates models and fits scaling laws to draw figures and tables. All materials are released under CC BY-NC-SA 4.0. Models and dataset are hosted on Hugging Face. The repository includes scripts to reproduce every figure and table. It also provides commands to retrain or rescore models. The paper is available as an arXiv preprint.
What it unlocks
Reproducing the paper's scaling laws for AI-generated web text using released code, models, and dataset.
- 96M English web documents
- 800-model grid
- 268M parameters
- 20 human tokens per parameter
What you need to act on it
- Python 3.12
- CUDA 12.8 for training
- Pangram API key for labeling
Sources