Andrej Karpathy recently shared a striking comparison highlighting the blistering pace of progress in AI efficiency. In 2019, OpenAI required 32 TPU v3 chips running for a full week to train GPT-2, costing approximately $43,000. By 2026, Karpathy demonstrates that the same performance can be achieved in just three hours on a single node of H100 processors. This 600x reduction in cost illustrates how rapidly the barriers to entry for high-performance modeling are dissolving.

The benchmark used for this comparison is the CORE score, an ensemble metric (a combined score from multiple specialized tests) introduced in the DCLM research paper. While the original GPT-2 hit a specific score threshold, modern optimizations—many originating from the community-driven "modded-nanogpt" project—allow developers to reach that same milestone for the price of a dinner. This trend suggests the financial investment required to develop a capable Language Model is falling by roughly 2.5x every year.

This evolution is driven by both hardware gains and software refinements that maximize silicon performance. As training becomes affordable, sophisticated models can be developed by small teams, shifting power away from tech giants. This 600x improvement reminds us that yesterday's cutting-edge breakthrough is today's affordable weekend project.

Andrej Karpathy recently shared a striking comparison highlighting the blistering pace of progress in AI efficiency. In 2019, OpenAI required 32 TPU v3 chips running for a full week to train GPT-2, costing approximately $43,000. By 2026, Karpathy demonstrates that the same performance can be achieved in just three hours on a single node of H100 processors. This 600x reduction in cost illustrates how rapidly the barriers to entry for high-performance modeling are dissolving.

The benchmark used for this comparison is the CORE score, an ensemble metric (a combined score from multiple specialized tests) introduced in the DCLM research paper. While the original GPT-2 hit a specific score threshold, modern optimizations—many originating from the community-driven "modded-nanogpt" project—allow developers to reach that same milestone for the price of a dinner. This trend suggests the financial investment required to develop a capable Language Model is falling by roughly 2.5x every year.

This evolution is driven by both hardware gains and software refinements that maximize silicon performance. As training becomes affordable, sophisticated models can be developed by small teams, shifting power away from tech giants. This 600x improvement reminds us that yesterday's cutting-edge breakthrough is today's affordable weekend project.