Why Evaluation Is Becoming Unstable

Evaluation has always been the foundation of AI progress. Without reliable ways to measure performance, improvement cannot be tracked or compared.

Today, this foundation is becoming unstable.

Benchmarks are created, quickly saturated, and replaced. Metrics lose meaning faster than before. Results depend more on the specific test used than on any stable notion of capability.

This creates a new situation: evaluation is no longer a fixed background. It becomes part of the system that is constantly changing.

When evaluation changes too quickly, it cannot serve as a stable reference. Scores become difficult to interpret. Comparisons lose clarity. Confidence in results decreases.

The problem is not only that models are improving. It is that the tools used to measure them cannot keep up.

AI progress is therefore no longer just a question of capability. It is increasingly a question of whether evaluation itself can remain reliable under rapid change.

Read the Structural Note (SN-01) for the complete framework.