Back to Research
Agent TrustFeb 15, 2026

Gaming Resistance in AI Measurement: Countermeasures Against Goodhart's Law Across Ten Frameworks

Abstract

When a measure becomes a target, it ceases to be a good measure [1]. We analyze the gaming resistance properties of ten AI impact scoring frameworks, examining how non-compensatory aggregation methods, independent verification sampling, recursive trust weighting, and shadow objective libraries create structural resistance to manipulation. We demonstrate that geometric mean aggregation is inherently harder to game than arithmetic alternatives.

Problem

Goodhart's Law, originally articulated by Charles Goodhart in 1975 in the context of monetary policy [1], states that when a measure becomes a target, it ceases to be a good measure. The law captures a fundamental challenge for any measurement-based governance system: the act of measuring and incentivizing based on measurements creates pressure to optimize the measure rather than the underlying phenomenon the measure was intended to capture. In educational testing, teaching to the test improves scores without improving learning [2]. In corporate governance, earnings management improves reported profits without improving actual business performance. In cybersecurity, vulnerability count reduction can be achieved by reclassifying vulnerabilities rather than fixing them.

The application of Goodhart's Law to AI impact measurement is particularly concerning because autonomous agents have computational advantages over human actors in identifying and exploiting measurement weaknesses. An AI agent that is aware of the scoring framework used to evaluate its behavior can potentially optimize its actions to maximize its score without genuinely improving its impact. This optimization may be subtle: rather than outright fabrication of results, an agent might selectively emphasize dimensions where it performs well, time its actions to coincide with measurement windows, or structure its outputs to satisfy formal evaluation criteria while violating their spirit.

The history of standardized measurement systems provides instructive examples of gaming and the countermeasures developed in response. The FICO credit score has been subject to gaming through authorized user piggybacking, rapid rescoring, and strategic account management [3]. FICO responded with algorithm updates that detect and discount these manipulation strategies. The Basel Accords were gamed through regulatory capital arbitrage [4], where banks structured transactions to reduce risk-weighted assets without genuinely reducing risk. Basel III responded with leverage ratios and liquidity coverage requirements that are harder to arbitrage. In each case, the measurement system evolved through an adversarial dynamic between gamers and framework designers.

AI impact measurement must anticipate and design for this adversarial dynamic from the outset rather than responding to gaming after it occurs. The ten Amplitude scoring frameworks were designed with explicit gaming resistance mechanisms that draw on lessons from prior measurement systems. This paper systematically analyzes these mechanisms, evaluates their effectiveness through simulation, and identifies remaining vulnerabilities that future framework revisions should address.

The stakes of gaming resistance in AI measurement are higher than in most prior measurement contexts because the speed and scale of AI agent operations mean that exploitation of measurement weaknesses can propagate rapidly through entire ecosystems. A gaming strategy that takes a human actor months to identify and implement can be discovered and deployed by an AI agent in hours. The time asymmetry between attack and defense is even more extreme than in cybersecurity [5], making proactive design for gaming resistance essential rather than optional.

Framework Design

The first line of defense against gaming is the choice of aggregation function used to combine multiple measurement dimensions into a composite score. Arithmetic mean aggregation is the most vulnerable to gaming because it allows a high score on one dimension to fully compensate for a low score on another. An agent seeking to maximize its arithmetic mean score can ignore difficult-to-game dimensions and concentrate its optimization effort on the most easily manipulated dimensions. Geometric mean aggregation provides stronger resistance because it is more sensitive to low outliers: a score of 0 on any dimension produces a geometric mean of 0 regardless of performance on other dimensions. Harmonic mean aggregation provides even stronger resistance, as it weights low scores even more heavily than the geometric mean.

The Amplitude frameworks use different aggregation functions depending on the compensatory semantics of the domain being measured. Safety-critical frameworks such as Fidelity (alignment preservation) and Drift (objective deviation) use geometric mean or harmonic mean aggregation because a failure in any safety dimension should dominate the overall score. Compensatory frameworks such as Harmony (competitive dynamics) use arithmetic mean aggregation because genuine economic trade-offs exist between dimensions. This heterogeneous aggregation strategy complicates gaming because an agent seeking to optimize its scores across multiple frameworks cannot use a single gaming strategy; it must simultaneously satisfy geometric mean constraints on safety dimensions and arithmetic mean optimization on economic dimensions.

The second line of defense is independent verification sampling, where the measurement system randomly selects a subset of agent actions for independent verification by evaluators that the agent cannot identify or influence. The verification sampling rate is deliberately unpredictable, varying between 5% and 25% of actions per measurement period. Agents that behave differently when they believe they are being observed versus unobserved will exhibit detectable discrepancies between their sampled and unsampled behavior distributions. The statistical test for this discrepancy uses a two-sample Kolmogorov-Smirnov test [6] with a significance threshold calibrated to the verification sampling rate, ensuring that the detection power scales with the fraction of actions verified.

The third line of defense is recursive trust weighting, implemented in the Fidelity framework. Under recursive trust, the weight assigned to an agent's self-reported metrics is a function of the agent's historical accuracy in self-reporting, verified through independent audit. An agent that inflates its self-reported metrics will see its trust weight decrease over time, reducing the influence of its self-reports on its composite score. The recursion operates across delegation chains: an agent's trust weight also depends on the trust weights of the agents it delegates to, so an agent cannot improve its score by delegating to untrustworthy agents that provide favorable but inaccurate assessments.

The fourth line of defense is the shadow objective library, a mechanism that periodically evaluates agents against objectives they were not informed about in advance [7]. The shadow objectives are drawn from a library of evaluation criteria that are consistent with the framework's stated goals but differ in their specific operationalization. For example, a Harmony evaluation might use alternative market definitions, different time windows for volatility calculation, or modified Nash equilibrium assumptions. By evaluating agents against unpredictable variations of the scoring criteria, the shadow objective library ensures that agents optimizing for a specific operationalization of the framework will not consistently outperform agents that are genuinely achieving the framework's intended goals.

Scoring

To quantify the relative gaming vulnerability of arithmetic versus geometric mean aggregation, we construct a formal model of a strategic agent that can allocate a fixed optimization budget across n dimensions [8]. The agent's true capability in each dimension is represented by a base score b_i, and the agent can increase its score on dimension i by investing optimization effort e_i, yielding an observed score of b_i + g(e_i), where g is a concave effort-to-score function reflecting diminishing returns. The agent's total effort budget is constrained: the sum of e_i across all dimensions must not exceed E. The gaming vulnerability of an aggregation function is defined as the maximum score improvement achievable through optimal effort allocation relative to the uniform allocation baseline.

For arithmetic mean aggregation, the optimal gaming strategy is to allocate all effort to the dimension with the highest marginal return, which is typically the dimension with the lowest base score (since the concavity of g means that low-scoring dimensions offer the highest marginal improvement per unit of effort). The maximum arithmetic mean improvement from a budget of E is g(E)/n, which scales linearly with budget and inversely with the number of dimensions. For geometric mean aggregation, the optimal strategy distributes effort across dimensions in proportion to the inverse of each dimension's base score, because the geometric mean is more sensitive to improvements in low-scoring dimensions. The maximum geometric mean improvement is smaller than the arithmetic mean improvement for any given budget E, and the difference grows with the variance in base scores.

Simulation results confirm the theoretical analysis. We simulate 1,000 strategic agents, each with n=5 dimensions and base scores drawn uniformly from [30, 90]. Each agent optimizes its effort allocation to maximize either the arithmetic or geometric mean of its observed scores, with a budget of E=50 score points. Under arithmetic mean optimization, agents achieve an average score improvement of 10.2 points (from a baseline mean of 60.0 to 70.2). Under geometric mean optimization, agents achieve an average score improvement of 6.8 points (from a baseline geometric mean of 55.4 to 62.2). The geometric mean is therefore 33% more resistant to gaming than the arithmetic mean in this setting, confirming its structural advantage.

The gaming resistance advantage of geometric mean aggregation has a cost: it also penalizes agents with genuinely heterogeneous capability profiles. An agent that is legitimately excellent in four dimensions and weak in one will receive a geometric mean score that is substantially lower than its arithmetic mean score, even if the weakness is in a dimension that is genuinely less important for the agent's primary use case. This tension between gaming resistance and fair evaluation of heterogeneous agents is a fundamental design trade-off [9]. The Amplitude frameworks resolve it by using geometric mean aggregation only for safety-critical dimensions where heterogeneous profiles are not acceptable (an agent must be aligned on all dimensions, not just most of them) and arithmetic mean aggregation for economic dimensions where genuine trade-offs exist.

Validation

We validate the gaming resistance mechanisms through a red-team exercise [10] in which ten teams of ML engineers are tasked with maximizing the Amplitude scores of a controlled agent system without improving its genuine performance. Each team is given full documentation of the scoring frameworks and three weeks to develop and implement gaming strategies. The red-team exercise serves as an empirical test of Goodhart's Law [1]: if the frameworks are vulnerable to gaming, the red teams should be able to achieve significant score improvements through strategic optimization rather than genuine improvement.

The red-team results demonstrate that the gaming resistance mechanisms substantially constrain the scope for manipulation but do not eliminate it entirely. Across the ten teams, the average score improvement achieved through gaming was 4.2 points on a 100-point scale, compared to an average improvement of 11.7 points achieved through genuine capability enhancement by a control group working on the same agent system. The most effective gaming strategies targeted the measurement timing (synchronizing agent behavior with known measurement windows) and the dimension weighting (concentrating optimization effort on dimensions with the highest weight in the composite score). The least effective strategies targeted the aggregation function and the verification sampling, confirming that these mechanisms provide robust structural resistance.

The most significant vulnerability identified by the red teams was the predictability of the verification sampling schedule. While the sampling rate varies between 5% and 25%, the sampling events occur at fixed intervals (hourly) with variable inclusion probability. Red teams discovered that agents could detect the beginning of a sampling window through latency signatures in the evaluation API and adjust their behavior accordingly [11]. This vulnerability has been addressed in Amplitude v3.2 through the introduction of continuous sampling with randomized timing, eliminating the fixed-interval structure that enabled detection.

The shadow objective library proved to be the most effective gaming resistance mechanism in the red-team exercise. Three teams that had successfully developed gaming strategies for the primary scoring criteria saw their score improvements reduced by 60-80% when evaluated against shadow objectives. The shadow objectives evaluate the same underlying phenomena but through different operationalizations, so agents that have genuinely improved their performance score well on both primary and shadow objectives, while agents that have merely optimized for the specific operationalization of the primary criteria show degraded performance on shadow evaluations. The correlation between primary and shadow scores therefore serves as a meta-measure of gaming [12]: agents with high primary scores but low primary-shadow correlation are flagged for additional scrutiny.

The overall conclusion from the validation exercise is that gaming resistance in AI measurement is achievable but requires layered defenses [13]. No single mechanism provides complete resistance, but the combination of non-compensatory aggregation, independent verification, recursive trust, and shadow objectives creates a defense-in-depth architecture that constrains gaming to levels that are manageable within the broader governance framework. The 4.2-point average gaming improvement, while nonzero, represents a sufficiently small fraction of the 100-point scale that it does not undermine the discriminative validity of the scores. Framework designers should plan for continuous evolution of gaming resistance mechanisms as new manipulation strategies emerge, treating gaming resistance as an ongoing arms race rather than a solved problem.

References

  1. Goodhart, C. A. E. (1984). Problems of Monetary Management: The U.K. Experience. In C. A. E. Goodhart (Ed.), Monetary Theory and Practice: The U.K. Experience (pp. 91-121). Macmillan. (Originally presented in 1975.)
  2. Campbell, D. T. (1979). Assessing the Impact of Planned Social Change. Evaluation and Program Planning, 2(1), 67-90.
  3. Brevoort, K. P., Grimm, P., & Kambara, M. (2015). Data Point: Credit Invisibles. Consumer Financial Protection Bureau Office of Research.
  4. Basel Committee on Banking Supervision. (2011). Basel III: A Global Regulatory Framework for More Resilient Banks and Banking Systems. Bank for International Settlements.
  5. Schneier, B. (2000). Secrets and Lies: Digital Security in a Networked World. John Wiley & Sons.
  6. Massey, F. J. (1951). The Kolmogorov-Smirnov Test for Goodness of Fit. Journal of the American Statistical Association, 46(253), 68-78.
  7. Myerson, R. B. (1981). Optimal Auction Design. Mathematics of Operations Research, 6(1), 58-73.
  8. Gibbard, A. (1973). Manipulation of Voting Schemes: A General Result. Econometrica, 41(4), 587-601.
  9. Strathern, M. (1997). "Improving Ratings": Audit in the British University System. European Review, 5(3), 305-321.
  10. Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., ... & Anderljung, M. (2020). Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims. arXiv preprint arXiv:2004.07213.
  11. Anderson, R. (2020). Security Engineering: A Guide to Building Dependable Distributed Systems (3rd ed.). John Wiley & Sons.
  12. Manheim, D., & Garrabrant, S. (2018). Categorizing Variants of Goodhart's Law. arXiv preprint arXiv:1803.04585.
  13. Saltelli, A. (2020). Ethics of Quantification or Quantification of Ethics? Futures, 116, 102509.