Back to Research
Data ScienceAug 25, 2025

Measuring Data Source Value for AI Agents: A Counterfactual KL-Divergence Approach

Abstract

We propose a counterfactual protocol for measuring the decision impact of individual data sources on autonomous agent behavior. By comparing the KL divergence of an agent's output distribution with and without access to a specific data source, gated by a binary criticality test, we reduce the computational complexity of counterfactual evaluation from O(2^n) to O(1). This enables real-time assessment of data source value during agent inference, providing the quantitative foundation for data quality scoring in agentic AI systems.

Challenge

Autonomous AI agents consume data from multiple external sources during inference-time operations. A financial analysis agent may pull real-time market data, company filings, analyst reports, and macroeconomic indicators before producing an investment recommendation. A customer service agent may access CRM records, product documentation, order history, and knowledge base articles before generating a response. In each case, the agent's output is a function of the combined input from all data sources, and a natural question arises: how much does each individual data source contribute to the agent's decision?

The naive approach to answering this question is exhaustive counterfactual analysis [1]. For an agent consuming n data sources, the complete counterfactual evaluation requires computing the agent's output distribution for every possible subset of data sources: 2^n evaluations in total. Each evaluation requires a full inference pass through the agent, making the computational cost exponential in the number of data sources. For an agent consuming even a modest ten data sources, the complete counterfactual analysis requires 1,024 inference passes, which is computationally prohibitive for real-time applications and economically impractical for continuous monitoring.

Shapley value analysis, borrowed from cooperative game theory [2], offers a principled approach to attribution but suffers from the same computational complexity. Computing exact Shapley values for n data sources requires evaluating all possible orderings of source inclusion, which is factorial in n [3]. Approximation methods such as Monte Carlo sampling reduce the computational burden but introduce sampling variance and still require multiple inference passes per data source, making them unsuitable for real-time scoring during agent inference.

The challenge, therefore, is to develop a methodology that measures the decision impact of individual data sources with computational complexity that is constant in the number of sources, enabling real-time evaluation during agent inference without observable latency impact. This is not merely an optimization problem; it is a requirement for any measurement methodology that claims to operate at the speed of agentic AI systems.

Architecture

Our approach rests on two key innovations: the use of KL divergence as the measure of decision impact, and a binary criticality gate that reduces the evaluation space from combinatorial to constant. The KL divergence, D_KL(P || Q), measures the information-theoretic distance between two probability distributions [4]. In our framework, P represents the agent's output distribution with access to all data sources, and Q represents the output distribution without access to a specific data source. A high KL divergence indicates that removing the data source substantially changes the agent's decision, implying high decision impact.

The critical insight that enables constant-time evaluation is the binary criticality gate. Rather than computing the full counterfactual for every data source, we first apply a lightweight binary test that determines whether a data source is "critical" to the agent's current decision. The criticality test examines the attention weights [5], retrieval scores, or context window position of data from each source during the agent's standard inference pass. If a data source is not critical - meaning it receives negligible attention or contributes minimally to the retrieval context - its decision impact is set to zero without requiring a counterfactual inference pass.

For data sources that pass the criticality gate, we compute the KL divergence using a single additional forward pass through the agent with the critical source masked. This reduces the typical evaluation from 2^n full counterfactual passes to a small constant number of passes: one baseline pass plus one masked pass for each critical source. Empirically, the number of critical sources per inference is bounded by a small constant (typically 2-4 out of potentially dozens of available sources), making the total computational overhead O(1) in practice.

The mathematical formulation is as follows. Let S = {s_1, s_2, ..., s_n} be the set of data sources available to the agent. Let P(y | S) be the agent's output distribution over possible decisions y given access to all sources. For each source s_i, let C(s_i) be the binary criticality function that returns 1 if the source passes the criticality gate and 0 otherwise. The decision impact score for source s_i is then defined as D(s_i) = C(s_i) * D_KL(P(y | S) || P(y | S \ {s_i})) [6]. This formulation ensures that non-critical sources receive a score of zero without requiring counterfactual evaluation, while critical sources receive a score that quantifies their actual decision impact.

Implementation

The reduction from O(2^n) to O(1) complexity relies on two empirical properties that we validate across multiple agent architectures and task domains. The first property is criticality sparsity: in practice, only a small number of data sources are critical to any given agent decision. This is a consequence of the attention mechanisms [5] and retrieval algorithms [7] that govern how agents process multi-source input. Even when an agent has access to dozens of data sources, its decision for a specific query typically depends heavily on only a few sources, with the remainder contributing negligible information.

We validated criticality sparsity across three agent architectures: a retrieval-augmented generation (RAG) agent [7] consuming financial data sources, a multi-tool agent performing research tasks, and a decision-support agent processing healthcare information. In each case, the average number of critical sources per inference was between 2 and 4, with the distribution exhibiting a heavy tail - most inferences had exactly 2 critical sources, and fewer than 5% had more than 5. This empirical distribution confirms that the O(1) complexity claim holds in practice, as the constant factor is bounded by the expected number of critical sources.

The second property is criticality predictability: the binary criticality gate can accurately identify critical sources without requiring counterfactual evaluation. We implemented three variants of the criticality gate and compared their precision-recall characteristics against ground-truth criticality determined by exhaustive counterfactual analysis. The attention-weight variant examines the normalized attention scores assigned to tokens from each data source during the agent's standard inference pass [8]. The retrieval-score variant examines the relevance scores assigned by the retrieval mechanism during source selection. The gradient-based variant examines the gradient magnitude of the agent's output with respect to tokens from each data source [9].

All three variants achieved precision above 0.95 and recall above 0.90, meaning that the criticality gate correctly identifies the data sources that actually matter for the agent's decision while rarely flagging non-critical sources as critical. The attention-weight variant offered the best trade-off between accuracy and computational overhead, as attention weights are a byproduct of the standard inference pass and require no additional computation to extract. This implementation detail is significant because it means the criticality gate adds zero latency to the agent's primary inference operation.

The end-to-end evaluation pipeline operates as follows. During the agent's standard inference pass, attention weights for each data source are extracted and normalized. Sources whose normalized attention exceeds the criticality threshold are flagged as critical. For each critical source, a single masked inference pass is executed concurrently with the agent's primary inference, and the KL divergence [4] between the baseline and masked output distributions is computed. The resulting decision impact scores are available within the same inference cycle, enabling real-time data quality assessment during agent operation.

Applications

The counterfactual KL-divergence protocol provides the quantitative foundation for several applications in AI governance and data quality management. The most immediate application is real-time data quality scoring during agent inference. When an agent consumes a data source with high decision impact, any quality issues in that source have outsized consequences for the agent's output [10]. By combining the decision impact score with traditional data quality metrics (accuracy, completeness, freshness), organizations can compute a decision-weighted quality score that reflects the actual importance of each data source to the agent's current operation.

A second application is data source valuation in AI-mediated markets [11]. As AI agents increasingly mediate the consumption of external data, the economic value of a data source is no longer a static property; it is a function of how much the data source affects agent decisions. The decision impact score provides a quantitative basis for dynamic pricing: data sources that consistently exhibit high decision impact across a population of agents are demonstrably more valuable than sources that are rarely critical. This enables market mechanisms that align pricing with actual utility rather than with proxies such as data volume or update frequency.

A third application is dependency risk assessment. When an agent consistently relies on a small number of critical data sources, it creates a concentration risk analogous to single-supplier dependency in physical supply chains. The decision impact scores, aggregated over time, reveal which data sources are systemically critical to an agent's operations. This information enables risk managers to identify concentration risks, evaluate the availability and substitutability of critical sources, and develop contingency plans for source disruption.

Finally, the protocol enables attribution in multi-source agent decisions [12]. When an agent produces an output that requires explanation or audit, the decision impact scores provide a quantitative decomposition of which data sources contributed to the decision and by how much. This is more informative than post-hoc explanation methods that examine the agent's reasoning trace, because it is grounded in a counterfactual analysis of what the agent would have decided without each source [1]. The result is an attribution methodology that is both quantitatively rigorous and practically useful for compliance, auditing, and dispute resolution.

References

  1. Pearl, J. (2009). Causality: Models, Reasoning, and Inference (2nd ed.). Cambridge University Press.
  2. Shapley, L. S. (1953). A Value for n-Person Games. In H. W. Kuhn & A. W. Tucker (Eds.), Contributions to the Theory of Games II, Annals of Mathematics Studies, 28, 307-317. Princeton University Press.
  3. Roth, A. E. (Ed.). (1988). The Shapley Value: Essays in Honor of Lloyd S. Shapley. Cambridge University Press.
  4. Kullback, S., & Leibler, R. A. (1951). On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1), 79-86.
  5. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 30, 5998-6008.
  6. Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). Wiley-Interscience.
  7. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 33, 9459-9474.
  8. Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019). What Does BERT Look At? An Analysis of BERT's Attention. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 276-286.
  9. Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic Attribution for Deep Networks. Proceedings of the 34th International Conference on Machine Learning (ICML), 70, 3319-3328.
  10. Wang, R. Y., & Strong, D. M. (1996). Beyond Accuracy: What Data Quality Means to Data Consumers. Journal of Management Information Systems, 12(4), 5-33.
  11. Agarwal, A., Dahleh, M., & Sarkar, T. (2019). A Marketplace for Data: An Algorithmic Solution. Proceedings of the 2019 ACM Conference on Economics and Computation, 701-726.
  12. Shannon, C. E. (1948). A Mathematical Theory of Communication. The Bell System Technical Journal, 27(3), 379-423.