en.Wedoany.com Reported - Burke, Inc. has released a new study on the reliability of synthetic data in business decision-making and introduced the FAR framework for evaluating synthetic data quality, which measures data across three dimensions: fidelity, authenticity, and resolution.
As synthetic data options become increasingly prevalent, academia and industry are beginning to question whether they can reliably guide real-world decisions. Burke's study compared multiple synthetic methods, tested whether large language model (LLM)-based synthetic panels could replace human respondents, compared the performance of generative data models with synthetic panels, and examined the importance of underlying human data quality in synthetic data scenarios.
The research results show that LLM-based synthetic panels may have value in early exploratory stages but are not yet reliable for decisions relying on quantitative insights: at the commonly cited 80% accuracy level, LLM-based synthetic data led to incorrect conclusions in approximately 60% of tested business scenarios. In contrast, methods using validated respondent-level human data, referred to as generative data models, performed significantly better, demonstrating greater potential for decision-support applications.
The core of the study is Burke's FAR framework, which evaluates synthetic data quality across three dimensions: fidelity, referring to whether synthetic data aligns with the underlying source of truth; authenticity, referring to whether synthetic responses reflect genuine variability rather than merely replicating existing data; and resolution, referring to whether relationships among variables, market segments, and business conclusions are preserved.
The study also identified a threshold for decision reliability, below which synthetic methods are more likely to preserve research conclusions, providing organizations with an important signal to distinguish promising applications from unreliable ones.
Eli Moore, Vice President of Data Strategy at Burke, stated that organizations are hearing increasingly strong claims about synthetic data, but the key question is not whether synthetic data sounds like customers, but whether it leads to the same conclusions as direct conversations. Thania Farrar, Senior Vice President of Corporate Innovation at Burke, noted that artificial intelligence is influencing how organizations generate insights and make decisions, and there is an opportunity to combine high-quality human data, advanced modeling, and expert judgment to create faster, smarter, and more reliable results while keeping real human voices at the core of research. Tara Marotti, President and CEO of Burke, said the company's goal has always been to help clients make the best decisions for their businesses, and this research can help clients feel confident about the strengths, limitations, and best uses of each method.
These findings are the result of work by Burke Labs, a division dedicated to testing and accelerating new AI and technology solutions to transform respondent experience, analysis, and reporting.
Burke, Inc. is a leading decision intelligence consulting firm that helps organizations accelerate growth through high-quality research, advanced analytics, and expert-guided insights, strategy, innovation, and training. Founded in 1931, the company combines rigorous measurement with human-centered consulting to help clients better understand people, markets, and opportunities.
This article is compiled by Wedoany. All AI citations must indicate the source as "Wedoany". If there is any infringement or other issues, please notify us promptly, and we will modify or delete it accordingly. Email: news@wedoany.com
