Testing studies

Verified 4 September 2026.

Contents

These are the studies that matter for pilot design because they report acceptance rates, coverage deltas, classification accuracy and repair outcomes on real or benchmark code. None reports a dollar saving. Most are vendor-internal deployments, which is a strength for realism and a weakness for generalization.

AI-generated tests, test maintenance and failure triage

Meta, “Automated Unit Test Improvement using Large Language Models at Meta” (TestGen-LLM), Feb 2024, FSE 2024 industry track

Finding: 75% of target test classes had at least one generated test that built; 57% had one that also passed; 25% had one that also increased coverage. These are cumulative class-level yields, not percentages of individual generated tests. Cohort: 86 Kotlin test classes (31 Stories, 55 Reels), §3.3. In separate Instagram/Facebook test-a-thons, engineers accepted 73% of recommendations; 11.5% of all classes to which the tool was applied were improved. The cohorts and denominators differ. Neither acceptance rate is a predicted result for a bank pilot. [E04]

Sponsor and caveats: Vendor-internal; improves existing tests, does not create suites from scratch

Pilot use: Use cumulative build, pass and useful-test gates; measure acceptance on the bank’s own candidate and class denominators. Do not transfer the Meta acceptance rate as a pilot expectation.

Sample and method details
  • Setting and method: LLM extends existing human-written unit tests; filters require build, reliable pass and coverage increase; deployed in test-a-thons

Uber and UT Austin, “FlakyGuard: Automatically Fixing Flaky Tests at Industry Scale”, Nov 2025

Finding: 47.6% of reproducible flaky tests received fixes; 51.8% of generated fixes accepted by developers; only 71.6% of flaky tests were reproducible; developers self-estimate under one day to 2-4 days saved per flaky test

Sponsor and caveats: Single company and language; time savings self-estimated

Local measurement: How many eligible failures can we reproduce, repair without weakening assertions, and have independently accepted? In the reported Uber deployment, FlakyGuard produced fixes for 47.6% of reproducible flaky tests; developers accepted 51.8% of generated fixes. Reproducible tests and generated fixes are different denominators. This Go-monorepo case does not predict bank repair yield, acceptance or savings. FlakyGuard study ↗

Sample and method details
  • Setting and method: LLM-guided call-graph exploration on a Go monorepo

Google, “LLM-Based Automated Diagnosis of Integration Test Failures at Google”, ICSE 2026

Finding: 90.1% accuracy on 71 manually evaluated failures; 63% helpful rate from explicit feedback; 5.8% “not helpful”

Sponsor and caveats: Small manual evaluation (three evaluators); feedback subset self-selected

Pilot use: Reference for failed-test triage use cases: classification accuracy target, helpfulness measurement design

Sample and method details
  • Setting and method: Gemini 2.5 Flash; deployed on 52,635 failing tests and 91,130 code changes

Tianjin University and Huawei Cloud, “Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation” (CATGen), Jul 2026

Finding: Compilation success 91.8% vs 51-76% for baselines; 70.1% line coverage; token use down 67-84%

Sponsor and caveats: Java only; proprietary benchmark; no developer acceptance rate

Pilot use: Context engineering (project structure, dependencies) is the difference between compiling and non-compiling generated tests

Sample and method details
  • Setting and method: Industrial Java benchmark plus Defects4J

“Automated Software Test Generation at Industry Scale Using a Multi-Agent Architecture”, ICSE-SEIP 2026

Status: Not verified: publisher returned 403

Sample and method details
  • Setting and method: Not verified: publisher returned 403