Quality engineering in the AI era

Gartner forecasts broader AI assistant use. McKinsey’s survey connects reported impact with changes across delivery and the operating model. DORA describes AI as an amplifier of the surrounding system.

Invest in shared evaluation assets, platform controls and operational learning that serve both AI-assisted engineering and AI products. These sources do not establish a bank-specific return.

AI for QE and QE for AI share testable intent, independent assurance and operational learning. That common capability supports validated delivery, trustworthy AI behavior and reuse across teams.

Cui et al. report a 26.08% increase in completed tasks across three field experiments. METR’s early-2025 trial reports a 19% increase in completion time for experienced developers on familiar repositories.

Segment pilots by task and user population, retain verification effort, and measure quality alongside throughput. METR’s later follow-up has selection concerns; it does not supply a clean causal time trend.

Cui: 4,867 developers, completed tasks +26.08%, standard error 10.3 percentage points. METR: 16 developers, 246 tasks, completion time +19%, 95% confidence interval +2% to +39%. The studies measure distinct outcomes and should not be pooled.

Industrial cases demonstrate bounded test generation and diagnosis. Agent guidance adds the consequences of actions to the evaluation problem.

Select an initial workflow with an independent check and limited authority; assess judgment-heavy and consequential workflows separately.

A two-by-two screen separates easy-to-verify results from expert judgment, and advice from greater action authority. Test drafts and log-linked diagnosis are candidate starting points.

The sources address different failure modes: verification effort, weak generated tests, lifecycle change and the consequences of agent actions.

Translate each finding into an observable design requirement. Review burden needs effort telemetry; test generation needs a quality gate; changing AI needs repeatable evaluation; tool use needs enforceable permissions.

DORA informs joined effort measurement. Meta informs independent test-quality gates. NIST informs versioned evaluation and feedback. OWASP and OSFI inform external permission enforcement. Each design response has an associated evidence record.

NIST treats evaluation and risk management as lifecycle activities. Evaluation product documentation supports versioned datasets and comparisons; judge calibration still requires local evidence.

Compare baseline and candidate configurations against held-out cases, gate releases on agreed floors, then turn reviewed production failures into regression cases.

Versioned application configurations and a held-out evaluation corpus feed comparable runs and independent scoring. A failed release gate holds the change for investigation. A passing gate leads to controlled deployment and production monitoring; curated failures refresh the corpus.

DORA and McKinsey place AI adoption in the surrounding delivery organization. NIST frames governance across the lifecycle.

Assign a product owner for outcomes, a platform owner for services and a separate challenge function; include operations and value capture.

Product and QE accountability flows to the delivery team. Shared platform services enable delivery; delivery and risk owners challenge its evidence. Operations and value owners use the resulting outcomes.

Organizational research emphasizes delivery foundations. Evaluation tools expose datasets and run comparisons, but task fit still requires local validation.

Fund reusable services centrally while product teams own domain-specific quality assets. Use adapters and evidence export requirements to support substitution.

Three example domains contribute different quality assets. They use a common assurance layer, which connects through replaceable model and tool adapters.

Industry research places AI adoption within delivery-system and operating-model change. Assurance guidance makes the consequences of actions and ongoing monitoring relevant to expansion.

Develop bounded assistance first, connect shared assurance services next and expand agent authority only after local quality, repeatability and recovery evidence supports it.

Horizon one is trusted assistance, gated by local quality and net effort. Horizon two is connected assurance, gated by repeatability across teams. Horizon three is controlled agents, gated by safe actions and tested resilience.

Studies use different tasks, populations and outcome measures; verification effort can offset generation gains.

Tie each strategic objective to observable evidence and an explicit decision. Keep quality and control floors alongside capacity measures.

Four rows connect delivery capacity, trustworthy outcomes, reusable capability and controlled autonomy to evidence and leadership decisions.

NIST links governance to lifecycle risk management. OSFI discusses AI accountability and resilience; revised E-23 has a future effective date.

Maintain an inventory and an explicit authority boundary, then revisit them after incidents or material changes. This diagram is not a compliance certification.

Inventory informs authority limits. Evidence review supports an approved scope. Operations feed incidents and service changes back into the inventory and authority decision.

Industry outlook and lifecycle guidance support broader AI participation and independent assurance.

Use shared assurance as the strategic center; delegate actions only as boundaries and evidence mature.

Local assistants accelerate individual tasks. Shared assurance makes expectations, evaluation and evidence reusable. Bounded agents add delegated actions and stronger recovery obligations. These are design choices, not measured maturity scores.

Test-generation studies use quality gates; verification work remains part of the workflow.

Join a domain-owned payment contract to shared evidence services and existing release authority.

The current-state hypothesis has local interpretation, test writing, review and late evidence assembly. The target versions the approved payment rule, generates tests in a scoped sandbox, independently checks fault detection and retains the reviewer decision and release evidence.