Approved QA context
Azure Boards / Test Plans + repository
Product + domain QA
Story and test revisions → 2
Our Banking Client: implementation plan
Open the interactive case or the technical briefing.
Fictional case, edition v1.24.0. This is a proposed delivery design and a working browser teaching simulation. It is not a deployed fintech QA platform. All company details, effort inputs, payment rules and pilot targets below are authored assumptions. The sources document technology capabilities, not the case’s results.
QE testing scenario: timeout and retry. PAY-142 deliberately injects a duplicate-posting defect to test whether the journal-count and balance assertions catch it. The expected outcome is one transfer. This is a test design, not a reported client incident; a timeout alone does not imply two debits.
The manual QE starting point
The scenario begins with a large offshore QA organization whose delivery is constrained by its test system. Most QE work is manual. Internal test environments are limited, backend services are not virtualized, and vendor-provided test-environment availability and booked slots limit how many tests can run at once. Shared accounts and resets create further contention. QA engineers manually rerun tests and assemble screenshots, logs and release evidence.
Some Java API, Selenium and Appium automation and partial CI exist. They do not remove the vendor environment limit or supply independently resettable data. More QE engineers alone cannot create more test-environment capacity. Establish the vendor environment count, concurrent-run limits, booking rules and waiting time during discovery.
QE modernization before wider execution
The modernization guide develops the foundation backlog behind this case. Build and prove the target for one payment journey: reviewed assertions, a reproducible environment, isolated data, a versioned provider substitute and durable evidence. These are modernization deliverables, not existing baseline capabilities. Compare its behavior with the real provider and settlement path. Containerize suitable components; reserve supported environments for systems that cannot be made disposable.
Reviewed AI drafts can begin where their own prerequisites are satisfied. Wider execution depends on demonstrated test reliability, environment repeatability and supported ownership. The 480-hour setup allowance below does not price this modernization backlog.
Assumption register
| Area | Assumption to validate in discovery |
|---|---|
| Organization | 75 offshore QA staff across five application squads. Primary roles: 45 manual/domain testers, 15 automation engineers, eight data/environment engineers and seven leads/coordinators. These groups do not overlap. |
| Pilot team | Eight existing QA staff: one lead, four domain testers, two automation engineers and one data/environment engineer. Product, development and platform support are additional contributors whose effort must be recorded. |
| Application portfolio | Payments, web, mobile, onboarding and settlement. Java/Spring Boot payment services, React web, PostgreSQL journal and an existing Azure test environment. Legacy settlement shares a nightly batch environment. |
| Current environment constraints | Limited shared internal test environments; backend services are not virtualized. Vendor-provided test-environment availability and booking slots cap parallel integration testing. Shared accounts, resets and settlement windows further restrict runs. Inventory actual capacity and restrictions before sizing the target. |
| Current manual workflow | Testers prepare cases and data, wait for environments, run and rerun checks, and manually reconcile screenshots, logs, defects and release status. Queue time is recorded separately from active effort. |
| Current delivery | Fortnightly releases. A bounded release test pack uses 300 person-hours across 10 business days; the pilot team also performs other work. This is not 10 days of full-time effort by all eight people. |
| Current automation | Some usable Java API tests, an uneven Selenium UI suite, existing Appium mobile coverage and partial CI. These islands of automation still depend on shared environments and vendor slots. Framework and language versions must be checked against the actual repositories. |
| Payment contract | PAY-142 transfers CAD 100.00 between synthetic accounts starting at CAD 1,000.00 and CAD 0.00. No fees, FX, overdraft, settlement timing or unrelated account activity is modeled. |
| Expected behavior | The same request key and payload return the original transfer identity. Conflicting payloads are rejected. Retries and duplicate callbacks create one journal with two balancing entries. |
| Evidence status | Hours, maturity profiles, staffing and simulated outcomes are assumptions. No client baseline or actual pilot result has been collected. |
Client adoption assumption register
The platform readiness hub makes twelve prerequisites explicit. Our Banking Client has not been assessed; the current-state statements below are hypotheses for discovery. For each one, record the evidence reference, accountable person, remediation and review date. A modern API does not establish readiness for mobile, legacy or batch applications.
| Dependency | Our Banking Client scenario starting point | Accountable role |
|---|---|---|
| Requirements & expected outcomes | PAY-142 exists; product owners must confirm retry, concurrency and journal rules. | Product owner + domain QA |
| Infrastructure & environments | Few shared test environments and shared data constrain the QA teams. Vendor-owned test environment capacity caps parallel integration runs; independent reset is not established. | Infrastructure / environment lead |
| DevOps & delivery pipelines | CI is available; deployment remains partly manual and artifact links need integration. | DevOps lead + application developer |
| Service virtualization & dependencies | Backend services are not virtualized at baseline. Versioned provider substitutes are proposed work; limited vendor test environments and the shared settlement batch still need reserved real-integration windows. | Integration lead + dependency owner |
| Test data & reset | Synthetic account fixtures and scripted reset are backlog items; shared records cannot establish repeatability. | Test-data lead + application owner |
| Application & integration testability | Payment APIs are the candidate surface. UI reliability and batch independence require separate readiness checks. | Application architect + developer |
| Frameworks, runners & devices | Some Java tests are usable; noisy UI tests and device scheduling require their own readiness decisions. | Automation lead + application QA |
| Observability & release evidence | Evidence is spread across tools; a joined run manifest and retention rules must be demonstrated. | QA lead + observability / release owner |
| AI access, data rules & evaluation | An approved endpoint and permission to use repository context are assumptions to validate, not installed capabilities. | AI platform owner + data owner |
| People, skills & offshore handoffs | 75 QA staff include 45 manual/domain testers; automation skills and working-hour overlap are uneven. Support effort is additional to the eight QA pilot roles. | QA delivery director + supplier lead |
| Shared platform ownership & service capacity | A shared QA service is proposed. Cross-team reuse and support coverage have not been demonstrated. | QA platform product owner |
| Baseline, funding & adoption evidence | The 300-hour baseline, 480-hour setup and 12-hour operating allowance are unverified planning assumptions, not a readiness budget. | Sponsor + delivery measurement lead |
No required capability can be compensated for by a higher maturity score elsewhere. The adoption worksheet applies requirements separately to each workflow; second-team rollout additionally requires supported platform operation, evidence reuse and trained handoffs. The illustrative effort profiles below are not adoption approval.
QE modernization is an explicit adoption dependency within this register. For the selected workflow, confirm test strategy, environment lifecycle, service virtualization and contract fidelity, data, CI evidence and shared ownership. Record remediation cost, an owner and acceptance evidence before committing to broader AI execution. The fintech trial dependencies connect these workstreams to each proposed trial.
Stack
Context assembly
Versioned templates + context adapter
QA platform owner
Prompt + source IDs → 3
AI workbench
Approved assistant / model endpoint
AI platform owner
Candidate test + provenance → 4
Human review
Domain check + reviewed pull request
Domain QA + automation engineer
Approved test commit → 5
CI and test runners
Azure Pipelines · JUnit / REST Assured · Playwright
DevOps + application engineers
Assertions + run manifest → 6
Evidence catalog
Pipeline artifacts + Test Plans links
QA + release owner
Accountable release review
Execution boundary: approved commits enter CI. REST Assured/JUnit test the payment API; Playwright covers the web journey. Real PostgreSQL + a WireMock provider model run in the test environment. Mobile devices and real provider/settlement checks retain separate lanes.
The proposed design extends tools that the team in this scenario already knows. WireMock mappings, isolated fixtures, repeatable environments and joined evidence are new work to implement and verify. A discovery assessment can substitute the customer’s equivalents without changing the workflow responsibilities.
| Capability | Example technology | Integration to implement |
|---|---|---|
| Requirements and cases | Azure Boards and Azure Test Plans | Read approved story/test revisions, link candidates back to their source, record unresolved questions and reviewer decisions. An AI connector is proposed work. |
| Shared QA context | Repository templates, a versioned scenario catalog and a small context assembly service | Filter by application and release, attach source IDs and revision hashes, remove obsolete examples and record the model/prompt configuration used. Start with selected documents; a vector database is optional. |
| AI assistance | Approved coding assistant or model endpoint | Stage-specific prompts, structured candidate artifacts, bounded retries and reviewer feedback. The model does not set the authoritative expected financial behavior. |
| API tests | REST Assured and JUnit in the Java repository | Reuse authentication fixtures, request helpers and assertions. Add retry, concurrency and journal outcome checks around the existing service. |
| Browser tests | Playwright for the new React payment journey | Reuse a seed fixture and approved plan, review generated tests and store traces from failed CI runs. Retain existing Selenium coverage with named maintainers. |
| Mobile tests | Existing Appium suite and device pool | Keep device scheduling, app build identity, account reset and evidence collection explicit. API-level coverage does not replace mobile interaction testing. |
| Data | PostgreSQL synthetic fixtures; Testcontainers for service tests | Assign unique accounts per run, seed exact balances and validate relationships. Container tests require a supported runtime and do not provision the whole estate. |
| Provider dependency | WireMock test stub plus separate provider sandbox tests | Introduce service virtualization for timeout, delayed-callback and duplicate-event scenarios. Version and validate mappings; reserve vendor slots for real integration and fidelity checks. Virtual runs do not replace provider acceptance or settlement validation. |
| Execution | Existing Azure Pipelines and test runners | Pin build/test/fixture/stub revisions, run preflight checks, publish original runner evidence and retain the mandatory payment suite. |
| Reporting | Test Plans and an artifact store | Link requirement, candidate, review, manifest, result and defect disposition. AI drafts a summary with evidence links; the release owner decides. |
Primary documentation: Azure Test Plans, REST Assured, Playwright Test Agents, Trace Viewer, Appium, Testcontainers and WireMock. Reviewed 7 September 2026. Product choices are examples, not procurement recommendations.
A concrete generated-test review
The candidate should test the financial outcome independently of the success message. The following is illustrative Java pseudocode. payments, provider and journal are application-specific adapters to implement. It is not a runnable integration supplied by this site.
// Fixture: payer=100000, recipient=0, amount=10000 CAD minor units.
provider.acceptThenTimeout("pay-142");
payments.submit("pay-142", 10000); // response may be unknown
Transfer retry = payments.submit("pay-142", 10000);
assertEquals(originalTransferId("pay-142"), retry.id());
assertEquals(1, journal.countForTransfer(retry.id()));
assertEquals(90000, accounts.balance(payer));
assertEquals(10000, accounts.balance(recipient));
assertEquals(0, journal.sumSignedEntries(retry.id()));
The product owner and domain QA define these expected outcomes before generation. The automation reviewer checks that the test fails against a deliberate duplicate-posting defect. Separately test concurrent same-key submissions, the same key with a changed amount, and repeated callbacks with the same event identity. The browser simulation simplifies these mechanisms to teach the outcome; it does not model database transactions, race timing or provider protocol guarantees.
For a real implementation, the developer must resolve request-key scope, retention, concurrent inserts, transactional posting, event identity and recovery after partial failure. The QA team needs fault-injection points and read-only observation of the journal in its test environment. A balanced journal alone is insufficient: duplicated balanced journals can still move money twice.
Playwright’s documented healer may return a skipped test when it believes behavior is broken. In this proposal, an unexplained skip or weakened assertion cannot satisfy a required case. Such candidates return to review. Playwright Test Agents.
Workflow artifacts and handoffs
Worked example / PAY-142 / Authored samples
Watch the evidence travel with the work
A CAD 100 payment times out after provider acceptance. Follow the same requirement through all eight stages. These samples illustrate the proposed artifacts; they are not a live integration or client records.
- 1. Requirements
- 2. Test design
- 3. Test data
- 4. Environment
- 5. Automation
- 6. Execution
- 7. Triage & retest
- 8. Release evidence
Requirements → Accepted payment rule
Manual work
Clarify what a retry means and chase answers across time zones.
AI contribution
Draft ambiguity questions and identify affected payment journeys.
Requirement: retry an accepted payment after a timeout.
Key: same request key + same amount → same transfer ID.
Expected: one journal; payer 90000, recipient 10000 minor units.
Open question: key-retention period requires product agreement.Product owner + domain QA checks: Expected financial behavior is agreed independently of the generated test. Open questions remain visible.
Handoff: Approved rule-r3 → domain QA; unresolved rules return to the product owner.
Test design → Scenario matrix
Manual work
Copy cases between spreadsheets, fill gaps and review duplicates.
AI contribution
Draft positive, boundary, retry and concurrency cases with requirement links.
C1 Timeout → same-key retry → one transfer
C2 Concurrent same-key calls → one transfer
C3 Same key, different amount → reject conflict
C4 Repeated provider callback → one postingDomain QA lead checks: The matrix includes independent outcomes, concurrency and changed-payload behavior.
Handoff: Reviewed scenarios-r2 → automation and fixture owners.
Test data → Synthetic fixture
Manual work
Find eligible accounts, prepare balances and coordinate shared data resets between squads.
AI contribution
Draft seed scripts and scenario combinations using synthetic records.
run_id: sample-142
currency: CAD
payer_start: 100000 minor units
recipient_start: 0 minor units
amount: 10000 minor units
reset: isolated accounts recreated per runTest-data engineer checks: Balances and account relationships are valid; separate runs do not share mutable fixtures.
Handoff: fixtures-r4 + account IDs → the environment preflight.
Environment → Run preflight
Manual work
Check deployments, book scarce vendor test-environment slots and wait for shared resets. Backend services are not virtualized.
AI contribution
Summarize readiness failures and draft stub mappings for engineers to implement and validate.
app_build: payment-example-a1
fixture_rev: fixtures-r4
provider_model: accepted-then-timeout-r2
readiness: app + database health checks required
retention: evidence stored outside disposable resources
cleanup_owner: platform on-callPlatform engineer + QA checks: The provider substitute reproduces the intended failure. The real provider and settlement still need separate checks.
Handoff: Pinned manifest + usable environment → CI runner.
Automation → Reviewed test change
Manual work
Write selectors, API requests, assertions and reusable helpers.
AI contribution
Draft tests in the existing framework and propose maintenance patches.
AI candidate r1: assert response.status == 200
Reviewer: insufficient; this can miss a duplicate posting.
Revision r2: assert same transfer ID on retry
assert journal count == 1
assert balances == 90000 / 10000
Challenge: deliberate duplicate-posting variant MUST FAILAutomation engineer checks: A candidate cannot weaken required assertions or skip the deliberate-defect check. This is sample pseudocode.
Handoff: Reviewed commit → CI; rejected candidates return to the author.
Execution → Original run receipt
Manual work
Run regression packs by hand alongside partial scripts, collect screenshots and queue reruns for the next available environment slot.
AI contribution
Recommend affected tests and summarize run evidence.
Candidate: review-r2
Correct behavior fixture: required assertions PASS
Deliberate duplicate variant: journal count 2 → FAIL (expected)
Candidate disposition: challenge detects the known defect
Required suite: other cases must still complete
Evidence: manifest + assertions + logsApplication QA team checks: These are illustrative receipts, not executed results. Detecting the injected defect does not establish complete coverage.
Handoff: Original receipt + artifact links → QA triage and reviewer.
Triage & retest → Evidence-linked disposition
Manual work
Compare failures, recreate shared data and coordinate fixes and retests around vendor slots and time zones.
AI contribution
Cluster related failures and draft evidence-linked defect or repair proposals.
AI draft: duplicate journal observed after retry.
Evidence: sample-run-142 / assertion journal_count=2
Uncertainty: root cause not established by this assertion.
Human disposition: developer investigates transaction handling.
Retest: new build + fresh run; old failure stays retained.QA lead + developer checks: The diagnosis is traceable and uncertainty is preserved. A suggested fix does not close the defect.
Handoff: Named developer + reproducible failure → fix, fresh retest and domain review.
Release evidence → Release evidence receipt
Manual work
Reconcile screenshots, logs and spreadsheets by hand, then chase owners for release status.
AI contribution
Draft the release summary with links and unresolved evidence gaps.
Requirement: rule-r3
Scenarios: scenarios-r2 / all required cases accounted for
Test + fixture + build: linked in run manifest
Defects: dispositions + fresh retest links required
Provider / settlement: separate evidence required
Decision: HOLD while required evidence is missingRelease owner + QA lead checks: The release owner resolves required checks and exceptions. An AI summary cannot override a missing result.
Handoff: Reviewed evidence + explicit decision → accountable release owner.
1. Requirements
Input: PAY-142 story, API contract and past payment defects.
Current manual work: Clarify what a retry means and chase answers across time zones.
Assistance: Draft ambiguity questions and identify affected payment journeys.
Output: Versioned acceptance criteria and an open-question log. Accountable reviewer: Product owner + domain QA.
Acceptance check: Product owner confirms that a retry uses the same key and returns the original transfer.
Adoption prerequisites: Inspect the required foundations for requirements.
2. Test design
Input: Approved behavior, prior scenarios and defect history.
Current manual work: Copy cases between spreadsheets, fill gaps and review duplicates.
Assistance: Draft positive, boundary, retry and concurrency cases with requirement links.
Output: Reviewed test matrix with independently defined expected results. Accountable reviewer: Domain QA lead.
Acceptance check: Include same-key/different-payload, parallel retry and duplicate callback cases.
Adoption prerequisites: Inspect the required foundations for test design.
3. Test data
Input: Account schema, balance rules and fixture conventions.
Current manual work: Find eligible accounts, prepare balances and coordinate shared data resets between squads.
Assistance: Draft seed scripts and scenario combinations using synthetic records.
Output: Versioned fixtures with isolated accounts and known starting balances. Accountable reviewer: Test-data engineer.
Acceptance check: Verify relationships and exact balances before running tests.
Adoption prerequisites: Inspect the required foundations for test data.
4. Environment
Input: Build ID, deployment manifest and dependency contracts.
Current manual work: Check deployments, book scarce vendor test-environment slots and wait for shared resets. Backend services are not virtualized.
Assistance: Summarize readiness failures and draft stub mappings for engineers to implement and validate.
Output: Reproducible test run with pinned build, data and stub versions. Accountable reviewer: Platform engineer + QA.
Acceptance check: A deterministic preflight checks health, schema and fixture readiness.
Adoption prerequisites: Inspect the required foundations for environment.
5. Automation
Input: Reviewed scenarios, seed tests and repository conventions.
Current manual work: Write selectors, API requests, assertions and reusable helpers.
Assistance: Draft tests in the existing framework and propose maintenance patches.
Output: Reviewed pull request linked to the accepted test matrix. Accountable reviewer: Automation engineer.
Acceptance check: A deliberately broken duplicate-posting variant must fail. No weakened assertions or silent skips.
Adoption prerequisites: Inspect the required foundations for automation.
6. Execution
Input: Approved tests, pinned build and ready environment.
Current manual work: Run regression packs by hand alongside partial scripts, collect screenshots and queue reruns for the next available environment slot.
Assistance: Recommend affected tests and summarize run evidence.
Output: Runner results, traces and mandatory-suite completion record. Accountable reviewer: Application QA team.
Acceptance check: Keep the mandatory payment suite while validating selection quality.
Adoption prerequisites: Inspect the required foundations for execution.
7. Triage & retest
Input: Failed assertions, logs, traces, changes and environment state.
Current manual work: Compare failures, recreate shared data and coordinate fixes and retests around vendor slots and time zones.
Assistance: Cluster related failures and draft evidence-linked defect or repair proposals.
Output: Reviewed defect disposition, fix and fresh rerun evidence. Accountable reviewer: QA lead + developer.
Acceptance check: Separate product defects, flaky tests and environment faults. Recheck original intent after a patch.
Adoption prerequisites: Inspect the required foundations for triage & retest.
8. Release evidence
Input: Results, open defects, waivers and requirement coverage.
Current manual work: Reconcile screenshots, logs and spreadsheets by hand, then chase owners for release status.
Assistance: Draft the release summary with links and unresolved evidence gaps.
Output: Versioned evidence pack and accountable release decision. Accountable reviewer: Release owner + QA lead.
Acceptance check: Missing evidence remains unknown. A failed money-movement assertion blocks this case’s release.
Adoption prerequisites: Inspect the required foundations for release evidence.
A run manifest should identify the requirement revision, scenario ID, test commit, application build, fixture revision, dependency-stub revision, environment and runner result location. Candidate-generation records additionally retain prompt/model configuration and reviewer disposition. Record missing links as gaps; do not infer a passing run from an AI summary.
Application and delivery maturity
Assess requirements, DevOps, cloud/environment readiness, application testability, automation and offshore delivery for each application. The case’s three profiles are discussion aids, not a certified maturity score.
| Condition | First work to commission | Evidence before expanding |
|---|---|---|
| Shared environments, manual deployment and fragile scripts | Requirements clarification, reviewed scenario drafts, stable fixtures, a reproducible run and clear ownership | Another engineer can repeat the same test with the same result. |
| Usable API automation and partial CI, but no backend virtualization, limited vendor slots and shared data | The payments API pilot, one web journey, validated provider stubs and evidence integration | Full effort accounting and trustworthy required tests across comparable release packs. |
| Repeatable deployment, isolated data and maintained tests | Expand generation and triage, then evaluate selective regression in shadow mode | A second team reuses the pattern and selection retains important defect detection. |
Cloud hosting does not establish maturity by itself. Readiness depends on deployment reproducibility, reset speed, dependency control and observability. DORA’s findings motivate attention to the surrounding delivery system; they do not supply this case’s numerical assumptions. DORA 2025 report.
Offshore delivery adds practical dependencies: repository and environment access, coding skills, a shared review window, accountable next owners, training capacity and supplier incentives. Reuse requires time and ownership. Coaching and supporting-engineer effort belong in the economics.
Quality outcomes beyond hours
Compare the architecture and inspect the outcome register. Three primary measures cover critical payment-fault detection, critical-scenario coverage and complete decision evidence. Repeatability, real-provider fidelity and second-squad reuse supply supporting proof. Freeze scope, denominators, exclusions and owners before collecting baseline and assisted observations.
Distinguish current work, modernized QE without AI and the same foundation with AI where feasible. Without a comparable foundation-only reference, report combined contribution rather than isolated AI impact. Seeded-fault detection does not prove reduced production incidents or financial loss.
Measurement
The unit is one comparable, bounded release test pack, not a person, application portfolio or organization. Capture task-level active effort in mutually exclusive workflow stages. Use timestamps and reason codes separately for waits and elapsed release time. Record vendor-slot waits, internal environment queues, data/reset waits, blocked scenarios and successful concurrent runs, with the same scope and observation window before and after. Do not add days of waiting to person-hours or claim that every saved task hour shortens the release’s critical path.
| Stage | Baseline hours | Mixed-profile work | Mixed-profile review | Assisted total |
|---|---|---|---|---|
| Requirements | 20 | 12 | 4 | 16 |
| Test design | 40 | 20 | 8 | 28 |
| Test data | 30 | 20 | 4 | 24 |
| Environment | 25 | 20 | 2 | 22 |
| Automation | 55 | 30 | 10 | 40 |
| Execution | 80 | 52 | 4 | 56 |
| Triage & retest | 35 | 20 | 6 | 26 |
| Release evidence | 15 | 8 | 2 | 10 |
| Total | 300 | 182 | 40 | 222 |
Assisted work excludes the review column. The baseline includes its ordinary review and rework. Record corrections and retests in the appropriate stage once. The target combines AI assistance with fixtures, automation and environment changes; it does not isolate AI’s causal contribution.
The mixed-profile scenario calculation is:
- 300 baseline hours − 222 assisted hours = 78 hours of gross capacity.
- Subtract 12 hours of recurring platform operation per pack = 66 net hours.
- Redeploy 50% in this scenario to named other work = 33 usable hours.
- The 480-person-hour setup input, divided by 33, takes 15 comparable packs, rounded up, to recover in capacity terms.
The twelve-dependency remediation backlog is not priced by the 480-hour setup assumption. Re-estimate infrastructure, provider virtualization, data, licenses and support effort for the actual client. Setup includes integration, initial fixtures, coaching and supporting product/development/platform effort. Avoid charging initial setup again in each assisted pack. Conversely, recurring operation must not disappear into the setup bucket. The low-maturity profile has 315 assisted hours plus 12 operating hours: 27 additional hours per pack. The model retains that full cost even at a low capture percentage. At zero positive capacity or zero capture, there is no finite capacity payback.
Tool, cloud and vendor charges are excluded. Staff hours recovered are not automatically budget savings. Do not extrapolate the pack to all 75 people or annualize it until eligible volume, adoption, maintenance and redeployment are observed. A cash business case requires actual commercial costs and an agreed mechanism for realizing financial benefit.
Pilot backlog and acceptance
| Phase | Deliverable | Acceptance evidence |
|---|---|---|
| Weeks 1–2: workflow assessment | Current release map, effort and wait baseline, application readiness, test intent and bounded pilot backlog | QA, product and platform leads agree on scope, the expected payment behavior and who supplies missing evidence. |
| Weeks 3–8: payment pilot | Context adapter, test matrix, synthetic fixtures, provider stub, reviewed framework changes and pipeline evidence | At least two comparable assisted packs, with matched conventional tasks as a reference. All required payment checks run; the duplicate-posting fault fails. |
| Weeks 9–12: reuse trial | Second-squad onboarding, maintained templates, operating ownership and rollout backlog | The second squad runs and maintains the pattern without its original authors. Net capacity remains positive and quality evidence remains trustworthy. |
The API pilot windows are illustrative planning inputs, not observed client durations. Extend the measurement period if release volume, environment incidents or task differences make the evidence inconclusive. Pause expansion for an escaped money-movement defect, unreliable fixtures or a test that passes the deliberately broken variant. Small samples inform the next delivery decision with uncertainty; they do not establish an industry productivity rate.
This offers a concrete progression of work: an assessment with a usable baseline, a pilot with reviewable artifacts and measured outcomes, then a rollout tied to application readiness. It avoids a headcount-based savings promise and gives the team a specific delivery decision at each phase.
Observed time framework for the API pilot
A framework for collecting actual API pilot timings. The windows below are illustrative planning inputs, not observed client durations.
Observation status: not yet recorded
- Planning window · weeks 1–2
Observe the baseline
Record API test preparation, active effort, waits and rework for one comparable pack.
Evidence gate: Reviewed API scope + usable baseline - Planning window · weeks 3–8
Run the API pilot
Record setup separately; measure review, execution, triage and retest over repeated packs.
Evidence gate: Required assertions pass + comparable run evidence - Planning window · weeks 9–12
Confirm repeatability
Measure second-team setup, support effort and repeat runs before choosing the next scope.
Evidence gate: Reusable pattern + evidence-based next step
Use one record per comparable API test pack. Retain failed runs and foundation work, and record the observation source instead of treating elapsed weeks as effort saved.
| Observation | What to record | Current status |
|---|---|---|
| Scope and comparability | Pack ID, acceptance rules, test count, complexity and conventional or assisted approach | Not recorded |
| Calendar time | Actual start/end dates and elapsed hours for each phase | Not recorded |
| Human effort | Preparation, review, correction, triage and retest in person-hours | Not recorded |
| Waiting and prerequisites | Environment, data, provider and reviewer waits with reasons | Not recorded |
| Foundation and operation | One-time setup effort, recurring support and runner time separately | Not recorded |
| Quality and reuse | Mandatory assertions, detected faults, escaped defects and second-team handoff | Not recorded |
Readiness controls the pace: environment, data, virtualization, CI and reviewer gaps can extend a phase. Advance on evidence, not the calendar. Compare like-for-like packs and report modernization effects separately from incremental AI assistance.
External evidence and the client trial
The financial-services evidence explorer separates direct AI testing cases from modernization and adjacent engineering results. Use its trial selector to agree the first workflow, reviewer and acceptance criteria. Peer figures do not replace Our Banking Client’s illustrative inputs or supply a client forecast. Record actual API pilot dates and effort against the illustrative planning windows; readiness and sufficient evidence determine progression.
From a unit-test candidate to API proof
The nine-step walkthrough follows one authored payment scenario. Its identifiers connect the handoffs; they are not records from a deployed bank. Developer unit testing is shown alongside the existing eight-stage QA effort model, without adding an unsupported savings estimate.
Approved input: AC-142 requires the same idempotency key and payload to return the original transfer without another posting. CASE-142-R exercises a timeout after provider acceptance and a same-key retry. Domain QA sets the expected outcome before AI drafts the code.
An AI assistant can draft this JUnit-style pseudocode from the accepted behavior and existing test patterns. The handler, request and fake repository below illustrate interfaces that the application team must implement; this snippet is not an executable bank integration.
@Test void sameKeyReturnsOriginalTransfer() {
var repository = new RecordingFakeTransferRepository();
var handler = new TransferHandler(repository);
var request = transfer("pay-142", "payer", "recipient", 10000);
var original = handler.create(request);
var retry = handler.create(request);
assertEquals(original.id(), retry.id());
assertEquals(1, repository.insertions());
}
Developer review: verify that the fake records real calls, that assertions reflect AC-142, and that the test fails when deduplication is deliberately bypassed. This unit test does not exercise network timeouts, actual database constraints, provider acceptance, or concurrent transactions. Those need separate API, concurrency and integration cases.
API handoff: PR-142 carries the reviewed executable test. FIX-142 supplies synthetic accounts starting at 100000 and 0 minor units. STUB-142 models provider acceptance followed by a lost response. A REST Assured/JUnit test must check the original transfer ID, exactly one balanced journal, payer 90000 and recipient 10000 minor units after retry. Selected Playwright checks cover the pending state and customer retry action.
Failure and repair: the deliberately faulty variant produces two journals in RUN-142. AI can draft DEF-142 with assertion, trace and journal references; an engineer confirms the cause. Keep the original FAIL record. After a reviewed production fix, RERUN-143 uses fresh data and unchanged expected behavior. The corrected build must pass and the deliberate duplicate-posting variant must still fail. These are expected sample outcomes, not executed client results.
Release handoff: PACK-142 links AC-142, CASE-142-R, the test commit, fixture/stub revisions, original run, reviewed defect, fix and rerun. Required real-provider checks and settlement evidence remain release gates. AI drafts the summary; the release owner decides.
Measure three states: compare manual QE with repeatable automated QE, then compare equivalent automated packs with and without AI assistance. Record critical-scenario coverage, candidate acceptance after review, injected-fault detection, environment wait, feedback time and evidence completeness. Include review, rework and platform overhead. Service virtualization benefits must not be counted as an isolated AI gain.
Mechanism references: Copilot test generation and review, WireMock service virtualization, and Azure Pipelines test-result publishing.