The language behind the system
Understand the terms. Follow the architecture.
Explain the idea in your own words.
62 plain-language definitions, with examples and links to the research. Start with AI for QE, QE for AI or released capacity.
62 terms · A–Z
A simplified path through the proposed design. Decisions and results feed the evidence log. Explore all 11 components in 3D →
Try a shorter phrase, an acronym, or choose All topics. Clear resets both filters.
Also found as: Permission gate · Authorization gateway
The independent control that checks whether a specific identity may perform a specific action on a specific resource before a tool executes it.
Allow a sandbox test run; deny a request to change production data.
Read in context → · View component in 3D ↗
Related: Authorization · Least privilege · Sandbox checks
Also found as: Adoption · Utilization · Actual usage
How much eligible work actually uses the AI-assisted process. Tool access or license counts alone do not measure this.
Half of eligible test-writing work uses the assistant during the measured period.
Related: Eligibility · Net task saving · Released capacity
Also found as: Agent · Agentic AI
A system that uses an AI model to choose steps and call tools toward a task. Its ability to propose an action is separate from permission to execute it.
A test author chooses a repository search, then asks to run a test in a sandbox.
Related: Action gateway · Least privilege · QE runtime
Also found as: AI applications · System under evaluation
The AI-enabled product being evaluated, including its model, prompts, retrieval and tools. It is distinct from the assistant that creates tests.
A customer-support agent is the application under test.
Read in context → · View component in 3D ↗
Related: QE for AI · Evaluation service · Evaluation manifest
Also found as: AI-assisted quality engineering
Using AI to help people design tests, investigate failures and maintain automation. People still need evidence that the resulting work is correct.
An assistant drafts a payment test; an engineer checks its expected result.
Related: QE for AI · Quality engineering (QE) · Test oracle
Also found as: API · Tool interface
A defined way for software to request data or actions from another system. In this design, a tool call uses a bounded interface rather than unrestricted system access.
A sandbox API accepts a test artifact and returns a run identifier.
Related: Action gateway · Idempotency key
Also found as: Identity verification · AuthN
Checking who or what is making a request. Establishing identity does not establish permission for every action.
Verify the service identity that submitted a sandbox run request.
Related: Authorization · Non-human identity
Also found as: Access decision · AuthZ · Permission
Deciding whether an identity is allowed to perform the requested action on the requested resource under the current policy.
A known service identity may read test fixtures but may not access production accounts.
Related: Authentication · Action gateway · Least privilege
Also found as: Before measurement · Comparison baseline
The measured starting point used for a comparison, including the work, people, quality and effort counted.
Measure current test-writing and review time before introducing the assistant.
Related: Net task saving · Quality floor · Transferability
Also found as: Benchmark study · Reference dataset
A defined task set and measurement method used to compare performance. Results apply to those tasks and conditions before any broader inference is made.
Success on a public coding set does not establish savings on the bank’s regression work.
Related: Held-out set · Transferability · Causal evidence
Also found as: Capture rate · Finance-approved capture
The share of positive released capacity that can become a financial benefit through an agreed spending change in the relevant period.
An approved reduction in contractor scope converts part of the released capacity into lower spending.
Related: Released capacity · Hard-dollar savings · Net benefit
Also found as: Causality · Randomized experiment · RCT
Evidence supporting the claim that an intervention changed an outcome, using a design that addresses alternative explanations. Its scope still depends on the study.
A randomized comparison estimates a tool’s effect for the studied participants and tasks.
Also found as: Continuous integration · Continuous delivery · Continuous deployment
Automated steps that integrate code and prepare it for release. CD can mean delivery or deployment; this site’s release path retains an explicit approval decision.
A code change triggers tests and evidence collection before an approved release.
Also found as: CI · Statistical uncertainty
A range calculated by a statistical method to express uncertainty in an estimate. A 95% method is designed to cover the true parameter in 95% of repeated samples under its assumptions.
A wide interval signals that the data allow a broad range of effect sizes.
Related: Point estimate · Causal evidence
Also found as: Approved context · Context retrieval
The proposed component that supplies relevant, versioned material the task is allowed to access. Retrieved text remains untrusted as instructions.
Supply the approved API contract, excluding another team’s restricted customer records.
Read in context → · View component in 3D ↗
Related: Retrieval-augmented generation (RAG) · Prompt injection · Least privilege
Also found as: Avoided cost · Avoided hiring
Preventing an otherwise planned cost, measured against an approved counterfactual spending plan. It differs from reducing current spending.
Absorb planned demand without an additional hire that was in the approved workforce plan.
Also found as: Source systems · Repository · Pipeline
The existing tools that hold requirements, code, builds, tests and change records. They supply versioned inputs to the proposed platform.
The repository provides a commit; the ticket system provides its acceptance criteria.
Read in context → · View component in 3D ↗
Related: Context service · CI/CD
Also found as: Deterministic validation · Rule-based check
A check that applies fixed rules to the same inputs and conditions, producing a repeatable result. Its usefulness still depends on choosing the right rule.
Assert that an invalid payment returns status 422 and writes zero payment records.
Related: Test oracle · LLM judge
Also found as: Hash · Checksum · Artifact digest
A compact value calculated from content to identify a particular version and detect changes. It does not show that the content is safe or correct.
A changed test file has a different digest, so its earlier evaluation cannot be reused as proof.
Related: Evaluation manifest · Evidence store
Also found as: Distribution shift · Behavior drift
A change in the inputs, environment or behavior that can make earlier evaluation results less representative of current use.
Customers begin asking about a new product that the evaluation set does not cover.
Related: Held-out set · Regression corpus · Telemetry
Also found as: Eligible work · Addressable tasks
The share of an activity that fits the agreed AI use case and its control constraints. It is an input to the site’s savings model.
Exclude tasks needing restricted data when the pilot cannot access that data.
Also found as: Acceptance criteria · Evaluation plan
The written agreement on what is being tested, for whom, which failures matter, how results are scored and what thresholds must be met.
Require the payment system to reject negative amounts and create no payment record.
Related: Test oracle · Quality floor · Evaluation manifest
Also found as: Manifest · Versioned configuration
A record of the exact artifacts, configuration, evaluation data and checks used in an evaluation. Changes can invalidate earlier results.
Record the model, prompt, dataset, runner and candidate digests for run EVAL-042.
Related: Digest · Evidence store · Evaluation contract
Also found as: AI evaluator · Evaluation harness
The proposed component that runs an AI application against defined cases and scoring rules, then records its behavior and results.
Check answer correctness, permitted tool use and refusal behavior for a support agent.
Read in context → · View component in 3D ↗
Related: Evaluation contract · LLM judge · QE for AI
Also found as: Evidence log · Audit trail · Evidence record
The proposed store of linked inputs, decisions and results that lets a reviewer reconstruct what happened. Recording an outcome does not by itself make that outcome trustworthy.
Link the task, permission decision, artifact version and sandbox result in one reviewable record.
Read in context → · View component in 3D ↗
Related: Evaluation manifest · Digest · Release authority
Also found as: Safe fallback · Rollback · Recovery
A predefined alternative when an AI step or dependency cannot safely continue. A rollback specifically restores an earlier state; a fallback may route work to a person instead.
Stop the agent and send the task to a reviewer when the required context is unavailable.
Related: Release authority · Quality floor · Telemetry
Also found as: Missed detection · FN
A result that fails to flag a problem or positive condition that is actually present.
A scan reports no issue even though an exploitable flaw exists.
Related: False positive · Recall · Quality floor
Also found as: False alarm · FP
A result that flags a problem or positive condition when that condition is absent. Interpret “positive” according to the task.
An automated test reports a payment defect, but investigation finds an invalid fixture rather than a product defect.
Related: False negative · Precision
Also found as: Flakiness · Non-deterministic test
A test that sometimes passes and sometimes fails without a relevant change to the code under test. Investigate causes before changing assertions or suppressing it.
A timing-sensitive test fails intermittently when a shared environment is busy.
Related: Regression corpus · Telemetry
Also found as: Grounded response
Connecting an AI output to the supplied source material so its claims can be checked against that material. Source support alone does not prove the source is correct.
A generated test cites the exact requirement version that defines the expected behavior.
Also found as: Cash savings · Realized savings · Budget savings
A reduction in spending recognized through an agreed Finance process. The site distinguishes this from time saved or capacity redeployed.
An approved contract renewal costs less because less work is purchased.
Related: Released capacity · Cost avoidance · Capture factor
Also found as: Holdout · Held out · Test set
Evaluation cases kept separate from the examples used to develop or tune the system, so the assessment is less influenced by those choices.
Reserve unseen payment edge cases for the final pilot evaluation.
Related: Regression corpus · Benchmark · Evaluation contract
Also found as: Idempotency · Duplicate request
An identifier a service uses to recognize repeated requests for the same operation and avoid duplicate effects under its defined contract.
After a timeout, check the original run using the same key before attempting a retry.
Also found as: Calibration · Scoring rubric · Human agreement
Comparing a model judge’s scores with expert ratings on representative cases, then checking disagreement, bias and consistency.
Inspect cases where the judge passes an answer that experts rate as incorrect.
Related: LLM judge · Held-out set · Evaluation contract
Also found as: LLM · Language model · Generative AI
An AI model that generates or analyzes language and code from its input. A fluent response can still be incorrect.
A model writes a test from a requirement but may invent the expected result.
Related: AI agent · Test oracle · LLM judge
Also found as: Scoped permission · Scope · Bounded access
Giving an identity only the access needed for its task, limited by resource, action and time where possible.
Permit one test run in an isolated project until the task’s authorization expires.
Related: Authorization · Non-human identity · Action gateway
Also found as: Model-based judge · AI judge
A language model used to score an output against a rubric. Its scores need checks against expert judgment and known failure cases.
A judge scores answer completeness; experts verify whether it overlooks unsupported claims.
Related: Judge calibration · Test oracle
Also found as: Mutation score · Seeded fault · Mutant
Deliberately introducing small faults to check whether tests detect them. Detection adds evidence about test strength; it does not prove all real defects will be caught.
Change “reject negative amounts” to “accept negative amounts” and check that the test fails.
Related: Test oracle · Test coverage · Sandbox checks
Also found as: Net economic impact · Net captured impact
Benefits remaining after the relevant AI, platform, evaluation, governance and other costs are deducted. State whether the result values effort or represents cash impact.
Subtract pilot costs from captured benefit; value any added effort separately from actual cash spending.
Related: Capture factor · Net task saving · Scenario
Also found as: Task efficiency · Review burden · Net effort reduction
The share of baseline human effort removed after counting AI-assisted work, review, correction and additional controls. It can be negative.
A 10-hour task becomes 6 hours of work plus 2 hours of review: net task saving is 20%.
Related: Baseline · Released capacity · Quality floor
Also found as: NHI · Service identity · Workload identity
An identity assigned to a service or workload rather than a person, with an owner and managed permissions.
The test runner uses its own scoped identity instead of an engineer’s broad credentials.
Related: Authentication · Least privilege
Also found as: Correlation · Association · Survey
Evidence from observed behavior or reported experience without assigning the intervention. Differences may reflect other factors as well as AI use.
Teams using AI report faster delivery, but may also differ in skills or platform maturity.
Related: Causal evidence · Transferability
Also found as: Effect size estimate · Single estimate
One numerical estimate of an outcome or effect. It should be read with the study’s method, uncertainty and measurement unit.
An estimated time reduction needs its task definition and uncertainty, not only a percentage.
Related: Confidence interval · Benchmark
Also found as: Positive predictive value
The share of positive findings that are correct: true positives divided by all reported positives. High precision means fewer false alarms among the findings.
Of 10 reported payment defects, 8 are confirmed: precision is 80%.
Related: Recall · False positive
Also found as: Indirect prompt injection · Malicious instructions
An attempt to make an AI system follow attacker-supplied instructions, including instructions embedded in retrieved material or tool results.
A repository file tells the assistant to ignore its task and disclose a secret.
Related: Context service · Action gateway · Least privilege
Also found as: Quality engineering for AI
Testing and monitoring an AI application itself: its answers, use of tools, resistance to attacks and behavior when something fails.
Check whether a support agent retrieves an allowed document and refuses an unauthorized action.
Related: AI for QE · Evaluation service · Prompt injection
Also found as: AI test author · Generation runtime
The proposed execution environment that coordinates AI-assisted QE work and drafts candidate artifacts within time and cost limits.
Draft a test, request a sandbox run, and revise it using the failure evidence.
Read in context → · View component in 3D ↗
Related: AI agent · Action gateway · Test oracle
Also found as: QA · Quality assurance
Activities that build confidence that quality requirements will be met, including agreed processes, reviews and checks. This site uses QE for the broader engineering approach and retains QA where sources or cost models use that label.
Review whether a release followed its agreed testing and approval process.
Also found as: QE
Building quality into delivery through clear requirements, prevention, testing and feedback. On this site, QE includes both conventional software and AI behavior.
Define the payment rule before writing code, test it, then monitor failures after release.
Related: AI for QE · QE for AI · Quality floor · Quality assurance (QA)
Also found as: Minimum quality · Release threshold · Guardrail
The minimum acceptable quality and safety conditions that must hold while testing a productivity improvement. Mandatory gates cannot be offset by time saved.
Hold the pilot when a required payment assertion fails, even if tests are produced faster.
Also found as: Sensitivity · Detection rate
The share of actual positive cases detected: true positives divided by all actual positives. Measuring it requires knowing which cases truly contain the condition.
A seeded set contains 10 payment faults and the test suite detects 8: recall is 80%.
Related: Precision · False negative
Also found as: Regression cases · Regression suite · Test corpus
A curated, versioned collection of cases rerun after changes to detect previously addressed failures returning.
Add a reviewed negative-payment case after a defect investigation.
Read in context → · View component in 3D ↗
Related: Held-out set · Evaluation service · Drift
Also found as: Human approval · Release decision
The accountable person or process that decides whether the evidence and required approvals support release. In the demo, human approval is independent of generation.
Hold the change when a mandatory check fails, even if the generated code looks plausible.
Read in context → · View component in 3D ↗
Related: Quality floor · Evidence store · Fallback
Also found as: Capacity released · Productivity capacity
Human time made available after net task savings, eligible work and actual adoption are accounted for. It may be redeployed without reducing spending.
A team uses freed hours to reduce its backlog while the payroll stays the same.
Also found as: RAG · Retrieval augmented generation
Finding relevant material and supplying it to a model as context for its response. Retrieval must respect the requesting identity’s permissions.
Retrieve the approved payment specification before drafting its tests.
Related: Context service · Grounding · Prompt injection
Also found as: Test sandbox · Sandbox · Verification
Execution and validation inside an isolated environment with limited access and resources. Passing these checks provides evidence for review.
Run the candidate test against known good behavior and a deliberately faulty implementation.
Read in context → · View component in 3D ↗
Related: Mutation testing · Test oracle · Release authority
Also found as: Sensitivity analysis · Downside · Base case · Upside
An explicit set of assumptions used to explore possible outcomes. A scenario is not a measured result or a forecast unless evidence supports that use.
Compare lower adoption or higher review effort while keeping the other model inputs fixed.
Related: Net benefit · Baseline
Also found as: Team request · Experience layer
The place where a person submits a scoped task and inspects the result. It records the request and its owner.
An engineer asks for tests for one payment endpoint.
Also found as: Observability · Operational signals
Recorded operational measurements and events used to understand system behavior. Logs, timing and tool-use records can support investigation and monitoring.
Track failed requests, tool denials, evaluation scores and cost per run.
Related: Evidence store · Drift
Also found as: Code coverage · Branch coverage · Coverage
A measure of which parts of a system or requirement set tests exercise. High coverage does not establish that the assertions detect incorrect behavior.
A test reaches the rejection branch but never checks whether a payment was created.
Related: Test oracle · Mutation testing
Also found as: Oracle · Expected behavior · Independent expected result
The basis for deciding whether observed behavior is correct. It should come from an independent rule or trusted reference, rather than simply repeat the generated code’s assumptions.
The payment specification requires HTTP 422 and no new payment for a negative amount.
Related: Mutation testing · Evaluation contract
Also found as: External validity · Generalizability
How well a finding may apply to a different population, task or operating setting. This site requires local measurement before scaling a claim.
A student programming study may not predict an experienced team’s regression-maintenance effort.
Related: Benchmark · Baseline · Causal evidence
These are original editorial explanations of how terms are used on this site. Architecture component names describe a proposed design; examples are illustrative. “Read in context” opens the relevant explanation and its sources. For original publications, visit the document library.