What Is Compliance Testing? A Software Testing Guide

Compliance testing in software testing checks whether an application, system, or development process meets specified regulations, standards, contracts, or internal policies. Also called conformance testing, it turns those requirements into verifiable checks of behavior and controls. A compliance audit is different: an auditor evaluates the broader body of evidence and processes, often independently, rather than simply executing the software tests.

Passing functional QA doesn’t mean your system is compliant. A valid user might log in and a payment might complete while an unauthorized path remains open or a required audit event goes missing. The engineering question is whether the relevant safeguards work under representative conditions and whether the team can prove it.

Redefining Compliance Testing for Engineering Teams

The ISTQB glossary uses conformance testing as another name for compliance testing: testing whether a component or system meets specified requirements. In a DevOps environment, those requirements can apply to application behavior, infrastructure configuration, data handling, release approvals, and operational safeguards.

A policy document can say that administrators must use strong authentication. A functional test can show that an administrator successfully signs in. Neither result proves that the deployed system blocks a weak authentication path, applies the rule to every privileged service, or records the event in an audit log.

That gap between declared control and observed behavior is where compliance testing earns its value. NIST’s Special Publication 800-53 Revision 5 organizes security and privacy requirements into 20 control families. Its assessment guidance asks whether controls are implemented correctly, operating as intended, and producing the required outcome.

A policy is not evidence of operation

In an audit, “we have a policy” answers a design question. It doesn’t answer an operating question. For an authentication control, engineering evidence might include:

• Configuration state: The identity provider and application settings enforce the intended authentication policy.
• Execution records: Access logs show successful and rejected attempts, including privileged access.
• Negative-path results: A deliberately invalid or incomplete authentication attempt is rejected as expected.
• Change history: The team can connect the deployed configuration to an approved change and identify who reviewed it.
• Repeatable output: Another qualified person can reproduce the test and reach the same conclusion.

The same principle applies to encryption, authorization, change management, backup restoration, logging, and incident response. A system may contain the right code and still fail because a deployment variable disables it, a service uses a different path, or an exception was never removed.

Practical rule: Test the control where it actually runs, not where the architecture diagram says it should run.

Compliance becomes an engineering artifact

The useful unit of compliance isn’t a binder of screenshots. It’s a relationship between a requirement, a control objective, a test scenario, expected behavior, collected evidence, and a remediation record.

That relationship makes compliance testing repeatable. Teams can run the same check after an infrastructure change, compare results over time, investigate deviations, and show an auditor why a finding was opened or closed. Developers own application behavior, platform engineers own deployment and access configuration, security teams define assurance needs, and control owners approve exceptions. Independent review still matters: it can challenge scope, assumptions, and the sufficiency of the evidence.

Types of Compliance Testing

The type of compliance testing depends on the obligation and the software or process in scope. A category tells you what to check, not that one test automatically satisfies every requirement within it.

Type	Examples	What the tests check
Regulatory and attestation requirements	GDPR, HIPAA, PCI DSS, SOX, SOC 2	Relevant privacy, payment, financial-reporting, and security controls, such as access, retention, audit trails, approved changes, and operating evidence; applicability and assessment criteria vary.
Industry standards and specialized requirements	ISO/IEC 27001, WCAG accessibility, GxP and 21 CFR Part 11	Whether systems support the applicable security-management controls, accessible user flows, or validated electronic records, signatures, and audit trails.
Internal policy and coding standards	Secure coding rules, retention policy, approved dependencies	Whether code, configurations, release steps, and data handling follow organization-approved rules.
Security compliance	Access control, encryption, segmentation, logging	Whether required safeguards are configured and actually enforce expected allow, deny, and detection outcomes.
Contractual and SLA compliance	Customer security terms, uptime and incident-response commitments	Whether agreed technical behavior, monitoring, reporting, and response procedures meet contractual commitments.

For example, an accessibility check may need both automated scans and manual keyboard or assistive-technology testing; a financial-reporting control may need dated approval and operating evidence. An ISO/IEC 27001 assessment, a SOC 2 examination, and a product test also have different scopes even when they draw on the same logs.

Compliance Testing vs Conformance Testing vs Audit

Term	Primary question	Typical output
Compliance testing	Does the software or control meet the applicable requirement when checked?	Test result, logs, configuration evidence, and defects.
Conformance testing	Does the component or system conform to a defined specification or requirement?	The same kind of test result; in software-testing terminology, it is generally a synonym for compliance testing.
Compliance audit	Does the organization have sufficient, appropriate evidence that the scoped requirements and controls were met?	An assessment or audit report covering evidence, processes, scope, and findings.

The terms compliance and conformance sometimes emphasize different sources of requirements in everyday usage, but ISTQB treats conformance testing as compliance testing. An audit may use test results, inspect approvals, and interview staff; a passing technical test alone does not guarantee a favorable audit outcome.

Navigating Regulatory Privacy and Security Frameworks

Compliance requirements depend on the data handled, services operated, contracts signed, and systems included in scope. A privacy assessment asks different questions from a payment-card assessment, even when both review access control, logging, and encryption. Start with the obligation, not the testing tool: identify the applicable framework or contract, map the assets and data flows in scope, then translate the wording into control objectives and observable behavior.

Compare the purpose before choosing the test

Framework	Primary Focus	Key Technical Evidence Required
HIPAA	Protection and handling of health information	Access records, authorization behavior, audit trails, risk responses, and evidence that safeguards operate in relevant workflows
GDPR	Privacy rights, lawful data handling, and protection of personal data	Data-flow records, access controls, retention behavior, masking or deletion results, and evidence that processing safeguards work as designed
SOC 2	Trust-service controls across security and related operational practices	Control descriptions, configuration records, tickets, approvals, monitoring output, and repeatable operating evidence
PCI DSS	Protection of payment-card data and the cardholder data environment	Segmentation tests, access records, configuration evidence, vulnerability and security testing output, and remediation records
Internal policy	Organization-defined risk and operating requirements	Approved procedures, system settings, execution logs, exceptions, ownership records, and retest results

The categories overlap, but their scope and evidence requirements differ. One access-control test may support several frameworks, while each can define different populations, systems, review duties, or retention rules. Reuse evidence only when it demonstrates the specific obligation. A screenshot of a setting rarely proves how the service behaves under real traffic.

Production-traffic replay makes that distinction visible. Replay sanitized requests through the same gateways, services, and authorization paths used in production, then compare expected and observed behavior. Masking must remove sensitive values while preserving the conditions that exercise the control. The resulting request, response, configuration, and log trail can show whether a safeguard operated across the workflow. Store the evidence with restricted write access and integrity checks so later changes are detectable.

PCI DSS shows why precise requirements matter. The PCI DSS assessment history and testing requirements describe penetration testing of segmentation controls at least every six months and after changes to the segmentation method for applicable service providers. A general application regression suite cannot satisfy a requirement to test the boundary itself.

Build scope before test coverage

Include services that store, process, transmit, administer, or protect regulated data, plus deployment paths and third parties that can change those services. Record exclusions with a reason, owner, and risk decision. Leaving an asset off a spreadsheet does not remove its risk.

The output should be a framework-specific control matrix: obligation, affected asset, control owner, test frequency, evidence location, and events that trigger reassessment. Developers then get a concrete release question: which behavior must this release prove?

The Three Pillars of Control Assessment

NIST describes three complementary assessment methods: examine, interview, and test. Documentation shows how the organization says a process should work. Interviews reveal how people understand and perform it. Technical testing shows what the mechanism does under specified conditions.

Examine the designed control

Examination covers specifications, architecture documents, policies, configurations, records, tickets, and other artifacts. It can answer whether a control has been defined, whether a service is supposed to emit audit logs, and whether a change received the required approval.

It can’t reliably prove that the system followed the design during execution. A configuration export may show that logging is enabled, but it won’t necessarily show that every relevant request generated an event, that events reached the retention system, or that an operator could not alter them without detection.

Interview the people who operate it

Ask the service owner how access is granted, the on-call engineer what happens during an alert, and the release manager how emergency changes are reviewed. Their answers can reveal undocumented dependencies, manual workarounds, and controls that exist only because one experienced person remembers a procedure. Interview evidence should shape test cases and identify evidence sources, not replace execution evidence.

Teams that are still defining boundaries should settle the scope and the controls that address their compliance risk before writing test cases. Clear scope gives interviews a useful target and keeps technical tests from drifting into unrelated assurance work.

Test the mechanism under realistic conditions

Technical testing compares observed behavior with expected behavior. For an authorization control, don’t stop at a successful request from an approved user. Test an unauthorized role, an expired session, a direct object reference, a background worker, and a failure path that should still create an audit record. For backups, test restoration rather than merely checking that backup jobs report success. For change management, verify that a production-relevant configuration change creates the required approval and traceability.

NIST’s assessment methods are complementary. A policy and an interview may establish intent; a controlled test establishes whether the deployed mechanism delivered the expected outcome.

Control Testing Methods

In practical control-testing language, teams often use inquiry, observation, inspection, re-performance, and automated continuous control monitoring. These are useful techniques, but they are not five separate top-level NIST assessment methods. NIST’s assessment guidance groups activities under examine, interview, and test.

Technique	How to use it	Example
Inquiry	Ask owners how a control operates; corroborate their answers.	Ask how emergency access is approved and revoked.
Observation	Watch a control or process as it operates.	Observe an operator completing an access review.
Inspection	Examine records, settings, code, and approvals.	Inspect an identity-provider rule and its change ticket.
Re-performance	Independently repeat the procedure or exercise the mechanism.	Attempt an unauthorized request and verify denial and logging.
Automated continuous control monitoring	Run scheduled or change-triggered checks and review exceptions.	Detect configuration drift and retain dated pipeline results.

Observation and inspection can support NIST’s examine method; inquiry commonly supports interview; re-performance is a way to test. Continuous monitoring brings these checks into an ongoing cadence but does not eliminate manual review of scope, exceptions, or ambiguous results.

Designing an Audit-Ready Testing Workflow

An audit-ready test starts with a requirement and ends with evidence that someone else can understand without a meeting. Requirements become testable objectives, tests produce artifacts, failures become tracked work, and retests close the loop.

A step-by-step compliance testing process

1. Scope the requirements. Identify applicable obligations, systems, data flows, assessment periods, owners, and justified exclusions.
2. Map controls to tests. Translate each obligation into a control objective, positive and negative scenarios, expected results, and evidence sources.
3. Test design and operating effectiveness. Inspect whether the control is appropriately designed, then exercise or observe it to determine whether it works in the relevant environment and period.
4. Collect evidence. Retain dated inputs, outputs, configurations, logs, approvals, environment and deployment identifiers, and tester or pipeline identity.
5. Remediate failures. Record the failed condition, affected asset, owner, corrective change, approval, and risk decision for any exception.
6. Retest. Run the failed case and relevant regression cases after the fix; preserve both the original result and the retest.
7. Report. Summarize scope, test coverage, outcomes, open exceptions, evidence references, and limitations for reviewers and auditors.

Start with a control contract

For each requirement, record five things:

1. Control objective: State the protection or outcome in operational terms.
2. Test scenario: Define the request, event, configuration, or process to execute.
3. Expected result: Write objective pass and fail conditions.
4. Evidence artifact: Identify the logs, records, outputs, approvals, or screenshots to retain.
5. Ownership and remediation: Assign the control owner and the person responsible for resolving a failure.

NIST’s assessment procedures for Revision 5 support selecting procedures, methods, objects, depth, and coverage that match the control and system environment. A vague case such as “verify encryption” is not executable. A stronger case identifies the data flow, checks effective configuration, sends a representative transaction through the path, verifies behavior, and stores the result with a timestamp and environment identifier.

Capture evidence as the test runs

Evidence collection should be part of the test harness, not a manual task left for the end of the quarter. Store the test definition, input or fixture identifier, environment, commit or deployment reference, execution time, result, and relevant logs together. Where screenshots are needed, supplement them with machine-readable output because screenshots are difficult to search, compare, or validate.

A finding should state the failed condition, affected asset, severity, evidence reference, corrective action, owner, and retest result. Don’t write “authentication issue” when the actual failure is “the service accepted a session after the required expiration condition.”

Control the test environment

Use an isolated or production-like environment with side effects disabled, sanitize sensitive inputs, and define who can approve and execute the run. If replayed requests can send email, create orders, charge accounts, or modify customer records, add explicit guards before execution.

For cryptographic controls, teams often need a separate evidence trail for key creation, approval, rotation, access, and retirement. A focused resource on how to build a crypto audit process can help connect those events to the broader control matrix.

Preserve evidence against casual alteration. Restrict write access, retain hashes or equivalent integrity indicators where appropriate, and keep the relationship between the original result and any exported report.

How to Automate Compliance Checks in CI/CD

Automation works best when a specific control objective can be expressed as a repeatable assertion. Put requirements and policies under version control, attach test IDs to control IDs, and choose checks based on what a change could affect. NIST guidance on policy as code describes executable policy logic that can run in CI/CD and at runtime.

• Policy-as-code gates: Evaluate infrastructure definitions, access rules, approved configuration baselines, and deployment restrictions before release. A failing rule should identify the control, asset, and condition, not merely return a red build.
• Application and API assertions: Test permission boundaries, retention and deletion behavior, required logs, and relevant accessibility flows. A scan alone cannot cover every manual or operational requirement.
• Change-triggered coverage: Run authorization checks for permission changes, boundary tests for network changes, and logging assertions when observability configuration changes.
• Audit-trail evidence: Save the policy version, commit, approver, pipeline identity, environment, timestamps, test input identifier, results, and remediation or exception reference in a controlled evidence store.
• Replay-based checks: Send masked, production-shaped requests to an isolated candidate service. Verify that sensitive fields were transformed before replay, unauthorized paths stay denied, expected audit events appear, and no external side effects occur.

A passing pipeline proves only the conditions it actually tested. Runtime drift, manual procedures, new data flows, and scope decisions still need monitoring and named human owners.

Safely Replaying Production Traffic for Realistic Validation

Synthetic tests are clean by design. Production traffic isn’t. Real requests contain unusual sequences, optional fields, stale sessions, retries, malformed input, authorization differences, and state transitions that scripted happy paths rarely reproduce. Those behaviors often expose the difference between a control that works in theory and one that survives actual application use.

Copying live traffic and data into staging creates a different problem. PCI DSS v4.0.1 Requirement 6.5.5 prohibits live primary account numbers (PANs) in pre-production environments unless those environments are in the cardholder data environment and protected accordingly, so realistic validation must separate traffic realism from raw data. That is why masking, anonymization, or pseudonymization matters before sensitive data enters a non-production environment.

Preserve behavior without preserving identity

A useful transformation retains the properties the test needs:

• Protocol structure: Methods, paths, headers, content types, and response relationships remain valid.
• State transitions: A login can precede a session request, and a session can precede an authorized action.
• Referential integrity: A masked customer identifier remains consistently linked across related requests.
• Edge-case patterns: Invalid values, retries, missing fields, and error paths remain available for testing.
• Privacy boundaries: Personal, payment, credential, and account data are tokenized, redacted, or replaced with controlled fixtures.

Masking isn’t automatically safe because a field was renamed or partially hidden. Evaluate re-identification risk, access controls, retention, purpose limitation, and whether transformed values can be joined with another dataset. Verify that masking didn’t remove the relationship that makes the test meaningful.

A record-and-replay workflow can capture HTTP requests, apply filtering and transformation, and send the sanitized workload to an isolated service. The production-data masking workflow for testing is relevant when the objective is to preserve request behavior while removing sensitive values from the test corpus.

Make replay evidence reproducible

A replay becomes compliance evidence only when the team can explain what was captured, transformed, excluded, and run, and what result was expected. Record masking rules, fixture version, test environment, deployment identifier, timestamps, response comparisons, and application-state checks. Keep the original sensitive capture out of the test system, and limit access to controlled source material.

Replay also needs guardrails. Disable external side effects, isolate queues and databases, prevent outbound notifications, and use test credentials with narrowly defined permissions. A realistic request without realistic consequences is the right balance for most validation work.

The video below provides additional context for using traffic replay as part of controlled testing.

Executing a Shadow Test for Network Segmentation

A network-segmentation test passes only when it proves the boundary under realistic application traffic. Healthy responses are not enough. The evidence must show that an allowed path reaches the intended service while requests outside the permitted boundary cannot access the cardholder data environment.

Start by defining the boundary and exclusions. Identify the candidate service, protected service, permitted routes, denied routes, and production side effects that must never occur. Obtain authorization from system owners before capturing or replaying traffic. Sensitive fields must be sanitized before the workload leaves the controlled source environment.

Capture at the right observation point

If the application terminates TLS at a trusted internal boundary, the team can observe the plaintext HTTP request after termination, subject to authorization and data-protection controls. That request includes the method, path, headers, body structure, and session context received by the application. Capturing earlier may leave the payload encrypted. Capturing later may omit the network behavior required to validate segmentation.

Filter the workload to relevant routes, then transform regulated values while preserving meaningful relationships. Run the candidate service in an isolated environment with writes, notifications, payment actions, and external integrations disabled. Replay ordinary requests alongside sequences that exercise authorization, session handling, error behavior, and logging.

A shadow test sends captured traffic to a candidate service without replacing the live response path. The shadow testing documentation describes this pattern. Engineering and security teams still define the control objective, masking rules, isolation boundaries, and acceptance criteria.

Compare more than status codes

A successful status code does not demonstrate that segmentation worked. Compare response codes and bodies where safe, then inspect application state, audit events, downstream calls, and network telemetry. A denied request should remain denied, expose no protected content, and produce the expected security record.

Store the capture identifier, transformation rules, fixture version, deployment identifier, timestamps, response comparisons, and application-state checks in controlled, append-only records. Preserve access history and approvals so another engineer can reproduce the test and identify alteration.

For each route, preserve:

• The intended boundary: Which source and destination relationship the test represents.
• The expected decision: Allow or deny, with the control objective stated plainly.
• Observed behavior: Response, logs, state changes, and relevant network evidence.
• Failure interpretation: Whether the issue is routing, authorization, service configuration, logging, or test setup.
• Remediation status: The change made, approval reference, and retest result.

Applicable service providers using network segmentation must penetration-test those controls at least every six months and after changes to the segmentation method, as described in PCI DSS testing requirements. A shadow test can strengthen the evidence package by exercising realistic application paths, but it does not replace a required penetration test. Compliance testing, penetration testing, performance testing, resilience testing, and ordinary QA can support the same assurance effort while retaining different objectives.

Compliance Testing Checklist

Use this checklist for a release, scheduled assessment, or change-triggered review:

• Identify applicable laws, standards, contracts, and internal policies; document scope and exclusions.
• Map each in-scope requirement to a control owner, test case, expected result, and evidence source.
• Check both control design and operating effectiveness, including denied and failure paths.
• Confirm test environments, credentials, masking, and side-effect controls are safe.
• Run relevant automated checks in CI/CD and document required manual checks.
• Save dated, attributable results with the policy version, deployment reference, and logs.
• Record failures, approvals, exceptions, and corrective actions with named owners.
• Retest fixes, assess related regressions, and retain original and follow-up results.
• Report coverage gaps and evidence limitations; schedule checks for changes and control drift.

Avoiding Common Pitfalls and Building Continuous Assurance

The most damaging compliance mistake is treating the audit date as the start of testing. By then, evidence may be scattered, control owners may have changed, and a passing functional suite may say nothing about the configuration that reached production.

Replace the common shortcuts

One-time certification mindset creates gaps between assurance activities. A control can drift after an infrastructure change, identity-provider update, vendor integration, or emergency release. Schedule recurring checks alongside deployment and operational work.

Automated functional QA alone misses governance, evidence, and configuration failures. A test may confirm a workflow succeeds without verifying its audit event, denied path, or approval record. Add control-specific assertions and retain their output.

Scattered evidence turns an audit request into an archaeology project. Store test definitions, results, approvals, logs, exceptions, and remediation records with stable identifiers.

Uncontrolled realism can expose production data or trigger real side effects. Mask inputs, isolate environments, restrict credentials, and verify that transformed traffic still represents the regulated workflow.

Put assurance into the delivery system

A practical pipeline doesn’t run every possible compliance test on every commit. It selects checks by risk and change impact. An authorization change should trigger permission-path and audit-log tests; a network change should trigger boundary validation; a logging change should verify event generation, delivery, retention, and access restrictions.

Keep human review where judgment matters. Automated checks can compare responses, inspect configuration, and collect logs. They can’t decide on their own whether a new data flow changes regulatory scope or whether an exception remains acceptable. Assign those decisions to named owners and record approvals.

DORA has moved resilience expectations toward ongoing assurance. The ISACA discussion of NIS2 and DORA requirements notes that financial entities must maintain operational-resilience testing and address vulnerabilities identified through testing. That favors teams that continuously connect requirements, representative scenarios, evidence, and remediation.

The engineering view is simple: compliance belongs in the system’s feedback loops. Build the control matrix beside the service, run representative tests after meaningful changes, protect the test data, and make failures actionable. When an auditor asks whether a safeguard works, the answer should be a reproducible result, not a promise that someone checked the policy.

Compliance Testing FAQ

What is compliance testing in software testing?

It is testing whether software and its supporting controls meet applicable regulatory, standards-based, contractual, or internal requirements. It is also called conformance testing; tests should specify the requirement, expected outcome, and evidence.

Who performs compliance testing?

Developers and QA engineers can run application checks; security, platform, privacy, and control owners can assess their respective controls. Internal or external auditors may independently review the results and other evidence, but audit and test responsibilities are not identical.

What is an example of compliance testing?

For a requirement to restrict access to sensitive records, test that an authorized role can read a record, an unauthorized role cannot, and both attempts generate the required audit events. Save the test inputs, deployment reference, outputs, and logs.

Is compliance testing functional or non-functional?

It can involve both. A test might check a functional outcome, such as rejecting an unauthorized request, or a non-functional requirement, such as audit logging, accessibility, or retention. The defining feature is the requirement being verified, not a single testing category.

---

GoReplay captures and replays live HTTP traffic into isolated test environments, giving engineering teams a way to validate real workflows, compare behavior, and support compliance evidence when paired with masking and a control matrix. Visit GoReplay to evaluate how traffic replay can fit into your audit-ready testing workflow.