Substantive testing
Also known as: Substantive procedures, Detail testing, Transactional testing
This process involves verifying the actual accuracy and validity of data or transactions to ensure there are no material misstatements. Unlike checking if a rule exists, it examines the output of that rule to confirm the result is correct. In an IT audit, this means inspecting specific records rather than just reviewing the policy manual.
Why it matters
Relying solely on process checks can hide systemic failures where a policy exists but is ignored in reality. If these tests are skipped or failed, an auditor concludes that the organization's reported data cannot be trusted. This often results in a "qualified opinion" or a significant finding in a SOC 2 or ISO 27001 report. The CISO or CFO typically holds accountability for these failures and must sign off on remediation plans.
In practice
An external auditor performs this step after the initial walkthroughs and control testing phases. They request evidence such as system-generated logs, invoices, or user lists from a specific period. This work produces an Audit Workpaper that documents every sampled item and its pass/fail status. For first-year engagements, auditors typically use larger sample sizes to establish a baseline of trust. In repeat engagements, they may reduce the number of samples if previous years showed zero errors.
Worked example
FinTechFlow Inc. underwent a SOC 2 audit regarding their access management controls. The auditor requested a list of all employees terminated between January and June 2023 alongside their Active Directory account deletion dates. Upon reviewing 25 sampled records, the auditor found that five former employees still had active VPN access months after leaving. This discrepancy proved that while the offboarding policy was documented, it was not being executed. The outcome was a "deficiency" finding in the final report, requiring FinTechFlow to implement an automated ticketing system for offboarding.
Common mistakes
- Confusing this with control testing by verifying that a manager signed a form rather than checking if the data on the form is actually correct.
- Selecting samples provided by the client instead of pulling them independently from the raw database, which risks "cherry-picked" evidence.
- Using an insufficient sample size that fails to meet statistical confidence levels required by standards like AICPA.
- Performing these checks before controls are stabilized, leading to a high volume of avoidable errors and wasted audit hours.
Frequently asked questions
What is the difference between substantive testing and control testing?
Control testing verifies if a process is designed and operating effectively (e.g., "Is there a password policy?"). Substantive testing verifies the actual data resulting from that process (e.g., "Do any current passwords violate the policy?").
When should I use sampling versus testing the entire population?
Sampling is used for large datasets where checking every item is impractical, following a calculated sample size based on risk. Full population testing is performed using CAATs (Computer Assisted Audit Techniques) when the auditor can run a script against 100% of the data.
How do I determine the correct sample size?
Practitioners typically refer to an audit sampling table provided by their firm or standards like those from the AICPA. The number depends on the population size and the "tolerable error rate" defined in the audit plan.
What counts as valid evidence for these tests?
Valid evidence includes system-generated reports, screenshots of configurations, signed contracts, and bank statements. Manually created spreadsheets are generally considered weak evidence unless they can be tied back to a source system.
At what stage of the audit engagement does this occur?
It usually happens after the "design effectiveness" phase. Once an auditor confirms that controls are designed correctly and operating consistently, they perform these tests to ensure the resulting data is accurate.
More terms
Common controls framework · Statement of Applicability · Scoping · Gap assessment · Audit evidence · Crosswalk · Management representation letter · Inherent risk