Prompt history is not an external-audit evidence file

Auditors need a controlled evidence chain, not a chat transcript

A September 17 discussion among CFOs and controllers asked what happens when external auditors inspect AI used in finance. The most useful answers were not about model internals. They were about ordinary accounting discipline: no black-box entries, formula-backed workpapers, source-to-ledger traceability, execution logs, named review, exception evidence, and a human gate before anything posts.

One controller described rejecting an AI-built payroll workpaper because it was hard-coded. The replacement needed formulas and visible work so another person could pick it up and understand it. Another practitioner summarized the durable rule: AI may process, but it cannot approve. That is a more useful design target than preserving thousands of tokens of prompt history.

Prompt and response history can help reconstruct what a tool saw and said. It rarely proves that the input population was complete, that the source was reliable, that the correct model and configuration ran, that formulas agree with the source, that exceptions were resolved, or that a qualified reviewer approved the exact artifact used in financial reporting. A screenshot is even weaker: it usually omits report parameters, hidden filters, run identity, later changes, and control totals.

The regulatory direction reinforces this control view. The FRC's generative and agentic AI guidance keeps the human auditor accountable and asks how appropriate confidence in AI outputs is obtained. PCAOB AS 1105 centers sufficient, appropriate, relevant, and reliable evidence, including the controls over electronic information produced or received by the company. FEI's 2026 AI and ICFR framework emphasizes outcome validation, human review, curated ground truth, challenger comparison, analytics, and outlier resolution.

Freeze the AI-touch population before selecting evidence

Start with the process, not the vendor list. Map where AI can affect initiation, authorization, recording, processing, classification, reconciliation, review, disclosure, or monitoring. Include obvious agents and chat tools, but also embedded features inside close, ERP, consolidation, reporting, spreadsheet, data-room, and workflow products.

Then build a complete AI-touch population for the audit period. A population is not only successful model calls. Include failed runs, retries, fallbacks, overrides, rejected recommendations, manually completed cases, direct postings, draft workpapers, and items the system excluded. These records expose whether a control operates consistently and whether management selected only clean examples.

Reconcile the stages. If 10,240 transactions were eligible, 10,100 reached the AI service, 9,950 produced outputs, 9,930 were reviewed, and 9,925 posted, the packet needs an explanation for every difference. Do not net failures against retries. A failure later corrected remains relevant operating evidence.

Scope does not mean every AI use automatically becomes a key control. Inventory broadly, then map each use to the process, reliance, assertion, risk, and control. A tool that improves grammar in a management memo differs from an agent that prepares a revenue journal or performs a SOX control. Qualified management and the auditor determine significance under the applicable framework and engagement.

Map the AI touch to assertions and control objectives

An AI risk register can become detached from financial reporting. Connect each use to what could be wrong in the accounts or disclosures. For journal preparation, relevant assertions may include occurrence, completeness, accuracy, classification, cutoff, and authorization. For anomaly detection over a complete population, completeness and reliability of the input may be more important than the elegance of the model explanation.

Separate AI-dependent controls from controls around AI. A reviewer may rely on a model classification as part of the control. Access management, change approval, source completeness, posting restrictions, and exception review are controls around the system. Test both when they are relevant. A perfect model cannot compensate for unauthorized users or an incomplete population.

Document model and workflow identity precisely. Record provider, resolved model version, application version, prompt or question-schema version, retrieval sources, tools, thresholds, deployment, and change approval. If the workflow used a moving alias, capture the resolved version returned by the provider. If it cannot be reconstructed, treat that as a gap rather than guessing.

Build one reperformable packet per sampled or material item

A packet should allow a qualified person to follow the item without asking the preparer to narrate it from memory. Keep the packet narrow enough to inspect and structured enough to compare across a sample.

request and scope
  -> authoritative source + extract parameters + control total
  -> transformation and deterministic calculation
  -> provider / model / workflow / schema identity
  -> AI output clearly labelled
  -> reviewer challenge + exceptions + corrections
  -> approved workpaper or journal
  -> posting / ledger / report / disclosure destination
  -> assertion and control conclusion by named human
  -> immutable packet index and retrieval check

The source layer needs more than a filename. Retain the source system, report name, parameters, period, extraction time, row count, amount total, owner, and an immutable reference or hash where appropriate. Preserve original records and treat cleaned or transformed data as a new version with explicit rules.

The calculation layer should be independently executable. Use formulas, query logic, scripts, or clearly documented transformations. A workpaper that contains only hard-coded AI results is difficult to reperform and easy to detach from its source. Preserve before/after counts and totals for filters, joins, sign changes, currency conversions, grouping, and thresholding.

def reconcile_population(eligible, runs, reviews, postings):
    assert unique(eligible, "source_item_id")
    assert unique(runs, "run_id")
    assert unique(postings, "posting_id")

    joined = eligible.left_join(runs, on="source_item_id")
    joined = joined.left_join(reviews, on="run_id")
    joined = joined.left_join(postings, on="approved_item_id")

    return {
        "eligible": len(eligible),
        "not_submitted": count(joined.run_id.is_null()),
        "failed": count(joined.run_status == "failed"),
        "unreviewed": count(joined.review_id.is_null()),
        "approved_not_posted": count(joined.approved & joined.posting_id.is_null()),
        "posted_amount": sum(joined.posted_amount),
        "unexplained_differences": list_unowned_gaps(joined),
    }

The review layer distinguishes preparation from approval. Record the reviewer's competence and authority, procedures performed, questions, corrections, exceptions, and conclusion on the exact artifact version. A checkmark without a packet version or procedure is weak evidence. Direct posting or automated control execution needs an independent gate that the AI cannot satisfy itself.

Prompt history belongs in the packet when it is relevant to how the output was created, but it is supplementary to source and control evidence. Long histories may contain private data, unrelated context, or privileged material. Apply approved retention, access, minimization, and legal-review rules rather than exporting everything by default.

Worked example: an AI payroll journal that looks right

A payroll team uses an approved AI workflow to prepare a monthly accrual journal. The output totals $2.42 million and matches the amount expected by the preparer. The first workpaper contains a prompt, a response, and a pasted journal table. It looks clean, but another reviewer cannot see the population, formulas, excluded employees, exchange rates, mapping rules, or how the total reaches the ledger.

The team rebuilds the evidence. The eligible population comes from the payroll register with 8,412 employees and a control total of $2.47 million. The workflow log shows 8,410 items submitted. Two records with invalid department codes were excluded before the model call but were not in the exception log. The AI output also hard-coded a $14,000 currency adjustment rather than linking to the approved rate table.

The corrected journal may equal the original amount. That does not make the initial packet acceptable. The control improvement is not that the AI produced a better explanation. It is that the population, formulas, exceptions, version, approval, and posting are now reperformable.

The external auditor may still request different samples, test controls directly, challenge source reliability, or decide not to rely on management's procedure. Management should log that disposition separately. The workflow prepares evidence; it does not pre-negotiate an audit conclusion.

Failure modes to find before the auditor does

Do not hide contrary evidence. If the AI missed an anomaly, a reviewer overrode it, a fallback model produced a different answer, or a post-close adjustment exposed a weak control, retain the event and link it to remediation. Negative evidence helps a reviewer understand the operating boundary.

Run a 30-day audit-readiness pilot

Week 1: select one bounded finance process and one completed period. Inventory every AI touch, map assertions and controls, identify ITGC dependencies, and agree the evidence-request owner. Do not start with a process where the agent autonomously posts material entries.

Week 2: build the complete population and reconcile stages. Select all material items plus a risk-based sample supplied by the authorized owner. Build item packets and attempt independent reperformance without help from the original preparer. Log every question needed to reproduce the result.

Week 3: seed failures: missing source row, stale extract, changed model alias, hidden retry, hard-coded formula, unauthorized user, unlogged override, unresolved exception, duplicate posting, and a prompt containing unrelated sensitive data. Confirm the workflow stops, routes, or exposes each problem.

Week 4: run a mock request with Controller, process owner, ITGC, Internal Audit, privacy/security, and audit coordination. Measure time to retrieve a packet, population reconciliation differences, reperformance differences, missing versions, exception aging, reviewer rework, unauthorized data exposure, and unanswered request items.

A successful pilot does not guarantee external-auditor reliance. It demonstrates that management can identify AI-touched work, produce controlled evidence, reperform material procedures, preserve exceptions, and answer a request without reconstructing the process from memory.

For account-level source and exception preparation, use the governed reconciliation workflow. For document authenticity and lineage, use the source-document verification workflow. This page adds the cross-process population, control, and audit-request layer.

FAQ

Do we need to retain every prompt?

Retention depends on the workflow, control objective, policy, privacy, legal requirements, and engagement facts. Preserve enough to reconstruct relevant behavior, but do not treat indiscriminate prompt retention as a substitute for source lineage, calculations, controls, and approval evidence.

What if AI only drafts a workpaper?

Evaluate the reliance placed on the draft. If a qualified preparer independently verifies sources, calculations, assertions, and exceptions, retain that review. If the workpaper carries hard-coded outputs that nobody can reperform, the draft remains a weak artifact regardless of who generated it.

Can a second model validate the first model?

A challenger can identify disagreement, but it is not independent proof by itself. Use deterministic calculations, authoritative sources, ground-truth samples, qualified human review, and controls over both systems.

Who signs the final evidence package?

Each owner signs only within their authority: process owner for operational facts, Controller for accounting and financial reporting, ITGC/security for technical controls, Internal Audit/SOX for challenge, and audit coordination for delivery. The external auditor records its own conclusion separately.