Verify every AI knowledge assertion before someone acts on it

Self-updating knowledge makes verification more important, not less

Enterprise search used to return a list of documents. A knowledge agent can now assemble an answer, retain corrections, and act across connected systems. The interface feels more complete, but the underlying evidence can still be stale, unauthorized, contradictory, or authoritative for the wrong question.

Almanac’s current launch is a useful trigger. The company describes an agent that connects to tools such as Gmail, Slack, and GitHub, compiles company information into a self-updating wiki, links every line to a source, remembers corrections, and can take actions. Its Y Combinator launch says the agent pauses for a user at login, payment, or a decision. Those are product claims, not independent proof of accuracy, but they capture the operating shift: retrieval, memory, answer generation, and action are converging.

The August 31 Hacker News launch discussion reached 58 points and 48 comments. That is a meaningful product-specific conversation, not evidence of universal adoption. The durable problem is broader. Microsoft’s RAG guidance warns that connectors have different authentication, indexing, file, and security-trimming behavior. Its security guidance treats every retrieved chunk and memory write as untrusted. NIST’s Generative AI Profile calls for documenting origins and retrieval processes, tracking provenance, and monitoring risk.

A citation is necessary but incomplete. A Slack message may be newer than a policy but lack authority to change it. A signed contract may govern one customer and not another. A runbook may describe production version 4 while the team operates version 5. A source can be accurate when authored and wrong for today’s date. An employee may be allowed to discover a policy title but not read the attached investigation.

Store atomic assertions, not one polished paragraph

Operations cannot govern an answer as a blob. Split it into claims that can be tested independently. “The customer renews automatically next month at the current rate” contains at least four assertions: the renewal is automatic, the date is next month, the rate is unchanged, and no notice or exception alters the result. Each can come from a different source and have a different status.

assertion_id: KA-2026-0903-017
claim: "Customer ACME renews automatically on 2026-10-01."
claim_type: contractual_fact
subject: contract/ACME-2024-88
sources:
  - id: CLM-8871
    authority: executed_contract
    version: amendment-2
    anchor: section-8.2
access_scope: legal-sales-ops
observed_at: 2026-09-03T09:14:00+08:00
effective_from: 2026-04-01
effective_to: null
applicability: {entity: US-Co, product: Enterprise, customer: ACME}
support: direct
conflicts: [CRM-5541]
action_eligibility: blocked_pending_conflict
review_owner: commercial-operations

The record needs two kinds of time. observed_at says when the system retrieved the source; effective_from and effective_to say when the assertion governs. “Updated yesterday” is not the same as “effective today.” Retain the original source version or a controlled immutable reference so a later edit cannot silently change the basis for yesterday’s action.

Authority is claim-specific. The CRM may be authoritative for the account owner, the executed contract for renewal terms, the billing platform for invoices already issued, and the approved pricing system for list price. A single global “trusted source” rank is too crude. Model authority as a matrix of source, claim type, entity, environment, and time.

Action eligibility is also explicit. An assertion can be informational, draft-only, reviewer-ready, approved for a named action, blocked, or expired. “High confidence” is not an action state. The AI may calculate a retrieval score or propose eligibility, but only a policy service and named reviewer can approve a consequential transition.

Separate retrieval, verification, decision, and action

Use identity-bound retrieval. Microsoft’s Retrieval API states that it returns content a user is authorized to access, with Microsoft 365 security and compliance controls in place. That design principle should continue through caches, embeddings, memories, and exports. Never use a privileged service account to retrieve a broad corpus and then rely on the model to hide unauthorized passages.

Retrieved content remains untrusted even when permission is valid. A compromised wiki page, prompt injection in a ticket, stale incident note, or malicious attachment can instruct the agent to ignore policy or disclose secrets. Strip active instructions from the control channel, preserve them as quoted evidence, label origin and trust, and keep tool authorization outside retrieved text.

for claim in decompose(answer_candidate):
    evidence = retrieve_with_user_identity(claim, request.scope)
    assert no_permission_expansion(evidence, request.user)

    mapped = map_exact_support(claim, evidence)
    authority = authority_matrix.evaluate(claim.type, mapped.sources)
    time_state = temporal_policy.evaluate(claim, request.decision_date)
    conflicts = find_material_conflicts(claim, evidence)

    if mapped.unsupported or authority.unknown or time_state.stale:
        claim.eligibility = "blocked"
    elif conflicts.material:
        claim.eligibility = "blocked_pending_conflict"
    else:
        claim.eligibility = "reviewer_ready"

release_only_after_human_approval(hash(all_claims), exact_sources)

Bind approval to the exact answer, assertion set, source versions, decision, audience, and expiry. If the model rewrites a material sentence or the source changes, invalidate the permit. This prevents a reviewed renewal brief from becoming authority to send a notice with different terms.

Run five gates from question to release

1. Scope gate: freeze the decision before searching

Record the exact question, requester, intended audience, relevant entity and dates, business consequence, source permissions, and downstream action. “What is our refund policy?” is underspecified. The answer can vary by product, geography, purchase channel, contract, customer type, and transaction date.

2. Evidence gate: preserve what retrieval actually returned

Store source IDs, versions, anchors, timestamps, access identity, query, filters, and rankings. Record excluded systems and failed connectors. A failed search is unknown coverage, not evidence that no policy exists. Deduplicate repeated copies without losing lineage.

3. Assertion gate: make every sentence testable

Separate source facts, arithmetic, interpretation, recommendation, decision, and action instruction. Link every material claim to an exact span. If the answer says “therefore,” inspect whether the conclusion is supported or merely plausible. Mark partial support and inference; do not hide them behind fluent prose.

4. Authority gate: resolve time, applicability, and conflict

Apply the reviewed authority matrix. Check supersession, effective dates, subject, environment, exclusions, and material conflict. When hierarchy cannot decide—such as two executed amendments with unclear order—route the evidence to the contract owner. The model should not average contradictory rules.

5. Release gate: approve one answer for one use

The human reviewer sees the proposed answer, atomic assertions, exact evidence, unsupported elements, conflicts, and intended action. Approval is narrow and expiring. Record the released hash and, if action follows, the target-system receipt. Revalidate when a source changes, the decision date passes, or an exception expires.

Worked example: prepare a customer-renewal brief

A customer success manager asks, “Is ACME renewing automatically next month, and can I tell Finance to invoice at the current rate?” The knowledge agent retrieves an executed master agreement, amendment two, a CRM renewal record, the pricing catalog, a Slack message from Sales, and an unresolved support ticket.

The first generated answer says, “Yes. ACME renews October 1 at the current $240,000 annual rate.” Decomposition finds six assertions: contract identity, renewal mechanism, renewal date, notice status, applicable price, and permission to instruct invoicing. The executed amendment supports the automatic-renewal clause and October 1 date. The CRM record shows September 1 because it uses a legacy field. The pricing catalog shows $260,000 list price, while the Slack message says Sales promised to hold $240,000. No executed price amendment is attached.

The system marks renewal mechanism and contract date as reviewer-ready, the CRM date as conflicted, the price as unsupported for action, and invoicing authority as outside the knowledge agent’s role. The support ticket adds another relevant fact: the customer disputes a service-level credit. It does not automatically alter renewal, but it may affect the invoice and requires Finance and Legal review.

The released brief says what is verified, names the conflict, requests the missing executed price evidence, and identifies Finance and Legal as decision owners. It does not manufacture a single confident answer. Once the commercial owner records the valid renewal date and Legal resolves the price commitment, the system regenerates the assertion set and requires a new approval because the evidence hash changed.

Use a claim-specific authority and conflict matrix

Recency does not automatically beat authority, and authority does not eliminate time. A controlled policy can be superseded. A chat message can accurately report an emergency exception. Preserve both, check whether the exception was approved and scoped, and let the named owner decide whether it governs the current case.

Corrections should improve retrieval without erasing history. Store the proposed correction, source evidence, reviewer, effective time, affected assertions, and whether the underlying system of record changed. A model memory that “learns” a correction without a controlled source can spread an unreviewed statement to future answers.

Measure verified decisions, not answer volume

Answer speed matters only after quality is visible. Report the unsupported assertion rate, partially supported rate, material-conflict rate, stale-source rate, permission exceptions, reviewer override rate, median time to verified answer, expired approvals, action reversals, and overdue source owners. Segment by source, claim type, business unit, language, and consequence.

Track coverage honestly. A 95% citation rate can coexist with weak verification if the citations are secondary, stale, or inapplicable. A lower initial release rate may be healthy because the system is abstaining where authority is missing. Publish the denominator: questions received, assertion count, sources searched, inaccessible sources, connector failures, and cases routed to humans.

Use action outcomes as feedback, but do not let the model rewrite authority from them automatically. If a released answer causes an invoice reversal or policy exception, investigate whether the source, retrieval, mapping, authority rule, reviewer, or downstream execution failed. Correct the responsible layer and retest affected assertions.

Failure modes a polished answer can hide

A 30-day pilot should stay read-only

1. Select one recurring, medium-consequence question such as renewal briefs or controlled runbook lookup.
2. Define the exact source register, permission model, authority matrix, decision owners, and forbidden actions.
3. Build 30–50 historical cases with known source versions, conflicts, and final human outcomes.
4. Run identity-bound retrieval and verify that inaccessible sources never enter prompts, caches, memories, or exports.
5. Require atomic assertions, exact evidence anchors, effective dates, applicability, conflict status, and eligibility.
6. Shadow qualified reviewers; do not send notices, change records, approve transactions, or trigger production work.
7. Inject stale policies, contradictory records, missing connectors, prompt injection, renamed customers, and superseded amendments.
8. Measure unsupported, conflict, stale, permission, override, timing, and reviewer-burden metrics by case type.
9. Fix authority rules and source ownership before tuning answer fluency.
10. Release one narrow action only after the read-only pilot passes and rollback, audit, and incident paths are tested.

The go-live review should inspect example receipts, not just aggregate accuracy. Pick correct, incorrect, uncertain, conflicted, inaccessible, and adversarial cases. Verify that each released statement can be reconstructed from the stored evidence and that every blocked case names the person who can resolve it.

The public AlmanacCode/codealmanac repository can be studied as an Apache-2.0 developer artifact related to the founders’ earlier codebase-wiki work. It should not be described as the current Almanac product’s backend. Reuse only in accordance with its license, verify dependencies, and keep product claims anchored to current first-party pages.

FAQ

Is a citation enough to verify an AI answer?

No. Verify that the source directly supports the claim, the requester may access it, it is authoritative for that claim type, it applies to the subject and date, and no material conflict remains.

Can the AI choose which policy wins?

It can apply a documented hierarchy and show the result. If authority, supersession, or applicability is unclear, the accountable policy owner resolves the conflict.

Should the knowledge agent take actions?

Start read-only. Later actions need a separate narrow permit bound to the approved assertion set, exact target, expiry, and human decision, plus target-system evidence and rollback.

How should corrections persist?

Attach each correction to source evidence, a reviewer, effective time, affected assertions, and a controlled source update. Keep prior versions recoverable and expire unsupported memory.

What is the minimum useful output?

A concise answer, atomic assertion register, exact source anchors, authority and freshness states, conflicts and gaps, action eligibility, named reviewer, and an approval record tied to one snapshot.