Before You Blame the AI, Have It Audit Its Own Rules

Before You Blame the AI, Have It Audit Its Own Rules

When an AI-run process underperforms, audit its instructions before changing models. Give the AI every active rule, SOP, prompt, and checklist, then ask for conflicts with exact passages. A human should resolve those conflicts, name the rule that takes precedence, and test the revised process before returning it to work.

THE PATTERN

Why can a capable AI still make the wrong decision?

When a capable model keeps making the wrong call, suspect the rulebook before the model. Fix the rules first.

In the engagements we run, a recurring pattern is a marketing process that will not increase a clearly winning budget. One instruction permits an increase when performance supports it, while another holds spending steady until a separate approval arrives. The process stalls because both rules appear active, and neither explains which should govern that decision.

People reviewed the results and blamed the model's judgment. Nobody saw the contradiction until the AI read every instruction side by side.

Vista calls this Harness-Over-Model: inspect the instructions, permissions, and checks around the model before assuming the model is the problem. A stronger model needs a clear operating contract so choosing between incompatible rules does not leave the business decision hidden inside a guess.

What you need. Gather the active instruction files, a recent example of a disappointing decision, and the person authorized to settle conflicting rules. Keep an untouched copy of the current instructions so you can compare changes.

Instruction sets grow one fix at a time. A sloppy invoice adds "check before sending"; a slow week adds "move faster." Eventually the AI is told to move quickly, wait for approval, keep the old process and improve it. Nothing says which duty wins.

A rule review is cheaper than a model swap, and it usually tells you more. It shows, in writing, why a decision stalled.

THE INVENTORY

What should you give the AI to audit?

Give the AI the instructions the process actually receives, with their file names and status. An audit of a tidy handbook will miss the extra directions attached to daily work.

1. Gather every active instruction source

Collect the SOPs, prompts, system rules, approval checklists, templates, and recurring task instructions used by this process.
Why it matters: a contradiction can cross document boundaries, so reading each file separately can conceal it.

Start with a recent task and trace the directions from the request through the approval and reporting stages. Ask the operator what they paste in, what they attach, and what they assume the AI already knows. Include those extra instructions in the inventory, with a clear label showing where they enter the process. For each item, record:

• Its file name or other stable source label.
• Whether it is active, a draft, or an archived reference.
• The part of the process it governs.
• The person responsible for keeping it current.

Label archived material as archived so nobody, human or AI, mistakes it for a live rule. If you cannot establish whether a rule is active, label that uncertainty for review. Treating every old note as binding recreates the very problem you are trying to solve.

Give the AI the original wording. A summary smooths out exactly the differences you are hunting for. Preserve the wording that specifies approvals, limits, exceptions, and deadlines because those details determine whether two rules can coexist. A general summary such as "be careful with spending" removes the condition you need to compare.

Use a working copy in the environment your business permits, and list inaccessible instructions as gaps in the review. Ask the AI to confirm what it could read before evaluating whether its findings cover the whole process. Check its report for:

• Which files it read.
• Which active sources it could not read.
• Which decisions those missing sources govern.

Writing SOPs faster with AI can help document the process. This audit serves a different purpose: checking whether those documents remain coherent when the work crosses from one stage to the next.

THE REPORT

How do you get findings you can actually verify?

Demand evidence in every finding. A general critique leaves you guessing what to change. The report should show the passage, the affected decision, and what needs review about the interaction between rules.

2. Request an evidence-backed audit

Ask the AI to compare all supplied instructions before suggesting edits.
Why it matters: premature rewriting can erase a useful safeguard before you understand what it protects.
Use the prompt below after providing the complete instruction set and marking each source as active, draft, or archived.

Audit the supplied instruction set as a whole. Do not edit any file.
First list the files you could read and any missing or unclear sources.
Distinguish active instructions from drafts and archived references.

Find conflicts, duplicates, potentially stale rules, and rules with no owner.
For each finding, report:
- Type and affected decision.
- File name and exact passage from each relevant source.
- Conditions under which the instructions collide or become unclear.
- Likely operational consequence.
- What a human must decide or verify.

Separate confirmed conflicts from possible conflicts.
Do not assume newer files outrank older files.
Leave precedence decisions and change approvals to the owner.
Finish with unresolved questions and suggested cases for testing.

Sample finding. Conflict. Ad-spend SOP: "Increase budget when results hold." Weekly checklist: "Hold all spend changes until the owner signs off." Decision needed: which rule governs a mid-week increase, and above what limit does approval apply?

Check every finding against the original passage. Trust the quote over the AI's confidence. If it cites a phrase you cannot find, reject the finding and ask for the correct reference.

3. Classify each finding before fixing it

Sort the findings by the kind of decision required.
Why it matters: an identical sentence, an obsolete condition, and an incompatible approval rule need different repairs.

Type	Example	How to resolve
Conflict	One rule permits an increase; another requires approval for that same increase.	Decide the approval requirement and state which rule governs that case.
Duplicate	The same reporting instruction appears in several active files.	Keep a canonical version and point other documents to it.
Stale	A checklist refers to a review stage the team no longer uses.	Verify the current process, then retire or replace the instruction.
No owner	A rule requires approval but names no accountable role.	Assign an owner, a backup, and a route when neither is available.

Treat a possible conflict as a question for the owner. For example, "publish quickly" and "check accuracy" can coexist if the process defines the check. They become an operational problem when the AI must choose between a deadline and an unfinished review with no escalation path.

Similarly, a rule is not stale merely because it is old, so ask its owner whether the original purpose still applies. A longstanding restriction may be doing essential work even if nobody has written down the reason for keeping it. The report should distinguish that undocumented safeguard from an outdated reference to a process the business no longer uses.

THE DECISION

Who decides which rule wins?

A human with authority over the process should settle each conflict. The AI can show the options and their consequences, but the choice belongs to the business.

4. Resolve conflicts with the accountable owner

Review each confirmed conflict with the person responsible for the affected decision.
Why it matters: choosing speed, approval, spending limits, or quality standards changes how the business operates.

In the stalled marketing example, decide whether the AI may increase spending under a defined condition or must wait for approval. "Be more proactive" does not answer that question or explain the owner's intended boundary between initiative and permission. Specify what it may change, what evidence it needs, and where it must stop before making a further commitment. Keep a short decision record:

• The conflict and the original passages.
• The approved behavior and its limits.
• The owner who approved it.
• The files changed and the cases to test.

Fix the condition. Do not add another broad warning. If approval is required above a business-defined limit, state that limit in the appropriate rule. If the business has not decided the limit, leave the task blocked and name the decision needed.

5. Add a precedence rule

State which document governs when active instructions disagree.
Why it matters: without precedence, the next exception can reopen the same argument.

For example, an owner-approved hierarchy might give the operating policy precedence over task checklists and routine requests in the process. Explain how a valid exception gets authorized and where it is recorded so that departures from the hierarchy stay visible. Without that exception route, people may work around the hierarchy whenever a routine request needs a different answer.

Unresolved conflict. If precedence does not settle a disagreement, stop the affected action and ask the named owner. Return the competing passages and the decision needed, rather than quietly picking a convenient interpretation.

Update every active location affected by the decision, including frequently used templates where an old rule can defeat a clean policy document. Remove duplicates where practical, and use references to a canonical instruction when several stages need the same rule.

Delegating work to AI safely depends on clear authority as well as clear instructions. This review should produce both: a usable rule and a defined point where the AI returns control to a person.

THE MAINTENANCE

How do you keep the rules from fighting again?

Test the revised instructions on actual decision cases, then repeat the audit monthly or after a substantial change. Test the rules before returning to work.

6. Re-run the audit and test the revised process

Repeat the comparison after applying approved edits, then test the affected decisions in a controlled review.
Why it matters: removing one conflict can expose another, and clearer wording can still produce an unexpected action.

Start with the disappointing decision that triggered the audit. Ask the AI what it would do under the revised rules, which passage authorizes that action, and when it would escalate. The owner should be able to follow the answer without supplying unwritten assumptions. If the explanation depends on an assumption, document or resolve it. Include cases that distinguish the intended approval limit:

• A routine action the AI should complete independently.
• An action requiring approval under the revised rule.
• A missing-input case where it should request clarification.
• An unresolved contradiction where it should stop and escalate.

Block about an hour for the first session. The finish line is every finding verified and every conflict assigned an owner.

For a small team, maintenance means naming a rule owner, keeping the active inventory current, and recording why each new instruction exists. When adding a rule, ask what it replaces and whether it overlaps an existing instruction already governing the process. Otherwise, each repair becomes another layer that the next audit must untangle before anyone can resolve the underlying decision.

Use confidence gates for AI output to handle uncertain results alongside this review of the rules that govern the process. Uncertainty about an answer and uncertainty about authority need separate responses from the people responsible for the work. One calls for checking the output; the other calls for settling what the process is allowed to do.

Bring the instruction inventory and one real stalled decision to Vista's Collective. Start the audit on the process that needs repair.

Maintenance record. Keep the active file inventory, approved decisions, and test cases together. The next audit should be able to distinguish a deliberate change from another unexplained exception.

COMMON QUESTIONS

Frequently asked questions

What does an AI instruction audit check?

An instruction audit checks the whole rulebook. It compares the rules, SOPs, prompts, and checklists for incompatible directions, repeated requirements, outdated references, and missing owners. Each finding identifies its source passage and affected decision so a person can verify the evidence and approve changes to the operating rules.

Should I change models before auditing the rules?

Audit the instructions first when a process stalls or behaves inconsistently without an obvious explanation. Clarify the rules and test the affected decisions, since a different model will receive the same unresolved directions. Then assess the model alongside other possible causes, including missing inputs or a task whose design needs repair.

Can the AI decide which conflicting rule wins?

The AI can identify competing passages and explain the consequences of different choices. When precedence leaves a conflict unresolved, the accountable human should decide. The audit prompt asks for a decision to review. It never authorizes the AI to rewrite policy.

What if the AI cannot read every instruction file?

Treat the audit as incomplete and list the missing sources explicitly, preserving that limitation in the report until the gap is closed. Review the available material while recognizing that inaccessible instructions may contain approvals or restrictions governing the decisions under review. Obtain the missing passages through an approved route, or ask the responsible owner to compare them.

How often should we audit AI instructions?

Run an instruction audit monthly or after a substantial change to the process, responsibilities, or approval rules, and when decisions stall repeatedly. Maintain an active file inventory and decision record so the review can focus on the instructions that govern current work. Recheck approved changes against examples of both permitted actions and required escalations.

Does a clean audit mean the automation is safe?

A clean audit means the supplied instructions showed no unresolved findings after review, without establishing that the process will always act correctly. Test representative decisions, inspect its explanations, and verify approval boundaries before returning it to work. Missing inputs, poor task design, and incorrect outputs still need checks because clear instructions alone cannot prevent every problem.