Do not call it an AI recruiting pilot until it passes the readiness gate

The pilot starts after the pre-pilot gate, not after the demo

A November 1 practitioner account described an AI recruiting tool that was piloted through the summer and then stopped by security review. It is one self-reported case, not prevalence data. The useful signal is procedural: HR can spend months testing a workflow that security, privacy, legal, procurement, or accessibility specialists were never given a fair chance to evaluate.

The fix is not another 100-question form. It is a small, decision-ready packet assembled before live candidate data enters the system: the exact use case, prohibited functions, field-level data flow, subprocessors, access roles, retention and deletion proof, model-change policy, incident path, candidate notice, accessible alternative, human decision boundary, baseline, metrics, and stop rules. Each reviewer sees the same configured workflow rather than a different slice of vendor marketing.

This ordering matches current official guidance. The UK government's responsible-AI recruitment guidance assigns actions across procurers, deployers, and assurance providers and calls for governance, risk assessment, impact assessment, data protection, transparency, human oversight, and monitoring. The UK's data regulator tells recruiters to identify the lawful basis for processing, complete a data-protection impact assessment where required, be transparent with candidates, minimize data, and check third-party providers. Canada's Public Service Commission similarly frames AI use around accountability, documentation, privacy, security, accessibility, bias, and meaningful human oversight.

Use three readiness states

Do not collapse those states into a single “approved” checkbox. A tool may be ready to draft outreach but not rank applicants; ready for de-identified shadow testing but not live candidate records; ready in one country but not another; or ready for one product version only. Approval attaches to the tested conditions, and it expires when a material condition changes.

Readiness packet: eight artifacts, one owner each

A 2026 systematized review of recruiting agents found that none of the coded evaluation frameworks jointly covered utility, fairness, privacy, and security. That does not prove every product is deficient; it explains why an employer should not accept one accuracy or productivity score as a readiness decision. The gate must combine operational usefulness with rights, data, security, and human-control evidence.

The hiring AI market is moving faster than HR's measurement model

LinkedIn's 2026 Hiring Release promises to surface relevant applicants, use trusted network signals, and streamline recruiting workflows. Those features may save time, but the product claim does not define an employer's acceptable decision boundary. HR still has to decide what the tool may influence, how results will be checked, and what evidence would justify continued use.

The July community signal is blunt. In a July 20 discussion on r/humanresources about video and AI interviews, the top comment said management can see a dashboard and a promise to reduce time-to-fill while carefully worded candidate-experience concerns disappear. The thread received 42 points and 33 comments in the research window. This is not a representative survey, but it identifies a real governance failure: efficiency is visible immediately, while candidate trust, accessibility, and decision quality are easier to ignore.

That failure is avoidable. A pilot is not a smaller deployment. It is a controlled comparison designed to reduce uncertainty. It needs a defined workflow, a baseline, a comparison cohort, pre-registered metrics, named owners, stop rules, and a final decision. If the team changes the success criteria after seeing the results, the pilot becomes a sales demonstration.

This guide complements our AI resume screening risk guide. The earlier page explains why automated screening can be high risk. This skill gives HR a reusable method for evaluating a specific tool or workflow without pretending every AI feature is the same. If the proposal involves Claude in Chrome or another browser agent on LinkedIn Recruiter, begin with the browser-agent platform boundary workflow before designing the pilot.

Start with the decision boundary, not the vendor feature list

Map what the tool actually does in the employer's process. "AI-assisted recruiting" can mean drafting outreach, suggesting search terms, scheduling interviews, summarizing notes, ranking applicants, scoring video responses, or automatically rejecting people. These are different workflows with different evidence and review needs.

Write the human boundary in operational language. "Recruiters remain in control" is too vague. A testable boundary says: the tool may group applicants by recruiter-authored, job-related criteria; it may not add criteria, hide applicants, or send rejection messages; a recruiter must inspect the full applicant list and record the reason for each advance or rejection.

Also determine whether the workflow may be covered by specific law or regulation. New York City's Department of Consumer and Worker Protection says covered automated employment decision tools cannot be used unless a qualifying bias audit has been completed within one year, information about it is publicly available, and required notices are provided. Not every AI recruiting feature is necessarily an AEDT. Coverage depends on the function and use, which is why legal analysis must follow the mapped workflow rather than a marketing label.

Turn vendor approval into an evidence register

A pilot should not begin with a generic security questionnaire and a product demo. HR needs an evidence register that connects each vendor claim to the exact workflow, population, product version, validation material, limitation, and owner who will decide whether the evidence is sufficient.

Start with intended use. Write one sentence that describes the allowed function and one that describes the prohibited function. For example: "The tool may transcribe and summarize structured interview notes for recruiter review. It may not infer emotion, honesty, personality, disability, or candidate fit; rank candidates; suppress evidence; or send employment decisions." This statement becomes the reference point for configuration, testing, notices, reviewer training, and incident handling.

Then break the product into components. A single vendor may provide scheduling, transcription, search, matching, video analysis, candidate ranking, and automated communication. Evidence for accurate scheduling says nothing about the validity of a video-derived score. Approve or reject each function separately. Disable functions that are outside the pilot instead of relying on recruiters to ignore their outputs.

Video and voice tools deserve an additional gate because the interface itself may introduce barriers unrelated to job performance. Illinois law, for covered AI-analyzed video interviews for Illinois-based positions, requires notice before the interview, information explaining how the AI works and the general types of characteristics it uses, and applicant consent. Other jurisdictions and legal duties may apply. The operational lesson is broader than one statute: candidate notice must match the actual feature set, not a generic sentence buried in a privacy policy.

The U.S. Department of Justice's ADA guidance gives a concrete failure case: facial and voice analysis can screen out qualified people with autism, speech impairments, or other disabilities even when those traits are unrelated to the job. An alternative process therefore cannot be a support email that candidates may or may not find. The pilot packet should specify how candidates are told about the technology, how they request accommodation or an alternative, who responds, the response deadline, and how the team verifies that requesting an alternative does not reduce opportunity.

Require a candidate-impact preview

Before launch, walk through the process as a candidate using the actual interface. Capture every notice, consent screen, timer, recording requirement, score-relevant instruction, retry rule, accessibility feature, error state, support contact, and data-deletion statement. Compare that experience with the internal workflow map. A vendor's administrative console may describe controls that a candidate never sees.

Use the register as a living control. Each item needs an owner and revalidation date. A document that was sufficient for one role, language, location, or product version may not support another. Approval attaches to the tested workflow and conditions, not to the vendor forever.

Design a comparison that can answer a business question

A strong question is narrow: "Can AI-assisted sourcing expand the qualified prospect pool for these two engineering roles without reducing recruiter agreement or candidate response quality?" A weak question is "Does AI improve recruiting?" Narrow questions determine the necessary baseline and evidence.

1. Freeze the baseline

Document the current workflow before introducing the tool: intake fields, recruiter steps, time spent, sources searched, applicant volume, stage conversion, review disagreements, candidate complaints, accommodations, and known data gaps. The baseline should use the same role family and similar labor-market conditions where possible.

2. Separate pilot and comparison cohorts

Choose a defensible comparison design with qualified analytics and legal input. Depending on scale, this might be a shadow evaluation on historical or synthetic records, a parallel recruiter review, a phased rollout across comparable requisitions, or a randomized process element that does not deny opportunity. Avoid a design where the AI first selects the only candidates humans are allowed to see; that prevents independent assessment of omissions.

3. Pre-register metrics and stop rules

Write the metric definition, data source, owner, threshold, and review frequency before launch. Do the same for stop rules. The team should not debate whether unauthorized rejection is serious only after it happens.

4. Run in shadow mode first

Where feasible, let the tool produce outputs without changing candidate treatment. Compare those outputs with the existing process. Shadow mode does not remove privacy or legal obligations, but it can expose unexplained ranking, missing candidates, unstable scores, or data-quality problems before the tool influences an employment decision.

5. Keep the vendor outside the final judgment

The vendor should explain product behavior, validation, data practices, known limits, changes, and audit support. The employer owns the pilot question and go/no-go decision. A vendor dashboard should not be the only evidence that the vendor's tool succeeded.

Measure quality, experience, and control alongside speed

Time saved is legitimate, but it is a guardrail-free metric. A tool can reduce recruiter minutes by hiding complexity, narrowing a pool, or pushing friction onto candidates. Pair every efficiency metric with quality and risk measures.

The EEOC's employment test guidance is a durable floor: selection procedures should be job-related and consistent with business necessity, and they should be updated when job requirements change. An AI score does not escape this logic because it is difficult to explain. If the vendor cannot show what job requirement the output represents, how it was validated, and where it fails, the employer lacks a basis for treating it as decision evidence.

NIST's AI Risk Management Framework adds an operating structure. Governance is continuous, leadership is responsible for AI risk decisions, and human-AI roles should be explicit. Translate that into the pilot by naming the process owner, data owner, reviewer, incident owner, legal/compliance contacts, and final approver. Avoid collective responsibility, which often means no one can stop the tool.

Failure modes and stop conditions

Set quantitative thresholds where the evidence supports them and qualitative zero-tolerance rules where it does not. Unauthorized rejection, use outside scope, missing legally required notice, and a confirmed data incident are reasonable examples of immediate-stop events. Small-sample fairness results require care, but "too small to conclude" is not the same as "safe." It may mean the pilot cannot support the proposed decision.

Include accessibility and alternative-process testing. Video, voice, timed, or chatbot workflows can create barriers unrelated to job performance. Candidates need a usable way to request accommodation or an alternative, and recruiters need a documented escalation path that does not penalize the request.

FAQ

What should an AI recruiting pilot measure?

Measure recruiter effort after review, workflow accuracy, candidate experience, reviewer disagreement, relevant subgroup outcomes where lawful and appropriate, incidents, and downstream quality signals. Time saved alone is not enough.

Should AI reject applicants during a pilot?

A safer first pilot focuses on drafting, coordination, search assistance, or shadow decision support. A qualified human should remain responsible for advancing or rejecting candidates.

Does every AI recruiting tool fall under NYC Local Law 144?

No. Coverage depends on the definition and the employer's actual use. Obtain qualified legal advice rather than relying on a vendor label or this guide.

Can historical hiring data serve as ground truth?

Historical decisions can contain inconsistency and bias. Use them as one source to examine, not unquestioned truth. Define job-related criteria independently and include qualified human, legal, and statistical review.

What if the sample is too small for reliable subgroup analysis?

Record the limitation and avoid claiming safety. Consider a longer shadow period, broader but comparable evidence, independent validation, or limiting the tool to lower-risk administrative support.