How to evaluate an AI interviewer — and where Mochi stands
The dimensions to evaluate any AI interviewing agent against, what each one measures, and our answer to all 34.
September 7, 2026
9 mins
Fabiana Giorgi
An AI interviewer sits at the sharpest point of the hiring funnel. It talks to candidates, and its scores shape who advances. That makes the buying decision unlike most software choices.
The right question is not which tool has the most features. It is which tool produces fair, valid, defensible decisions at scale — and whether you can still explain a rejection eighteen months later, to a works council, a regulator, or the candidate.
Feature lists are not built for that question, so we wrote a rubric instead. Below are the dimensions any AI interviewing agent should be assessed on, what each one measures, and where Mochi — Maki’s AI voice interviewer — stands against it.
Six sections, thirty-four dimensions. Take the whole thing to every vendor on your shortlist, including us.
FIRST, THE THING BEING MEASURED
What Mochi is
Mochi is Maki’s AI voice interviewer. It calls a candidate, runs a structured screening conversation, and returns a scored, auditable result to the recruiter — the step that used to be a 20-minute phone screen nobody had time to do consistently.
A voice conversationOn the web or by phone, in the candidate’s language.
Not a chatbotA fixed conversation graph, so every candidate is asked the same things.
Advisory, never decidingScores go to a recruiter, who can override them.
1InviteThe candidate gets a message with a link, or a call.
2ConsentAI use and recording are disclosed before anything starts.
3ConversationA structured screening on a fixed graph, with follow-ups when an answer is thin.
4ScoringAnswers are scored against fixed behavioural grids, not model intuition.
5ResultA scored, auditable result syncs to the ATS. The recruiter decides.
ACandidate experience
BInterviewer behaviour & conversation control
CSafety, guardrails & content handling
DScoring, fairness & oversight
ESelf-service configuration & operations
FCompliance, security & integration
SECTION A
Candidate experience
The screening is a brand moment whether you designed it as one or not. Most candidates who meet an AI interviewer will not get the job, and all of them will remember the conversation.
4.42/5
Candidate satisfaction
90.4%
Felt the process was fair
87.7%
Improved their view of the employer
78.3%
Started and finished
Satisfaction
How candidates rate the screening immediately afterwards (1–5).
86.6% rate it 4–5★, and 70.4% give it the maximum score. Measured across the portfolio, not a single flagship account.
Conversational realism
Whether the conversation feels natural and adaptive, not scripted or robotic.
Adaptive real-time dialogue on a fixed conversation graph. The dominant candidate theme is that it did not feel like an AI, and it does not invent beyond the role brief.
Fairness perception
Whether candidates feel they got a fair chance to show their ability.
90.4% of candidates found the process fair.
Employer-brand impact
Whether the experience lifts candidates’ opinion of the employer.
87.7% said it improved their view of the employer.
Completion
Share of candidates who finish the screening.
78.3% of candidates who started a Mochi screening finished it.
SECTION B
Interviewer behaviour & conversation control
This is the section nobody puts on a comparison grid, and it is where an AI interviewer either holds an interview or merely holds a conversation.
Structured probing
Whether the agent digs beneath surface answers, pressing for more when a response is thin or vague.
Follows a defined conversation graph and asks follow-ups when an answer is vague or missing an example.
Staying on track
Whether the interview stays on topic and on time, and works through the questions it needs to cover.
Redirects off-topic candidates, time-boxes sections, and covers every required question every time.
Pushback
Whether the agent tests inconsistent or evasive answers rather than accepting them at face value, while holding a professional, non-confrontational tone.
Challenges inconsistent or evasive answers without becoming adversarial.
Call control
How the agent manages the mechanics of a live conversation: closing, pauses, interruptions and recovery from connection problems.
Ends cleanly, handles silence, interruptions and barge-in, and reconnects or resumes after a drop.
Consistency across candidates
Whether every candidate gets the same coverage and depth regardless of who they are or how they respond. Also a fairness consideration.
The same coverage and depth regardless of who is talking.
The conversation adapts. The evaluation does not.
SECTION C
Safety, guardrails & content handling
An AI interviewer will be asked to do things nobody planned for — some by candidates who are curious, some by candidates who are gaming it, and some by candidates who are having a bad day.
Out-of-scope & manipulation resistance
Whether the agent refuses questions outside its remit and holds to its instructions under pressure.
Refuses out-of-scope questions — scores, outcomes, other candidates, prompt-injection attempts such as “ignore your instructions” — and resists manipulation.
Protected characteristics
Whether the agent avoids soliciting protected characteristics and handles them correctly when a candidate raises them.
Never asks protected-characteristic questions. When a candidate volunteers health, pregnancy, age or similar, it handles the moment and excludes that information from scoring.
Recruiter flags and escalation
Whether the agent surfaces situations a human needs to see.
Flags offensive language, threats, eligibility or salary mismatches, and requests to speak to a human.
Candidate distress
Whether the agent recognises and responds appropriately to a candidate in distress.
Recognises distress, de-escalates, and hands off to a human.
Disclosure & consent
Whether AI use and recording are disclosed and consented to up front, in line with applicable transparency law.
AI disclosure and recording consent at the start (EU AI Act transparency, Illinois AIVIA), with configurable disclaimer text.
SECTION D
Scoring, fairness & oversight
Everything above is experience. This is the part that produces a decision — and the part a lawyer will read.
Validity & reliability
Evidence that scoring predicts performance and is consistent across candidates.
Scores against fixed, science-designed BARS grids with observable behavioural anchors, calibrated against expert human ratings before go-live, keeping scoring consistent and explainable across candidates.
Bias & adverse impact
Testing for, and monitoring, differential impact across candidate groups.
Formal bias management: DIF and adverse-impact (four-fifths) analyses across demographic groups. A further independent audit with Holistic AI is underway. Demographic data is consent-based and never used for scoring.
Score explainability
How a score is produced and what sits behind it for the recruiter to see.
AI outputs are advisory only. BARS rubrics and recruiter dashboards, with full traceability via audit logs linking inputs, criteria and outputs. Candidate right-to-explanation supported.
Review & override
Whether a recruiter can review, adjust or override the AI’s scoring.
Human-in-the-loop by design. Recruiters can override AI scores and the AI never makes the final hiring decision. Escalation and interruption procedures documented.
Speech-recognition accuracy across accents
Whether the agent transcribes speech equally well across accents and dialects — a real bias vector for a voice interview, since transcription errors can depress scores for some groups.
Uses Deepgram streaming ASR, chosen for accuracy and multilingual coverage. Scoring ignores accent, fluency and conversational feel, so a candidate’s accent does not drive the score.
Every screening lands in the funnel as a scored, reviewable row — not a verdict.
{{BANNER}}
SECTION E
Self-service configuration & operations
The unglamorous section. It decides whether the thing is still in use in a year.
Language coverage & proficiency
Running the interview in the candidate’s language; validating spoken proficiency.
Conducts interviews in the candidate’s language — production-ready in English, Spanish and French, ~30+ supported. Language proficiency is one of four in-conversation scoring dimensions, with soft skills, reasoning and knowledge, assessed against language-specific criteria. A candidate can switch languages mid-interview to validate a second language.
Candidate Q&A and knowledge base
Whether the agent can answer candidates’ questions about the role or company.
Candidates can ask about the role, company and work environment. Mochi answers from the materials provided — job description, company materials — and if the answer is not available it says so rather than inventing one.
Integrity & proctoring
Identity verification, anti-cheating and monitoring during the screening.
Proctoring runs at the Maki platform level: candidates can be required to enable their camera, with continuous monitoring and AI cheating-detection generating integrity flags for recruiter review. Enabled per customer.
Branding & customisation
Whether the candidate-facing experience and the assessment content can be tailored to the client.
Fully branded to client guidelines — logo, background, agent name, voice — and configured per role family. Assessment content can be contextualised to the client (wording, terminology and scenario framing, for example aviation terminology for a leadership assessment) while preserving the validated construct. A visual avatar is available but off by default, for latency and cost.
Accessibility & accommodations
Accessibility (WCAG), neurodiversity support and accommodation requests.
WCAG 2.1 AA aligned and assistive-technology compatible. Extra time and transcript for spoken tasks; scoring on content, not delivery. Candidates can request an accommodation during the screening, captured as a dedicated state.
AI opt-out and alternative path
A route for candidates who prefer not to be screened by AI.
Customers can offer candidates an opt-out and can disable AI at tenant level. A candidate’s preference to speak to a person is flagged for recruiter follow-up. Mochi is AI-only, so disabling AI removes Mochi.
Interview flow build & testing
How the interview flow is built and validated before it goes live: who configures it, and how changes are tested.
The flow is built as a state machine and validated before go-live by simulating full conversations — logic, edge cases, recovery. Maki configures the flow from client-supplied rules. Broader automated simulation is on the roadmap, and client-facing edit permissions and version history are not established.
Voice quality & latency
Naturalness of the voice and responsiveness of the exchange.
Real-time, full-duplex voice with streaming ASR (Deepgram) and TTS (ElevenLabs) over enterprise CPaaS. A turn-based, finite-state orchestration holds response times around 800ms, stable across 30–45 minute calls, with natural barge-in and interruptions inside defined structural boundaries.
Channel: phone or web
Whether the screening can run over the phone, the web, or both.
Delivered as a real-time voice conversation over web call or phone call. Web is the recommended default — better branding, global consistency, lower cost, richer interaction. Phone is available where telephony is the better fit.
Time to result
How quickly scored results are available to the recruiter after the screening.
Fully automated: scoring runs after the screening and results sync to the ATS with no recruiter effort.
SECTION F
Compliance, security & integration
The section a procurement team reads first and a marketing team writes last.
Regulatory compliance
Alignment with hiring-AI regulation in the markets of use.
GDPR compliant — privacy-by-design, DPA, data-subject rights. EU AI Act: classified high-risk, core controls in place, conformity assessment underway with Holistic AI, Instructions for Use published. NYC Local Law 144 (AEDT): assessments audited with no evidence of systemic adverse impact; a further independent audit with Holistic AI is underway.
Security, privacy & data
How candidate data is secured, stored, located and retained.
ISO 27001 and Cyber Essentials certified. AES-256 at rest, TLS 1.2+ in transit. MFA and RBAC. Hosted on GCP, primary region Belgium with EU multi-region DR. Retention customer-defined, typically two years, with deletion on request. SOC 2 certified, June 2026.
ATS and HRIS integration
Native fit into the client’s ATS or HRIS, plus API and webhooks.
30+ ATS integrations including iCIMS, SAP SuccessFactors and Ashby, with SmartRecruiters native. Partner API with webhooks. Platform SSO (SAML 2.0 / OIDC) and SCIM 2.0 provisioning.
Reliability & availability
Error rate, uptime and round-the-clock availability.
Designed for >99% availability (RTO 6h, RPO 12h) with annual BC/DR testing. Available 24/7.
GDPR, the EU AI Act, ISO 27001 — the paperwork a procurement team asks for first.
How to use this
Send the six sections, unedited, to every vendor you are considering, and ask for a written answer per line rather than a demo. You will learn who has a measurement system and who has a good presentation in about a week.
Keep the replies. They become the evidence file for your works council, your legal team and your first candidate complaint — and that file is worth more than the pilot.
Anywhere a vendor answers “on the roadmap”, that is a fine answer. The question is whether they said it before you asked.
Mochi is Maki’s AI voice interviewer. Maki is the intelligence layer between your ATS and your recruiters — the ATS stays, the recruiter stays. Want this rubric as a document you can put in front of procurement? Ask us.
See what Maki Agents can do for you
Experience how Maki’s AI agents simplify, speed up, and elevate your hiring
