AI Call Quality Scorecard

Calls sampled per week

Pick a number you'll actually review. Twenty calls graded every Monday beats a hundred graded once a quarter. Pull them at random from the calling platform, not the ones the campaign owner remembers.

____ calls per week

Sample split by outcome

Force a mix so the sample isn't all short hang-ups. A working split is a quarter each of positive outcomes, negative outcomes, human handoffs and calls under 30 seconds.

Positive ____ Negative ____ Handoff ____ Short ____

Sample split by language and segment

A campaign calling Spanish, Arabic and English dormant traders needs each language in the sample, or a broken voice in one language hides behind a good average.

Languages: __________ Segments: __________

Named scorers and a second reviewer

Two people grade the first ten calls independently every week. If their totals differ by more than 10 points on any call, they listen together and settle the definition before scoring the rest.

Scorer 1: __________ Scorer 2: __________

Pass threshold for the weighted total

The weighted total runs 0 to 100. Set the number below which a call gets flagged and the number below which the campaign is paused for review. A workable starting point is 70 to flag and 55 to pause; tighten once you have four weeks of scores.

Flag below ____ Pause campaign below ____

Where scores live

One sheet or one CRM object per call, keyed on the call ID from the calling platform, so a score can always be traced back to a recording and a transcript.

Location: __________

Score what happened, not what the prompt says

The agent's instructions might include a perfect disclosure line. If the recording shows it was cut off by the trader and never repeated, that's a 2, not a 5.

Score 'not applicable' as N/A, not 5

A call that ends at the voicemail beep never reaches objection handling. Mark N/A and remove that metric's weight from the denominator for that call, or a week of voicemails inflates your average.

Compute the weighted total

Multiply each metric's score by its weight, add them up, divide by 5. Ten metrics at a perfect 5 give 100. With N/A metrics, divide by 5 times the weights that applied.

Record one sentence of evidence per score below 4

Quote the transcript timestamp. 'At 0:42 the agent said the bonus was still active; it expired last month' is fixable. 'Accuracy felt off' isn't.

1. Opening and disclosure

A 5: brokerage name, reason for calling and the required disclosure, all before the trader's first reply. A 1: the trader asks 'who is this?' or 'is this a robot?' and doesn't get a straight answer. Which calls need an AI disclosure depends on the jurisdiction (TCPA rules in the US, financial promotion rules under the FCA, local telemarketing law elsewhere); confirm the wording with your compliance officer and paste it into the agent instructions verbatim.

Score ____ /5

2. Right-party verification

A 5: 'Am I speaking with the account holder?' or a name check, answered yes, before any account detail. A 1: the agent tells whoever picked up that 'your account has been dormant since March with a balance of 1,200'. That's a GDPR problem and a trust problem in one sentence.

Score ____ /5

3. Comprehension accuracy

A 5: zero misheard turns. A 3: one misunderstanding the agent caught and repaired ('Sorry, did you say Thursday?'). A 1: the agent booked a callback for the wrong day or logged 'interested' after the trader said 'not interested'. Watch non-native speakers and noisy lines separately; if one language scores a point lower every week, that's a speech model issue, not a script issue.

Score ____ /5 Misheard turns: ____ of ____

4. Latency and turn-taking

A 5: replies land naturally and the agent stops when the trader speaks. A 2: the trader says 'hello? hello?' into a gap, or the agent keeps going through an interruption. Long pauses and talk-over are the two things traders quote when they say a call 'sounded fake'.

Score ____ /5 Gaps over 2s: ____ Talk-overs: ____

5. Factual accuracy

The heaviest metric in this group because a wrong fact on a finance call is a complaint waiting to happen. A 5: every claim checks out. A 1: the agent quoted an expired promotion, a wrong minimum deposit, a leverage figure your entity doesn't offer, or a KYC document the trader already submitted. Log the exact claim; it usually traces back to a stale knowledge base entry.

Score ____ /5 Wrong claims: ____

6. Compliance guardrails

A 5: no forbidden phrasing anywhere, risk wording delivered where your policy requires it, and a 'please don't call me again' answered with confirmation and an end to the call. A 1: any promise of returns, any 'you should buy' or 'now is a good time', or a stop request talked past. Which phrases are forbidden depends on where the trader is (FCA financial promotion rules, MiFID II conduct rules, TCPA and DNC rules in the US); keep the list with your compliance officer and copy it into the scorer's notes.

Score ____ /5 Flagged phrases: ____

7. Objection handling

Dormant traders push back with 'I lost money', 'the spreads were bad', 'I moved to another broker' or 'I don't have time'. A 5: the agent acknowledges the specific objection, answers it once, and either offers a concrete next step or closes politely. A 1: the same rebuttal three times, or an objection ignored and the pitch continued.

Score ____ /5 Repeated rebuttals: ____

8. Handoff and escalation

Score both directions. Missed handoff: the trader asked to speak to a person, raised a complaint, or mentioned a withdrawal problem and the agent kept going. Bad handoff: transferred a plain 'call me next week' to the sales desk with no context. A 5 hands off on your agreed triggers, states who the trader will speak to, and passes the reason along.

Score ____ /5 Handoff correct: Y / N / N/A

9. Disposition accuracy

Your reactivation rate, handoff rate and every other campaign number are built on this tag. A 5: 'callback requested, Thursday 2pm' in the CRM and that's what was said. A 1: 'interested' on a call where the trader hung up mid-sentence. Sample this one hardest in the first two weeks; disposition drift is the most common reason a dashboard lies.

Score ____ /5 Tag matches: Y / N

10. Next step clarity

A 5: the agent states the next step, the trader confirms it, and the CRM shows a matching task or link sent. A 3: a next step was mentioned but not confirmed. A 1: the call ends with no next step at all, or with a step that contradicts the disposition. For unfunded and KYC calls, the next step is the whole point, so weight this higher if those campaigns dominate your mix.

Score ____ /5

Agent promised, implied or hinted at guaranteed returns or a 'safe' trade

Includes soft versions: 'traders like you have been doing well', 'the market is favourable right now'. Confirm the exact test with compliance; when in doubt, fail it and let them downgrade.

Agent gave personal trading advice

Naming an instrument, a direction or a time to enter. Reactivation calls are about the account and the relationship, never the trade.

Agent discussed account details with someone who wasn't verified as the holder

Balance, open positions, deposit history, KYC status or the fact of dormancy itself, before a verification step.

Agent continued after the trader asked to stop, opt out, or not be called again

One more sentence of pitch after a stop request is a fail. The agent should confirm, end, and the number should hit your suppression list the same day.

Agent failed to give the recording or AI disclosure your policy requires for that jurisdiction

Recording notice requirements and AI disclosure rules vary by country and by US state. Score against the list your compliance officer signed off, not against a general rule.

Agent applied deposit pressure to a trader who mentioned financial difficulty or vulnerability

Any mention of debt, illness, job loss or 'I can't afford it' ends the deposit conversation. The right move is a courteous close or a handoff to a human trained for it.

Agent quoted a leverage, bonus or fee figure not available to that trader's entity

Multi-entity brokers run different terms for different regulators. The agent's knowledge base has to be entity-aware; if it isn't, this rule will catch it within a week.

Calls scored this week and how the sample was drawn

State the count and the outcome and language split from section 1. If the split drifted (say, no Arabic calls this week), note it, because the average isn't comparable.

____ calls Split matched plan: Y / N

Lowest scoring call and what caused it

Call ID, weighted total, one line of cause. The lowest call is where the next fix lives.

Call ID: ________ Total: ____ Cause: __________

Scorer agreement on the shared ten calls

Average gap between the two scorers' weighted totals. Over 10 points means the definitions need work before the numbers mean anything.

Avg gap: ____ points

Metric that moved most and the change that explains it

A prompt edit, a knowledge base update, a new voice, a new segment. If nothing changed and the score moved, say so; it usually means the sample shifted.

Metric: ______ Change: __________

Fixes shipped since last review and the metric each one targets

Tie every prompt or knowledge base change to a number on this sheet. A change without a target metric is a guess.

__________

Segments or languages below the flag threshold

Cut the weighted total by segment and language. A campaign averaging 78 with one language at 58 has a problem the average hides.

__________

Sign-off

The QA lead signs the sheet; compliance countersigns any week with an auto-fail. Keep the signed sheets; they're the record if a regulator or a trader complaint asks how calls were monitored.

QA lead: __________ Compliance: __________ Date: ____/____

Keep a reference call for each score level on the three heaviest metrics

One recording that is a clear 5, 3 and 1 on factual accuracy, compliance guardrails and comprehension. New scorers listen to those nine calls before they grade anything.

Re-score two old calls blind every month

Pull two calls scored eight weeks ago, hide the old scores, grade again. A gap over 5 points on the weighted total means the scale has drifted.

Review definitions whenever the agent instructions change

A new disclosure line, a new handoff trigger or a new offer changes what a 5 looks like. Update the metric description the same day the prompt ships.

Bring compliance into the calibration session each quarter

They own the list of forbidden phrases and the auto-fail rules. A quarterly hour with the recordings keeps the scorecard aligned with what they'd actually flag.

Compare AI scores with a human-agent sample once a quarter

Grade twenty human calls on the same ten metrics. It settles the 'the AI sounds worse' argument with numbers in either direction, and it often turns up disposition and disclosure gaps on the human side too.