Analyze VICIdial calls with AI safely: build a human-reviewed sales and QA workflow

Ask one small, auditable question

“Analyze the call” is too vague to test. Begin with facts a reviewer can check: Was a callback requested? Which product name was mentioned? Was the required lab greeting present? Separate extraction from judgment: extracting a spoken phrase is different from deciding whether an agent performed well.

Write the allowed output fields before selecting a model: call ID, exact supporting quote with timestamp when available, an uncertainty flag, and `needs_human_review`. If the source cannot support a field, the correct value is unknown—not a guess.

Good starter questions read like a checklist item, not a verdict: did the agent state the required disclosure, was a price quoted, was a callback requested, did the customer ask to be placed on a do-not-call list. Avoid a first question that already assumes a judgment, such as whether the agent was persuasive, because two careful reviewers can disagree about that without either one being wrong.

Make a tiny human-labeled test set

Use 10–20 approved test calls or scripts spanning clear positives, clear negatives, silence, interruptions and ambiguous wording. Two humans should label the expected result independently, then resolve differences. That gives you a baseline before the model ever sees a production-like queue.

Keep the test set separate from campaign reporting. A model that appears useful on a friendly sample may fail on accents, noise, jargon, transfers, multiple speakers or incomplete recordings.

Spread the sample across agents, campaigns, call lengths and days rather than pulling twenty consecutive calls from one queue, or the test set will only confirm that the model handles whatever pattern happened to be common that day. When the two human labels disagree, write down why before resolving it — that disagreement log is often more useful later than the final label.

Guided example: use a constrained review record

This JSON is a local contract for the reviewer. It does not call an AI service; it shows that the output must preserve evidence and uncertainty instead of producing a confident free-text score.

Keep the record's shape identical across every test call, including the ones where nothing relevant was said. An empty `supportingText` with `requestedCallback: 'unknown'` is a complete, honest result; do not let a later prompt revision quietly drop a field the reviewer still expects to see.

const review = {  testCallId: 'lab-001',  requestedCallback: 'unknown', // yes | no | unknown  supportingText: '',  reviewerDecision: 'pending',  modelOutputIsAdvisory: true};

Guided example: validate the shape, then the meaning

A structured response is easier to reject safely than an unbounded paragraph. Schema validation only proves the response has the right fields; it does not prove the call interpretation is correct. Compare every result with the human labels in your test set.

Version the prompt text itself, not only the model name. A one-line change to the prompt can shift results across the whole test set, and without a saved prompt version you cannot tell whether a new disagreement came from the model, the prompt, or the transcript.

function acceptForReview(result) {  const allowed = new Set(['yes', 'no', 'unknown']);  return typeof result?.testCallId === 'string' &&    allowed.has(result?.requestedCallback) &&    typeof result?.supportingText === 'string' &&    result?.modelOutputIsAdvisory === true;} // A true result enters a human review queue; it does not update VICIdial or a CRM.

Measure disagreements, not just attractive examples

Count false positives, false negatives and ‘unknown’ cases against the human gold labels. Review disagreements by call type and language. A useful workflow reports uncertainty and routes edge cases to people; it does not hide them behind a single average score.

Store the prompt version, model/version identifier offered by the provider, source transcript version, reviewer decision and date. That record makes it possible to reproduce a surprising result or pause a bad release.

In plain language: a false positive is the model claiming something happened that the human labels say did not; a false negative is the model missing something the human labels say did happen. Track both separately, because a workflow that hides its false negatives behind a high overall agreement rate is the one most likely to embarrass you in production.

Keep the production rollout reversible

If the test is acceptable, begin with an advisory queue for a small authorized team. Give reviewers an easy reject/correct action and monitor disagreement rates. Keep original reports and dispositions authoritative; AI notes are a separate attachment until governance explicitly approves another use.

Do not claim a model detects compliance, intent, sentiment, deception or sales quality reliably without a domain-specific evaluation and accountable policy owner. Those are high-impact interpretations, not simple transcript facts.

VICIdial's own Quality Control Manual documents a human call-review process that most installations already run in some form. Treat this workflow as a second, advisory layer beside that existing process, not a replacement for it — the manual's own review discipline is the baseline this workflow has to earn a place next to, not skip past.

Set a retention period for the test transcripts and the review records themselves, not just the customer recordings they came from. A QA experiment that quietly becomes a permanent, ungoverned archive of call content creates its own privacy liability, separate from whatever retention rule already governs the original recording.

• The rubric is limited and testable.
• Humans labeled a test set before review began.
• Every output remains advisory and reversible.