Scaling VICIdial: designing a multi-server cluster for 500+ agents

Why one VICIdial box runs out

Fast answer: a single VICIdial box runs Apache with mod_php, MariaDB, and Asterisk together, and all three compete for the same central processing unit (CPU), memory, and disk input/output (I/O) at once. Measure resource pressure under the actual workload; splitting web, database, telephony and reporting roles is one possible response when a shared host reaches a measured limit.

In plain language: an agent is the person logged into the browser-based calling screen; a campaign is one outbound or inbound calling project with its own dialing rules and script; a lead is one contact record with a phone number and status; a list is a named batch of leads loaded together and dialed as a unit; the hopper is the shared queue of leads a campaign is currently allowed to call next; a channel is one live audio path Asterisk is holding open, whether it is an agent's phone, an outbound call, or a conference leg; a carrier is the phone company or trunk provider that carries a call onto the public telephone network; and the dial level, or dial ratio, is how many outbound calls the system places for each agent who is free to take one.

Every one of those nouns lives in one shared MariaDB database. VICIdial does not have a separate message queue or event bus — the database itself is the command bus. An agent's browser posts state, a dialer process reads the hopper and writes a manager command row, Asterisk executes it and generates events, and a listener writes the result back. Splitting a cluster means deciding which of those steps can move to its own machine without breaking that loop.

The four roles that split off a cluster

A cluster is not one thing scaled up — it is four distinct roles, each with a different resource profile, that a single box was quietly doing all at once. The web/agent role serves Admin, the Agent screen, and the roughly one-second browser poll that keeps every logged-in agent's session current. The database role holds the shared configuration, live runtime state, and call history every other role reads and writes. The dialer/telephony role runs the hopper-fill and pacing processes alongside the Asterisk instance that actually places and bridges calls. The archive/reporting role runs the report pages, the VERM analysis modules behind them, and the scheduled jobs that move aged rows out of the live tables.

The installed servers table is where this split becomes configuration, not just concept. Admin lets you mark a server active for web/agent logins, active for Asterisk, active for recording, and separately size its outbound trunk ceiling and its outbound calls-per-second rate. None of those flags are automatic — each one is a real capacity decision you make after you know what that box can actually carry.

Splitting roles does not remove the shared database in the middle. Every role still ends up reading or writing the same tables — vicidial_live_agents, vicidial_hopper, vicidial_manager, vicidial_auto_calls — so the database is the one role that cannot simply be duplicated the same way you duplicate a web server. That asymmetry is the reason the rest of this piece keeps circling back to the database.

role: web/agentservers: 192.0.2.10, 192.0.2.11limited_by: one-second agent poll rate and PHP/Apache worker count, not campaign logic role: databaseservers: 198.51.100.20limited_by: MyISAM table locks, missing composite indexes, and the size of vicidial_log/vicidial_hopper role: dialer/telephonyservers: 198.51.100.31, 198.51.100.32limited_by: CPU and RTP capacity per box plus the outbound trunk channels a carrier actually grants role: archive/reportingservers: 203.0.113.40, 203.0.113.41limited_by: replica lag versus report freshness, and disk growth on log/archive tables

Why the database is usually first to hurt

The database is a shared dependency and a useful place to investigate, but the first bottleneck depends on the workload. The web tier polls it once a second per agent, the dialer writes a new manager row for every call, Asterisk-side listeners write channel and call state back after every event, and reports scan months of history from the same tables. Nothing else in the architecture has that many concurrent readers and writers landing on the same rows.

Check the actual table engines, indexes and query plans on the installed build before diagnosing contention. MyISAM uses table-level locking; MEMORY and InnoDB behave differently. Missing or poorly ordered composite indexes can expand scans, but the unexecuted sample below does not establish which indexes your deployment lacks.

History grows unless the installed maintenance policy moves or expires it. VICIdial ships archival workers, including ADMIN_archive_log_tables.pl and AST_twoday_tables_control.pl;inspect their installed arguments, schedule, retention and recent completion before proposing another worker or a purge. A scheduled filename is not proof that rows moved successfully.

None of this means every install needs a second database server on day one. Measure database pressure alongside the web, media and network tiers when agents report lag. If database contention or growth is confirmed, test indexing and archival policy before adding servers that still share that bottleneck.

SHOW INDEX FROM vicidial_hopper;SHOW INDEX FROM vicidial_log;SHOW INDEX FROM vicidial_live_agents; SELECT table_name, table_rows,  ROUND((data_length + index_length) / 1024 / 1024, 1) AS size_mbFROM information_schema.tablesWHERE table_schema = DATABASE()  AND table_name IN ('vicidial_log', 'vicidial_hopper', 'vicidial_auto_calls', 'vicidial_manager', 'vicidial_live_agents')ORDER BY size_mb DESC;

What a report slave is and when you actually need one

A report slave is a second MySQL or MariaDB server that continuously copies data from the primary through standard replication, so reporting queries run against a copy instead of against the same server the dialer is writing to. VICIdial's system_settings table has three fields built for exactly this: slave_db_server names the replica, reports_use_slave_db is a per-report list of which stock report pages should query it, and custom_reports_use_slave_db does the same for custom reports. Nothing routes to a report slave until you fill those in — an unconfigured install simply has the support present with the primary handling every report, which is the normal starting state.

You need a report slave once report traffic is measurably competing with the dialer for the same tables — supervisors running real-time or historical reports during a shift while agents complain the screen is lagging, or a report that used to finish in a couple of seconds now taking most of a minute against a growing vicidial_log. A report slave does not fix a missing index or an ever-growing table; it isolates the damage so a heavy report degrades its own replica instead of the primary the live call flow depends on.

The trade you are making is freshness for isolation. A report slave lags behind the primary by however long replication takes to catch up, which is normally a small number of seconds under healthy load but can grow under the same write pressure that made you want the replica in the first place. Do not point a real-time report, or anything a supervisor uses to decide whether to log an agent out, at the replica — route only the historical and analytical report pages that can tolerate a short delay, using the reports_use_slave_db list to be explicit about which ones.

1. Identify database product/version and the actual candidate replica;keep real-time call control on the primary.
2. Have the database owner use the least monitor privilege supported by that version. For current MariaDB consult REPLICA MONITOR;do not assume MySQL REPLICATION CLIENT is interchangeable.
3. Check replica I/O and SQL thread health as well as lag and freshness against a known primary event. A small/NULL lag field alone is not a complete pass.
4. Route only selected historical reports after verifying their tolerated delay. If replication stops or freshness exceeds the agreed bound, mark those reports stale and restore their previous source through the documented configuration;do not use stale data to control live agents.

SELECT slave_db_server, reports_use_slave_db, custom_reports_use_slave_dbFROM system_settings; SHOW SLAVE STATUS\G

How the dialer processes and Asterisk servers distribute load

Scaling the dialer/telephony role means adding Asterisk boxes, not just adding trunks to one box. Each server row carries its own outbound trunk ceiling (max_vicidial_trunks), its own pacing limit (outbound_calls_per_second), and its own Asterisk Manager Interface (AMI) accounts — separate credentials for sending commands, listening for events, and applying updates. Every Asterisk server added to the cluster needs its own manager-send, manager-listen, and auto-dial processes running against that box's own AMI, because those processes are scoped to the server they are paired with.

Hopper and pacing worker placement is version- and configuration-specific; inspect the installed scheduling and process design before changing worker counts. The hopper-fill worker refills the shared hopper for a campaign regardless of which Asterisk box will place the call, and the adaptive pacing worker adjusts a campaign's dial level by looking at agent availability across every server the campaign runs on. Running duplicate workers without understanding their coordination can cause contention or duplicate work — the same one-writer discipline that governs a single Originate or Redirect command applies to these background workers too.

For campaigns that outrun their own server's trunk supply, VICIdial supports dial-out-only balance servers: a server row can be flagged as a balance-active server with a reserved trunk count that a balance-fill worker draws on to place calls for a busier server's campaign without touching the trunks that server has reserved for its own work. This is a way to add outbound capacity horizontally without adding another full agent-facing Asterisk box.

None of the numbers in these fields — trunk ceilings, calls-per-second, reserved trunks — are ones the application can respect for you. They only work if you measure a box's real CPU and Real-time Transport Protocol (RTP) audio-channel headroom under your actual codec mix and conferencing engine, then set the field to match, instead of leaving the installer default in place.

#!/bin/bashset -euo pipefail CNF=/etc/vicidial-readonly.cnf mysql --defaults-extra-file="$CNF" -e "SELECT active_asterisk_server, max_vicidial_trunks, outbound_calls_per_second, vicidial_balance_active, balance_trunks_offlimits FROM servers WHERE active = 'Y';" asterisk -rx 'core show channels count' 

• Per-server: manager-send, manager-listen, and auto-dial processes, one live set per Asterisk box.
• Inspect the installed hopper-fill and adaptive-pacing worker topology before changing worker counts.
• Balance/dial-out-only servers add outbound trunk capacity without adding agent-facing channels.

What shared state must stay consistent across the cluster

A cluster only works if every server agrees on the state the database is holding for it. VICIdial keeps a handful of tables on the MEMORY storage engine specifically for live/runtime state — channel, session, hopper-adjacent, auto-call, and parked-channel data that changes constantly and does not need to survive past the current shift; the query below lists exactly which ones exist on your own install rather than asking you to trust a fixed count. Those tables live on one database server; a replica or a failover node has to be tested on its own for how it handles that volatile, non-persistent state, because MEMORY tables do not behave like ordinary durable tables during a restart or a failover.

Clock agreement across servers is not optional. Call-time and timezone compliance, caller-ID selection, and the timestamps every worker writes to vicidial_auto_calls and vicidial_manager all assume the whole cluster agrees on the time. A dialer server that drifts even a little can dial outside an allowed local window or make its rows impossible to correlate with what Asterisk and the database server logged for the same call.

Configuration generation is another place where two servers disagreeing is worse than one server being slow. Each server row's rebuild flag controls whether that box regenerates its own Asterisk configuration from the database. Carrier and trunk changes on a live multi-server install have produced real ordering bugs — one documented case required every generated outbound route to reopen the shared dialplan context, because an earlier route on the same server could otherwise capture calls meant for a route added later. The fix belongs to exactly one generator per server; never let two processes regenerate or reload the same server's configuration at the same time.

The practical rule underneath all of this: anything that issues a telephony command — Originate, Redirect, Hangup, a transfer, a recording start or stop — must have exactly one authoritative sender in a cluster, the same as it must on a single box. Shadow reads of live state for comparison or monitoring are safe; a second live copy of the thing that sends the command is not.

SELECT table_name FROM information_schema.tables WHERE table_schema = DATABASE() AND engine = 'MEMORY';

Sizing reasoning for 500 agents, without inventing a benchmark

There is no published number that safely tells you how many agents one web server, one database server, or one Asterisk box can carry, because that number depends on your hardware, your campaign mix, your codec, and your conferencing engine. What you can do is reason from what each role is actually doing and then measure your own install against that reasoning.

On the web/agent tier, the Agent screen's browser polling is documented as roughly a one-second refresh. Five hundred logged-in agents polling once per second would imply about 500 requests per second against the Agent state endpoint alone, before manager actions, dispositions, lead searches, or supervisor report traffic are added. That is arithmetic from a documented interval, not a capacity claim — it tells you the request rate you must load-test against, not the number of web servers you need.

On the database tier, the limiting resource is usually not raw CPU but lock wait time and I/O — how long a write to vicidial_live_agents or vicidial_auto_calls waits behind a table lock, and how many rows a query has to scan because an index is missing. Measure query latency and lock wait time directly rather than estimating agent capacity from memory size alone.

On the dialer/telephony tier, the limiting resource is channel capacity: how many simultaneous RTP audio paths a given Asterisk box's CPU and network can sustain at your codec, plus how many outbound trunk channels your carrier actually grants. That ceiling belongs in the server row's trunk and pacing fields, set from a measured test on that specific hardware, not copied from another deployment's number.

Put together, sizing 500 or more agents is an exercise in measuring four separate ceilings — request rate, lock wait time, channel capacity, and report I/O — on your own hardware, then adding capacity to whichever ceiling you hit first. It is not a single number you can look up once and trust afterward, because the campaign mix, disposition pattern, and report schedule that produced last quarter's ceiling will not be the same ones straining the cluster next quarter.

The operational cost of running a cluster instead of a box

Every role split out is a role you now have to patch, monitor, and recover on its own schedule. A single box fails as one unit and comes back as one unit; a cluster fails one role at a time, and a monitoring plan that only watches the database server misses a dialer box quietly running out of trunk channels or a web server slowly falling behind on agent polling.

High availability for the database is the part most often assumed rather than designed. VICIdial's stock architecture has no automatic database failover; running a multi-node replicated cluster behind a virtual IP (VIP) address managed by a load balancer is a reasonable direction for closing that gap, not a drop-in fix — it still has to be tested against the same MEMORY-table volatility and single-writer command-bus behavior described earlier, on a copy of your database, before you trust it with live agents.

Adding Asterisk servers multiplies the AMI accounts and connections you manage — each server row carries its own send, listen, and update credentials, and the Asterisk Manager Interface has its own configured connection ceiling on every box, one that a growing set of listener and update workers can approach faster than expected on a busy cluster. Track active AMI connections per Asterisk box the same way you track trunk channels, and confirm your own configured ceiling rather than assuming a number.

Recordings and archived history add their own operational weight once they live on a dedicated role: recordings stored only on the local disk of the server that captured them are a data-loss risk if that server is lost, and archive/report tables still need the same backup and restore discipline as the live tables they were moved out of. A cluster is worth the added weight once one box is the reason agents are waiting on the screen — it is not worth adopting piece by piece just because a guide says a 500-agent operation should have one.

• Monitor each role's own limiting resource, not only the database server.
• Test any database high-availability design against MEMORY-table and single-writer behavior before trusting it with live agents.
• Track Asterisk Manager Interface connection counts per server as you add listener and update workers.