Scaling Laravel Horizon: Managing Thousands of Background Jobs Without Memory Saturation

Balancing High-Priority vs Bulk Batch Jobs

Running user-facing notifications, invoice generation, and background data imports on a single generic queue leads to queue starvation. Segregating jobs into distinct priority queues ensures critical customer notifications are never delayed by bulk data synchronizations.

Horizon Auto-Scaling Configuration

'environments' => [
    'production' => [
        'supervisor-priority' => [
            'connection' => 'redis',
            'queue' => ['high', 'default'],
            'balance' => 'auto',
            'autoScalingStrategy' => 'time',
            'minProcesses' => 5,
            'maxProcesses' => 30,
        ],
    ],
]

Using autoScalingStrategy => 'time' dynamically allocates additional PHP worker processes when queue wait time exceeds defined thresholds.