Deploying Production LLM Inference on VPS with Auto-Scaling

Introduction

Deploying Large Language Model (LLM) inference services on your own VPS gives you complete control, data privacy, and cost efficiency — but it also introduces operational complexity that can quickly become overwhelming as traffic grows. This article provides a practical, end-to-end guide for building a production-ready LLM inference system on self-hosted VPS with intelligent auto-scaling and cost control mechanisms.

Whether you’re running business-critical applications, building custom AI assistants, or hosting open-weight models for community access, the patterns covered here will help you balance performance, reliability, and cost in a resource-constrained environment.

Why Self-Hosted LLM Inference on VPS?

Before diving into the technical details, let’s understand why choosing VPS over cloud-managed inference services makes sense:

Factor	Cloud API (OpenAI, Anthropic, etc.)	Self-Hosted VPS
Data Privacy	Data sent to third party	Fully private, data stays local
Cost predictability	Pay-per-token, hard to forecast	Fixed infrastructure cost, predictable OPEX
Customization	Limited to provider options	Full control over model, quantization, optimizations
Latency	Network-dependent	Local, minimal latency
Throughpe limitations	Provider-imposed quotas	Scale with your hardware
Model selection	Only provider’s models	Any open-weight model, fine-tuned versions

For workloads involving sensitive data, high-volume request patterns, or specialized domain models, self-hosting becomes not just an option but a necessity. However, managing inference elasticity on limited VPS resources requires careful architectural decisions.

Architecture Overview

The system follows a layered, microservices-inspired architecture designed for resilience and scalability at the single-VPS level:

┌─────────────────────────────────────────────────────────────┐
│                    Client Applications                      │
│   (Web UI, Mobile Apps, Third-party Services, Cron Jobs)    │
└──────────────────────────────┬──────────────────────────────┘
                               ▼
┌─────────────────────────────────────────────────────────────┐
│              Load Balancer & Request Router                 │
│         (NGINX/Traefik with rate limiting & auth)          │
└──────────────────────────────┬──────────────────────────────┘
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                API Gateway Service                          │
│  (Health checks, request validation, metrics, logging)     │
└──────────────────────────────┬──────────────────────────────┘
                               ▼
┌─────────────────────────────────────────────────────────────┐
│           Inference Controller + Auto-scaler               │
│  (Manages worker pods, monitors GPU/CPU/memory, scales up) │
└──────────────────────────────┬──────────────────────────────┘
          ┌────────────────────┼────────────────────┐        │
          ▼                    ▼                    ▼        │
┌──────────────┐      ┌──────────────┐      ┌──────────────┐  │
│  Inference   │      │  Inference   │      │  Inference   │  │
│    Worker 1  │─────▶│    Worker 2  │─────▶│    Worker 3  │◀─┤
│  (vLLM/TGI)  │      │   (Ollama)   │      │  (TensorRT)  │  │
└──────────────┘      └──────────────┘      └──────────────┘  │
          ▲                    │                             │
          └────────────────────┼─────────────────────────────┘
                               ▼
┌─────────────────────────────────────────────────────────────┐
│             Model Storage & Cache Layer                     │
│  (Local disk, SSD cache, optional remote storage)          │
└─────────────────────────────────────────────────────────────┘

Key Design Principles

Stateless Workers: Each inference container is stateless and replaceable, enabling horizontal scaling within a single host.
Decoupled Scaling Logic: The autoscaler decouples monitoring from decision-making, allowing independent evolution of health metrics and scaling policies.
Resource Isolation: Docker containers with explicit CPU/memory limits prevent noisy neighbor problems and ensure fair resource sharing.
Observability First: Every component emits structured logs, metrics, and traces for complete visibility into system behavior.
Progressive Rollout: New model versions can be deployed alongside old ones with traffic shifting, minimizing downtime during updates.

Choosing Your Inference Engine

Selecting the right inference engine depends on your workload characteristics, model format, and performance requirements. Here are the top contenders:

vLLM (High Throughput Serving)

Best for: High-throughput batch serving, long-context models, production workloads

Pros:

PagedAttention memory management enables high throughput with low memory overhead
Supports tensor parallelism across multiple GPUs
Continuous batching improves GPU utilization
OpenAI-compatible API

Cons:

Requires more RAM than other engines
Primarily GPU-accelerated (CPU-only mode limited)

# docker-compose-vllm.yml
version: '3.8'
services:
  vllm-engine:
    image: vllm/vllm-openai:latest
    ports:
      - "8000:8000"
    volumes:
      - models:/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [compute, utility]
        limits:
          memory: 32G
    environment:
      - VLLM_HOST=0.0.0.0
      - VLLM_PORT=8000
      - MODEL_NAME=qwen/qwen-7b-chat
      - MAX_OUTPUT_TOKENS=2046
volumes:
  models: {}

Text Generation Inference (TGI)

Best for: Rust-based production deployment, Docker-centric workflows

Pros:

Built by Hugging Face, battle-tested in production
Supports Shard/Parallel for multi-GPU setups
Efficient token generation with streaming support
Docker-native deployment

Cons:

Steeper learning curve for configuration
Higher baseline memory footprint

Ollama (Simple & Flexible)

Best for: Rapid development, smaller models, CPU/GPU hybrid workloads

Pros:

Extremely simple setup (ollama run llama3)
Automatic model downloading and caching
RESTful API with WebUI integration
Works on CPU-only machines well

Cons:

Lower throughput compared to vLLM/TGI
Less optimized for high-concurrency production

TensorRT-LLM (NVIDIA Ecosystem)

Best for: NVIDIA GPU-optimized deployments, maximum performance

Pros:

Highest throughput on NVIDIA GPUs
Advanced quantization support (INT8, FP8)
Deep integration with CUDA ecosystem

Cons:

NVIDIA hardware required
More complex build and deployment process

Production Deployment Pattern

For most VPS scenarios, I recommend a hybrid approach: use vLLL for heavy model workloads on dedicated GPU instances, Ollama for smaller models on general-purpose nodes, and front everything with a reverse proxy and API gateway.

Complete docker-compose Stack

Here’s a production-ready docker-compose.yml that orchestrates all components:

# docker-compose-production.yml
version: '3.8'

services:
  # ====================
  # Reverse Proxy Layer
  # ====================
  nginx-proxy:
    image: traefik:v2.10
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address:443"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./traefik.yml:/etc/traefik/traefik.yml"
      - ./traefik-certs:/certs
    restart: unless-stopped

  # ====================
  # API Gateway & Auth
  # ====================
  api-gateway:
    build: ./api-gateway
    image: selfvps/api-gateway:latest
    ports:
      - "8081:8081"
    environment:
      - VLLM_ENDPOINT=http://vllm-engine:8000
      - OLLAMA_ENDPOINT=http://ollama:11434
      - JWT_SECRET=${JWT_SECRET}
      - RATE_LIMIT=1000/hr
    depends_on:
      - vllm-engine
      - ollama
    restart: unless-stopped

  # ====================
  # Inference Engines
  # ====================
  vllm-engine:
    image: vllm/vllm-openai:latest
    command: ["--model", "mistralai/Mistral-7B-Instruct-v0.3", 
              "--port", "8000", "--host", "0.0.0.0",
              "--max-model-len", "32768"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [compute, utility]
        limits:
          memory: 24G
    volumes:
      - ./models:/models
    environment:
      - VLLM_HOST=0.0.0.0
      - VLLM_PORT=8000
    restart: unless-stopped

  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ./ollama-storage:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0
      - OLLAMA_PORT=11434
    restart: unless-stopped

  # ====================
  # Auto-scaling Controller
  # ====================
  autoscaler:
    image: selfvps/autoscaler:latest
    command: [--scale-down-interval=30s, --scale-up-threshold=0.7]
    environment:
      - SCALING_METRICS_URL=http://prometheus:9090
      - VLLM_CONTAINER_NAME=vllm-engine
      - MIN_WORKERS=1
      - MAX_WORKERS=4
      - COOLDOWN_PERIOD=300s
    depends_on:
      - prometheus
    restart: unless-stopped

  # ====================
  # Monitoring Stack
  # ====================
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GF_PASSWORD}
    restart: unless-stopped

volumes:
  models:
  ollama-storage:
  prometheus-data:

Auto-Scaling Implementation

The core challenge in VPS-scale inference is balancing resource contention against request latency. On a single server with finite GPU memory, spinning up too many workers causes thrification, while keeping too few leads to queuing delays.

Scaling Strategy Overview

I recommend a three-tier adaptive scaling approach:

Short-term Scaling (seconds): Based on queue length and wait time. If requests accumulate faster than they’re processed, spin up additional workers immediately.
Medium-term Scaling (minutes): Based on GPU utilization percentages. If average GPU usage exceeds 70% for sustained periods, increase capacity gradually.
Long-term Scaling (hours/days): Based on demand forecasting. Use historical patterns to pre-warm workers before predicted traffic spikes.

Python Autoscaler Example

Here’s a simplified example of an autoscaler that monitors worker metrics and adjusts scale accordingly:

#!/usr/bin/env python3
"""
LLM Inference Autoscaper for VPS Deployments
Monitors GPU memory, request latency, and queue depth
then triggers scale-up/scale-down decisions
"""

import subprocess
import time
import json
import psutil
from collections import deque
from datetime import datetime, timedelta

class InferenceAutoscaler:
    def __init__(self, min_workers=1, max_workers=4, 
                 gpu_threshold=0.7, cooldown=300):
        self.min_workers = min_workers
        self.max_workers = max_workers
        self.gpu_threshold = gpu_threshold  # 70% utilization
        self.cooldown = cooldown  # seconds between decisions
        self.last_decision = None
        self.latency_history = deque(maxlen=100)
        
    def get_gpu_memory_info(self):
        """Get GPU memory usage using nvidia-smi"""
        try:
            result = subprocess.run(
                ['nvidia-smi', '--query-gpu=index,memory.used,memory.total', 
                 '--format=csv,noheader,nounits'],
                capture_output=True, text=True, timeout=10
            )
            lines = result.stdout.strip().split('\n')
            if len(lines) >= 1:
                used, total = map(int, lines[0].split(','))
                return used / total if total > 0 else 0
        except Exception as e:
            print(f"GPU query error: {e}")
        return None
    
    def get_worker_status(self):
        """Check current number of running inference workers"""
        try:
            result = subprocess.run(
                ['docker', 'ps', '--filter', 'name=vllm-engine', 
                 '--format', '{{.Names}}'],
                capture_output=True, text=True, timeout=5
            )
            workers = [w.strip() for w in result.stdout.strip().split('\n') if w.strip()]
            return len(workers), workers
        except Exception as e:
            print(f"Docker check error: {e}")
            return 0, []
    
    def get_request_metrics(self):
        """Get latency and queue metrics from API gateway"""
        try:
            # In production, this would query Prometheus or a metrics endpoint
            response = subprocess.run(
                ['curl', '-s', 'http://localhost:9090/api/latency'],
                capture_output=True, text=True, timeout=5
            )
            if response.returncode == 0:
                return json.loads(response.stdout)
        except Exception as e:
            print(f"Metrics query error: {e}")
        return {'avg_latency': 0, 'queue_depth': 0}
    
    def should_scale_up(self, current_workers):
        """Determine if we need to add workers"""
        if current_workers >= self.max_workers:
            return False
            
        now = time.time()
        if self.last_decision and (now - self.last_decision) < self.cooldown:
            return False
        
        gpu_util = self.get_gpu_memory_info()
        if gpu_util is None:
            # Fallback: check CPU load
            cpu_load = psutil.getloadaverage()[0]
            avg_cpu_per_worker = cpu_load / max(current_workers, 1)
            if avg_cpu_per_worker > self.gpu_threshold * 4:  # Adjust threshold for CPU
                return True
            return False
        
        if gpu_util > self.gpu_threshold:
            return True
        
        metrics = self.get_request_metrics()
        if metrics.get('avg_latency', 0) > 500:  # ms
            return True
        if metrics.get('queue_depth', 0) > 20:
            return True
        
        return False
    
    def should_scale_down(self, current_workers):
        """Determine if we can remove a worker"""
        if current_workers <= self.min_workers:
            return False
            
        now = time.time()
        if self.last_decision and (now - self.last_decision) < self.cooldown:
            return False
        
        # Only scale down if stable for extended period
        metrics = self.get_request_metrics()
        if metrics.get('avg_latency', 0) < 200 and metrics.get('queue_depth', 0) < 5:
            gpu_util = self.get_gpu_memory_info()
            if gpu_util and gpu_util < self.gpu_threshold * 0.5:
                return True
        
        return False
    
    def make_scaling_decision(self, current_workers):
        """Make the final scaling decision"""
        now = time.time()
        
        if self.should_scale_up(current_workers):
            new_count = current_workers + 1
            print(f"[{datetime.now()}] SCALE UP: {current_workers} → {new_count} workers")
            self.trigger_scale_up(new_count)
            self.last_decision = now
            return new_count
            
        elif self.should_scale_down(current_workers):
            new_count = current_workers - 1
            print(f"[{datetime.now()}] SCALE DOWN: {current_workers} → {new_count} workers")
            self.trigger_scale_down(new_count)
            self.last_decision = now
            return new_count
        
        return current_workers
    
    def trigger_scale_up(self, target_count):
        """Scale up using docker compose or Kubernetes"""
        print(f"Scaling up to {target_count} workers...")
        # Implementation would call docker compose or kubectl
        pass
    
    def trigger_scale_down(self, target_count):
        """Scale down using docker compose or Kubernetes"""
        print(f"Scaling down to {target_count} workers...")
        # Implementation would call docker compose or kubectl
        pass
    
    def monitor_loop(self, interval=30):
        """Main monitoring loop"""
        print("Starting autoscaler monitor...")
        while True:
            current_workers, worker_names = self.get_worker_status()
            print(f"Current workers: {current_workers}")
            
            new_count = self.make_scaling_decision(current_workers)
            
            # Update any dependent services if needed
            if new_count != current_workers:
                self.notify_dependencies(new_count)
            
            time.sleep(interval)
    
    def notify_dependencies(self, new_worker_count):
        """Notify downstream services of scale change"""
        # Update load balancer config, metrics dashboards, etc.
        pass

if __name__ == '__main__':
    scaler = InferenceAutoscaler(min_workers=1, max_workers=4, gpu_threshold=0.7)
    try:
        scaler.monitor_loop(interval=30)
    except KeyboardInterrupt:
        print("\nAutoscaler stopped")

Docker-Based Scaling Using Compose

For simpler deployments without Kubernetes, you can leverage Docker Compose with external orchestration:

# Start initial replica
docker compose up -d --scale vllm-engine=2

# To scale up (manual or via script)
docker compose up -d --scale vllm-engine=3

# Monitor and script-based scaling example
#!/bin/bash
THRESHOLD=70
CURRENT=$(docker stats --no-stream --format "{{.Name}} {{.CPUPerc}}" | grep vllm | awk '{sum+=$2} END {print sum/NR}')

if (( $(echo "$CURRENT > $THRESHOLD" | bc -l) )); then
    COMPOSE_COUNT=$(docker compose ps vllm-engine | wc -l)
    if [ $COMPOSE_COUNT -lt 4 ]; then
        docker compose up -d --scale vllm-engine=$((COMPOSE_COUNT + 1))
        echo "Scaled up to $((COMPOSE_COUNT + 1)) workers"
    fi
fi

Cost Optimization Techniques

Running LLM inference on VPS involves significant costs for compute, memory, and storage. These strategies help optimize expenses:

1. Model Quantization and Downsizing

Use quantized versions of models to reduce memory footprint and increase throughput:

# Using GGUF format for CPU-friendly inference
ollama pull qwen:7b-q4_0  # 4-bit quantized version (~4GB vs ~14GB full precision)

# Or use AWQ/GPTQ quantized formats for GPU
ollama pull codestarcoder/starcodercode-15b-awq

Quantization typically reduces memory usage by 50-75% with minimal quality loss for most inference tasks.

2. Dynamic Batch Size Tuning

Adjust batch sizes based on available GPU memory and request patterns:

# vLLM configuration with dynamic batching
--enable-pipelined-scheduler  # Overlap compute and I/O
--max-num-seqs-per-batch 64   # Tune based on your GPU VRAM
--scheduler-policy FIRST     # Or LAST, FCFS,等不同策略

3. Idle Resource Management

Scale down during low-traffic periods to save costs:

# cron job for daily off-peak scaling
0 8 * * * /usr/local/bin/vps-autoscale --scale-down --min 1
0 20 * * * /usr/local/bin/vps-autoscale --scale-up --max 4

4. Model Caching Strategies

Implement aggressive model loading caching to avoid repeated disk I/O:

# Pre-load commonly used models on boot
sudo systemctl enable ollama-preload.service

# Use SSD-backed cache for frequently accessed models
mount -o noatime,discard /dev/ssd0 /var/ollama-cache

5. Spot Instance Considerations

If using cloud VPS providers that offer spot/preemptible instances:

Schedule model warm-up during spot instance provisioning
Implement checkpoint/resume logic for interrupted workloads
Build redundancy across multiple instances to handle preemption

Cost Comparison Table

Approach	Estimated Monthly Cost (with typical usage)	Throughput (tokens/sec)
Cloud API (pay-per-use)	$150-$500+	Limited by provider quotas
Single GPU (A100/VPS)	$100-$300 + electricity	50-200 tokens/sec
Multi-GPU Cluster (3x)	$300-$900 + electricity	300-800 tokens/sec
CPU-only (quantized)	$30-$80 + electricity	5-20 tokens/sec
Hybrid (GPU for hot paths, CPU for cold)	$80-$200 + electricity	Mixed

Note: Costs vary by region, hardware specifications, and usage patterns.

Performance Optimization Tips

Beyond basic deployment, these advanced techniques squeeze every bit of performance from your VPS:

Kernel and OS-Level Optimizations

# Set transparent huge pages for better memory performance
echo always > /sys/kernel/mm/transparent_hugepage/enabled

# Optimize network stack for low-latency inference
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.tcp_slow_start_after_idle=0
sysctl -w net.netfilter.nf_conntrack_max=1048576

# Set process affinity for inference workers
numactl --cpunodebind=0 --membind=0 python inference_worker.py

Docker Resource Configuration

# docker-compose.yml snippet for optimized resource allocation
deploy:
  resources:
    limits:
      cpus: '4.0'
      memory: 32g
    reservations:
      cpus: '2.0'
      memory: 16g
      devices:
        - driver: nvidia
          count: 1
          capabilities: [compute, utility]
  placement:
    constraints: [node.platform.os == linux]

Inference-Specific Optimizations

Flash Attention: Enable where supported by your model and framework
KV Cache Management: Configure appropriate cache sizes to balance memory and speed
Speculative Decoding: Use smaller fast models to draft responses verified by larger slow models
Continuous Batching: Keep GPU fully utilized by processing multiple requests in batches

Security Considerations

Security is paramount when exposing LLM endpoints publicly:

# NGINX security headers location / {
    add_header X-Frame-DENY always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    
    # Rate limiting at the edge
    limit_req zone=req_rate burst=5 nodelay;
    
    # Authentication
    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/.htpasswd;
}

Key security practices:

API authentication: Require API keys or JWT tokens for all endpoints
Rate limiting: Prevent abuse and DDoS attacks at the ingress layer
Input sanitization: Filter user inputs to prompt injection attempts
Model access controls: Restrict which models certain users can invoke
Audit logging: Log all inference requests for compliance and debugging
Network isolation: Place inference workers behind internal networks, accessible only through the gateway

Implementation Checklist

✅ [ ] Infrastructure Planning: Assess VPS specs (RAM, GPU, disk I/O) against model requirements
✅ [ ] Model Selection: Choose appropriate models for workloads (quantized vs full precision)
✅ [ ] Containerization Package: Create Docker images with pinned dependencies and base images
✅ [ ] Orchestration Setup: Configure docker-compose or Kubernetes manifests with resource limits
✅ [ ] Reverse Proxy: Set up Traefik/NGINX with TLS termination and health checks
✅ [ ] Monitoring Stack: Integrate Prometheus, Grafana, and alerting rules
✅ [ ] Autoscaler Configuration: Define scaling thresholds, cooldowns, and action hooks
✅ [ ] Security Hardening: Implement auth, rate limiting, and audit logging
✅ [ ] Backup Strategy: Schedule model snapshot backups and configuration versioning
✅ [ ] Documentation: Create runbooks for common failure scenarios and recovery procedures

Real-World Scenarios

Let’s see how this architecture handles specific use cases:

Scenario 1: Customer Support Chatbot

Requirements: Low latency (<500ms), moderate concurrency (50-100 RPM), high availability
Configuration: 2x vLLM workers on 24GB GPU, autoscale to 4 during peak hours, 7B parameter model
Cost: ~$150/month including VPS and electricity
Benefit: Personalized responses without sending customer data to third-party APIs

Scenario 2: Internal Knowledge Base Assistant

Requirements: Access to company documents via RAG, lower priority than web-facing apps
Configuration: 1x Ollama worker on CPU-only node, quantized Qwen-7B model, integrated with Pinecone vector store
Cost: ~$30/month on shared VPS
Benefit: Full data sovereignty, no external dependency

Scenario 3: Public API for Community Models

Requirements: High throughput, variable demand, potential traffic spikes
Configuration: Multi-node cluster (3 VPS instances), each with vLLM, global load balancer, autoscaling across nodes
Cost: ~$600/month
Benefit: Elastic capacity that adapts to unpredictable traffic patterns

Conclusion

Building a production-grade LLM inference service on a self-hosted VPS is entirely feasible with the right architectural choices. By implementing stateless containerized workers, intelligent auto-scaling, and comprehensive observability, you can achieve a system that balances performance, reliability, and cost effectively.

The key takeaway is that incremental improvement beats perfect planning. Start with a basic single-worker deployment, measure its behavior under real traffic, and progressively add scaling, monitoring, and optimization features as your needs grow. This approach minimizes upfront risk while ensuring that each architectural addition addresses actual pain points rather than hypothetical ones.

As LLM technologies continue to evolve, the principles outlined here—containerization, resource awareness, observability, and progressive scaling—will remain foundational to successful self-hosted inference deployments.