AI-Driven VPS Container Runtime Security: Behavioral Analysis, Threat Detection & Automated Response

Introduction

You have ten Docker containers running on your VPS: web services, databases, caches, message queues… Each container has passed image scanning and configuration audits. Everything looks secure. Until one day, an attacker exploits a zero-day vulnerability to enter your web container, then:

Quietly mines cryptocurrency in the background, consuming all CPU;
Encrypts files and exfiltrates data to an external C2 server;
Lateral-moves to the database container to steal sensitive information.

The blind spot of traditional container security is exactly runtime—the vulnerability scanner cannot detect an attacker who is already inside, and manual monitoring cannot keep pace with attack speed.

This is where AI-driven container runtime security solves the problem. By continuously monitoring process behavior, network traffic, and data access patterns within containers, AI systems can identify activities that deviate from normal baselines and automatically respond before the threat causes damage.

This article shows you how to build a complete runtime security protection system for a VPS running multiple containers, using entirely open-source tools at zero additional cost.

Why Runtime Security?

Container security is typically divided into three layers:

Security Layer	When It Covers	Typical Tools	Limitation
Image Security	Before build/deploy	Trivy, Clair	Cannot detect runtime-injected attacks
Configuration Security	Before deployment	Kube-bench, Checkov	Cannot detect runtime behavioral anomalies
Runtime Security	During execution	Focus of this article	Requires real-time analysis, higher resource overhead

The core challenge of runtime security is: what behavior counts as anomalous?

Traditional rule-based engines (like “block outbound network from containers”) are easily bypassed or generate excessive false positives. AI’s core advantage is learning each container’s normal behavior baseline, then detecting deviations—this is the value of user-space process monitoring.

Architecture Design

┌──────────────────────────────────────────────────────────────┐
│                    AI Security Orchestrator                   │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │ Baseline DB │  │ Anomaly     │  │ Automated           │  │
│  │ (Per-container│  │ Scoring    │  │ Response Engine     │  │
│  │  Baseline)  │  │ Engine)     │  │                     │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
│         ▲                  ▲                    ▲            │
│         │                  │                    │            │
│  ┌──────┴──────┐  ┌───────┴───────┐  ┌────────┴────────┐   │
│  │ Process Agent│  │ Network Agent  │  │ Filesystem Agent │   │
│  │ (Syscall    │  │ (eBPF          │  │ (Inotify +       │   │
│  │  tracing)   │  │  packet cap)   │  │  Hash tracking)  │   │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
└──────────────────────────────────────────────────────────────┘
                          │
              ┌───────────┼───────────┐
              ▼           ▼           ▼
         ┌────────┐  ┌────────┐  ┌────────┐
         │Container│  │Container│  │Container│
         │  A (Web) │  │  B (DB) │  │  C (Cache)│
         └────────┘  └────────┘  └────────┘

Core design principles:

Independent per-container modeling: Web and database containers have completely different normal behaviors—no shared rules
Separate learning and detection phases: Let the system observe for 3-7 days to build baselines, then switch to real-time detection
Tiered response: Low risk → alert only; medium risk → restrict operations; high risk → immediate isolation

Step 1: Deploy User-space Process Monitoring

We use falco as the core runtime security engine. Falco traces system calls via eBPF and can capture process behavior inside containers.

Install Falco

# Add Falco repository
curl -s https://falco.org/repo/falcosecurity-packages.asc | \
  sudo apt-key add -
echo "deb https://download.falco.org/packages/deb/ stable main" | \
  sudo tee -a /etc/apt/sources.list.d/falco.list

# Install
sudo apt-get update && sudo apt-get install -y falco

Define Per-Container Behavior Baselines

Falco’s core is its rule files. Default rules cover common threats, but we need to add custom baseline rules for each container.

Create /etc/falco/falco_rules.local.yaml:

# Web container behavior baseline
- rule: Web container normal processes
  desc: Allow only expected processes in web container
  condition: container.name=web-app and not proc.name in (nginx, node, python, bash)
  output: "Web container unexpected process: %proc.name (pid=%proc.pid)"
  priority: WARNING

# Database container behavior baseline
- rule: DB container unexpected network
  desc: Database should not initiate outbound connections
  condition: container.name=db-main and container.net.transport=tcp and container.net.direction=egress
  output: "DB container outbound connection blocked: %container.net.dst:%container.net.dstport"
  priority: CRITICAL
  prefilter: true

Integrate with Docker Compose

version: '3.8'
services:
  falco:
    image: falcosecurity/falco:latest
    container_name: falco
    restart: unless-stopped
    privileged: true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:rw
      - /dev:/host/dev:ro
      - ./falco_rules.local.yaml:/etc/falco/falco_rules.local.yaml:ro
    environment:
      - FALCO_K8S_AUDIT_LOG_ENABLED=false
      - OUTPUT_FORMAT=json  # Easy AI parsing
    networks:
      - security-net

  web-app:
    image: my-webapp:latest
    security_opt:
      - no-new-privileges:true
    deploy:
      resources:
        limits:
          memory: 512M
          cpus: '1'
    networks:
      - security-net

networks:
  security-net:
    driver: bridge

Step 2: Build AI Behavioral Analysis Engine

Falco outputs events as JSON streams. We need an AI engine to analyze these events.

System Architecture

# runtime_analyzer.py
import json
import time
import numpy as np
from collections import defaultdict, deque
from datetime import datetime, timedelta

class ContainerBehaviorAnalyzer:
    """Learn behavior baselines per container and detect anomalies"""
    
    def __init__(self, learning_days=7, alert_threshold=0.7):
        self.baselines = {}  # {container_id: baseline_data}
        self.learning_days = learning_days
        self.alert_threshold = alert_threshold
        self.learning_start = time.time()
        
    def process_event(self, event: dict) -> dict:
        """Process a single Falco event, return analysis result"""
        container = event.get('container', {}).get('name', 'unknown')
        ts = event.get('time', '')
        
        # Learning phase: accumulate baseline data
        if time.time() - self.learning_start < self.learning_days * 86400:
            self._update_baseline(container, event)
            return {'level': 'learning', 'container': container}
        
        # Detection phase: score and return anomaly level
        score = self._compute_anomaly_score(container, event)
        level = self._score_to_level(score)
        
        return {
            'level': level,
            'container': container,
            'anomaly_score': score,
            'rule': event.get('rule', ''),
            'timestamp': ts
        }
    
    def _update_baseline(self, container: str, event: dict):
        """Update container behavior baseline"""
        if container not in self.baselines:
            self.baselines[container] = {
                'processes': set(),
                'network_destinations': set(),
                'syscall_patterns': defaultdict(int),
                'file_access_patterns': set(),
                'time_profile': defaultdict(float)  # hour → event density
            }
        
        bl = self.baselines[container]
        
        # Record processes
        if 'proc' in event:
            bl['processes'].add(event['proc'].get('name', ''))
        
        # Record network destinations
        if 'output_fields' in event:
            nf = event['output_fields'].get('nf', {})
            if nf.get('dst_ip'):
                bl['network_destinations'].add(f"{nf['dst_ip']}:{nf.get('dst_port','')}")
        
        # Record syscall patterns
        if 'proc' in event and 'syscall' in event['proc']:
            bl['syscall_patterns'][event['proc']['syscall']] += 1
        
        # Record time distribution
        hour = datetime.now().hour
        bl['time_profile'][hour] += 1
    
    def _compute_anomaly_score(self, container: str, event: dict) -> float:
        """Compute anomaly score (0.0-1.0)"""
        if container not in self.baselines:
            return 0.5  # No baseline → medium score
        
        bl = self.baselines[container]
        scores = []
        
        # 1. Unknown process score
        if 'proc' in event:
            proc_name = event['proc'].get('name', '')
            if proc_name and proc_name not in bl['processes']:
                scores.append(0.6)
        
        # 2. Anomalous network score
        output = event.get('output_fields', {})
        nf = output.get('nf', {})
        if nf.get('dst_ip') and nf['dst_ip'] not in ['0.0.0.0', '127.0.0.1']:
            dest = f"{nf['dst_ip']}:{nf.get('dst_port','')}"
            if dest not in bl['network_destinations']:
                scores.append(0.8)
        
        # 3. Time anomaly score
        hour = datetime.now().hour
        if bl['time_profile']:
            avg_density = sum(bl['time_profile'].values()) / len(bl['time_profile'])
            current_density = bl['time_profile'].get(hour, 0)
            if avg_density > 0 and current_density < avg_density * 0.1:
                scores.append(0.5)
        
        return max(scores) if scores else 0.0
    
    def _score_to_level(self, score: float) -> str:
        if score >= 0.8:
            return 'CRITICAL'
        elif score >= 0.5:
            return 'HIGH'
        elif score >= 0.3:
            return 'MEDIUM'
        return 'LOW'

Run the Analysis Service

# Start analyzer listening to Falco's stdout
docker run -d --name runtime-analyzer \
  -v $(pwd):/app \
  -e FALCO_SOCKET=/var/run/falco.sock \
  python:3.11-slim \
  python /app/runtime_analyzer.py --input /var/log/falco/events.jsonl

Step 3: Automated Response Policies

After detecting a threat, the system must respond automatically. Response intensity should match the threat level:

# response_engine.py
class AutomatedResponseEngine:
    """Execute corresponding automated response based on anomaly level"""
    
    RESPONSE_ACTIONS = {
        'LOW': 'log',
        'MEDIUM': 'alert', 
        'HIGH': 'contain',
        'CRITICAL': 'isolate'
    }
    
    def __init__(self, docker_client):
        self.client = docker_client
    
    def execute(self, analysis_result: dict):
        level = analysis_result['level']
        container_name = analysis_result['container']
        action = self.RESPONSE_ACTIONS[level]
        
        if action == 'log':
            self._log_event(analysis_result)
            
        elif action == 'alert':
            self._send_alert(analysis_result)
            
        elif action == 'contain':
            self._limit_container(container_name)
            
        elif action == 'isolate':
            self._isolate_container(container_name, analysis_result)
    
    def _isolate_container(self, name: str, context: dict):
        """Isolate container: disconnect network but keep running for forensics"""
        # Method 1: Remove network interfaces
        self.client.containers.get(name).pause()
        
        # Method 2: Set network policies
        # docker network disconnect default <container>
        # docker network connect isolated-net <container>
        
        print(f"[CRITICAL] Isolated container: {name}")
        print(f"Context: {json.dumps(context, indent=2)}")
        
        # Create forensic snapshot
        self._create_forensic_snapshot(name)
    
    def _limit_container(self, name: str):
        """Limit container resources to reduce threat impact"""
        container = self.client.containers.get(name)
        container.update(blkio_weight=10, mem_limit='128m', cpus='0.5')
        print(f"[HIGH] Limited resources for container: {name}")
    
    def _create_forensic_snapshot(self, name: str):
        """Create container runtime snapshot for post-incident analysis"""
        config = self.client.containers.get(name).attrs
        logs = self.client.containers.get(name).logs(timespan=3600)
        print(f"[FORENSIC] Snapshot created for {name}")

Step 4: Integrate with VPS Monitoring Stack

Integrate runtime security into the existing VPS monitoring system:

# docker-compose.monitoring.yml
version: '3.8'
services:
  # Runtime security
  falco:
    image: falcosecurity/falco:latest
    privileged: true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:rw
      - /dev:/host/dev:ro
      - ./falco_rules.local.yaml:/etc/falco/falco_rules.local.yaml:ro
    networks:
      - monitoring

  # Behavioral analysis engine
  analyzer:
    build: ./analyzer
    depends_on:
      - falco
    networks:
      - monitoring

  # Alert routing
  alertmanager:
    image: prom/alertmanager:latest
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    networks:
      - monitoring

  # Visualization dashboard
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - ./dashboards:/var/lib/grafana/dashboards
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

Alert Rule Configuration

# alertmanager.yml
route:
  receiver: 'vps-alerts'
  group_by: ['container', 'level']
  routes:
    - match:
        level: 'CRITICAL'
      receiver: 'pagerduty-critical'
      repeat_interval: 5m
    - match:
        level: 'HIGH'  
      receiver: 'telegram-high'
      repeat_interval: 30m
    - match:
        level: ~
      receiver: 'slack-all'

receivers:
  - name: 'vps-alerts'
    webhook_configs:
      - url: 'http://analyzer:8080/webhook'

Real-World Demo: Cryptomining Detection

Let’s see how this system performs in a real attack scenario:

Normal Baseline (accumulated during learning period):

Web container processes: nginx, node, python3
Outbound destinations: api.service.com:443, cdn.example.com:443
Active hours: 9:00-22:00

Attack Occurs:

{
  "rule": "Drop CAP_SYS_PTRACE",
  "output": "Container <web-app> started process <xmrig> (pid=12345)",
  "priority": "CRITICAL",
  "time": "2026-09-08T03:15:22Z",
  "output_fields": {
    "proc.name": "xmrig",
    "container.name": "web-app",
    "nf.dst_ip": "pool.mining.com",
    "nf.dst_port": "3333"
  }
}

AI Analysis Result:

Unknown process: xmrig not in baseline → +0.6 score
Anomalous connection: pool.mining.com:3333 not in baseline → +0.8 score
Anomalous time: 03:15 AM, off-peak hours → +0.5 score
Combined score: 0.8 → CRITICAL

Automated Response:

Immediately pause container web-app
Send Telegram alert: “🚨 CRITICAL: Cryptomining process xmrig detected, container web-app isolated”
Create forensic snapshot for later analysis
Notify admin to investigate the entry vector

Advanced: LLM-Powered Root Cause Analysis

For complex attack scenarios, call a local LLM for deep analysis:

# llm_root_cause.py
from openai import OpenAI

def analyze_incident(event_history: list) -> str:
    """Use LLM to analyze attack chain and provide remediation advice"""
    client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
    
    prompt = f"""Analyze the following container security event sequence, identify the attack chain, and provide remediation suggestions:

{json.dumps(event_history, indent=2)}

Please answer in Chinese:
1. What type of attack is this?
2. How might the attacker have gained entry?
3. Recommended immediate remediation steps?
4. Long-term hardening plan?"""
    
    response = client.chat.completions.create(
        model="qwen2.5:7b",
        messages=[{"role": "user", "content": prompt}]
    )
    
    return response.choices[0].message.content

Summary

Building an AI-driven container runtime security system requires three layers:

Layer	Tool	Purpose
Data Collection	Falco + eBPF	Capture process, network, file behavior
Behavioral Analysis	Custom Python engine	Learn baselines, compute anomaly scores
Automated Response	Docker API + alerting integration	Auto-remediate based on threat level

The core value of this system is: shifting from “post-incident investigation” to “real-time interception”. When traditional security measures (image scanning, config checks) are already bypassed, behavioral analysis can still detect anomalies and respond automatically.

For VPS operators, this means even if a zero-day exploit succeeds, the attack can be detected and contained before it causes实质性 damage.