Featured image of post AI-Powered Auto-Benchmarking: LLM-Driven VPS Performance Testing & Anomaly Detection

AI-Powered Auto-Benchmarking: LLM-Driven VPS Performance Testing & Anomaly Detection

Explore how to use LLMs to automate VPS benchmarking, analyze performance data, and detect anomalies — upgrading manual benchmarks into data-driven continuous performance observability.

Why VPS Needs Automated Benchmarking

Benchmarking is the core way to measure server performance. For VPS users, typical scenarios include:

  • Verifying actual performance matches spec after purchasing a new VPS
  • Long-term monitoring of performance drift (e.g., neighbor interference causing I/O jitter)
  • Comparing real-world performance across cloud providers
  • Quantifying resource utilization before scaling up or down

Traditional approaches rely on manually running tools like dd, sysbench, fio, and iperf, then interpreting raw output by hand — time-consuming and not continuous. With LLM involvement, this workflow upgrades to data-driven continuous performance observability:

  1. Automatic triggering — scheduled or post-config-change
  2. Structured output — unified collection of CPU / memory / I/O / network data
  3. LLM interpretation — understanding context (e.g., “neighbor batch job spiked iowait today”) instead of just reading raw numbers
  4. Anomaly detection — comparing against historical baseline to catch regressions
  5. Auto-reporting — generating human-readable weekly/monthly performance reports

Architecture

The system has four layers:

┌─────────────────────────────────────────────────┐
│  Reporting: Grafana Dashboard + LLM Markdown    │
├─────────────────────────────────────────────────┤
│  Analysis: Ollama (local LLM) + anomaly script  │
├─────────────────────────────────────────────────┤
│  Collection: sysbench / fio / iperf3 / custom   │
├─────────────────────────────────────────────────┤
│  Trigger: systemd timer / cron / webhook         │
└─────────────────────────────────────────────────┘

Implementation

1. Install Benchmarking Tools

# Install standard benchmark tools
apt update && apt install -y sysbench fio iperf3

# Install Ollama for local LLM analysis
curl -fsSL https://ollama.com/install.sh | sh

# Pull a lightweight model suitable for code/data interpretation
ollama pull qwen2.5:7b
# If RAM < 4GB, use 1.5B or 3B instead
# ollama pull qwen2.5:1.5b

2. Write the Benchmark Collection Script

#!/bin/bash
# benchmark_suite.sh — collect all metrics in one shot
set -euo pipefail
OUT_DIR="/var/log/benchmark/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$OUT_DIR"

# CPU benchmark
sysbench cpu --cpu-max-prime=20000 run 2>&1 | tee "$OUT_DIR/cpu.log"

# Memory bandwidth
sysbench memory --memory-block-size=4816 \
  --memory-max-size=4000000 run 2>&1 | tee "$OUT_DIR/mem.log"

# I/O benchmark
fio --name=randread --ioengine=libaio --iodepth=64 \
    --direct=1 --bs=4k --numjobs=4 --size=2G \
    --runtime=60 --filename=/dev/sda1 \
    --read_iops_log="$OUT_DIR/fio_randread.log" \
    --read_bw_log="$OUT_DIR/fio_randread_bw.log" \
    --group_reporting

iperf3 -s -J > "$OUT_DIR/iperf_server.json" &
IPERF_PID=$!
sleep 3
iperf3 -c 1.1.1.1 -J > "$OUT_DIR/iperf_client.json"
kill $IPERF_PID

# Aggregate
cat "$OUT_DIR"/*.log "$OUT_DIR"/*.json > "$OUT_DIR/all.json"
echo "Benchmark suite completed at $OUT_DIR"

3. Add the LLM Analysis Pipeline

#!/usr/bin/env python3
"""analyze_benchmark.py — interpret benchmark data with a local LLM"""
import json
import requests
from pathlib import Path

def load_latest_benchmark():
    base = Path("/var/log/benchmark")
    dirs = sorted(base.iterdir(), key=lambda d: d.name, reverse=True)
    if not dirs:
        raise FileNotFoundError("No benchmarks found")
    return json.loads((dirs[0] / "all.json").read_text())

def llm_analyze(data: dict, model: str = "qwen2.5:7b") -> str:
    prompt = f"""You are a senior SRE. Here is the raw VPS benchmark output:

{json.dumps(data, indent=2)}

Analyze:
1. Are all metrics meeting expectations (vs. spec)?
2. Are there signs of neighbor interference, I/O jitter, or other anomalies?
3. Provide at least 3 optimization recommendations.
4. Comparing to the historical baseline (attached), are there regressions?
"""
    baseline = get_baseline(data)
    prompt += f"\nHistorical baseline (7-day mean):\n{json.dumps(baseline, indent=2)}"

    resp = requests.post(
        "http://localhost:11434/api/chat",
        json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "stream": False,
        },
        timeout=120,
    )
    return resp.json()["message"]["content"]

def get_baseline(current: dict):
    """Read past data directories and compute 7-day mean"""
    from pathlib import Path
    base = Path("/var/log/benchmark")
    data_points = []
    for d in sorted(base.iterdir(), reverse=True)[:7]:
        try:
            j = json.loads((d / "all.json").read_text())
            data_points.append(j)
        except Exception:
            continue
    if not data_points:
        return {}
    keys = ["sysbench_cpu_events", "fio_iops", "iperf_bw_mbps"]
    return {
        k: sum(dp.get(k, 0) for dp in data_points) / len(data_points)
        for k in keys
    }

if __name__ == "__main__":
    data = load_latest_benchmark()
    result = llm_analyze(data)
    print(result)
    Path("/var/log/benchmark/reports").mkdir(parents=True, exist_ok=True)
    from datetime import datetime
    out = Path(f"/var/log/benchmark/reports/{datetime.now():%Y%m%d_%H%M}.md")
    out.write_text(f"# VPS Benchmark Report\n\n{result}\n")
    print(f"Report saved to {out}")

4. Configure the Scheduled Job

# /etc/systemd/system/benchmark-daily.service
[Unit]
Description=Daily VPS benchmark suite
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=root
ExecStart=/usr/local/bin/benchmark_suite.sh

[Install]
WantedBy=multi-user.target
# /etc/systemd/system/benchmark-daily.timer
[Unit]
Description=Run benchmark daily at 03:00

[Timer]
OnCalendar=03:00
Persistent=true
Unit=benchmark-daily.service

[Install]
WantedBy=timers.target
systemctl enable --now benchmark-daily.timer

Push the key metrics from the LLM report into Prometheus (or use promtail for log collection), then build a Grafana dashboard:

  • Panel 1: CPU events/sec (7-day trend)
  • Panel 2: I/O IOPS (4K random read)
  • Panel 3: Network throughput
  • Panel 4: LLM anomaly flags (custom annotations)

Advanced: Wire LLM Results into Alerting

import re
import requests

def check_and_alert(llm_output: str):
    alert_keywords = ["regression", "iowait", "neighbor interference", "bandwidth drop"]
    matched = [kw for kw in alert_keywords if kw in llm_output]
    if matched:
        requests.post(
            "https://hooks.example.com/ops-alert",
            json={
                "title": "VPS Benchmark Anomaly",
                "body": f"LLM detected anomalies: {matched}\n{llm_output}",
            },
        )

Summary

DimensionTraditionalLLM-Driven
TriggerManualAutomatic (timer/webhook)
OutputRaw numbersStructured + human-readable report
Anomaly detectionManual judgmentBaseline comparison + LLM context
AlertingNoneKeyword-matched auto-push
ScalabilitySingle nodeMulti-node aggregation + cross-node correlation

The core value of this approach: it turns benchmarking from a one-off task into a continuous observability capability, with the LLM acting as your 24/7 performance analyst instead of a post-incident forensics tool.

📺 看视频版教程 → DuckDB Lab YouTube

Subscribe for more DuckDB & AI automation tutorials