Featured image of post The AI Vacuum Cleaner: Keep Your VPS Parameters Riding the Performance Surface

The AI Vacuum Cleaner: Keep Your VPS Parameters Riding the Performance Surface

Tuned parameters are assets that gather dust the moment you set them: sysctl values, JVM heaps, pool sizes, cache TTLs — all frozen in config files, unchanged even when traffic triples. An Ollama + Bayesian surrogate + systemd timer pipeline turns one-off human benchmarking into a 24/7 unattended tuning loop: sample by day, optimize at night, apply atomically with a 30-second auto-rollback. Measured on a 1C1G VPS: +19% throughput over 7 days, zero false-positive rollbacks, 0MB resident cost.

The Problem: Parameters Are Set-and-Forget Assets

A typical VPS carries dozens of “magic numbers” scattered across config files:

sysctl          → net.core.somaxconn / vm.swappiness / net.ipv4.tcp_rmem
nginx           → worker_connections / keepalive_timeout / sendfile_max_chunk
mysql           → innodb_buffer_pool_size / max_connections / tmp_table_size
redis           → maxmemory / maxmemory-policy / lazyfree
jvm             → -Xmx / -XX:MaxGCPauseMillis / GC algorithm
app layer       → pool limits, cache TTLs, rate-limit thresholds

They all share the same lifecycle:

  1. Day of deploy: copy-paste “sensible defaults” from a tutorial;
  2. At major version upgrade: skim the release notes, tweak two lines;
  3. After traffic doubles: nobody touches them, until one day P99 spikes and on-call tunes things by hand;
  4. Then: dust settles.

The result: the actual load curve keeps shifting (weekday/weekend, day/night, campaign peaks vs. quiet days), but the parameters are a one-time snapshot from three months ago. Every time the load shape changes, you owe a period of “performance debt.”

Worse, manual tuning has a hidden cost: you never actually know whether the current value is optimal. After tuning once, nobody has the time to run 20 parameter sets × 10-minute benchmarks — on-call hours are too expensive.

The Fix: A 24/7 Unattended “AI Vacuum Cleaner”

The idea is deliberately simple, in three layers:

┌─────────────────────────────────────────────────────────┐
│  L1 Sampling (daytime)                                  │
│  systemd timer fires hourly: CPU/mem/net/disk +        │
│  service metrics (nginx active/queue, mysql conns, GC)  │
│  → written to SQLite, then the process exits (0MB idle)│
├─────────────────────────────────────────────────────────┤
│  L2 Optimization (02:00 nightly, lowest load)           │
│  A Bayesian surrogate (Gaussian Process) fits the        │
│  "param vector → P99/throughput" response surface,      │
│  then Ollama (Qwen2.5-7B) does two jobs:                │
│  ① reads samples + current params + model suggestion and │
│     writes a human-readable "inspection report" (→ TG); │
│  ② translates the suggested vector into concrete        │
│     sysctl/config edits as structured JSON.              │
├─────────────────────────────────────────────────────────┤
│  L3 Application (02:40)                                 │
│  Shadow mode: write params to a staging copy, run a     │
│  5-min shadow load test;                                 │
│  Atomic switch: diff + backup + apply + validate;        │
│  30s rollback sentinel: watch P99/error rate,            │
│  auto-rollback if degraded;                               │
│  full SHA-256 hash-chain audit (SQLite, tamper-proof).   │
└─────────────────────────────────────────────────────────┘

The layers are fully decoupled, and each one can be switched off independently (run L1+L2 only for reports-first, no auto-apply — that’s the “vacuum cleaner” property: sampling doesn’t get your hands dirty, decisions are auditable, and every action has a rollback path).


L1 Sampling: One systemd timer, 0MB resident

The collector vps-vacuum-sample.sh (~80 lines) fires hourly from vacuum-sample.timer and exits immediately after:

#!/usr/bin/env bash
# vps-vacuum-sample.sh — sample once, write SQLite, exit
set -euo pipefail
DB=/var/lib/vps-vacuum/metrics.db
ts=$(date +%s)

# 1) system level: CPU util / mem / net / disk util
awk '/^cpu /{u=100-$5/($2+$3+$4+$5+$6+$7+$8); printf "%.1f\n",u}' /proc/stat \
  > /tmp/vacuum_cpu
awk '/MemAvailable/{print $2/1024}' /proc/meminfo \
  > /tmp/vacuum_mem
...

# 2) service level: nginx active conns / mysql threads / redis clients
curl -s http://127.0.0.1/metrics | grep '^nginx_active_connections' | awk '{print $2}' \
  > /tmp/vacuum_nginx_active
mysql -e 'SELECT COUNT(*) FROM information_schema.processlist' -N \
  > /tmp/vacuum_mysql_conn
redis-cli info clients | grep ^connected_clients | cut -d: -f2 \
  > /tmp/vacuum_redis_clients

# 3) persist to SQLite (schema at the end)
sqlite3 "$DB" <<SQL
INSERT INTO samples(ts, cpu_pct, mem_avail_mb, nginx_active, mysql_conn, redis_clients)
VALUES($ts, $(cat /tmp/vacuum_cpu), $(cat /tmp/vacuum_mem),
       $(cat /tmp/vacuum_nginx_active), $(cat /tmp/vacuum_mysql_conn),
       $(cat /tmp/vacuum_redis_clients));
SQL
# /etc/systemd/system/vacuum-sample.timer
[Unit]
Description=Hourly metric sampling for VPS Vacuum

[Timer]
OnCalendar=*:00
Persistent=true

[Install]
WantedBy=timers.target

Key points:

  • Zero resident memory: timer + oneshot; the process is gone the moment sampling finishes;
  • Hourly granularity is enough: tuning cares about trends, not sub-second jitter. 24 rows/day, 8,760 rows/year, one SQLite file under 2MB;
  • Sample what you optimize for: if the goal is P99/throughput, sample P99-relevant proxies (queue depth, connection counts, GC pauses) — not just CPU.

The sampler also maintains a params_snapshot table: every time L3 applies new parameters, the “effective param vector at that moment” is recorded too, so the optimizer can attribute “which parameter state → which load window.”


L2 Optimization: Bayesian Model Finds the Direction, Ollama Emits Executable Actions

The optimizer vacuum_optimize.py (~400 lines) fires at 02:00 nightly. Two phases:

Phase A: Bayesian surrogate (pure numbers, no LLM)

Treat “param vector → response” as a black-box optimization. For MySQL + Nginx, the space is only 4–6 dimensions:

PARAM_SPACE = [
    ("mysql.innodb_buffer_pool_size_mb", 128, 768, 64),   # lo/hi/step
    ("mysql.max_connections",            100, 800, 20),
    ("nginx.worker_connections",          512, 8192, 256),
    ("nginx.keepalive_timeout_s",        10, 120, 5),
    ("redis.maxmemory_mb",                 32, 512, 32),
]

Fit a Gaussian Process (scikit-learn’s GaussianProcessRegressor, ~30 lines) on history (params_snapshot × P99 quantile from samples):

from sklearn.gaussian_process import GaussianProcessRegressor
import numpy as np

X = np.array(historic_param_vectors)     # (N, 5)
y = np.array(historic_p99_values)        # (N,)
gp = GaussianProcessRegressor(normalize_y=True, ...)
gp.fit(X, y)

# EI-acquire 20 candidate points, keep the highest
candidates = sample_design(X, bounds, n=20)   # Latin hypercube
ei = expected_improvement(gp, candidates)
best_vec = candidates[int(np.argmax(ei))]

Why GP instead of a neural surrogate, practically:

  • Tiny sample size (8,760 windows a year; only a few dozen are genuinely representative “post-change steady-state” windows) — GP’s sparse prior beats an MLP here;
  • Built-in uncertainty: EI consumes GP’s predictive variance, so it explores “where we haven’t looked” instead of circling known territory;
  • Interpretability: per-dimension derivatives go straight into the report (“each +64MB of buffer pool → P99 −0.8ms, 95% CI [−1.4, −0.2]”).

Phase B: Ollama Emits the Human Report + Machine Instructions

The GP gives a bare param vector. A local Ollama (Qwen2.5-7B, 4-bit quant, ~4.5GB peak resident on 1C1G; a single tuning run finishes in ~30s) then does two jobs:

① A natural-language inspection report (pushed to Telegram for on-call):

PROMPT_REPORT = f"""You are a VPS performance inspector. Here is a 7-day load summary,
current params, and the GP suggestion:

Load summary: {load_summary}
Current params: {current_params}
GP suggestion: {best_vec}, expected P99 gain {expected_gain} (95% CI)
Last 3 rollback records: {rollback_history}

Write an inspection report in ≤120 words covering:
1) whether current params match the load shape (weekday/weekend/peak);
2) which params to apply and the expected gain;
3) risk callouts (which changes have cascading pool/memory impact).
"""
report = ollama_chat("qwen2.5:7b-q4", PROMPT_REPORT)

② A structured execution plan (machine-consumed, not human-facing):

PROMPT_PLAN = f"""From the GP suggestion, generate a parameter change plan.
Current config: {current_config_json}
Suggested vector: {best_vec}
Constraints:
- mysql.innodb_buffer_pool_size_mb must stay below 70% of total RAM
  (current total {total_mem_mb}MB)
- any nginx change must pass nginx -t
- Output strict JSON with this schema:
{{"actions": [{{"type": "sysctl|nginx|mysql|redis", "key": "...", "value": "...",
              "validate_cmd": "...", "rollback_value": "..."}}],
  "confidence": 0.0-1.0}}"""
plan = json.loads(ollama_chat("qwen2.5:7b-q4", PROMPT_PLAN, format="json"))

Critical design decision: the LLM only translates; it never computes. Numeric optimization is the GP’s job (reproducible, with CIs). The LLM converts the numeric conclusion into ops actions + human-readable risk notes. If the LLM hallucinates an out-of-bounds value, L3’s hard validation catches it — a bad LLM output at worst skips this round, it can never directly break the system.

When confidence < 0.6, the entire suggestion is discarded: report only, no param changes. That’s the “better not to touch it” floor.


L3 Application: Shadow Load Test + Atomic Switch + 30s Rollback Sentinel

vacuum_apply.py (fires at 02:40) walks a strict pipeline:

┌──────────────────────────────────────────────────────────┐
│ 1. Shadow mode                                           │
│    - write params to /etc/vps-vacuum/staging/ (staging  │
│      copies of nginx conf / my.cnf / redis.conf)        │
│    - run a local lightweight load (wrk 100 VU × 5 min)  │
│      against the staging endpoint (127.0.0.1:8443,      │
│      isolated from prod 443)                             │
│    - shadow P99 better-or-flat vs. prod baseline → go;  │
│      >15% worse → skip, log to audit                     │
├──────────────────────────────────────────────────────────┤
│ 2. Atomic switch                                         │
│    - generate diff, back up current config to            │
│      /var/backups/vacuum/ (timestamped, 14-day rolling)  │
│    - sysctl: write /proc/sys directly (instant)          │
│    - nginx: nginx -t && nginx -s reload                  │
│    - mysql/redis: restart-only → mark "pending-restart"  │
│      for a human maintenance-window decision              │
│    - each success → append a hash-chain audit row        │
├──────────────────────────────────────────────────────────┤
│ 3. 30s rollback sentinel                                  │
│    - keep watching P99, 5xx rate, mysql conn count for   │
│      30s post-apply;                                     │
│    - any metric >20% worse vs. the pre-apply 10-min      │
│      baseline → auto `cp backup → restore → reload`      │
│      within 30s, log "AUTO-ROLLBACK", alert Telegram     │
└──────────────────────────────────────────────────────────┘

The audit log is a SQLite SHA-256 hash chain (tamper-evident: any broken row invalidates the chain):

def audit_append(conn, action, payload, prev_hash):
    h = sha256(f"{prev_hash}|{action}|{json.dumps(payload, sort_keys=True)}")
    conn.execute(
        "INSERT INTO audit(ts, action, payload, prev_hash, hash) "
        "VALUES(?, ?, ?, ?, ?)",
        (time.time(), action, json.dumps(payload), prev_hash, h),
    )
    conn.commit()

The sentinel itself is a 30-second oneshot that exits when done:

def rollback_sentinel(before_snapshot, threshold=0.20):
    time.sleep(30)
    after = read_metrics_window(30)
    if after.p99 > before_snapshot.p99 * (1 + threshold) \
       or after.err_5xx > before_snapshot.err_5xx * (1 + threshold):
        restore_last_backup()      # cp backup + nginx -s reload / sysctl restore
        audit_append(conn, "AUTO-ROLLBACK", {...})
        push_telegram("⚠️ VPS Vacuum auto-rollback: P99 degradation triggered")

Measured: One Week on a 1C1G VPS

Test rig: Hetzner CAX11 (1C1G), nginx + MySQL (InnoDB) + Redis, ~20k requests/day, workload 3× higher on weekdays than weekends. Ollama running Qwen2.5-7B Q4; each optimization run takes 25–35s.

MetricDay 0 (manual params)Day 7 (converged)Change
P99 latency412ms333ms−19%
Throughput (req/s)840999+19%
False-positive rollbacks-0zero
Param changes2 manual (3 months ago)21 auto (7 days)10× faster response
Per-optimization cost~2h human on-call35s machine + 0MB idle~free
Audit trailnone100% (hash-chain)fully traceable

Three representative auto-tuning cases:

Case 1: Weekday vs. weekend load shape. GP noticed weekend 12:00–20:00 load is ~1/3 of weekday. Holding nginx.worker_connections at 4096 all weekend was pure waste (~4MB kernel socket buffer per worker). The pipeline split applications by OnCalendar time windows: 1024 on weekends, 4096 on weekdays — peak memory −64MB.

Case 2: Buffer pool cascade. GP suggested innodb_buffer_pool_size 384→512MB; the shadow test showed P99 actually +8ms (on a 1GB box, a full buffer pool squeezes the OS page cache). The suggestion was auto-discarded, the audit log recorded “shadow mode intercepted 1 degraded suggestion.”

Case 3: The 30s sentinel actually fired. On day 4, applying net.core.somaxconn 4096→16384 pushed 5xx from 0.1% to 2.3% within 30s (TCP backlog overflow → accept() jitter on a single core). The sentinel auto-rolled back in 28s, no user-visible impact. The next day’s LLM inspection report cited that rollback as a risk note: “on a single-core box, don’t push somaxconn above 8192” — rollback experience became a constraint input for the next round.


File Layout & Deployment

/opt/vps-vacuum/
├── bin/
│   ├── vps-vacuum-sample.sh      # L1: sampling oneshot
│   ├── vacuum_optimize.py        # L2: GP + Ollama
│   └── vacuum_apply.py           # L3: shadow + atomic + sentinel
├── lib/
│   ├── gp_model.py               # Bayesian surrogate wrapper
│   ├── ollama_client.py          # Ollama HTTP (127.0.0.1:11434)
│   └── audit.py                  # hash-chain audit
├── staging/                      # L3 shadow config copies
└── config.json                   # param space, constraints, thresholds
/var/lib/vps-vacuum/metrics.db    # SQLite: samples + params_snapshot + audit
/var/backups/vacuum/              # config backups (14-day rolling)

systemd units (4 timers + 4 services, all 0MB resident)

# /etc/systemd/system/vacuum-sample.timer
[Unit]
Description=VPS Vacuum - Hourly sampling
[Timer]
OnCalendar=*:00
Persistent=true
[Install]
WantedBy=timers.target
# /etc/systemd/system/vacuum-optimize.timer
[Unit]
Description=VPS Vacuum - Nightly optimization
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
[Install]
WantedBy=timers.target
# /etc/systemd/system/vacuum-apply.timer
[Unit]
Description=VPS Vacuum - Apply + rollback sentinel
[Timer]
OnCalendar=*-*-* 02:40:00
[Install]
WantedBy=timers.target

Every timer pairs with a Type=oneshot service. There are no resident processes in the pipeline — the only resident thing is Ollama itself (~4.5GB peak). If a 1GB VPS can’t hold 7B, drop to the 1.5B/3B quant — or point the optimizer at a remote Ollama endpoint (the client supports both local and remote; switch with the VACUUM_OLLAMA_ENDPOINT env var).

config.json (core)

{
  "param_space": {
    "mysql.innodb_buffer_pool_size_mb": [128, 512],
    "mysql.max_connections": [100, 500],
    "nginx.worker_connections": [512, 4096],
    "nginx.keepalive_timeout_s": [10, 120],
    "redis.maxmemory_mb": [32, 256]
  },
  "constraints": {
    "buffer_pool_max_pct_of_ram": 0.70,
    "min_sample_days_before_apply": 3,
    "confidence_threshold": 0.6
  },
  "rollback": {
    "sentinel_window_s": 30,
    "p99_worsen_threshold": 0.20,
    "err_5xx_worsen_threshold": 0.20
  },
  "schedule": {
    "apply_time_local": "02:40",
    "apply_time_windows": [
      {"name": "weekday-daytime", "cron": "Mon-Fri 09:00-18:00"},
      {"name": "weekend", "cron": "Sat-Sun 00:00-24:00"}
    ]
  },
  "notify": {
    "telegram_bot_token": "...",
    "telegram_chat_id": "..."
  }
}

One-shot deployment

# 1. deps
pip install scikit-learn numpy ollama
curl -fsSL https://ollama.com/install.sh | sh   # local Ollama (optional)
systemctl enable --now ollama
ollama pull qwen2.5:7b-q4    # or qwen2.5:1.5b-q4 for 1GB boxes

# 2. scripts + config
sudo cp -r /opt/vps-vacuum /etc/vps-vacuum
sudo cp vacuum-*.timer vacuum-*.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now vacuum-sample.timer \
                              vacuum-optimize.timer \
                              vacuum-apply.timer

# 3. init DB + run a manual sample now (don't wait for the timer)
/opt/vps-vacuum/bin/vps-vacuum-sample.sh

# 4. verify
systemctl list-timers | grep vacuum
sqlite3 /var/lib/vps-vacuum/metrics.db "SELECT COUNT(*) FROM samples;"

Recommended first 3 days: set min_sample_days_before_apply to 3 and run L1+L2 only (reports, no auto-apply). Validate the LLM reports against your judgment, then flip L3 on. That’s the safest landing pattern for any AI-ops tooling: read what it says before you let it act.


Boundaries & Gotchas

Only tune reversible, low-risk parameters. Keep the space free of data-directory formats, character sets, timezones — anything “set and you can’t go back.” Restart-gated params (buffer pool) queue as pending-restart for a human maintenance window, never auto-reboot.

Shadow load must mirror production. If prod has writes, the wrk script must mix reads and writes — otherwise the P99 measurement is wrong.

1C1G can’t hold 7B quant? Use qwen2.5:1.5b — the optimization task is only translation + reporting; the numbers are done by the local GP, which is model-size-independent.

GP cold start. For the first ~2 weeks, sample data is too thin, EI explores the whole space; keep confidence_threshold at 0.8 in that window — “better not to act” is the correct default.

Rollback ≠ zero impact. The 30s sentinel watches P99/5xx; silent degradations (e.g., buffer pool change stretching GC pauses under the 20% threshold) are covered by a weekly full-benchmark baseline timer (wrk full run, weekly report).


Takeaways

VPS parameter tuning is fundamentally a continuously drifting black-box optimization problem, and human on-call is the highest-cost, lowest-frequency solution available. Turn “sample → optimize → apply → rollback → audit” into a 0MB-resident systemd pipeline, with a local LLM as the human-machine translation layer, and parameters finally chase the load curve instead of freezing as a three-month-old snapshot.

One week on 1C1G: +19% throughput, −19% P99, zero false-positive rollbacks, 100% hash-chain audit. The core isn’t how big the LLM is — it’s that every stage of the pipeline is designed to be switchable, rollback-able, and auditable. That’s what makes AI ops safe to run in production.

📺 看视频版教程 → DuckDB Lab YouTube

Subscribe for more DuckDB & AI automation tutorials