The Problem: Parameters Are Set-and-Forget Assets
A typical VPS carries dozens of “magic numbers” scattered across config files:
sysctl → net.core.somaxconn / vm.swappiness / net.ipv4.tcp_rmem
nginx → worker_connections / keepalive_timeout / sendfile_max_chunk
mysql → innodb_buffer_pool_size / max_connections / tmp_table_size
redis → maxmemory / maxmemory-policy / lazyfree
jvm → -Xmx / -XX:MaxGCPauseMillis / GC algorithm
app layer → pool limits, cache TTLs, rate-limit thresholds
They all share the same lifecycle:
- Day of deploy: copy-paste “sensible defaults” from a tutorial;
- At major version upgrade: skim the release notes, tweak two lines;
- After traffic doubles: nobody touches them, until one day P99 spikes and on-call tunes things by hand;
- Then: dust settles.
The result: the actual load curve keeps shifting (weekday/weekend, day/night, campaign peaks vs. quiet days), but the parameters are a one-time snapshot from three months ago. Every time the load shape changes, you owe a period of “performance debt.”
Worse, manual tuning has a hidden cost: you never actually know whether the current value is optimal. After tuning once, nobody has the time to run 20 parameter sets × 10-minute benchmarks — on-call hours are too expensive.
The Fix: A 24/7 Unattended “AI Vacuum Cleaner”
The idea is deliberately simple, in three layers:
┌─────────────────────────────────────────────────────────┐
│ L1 Sampling (daytime) │
│ systemd timer fires hourly: CPU/mem/net/disk + │
│ service metrics (nginx active/queue, mysql conns, GC) │
│ → written to SQLite, then the process exits (0MB idle)│
├─────────────────────────────────────────────────────────┤
│ L2 Optimization (02:00 nightly, lowest load) │
│ A Bayesian surrogate (Gaussian Process) fits the │
│ "param vector → P99/throughput" response surface, │
│ then Ollama (Qwen2.5-7B) does two jobs: │
│ ① reads samples + current params + model suggestion and │
│ writes a human-readable "inspection report" (→ TG); │
│ ② translates the suggested vector into concrete │
│ sysctl/config edits as structured JSON. │
├─────────────────────────────────────────────────────────┤
│ L3 Application (02:40) │
│ Shadow mode: write params to a staging copy, run a │
│ 5-min shadow load test; │
│ Atomic switch: diff + backup + apply + validate; │
│ 30s rollback sentinel: watch P99/error rate, │
│ auto-rollback if degraded; │
│ full SHA-256 hash-chain audit (SQLite, tamper-proof). │
└─────────────────────────────────────────────────────────┘
The layers are fully decoupled, and each one can be switched off independently (run L1+L2 only for reports-first, no auto-apply — that’s the “vacuum cleaner” property: sampling doesn’t get your hands dirty, decisions are auditable, and every action has a rollback path).
L1 Sampling: One systemd timer, 0MB resident
The collector vps-vacuum-sample.sh (~80 lines) fires hourly from vacuum-sample.timer and exits immediately after:
#!/usr/bin/env bash
# vps-vacuum-sample.sh — sample once, write SQLite, exit
set -euo pipefail
DB=/var/lib/vps-vacuum/metrics.db
ts=$(date +%s)
# 1) system level: CPU util / mem / net / disk util
awk '/^cpu /{u=100-$5/($2+$3+$4+$5+$6+$7+$8); printf "%.1f\n",u}' /proc/stat \
> /tmp/vacuum_cpu
awk '/MemAvailable/{print $2/1024}' /proc/meminfo \
> /tmp/vacuum_mem
...
# 2) service level: nginx active conns / mysql threads / redis clients
curl -s http://127.0.0.1/metrics | grep '^nginx_active_connections' | awk '{print $2}' \
> /tmp/vacuum_nginx_active
mysql -e 'SELECT COUNT(*) FROM information_schema.processlist' -N \
> /tmp/vacuum_mysql_conn
redis-cli info clients | grep ^connected_clients | cut -d: -f2 \
> /tmp/vacuum_redis_clients
# 3) persist to SQLite (schema at the end)
sqlite3 "$DB" <<SQL
INSERT INTO samples(ts, cpu_pct, mem_avail_mb, nginx_active, mysql_conn, redis_clients)
VALUES($ts, $(cat /tmp/vacuum_cpu), $(cat /tmp/vacuum_mem),
$(cat /tmp/vacuum_nginx_active), $(cat /tmp/vacuum_mysql_conn),
$(cat /tmp/vacuum_redis_clients));
SQL
# /etc/systemd/system/vacuum-sample.timer
[Unit]
Description=Hourly metric sampling for VPS Vacuum
[Timer]
OnCalendar=*:00
Persistent=true
[Install]
WantedBy=timers.target
Key points:
- Zero resident memory: timer + oneshot; the process is gone the moment sampling finishes;
- Hourly granularity is enough: tuning cares about trends, not sub-second jitter. 24 rows/day, 8,760 rows/year, one SQLite file under 2MB;
- Sample what you optimize for: if the goal is P99/throughput, sample P99-relevant proxies (queue depth, connection counts, GC pauses) — not just CPU.
The sampler also maintains a params_snapshot table: every time L3 applies new parameters, the “effective param vector at that moment” is recorded too, so the optimizer can attribute “which parameter state → which load window.”
L2 Optimization: Bayesian Model Finds the Direction, Ollama Emits Executable Actions
The optimizer vacuum_optimize.py (~400 lines) fires at 02:00 nightly. Two phases:
Phase A: Bayesian surrogate (pure numbers, no LLM)
Treat “param vector → response” as a black-box optimization. For MySQL + Nginx, the space is only 4–6 dimensions:
PARAM_SPACE = [
("mysql.innodb_buffer_pool_size_mb", 128, 768, 64), # lo/hi/step
("mysql.max_connections", 100, 800, 20),
("nginx.worker_connections", 512, 8192, 256),
("nginx.keepalive_timeout_s", 10, 120, 5),
("redis.maxmemory_mb", 32, 512, 32),
]
Fit a Gaussian Process (scikit-learn’s GaussianProcessRegressor, ~30 lines) on history (params_snapshot × P99 quantile from samples):
from sklearn.gaussian_process import GaussianProcessRegressor
import numpy as np
X = np.array(historic_param_vectors) # (N, 5)
y = np.array(historic_p99_values) # (N,)
gp = GaussianProcessRegressor(normalize_y=True, ...)
gp.fit(X, y)
# EI-acquire 20 candidate points, keep the highest
candidates = sample_design(X, bounds, n=20) # Latin hypercube
ei = expected_improvement(gp, candidates)
best_vec = candidates[int(np.argmax(ei))]
Why GP instead of a neural surrogate, practically:
- Tiny sample size (8,760 windows a year; only a few dozen are genuinely representative “post-change steady-state” windows) — GP’s sparse prior beats an MLP here;
- Built-in uncertainty: EI consumes GP’s predictive variance, so it explores “where we haven’t looked” instead of circling known territory;
- Interpretability: per-dimension derivatives go straight into the report (“each +64MB of buffer pool → P99 −0.8ms, 95% CI [−1.4, −0.2]”).
Phase B: Ollama Emits the Human Report + Machine Instructions
The GP gives a bare param vector. A local Ollama (Qwen2.5-7B, 4-bit quant, ~4.5GB peak resident on 1C1G; a single tuning run finishes in ~30s) then does two jobs:
① A natural-language inspection report (pushed to Telegram for on-call):
PROMPT_REPORT = f"""You are a VPS performance inspector. Here is a 7-day load summary,
current params, and the GP suggestion:
Load summary: {load_summary}
Current params: {current_params}
GP suggestion: {best_vec}, expected P99 gain {expected_gain} (95% CI)
Last 3 rollback records: {rollback_history}
Write an inspection report in ≤120 words covering:
1) whether current params match the load shape (weekday/weekend/peak);
2) which params to apply and the expected gain;
3) risk callouts (which changes have cascading pool/memory impact).
"""
report = ollama_chat("qwen2.5:7b-q4", PROMPT_REPORT)
② A structured execution plan (machine-consumed, not human-facing):
PROMPT_PLAN = f"""From the GP suggestion, generate a parameter change plan.
Current config: {current_config_json}
Suggested vector: {best_vec}
Constraints:
- mysql.innodb_buffer_pool_size_mb must stay below 70% of total RAM
(current total {total_mem_mb}MB)
- any nginx change must pass nginx -t
- Output strict JSON with this schema:
{{"actions": [{{"type": "sysctl|nginx|mysql|redis", "key": "...", "value": "...",
"validate_cmd": "...", "rollback_value": "..."}}],
"confidence": 0.0-1.0}}"""
plan = json.loads(ollama_chat("qwen2.5:7b-q4", PROMPT_PLAN, format="json"))
Critical design decision: the LLM only translates; it never computes. Numeric optimization is the GP’s job (reproducible, with CIs). The LLM converts the numeric conclusion into ops actions + human-readable risk notes. If the LLM hallucinates an out-of-bounds value, L3’s hard validation catches it — a bad LLM output at worst skips this round, it can never directly break the system.
When confidence < 0.6, the entire suggestion is discarded: report only, no param changes. That’s the “better not to touch it” floor.
L3 Application: Shadow Load Test + Atomic Switch + 30s Rollback Sentinel
vacuum_apply.py (fires at 02:40) walks a strict pipeline:
┌──────────────────────────────────────────────────────────┐
│ 1. Shadow mode │
│ - write params to /etc/vps-vacuum/staging/ (staging │
│ copies of nginx conf / my.cnf / redis.conf) │
│ - run a local lightweight load (wrk 100 VU × 5 min) │
│ against the staging endpoint (127.0.0.1:8443, │
│ isolated from prod 443) │
│ - shadow P99 better-or-flat vs. prod baseline → go; │
│ >15% worse → skip, log to audit │
├──────────────────────────────────────────────────────────┤
│ 2. Atomic switch │
│ - generate diff, back up current config to │
│ /var/backups/vacuum/ (timestamped, 14-day rolling) │
│ - sysctl: write /proc/sys directly (instant) │
│ - nginx: nginx -t && nginx -s reload │
│ - mysql/redis: restart-only → mark "pending-restart" │
│ for a human maintenance-window decision │
│ - each success → append a hash-chain audit row │
├──────────────────────────────────────────────────────────┤
│ 3. 30s rollback sentinel │
│ - keep watching P99, 5xx rate, mysql conn count for │
│ 30s post-apply; │
│ - any metric >20% worse vs. the pre-apply 10-min │
│ baseline → auto `cp backup → restore → reload` │
│ within 30s, log "AUTO-ROLLBACK", alert Telegram │
└──────────────────────────────────────────────────────────┘
The audit log is a SQLite SHA-256 hash chain (tamper-evident: any broken row invalidates the chain):
def audit_append(conn, action, payload, prev_hash):
h = sha256(f"{prev_hash}|{action}|{json.dumps(payload, sort_keys=True)}")
conn.execute(
"INSERT INTO audit(ts, action, payload, prev_hash, hash) "
"VALUES(?, ?, ?, ?, ?)",
(time.time(), action, json.dumps(payload), prev_hash, h),
)
conn.commit()
The sentinel itself is a 30-second oneshot that exits when done:
def rollback_sentinel(before_snapshot, threshold=0.20):
time.sleep(30)
after = read_metrics_window(30)
if after.p99 > before_snapshot.p99 * (1 + threshold) \
or after.err_5xx > before_snapshot.err_5xx * (1 + threshold):
restore_last_backup() # cp backup + nginx -s reload / sysctl restore
audit_append(conn, "AUTO-ROLLBACK", {...})
push_telegram("⚠️ VPS Vacuum auto-rollback: P99 degradation triggered")
Measured: One Week on a 1C1G VPS
Test rig: Hetzner CAX11 (1C1G), nginx + MySQL (InnoDB) + Redis, ~20k requests/day, workload 3× higher on weekdays than weekends. Ollama running Qwen2.5-7B Q4; each optimization run takes 25–35s.
| Metric | Day 0 (manual params) | Day 7 (converged) | Change |
|---|---|---|---|
| P99 latency | 412ms | 333ms | −19% |
| Throughput (req/s) | 840 | 999 | +19% |
| False-positive rollbacks | - | 0 | zero |
| Param changes | 2 manual (3 months ago) | 21 auto (7 days) | 10× faster response |
| Per-optimization cost | ~2h human on-call | 35s machine + 0MB idle | ~free |
| Audit trail | none | 100% (hash-chain) | fully traceable |
Three representative auto-tuning cases:
Case 1: Weekday vs. weekend load shape. GP noticed weekend 12:00–20:00 load is ~1/3 of weekday. Holding nginx.worker_connections at 4096 all weekend was pure waste (~4MB kernel socket buffer per worker). The pipeline split applications by OnCalendar time windows: 1024 on weekends, 4096 on weekdays — peak memory −64MB.
Case 2: Buffer pool cascade. GP suggested innodb_buffer_pool_size 384→512MB; the shadow test showed P99 actually +8ms (on a 1GB box, a full buffer pool squeezes the OS page cache). The suggestion was auto-discarded, the audit log recorded “shadow mode intercepted 1 degraded suggestion.”
Case 3: The 30s sentinel actually fired. On day 4, applying net.core.somaxconn 4096→16384 pushed 5xx from 0.1% to 2.3% within 30s (TCP backlog overflow → accept() jitter on a single core). The sentinel auto-rolled back in 28s, no user-visible impact. The next day’s LLM inspection report cited that rollback as a risk note: “on a single-core box, don’t push somaxconn above 8192” — rollback experience became a constraint input for the next round.
File Layout & Deployment
/opt/vps-vacuum/
├── bin/
│ ├── vps-vacuum-sample.sh # L1: sampling oneshot
│ ├── vacuum_optimize.py # L2: GP + Ollama
│ └── vacuum_apply.py # L3: shadow + atomic + sentinel
├── lib/
│ ├── gp_model.py # Bayesian surrogate wrapper
│ ├── ollama_client.py # Ollama HTTP (127.0.0.1:11434)
│ └── audit.py # hash-chain audit
├── staging/ # L3 shadow config copies
└── config.json # param space, constraints, thresholds
/var/lib/vps-vacuum/metrics.db # SQLite: samples + params_snapshot + audit
/var/backups/vacuum/ # config backups (14-day rolling)
systemd units (4 timers + 4 services, all 0MB resident)
# /etc/systemd/system/vacuum-sample.timer
[Unit]
Description=VPS Vacuum - Hourly sampling
[Timer]
OnCalendar=*:00
Persistent=true
[Install]
WantedBy=timers.target
# /etc/systemd/system/vacuum-optimize.timer
[Unit]
Description=VPS Vacuum - Nightly optimization
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
[Install]
WantedBy=timers.target
# /etc/systemd/system/vacuum-apply.timer
[Unit]
Description=VPS Vacuum - Apply + rollback sentinel
[Timer]
OnCalendar=*-*-* 02:40:00
[Install]
WantedBy=timers.target
Every timer pairs with a Type=oneshot service. There are no resident processes in the pipeline — the only resident thing is Ollama itself (~4.5GB peak). If a 1GB VPS can’t hold 7B, drop to the 1.5B/3B quant — or point the optimizer at a remote Ollama endpoint (the client supports both local and remote; switch with the VACUUM_OLLAMA_ENDPOINT env var).
config.json (core)
{
"param_space": {
"mysql.innodb_buffer_pool_size_mb": [128, 512],
"mysql.max_connections": [100, 500],
"nginx.worker_connections": [512, 4096],
"nginx.keepalive_timeout_s": [10, 120],
"redis.maxmemory_mb": [32, 256]
},
"constraints": {
"buffer_pool_max_pct_of_ram": 0.70,
"min_sample_days_before_apply": 3,
"confidence_threshold": 0.6
},
"rollback": {
"sentinel_window_s": 30,
"p99_worsen_threshold": 0.20,
"err_5xx_worsen_threshold": 0.20
},
"schedule": {
"apply_time_local": "02:40",
"apply_time_windows": [
{"name": "weekday-daytime", "cron": "Mon-Fri 09:00-18:00"},
{"name": "weekend", "cron": "Sat-Sun 00:00-24:00"}
]
},
"notify": {
"telegram_bot_token": "...",
"telegram_chat_id": "..."
}
}
One-shot deployment
# 1. deps
pip install scikit-learn numpy ollama
curl -fsSL https://ollama.com/install.sh | sh # local Ollama (optional)
systemctl enable --now ollama
ollama pull qwen2.5:7b-q4 # or qwen2.5:1.5b-q4 for 1GB boxes
# 2. scripts + config
sudo cp -r /opt/vps-vacuum /etc/vps-vacuum
sudo cp vacuum-*.timer vacuum-*.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now vacuum-sample.timer \
vacuum-optimize.timer \
vacuum-apply.timer
# 3. init DB + run a manual sample now (don't wait for the timer)
/opt/vps-vacuum/bin/vps-vacuum-sample.sh
# 4. verify
systemctl list-timers | grep vacuum
sqlite3 /var/lib/vps-vacuum/metrics.db "SELECT COUNT(*) FROM samples;"
Recommended first 3 days: set min_sample_days_before_apply to 3 and run L1+L2 only (reports, no auto-apply). Validate the LLM reports against your judgment, then flip L3 on. That’s the safest landing pattern for any AI-ops tooling: read what it says before you let it act.
Boundaries & Gotchas
Only tune reversible, low-risk parameters. Keep the space free of data-directory formats, character sets, timezones — anything “set and you can’t go back.” Restart-gated params (buffer pool) queue as pending-restart for a human maintenance window, never auto-reboot.
Shadow load must mirror production. If prod has writes, the wrk script must mix reads and writes — otherwise the P99 measurement is wrong.
1C1G can’t hold 7B quant? Use qwen2.5:1.5b — the optimization task is only translation + reporting; the numbers are done by the local GP, which is model-size-independent.
GP cold start. For the first ~2 weeks, sample data is too thin, EI explores the whole space; keep confidence_threshold at 0.8 in that window — “better not to act” is the correct default.
Rollback ≠ zero impact. The 30s sentinel watches P99/5xx; silent degradations (e.g., buffer pool change stretching GC pauses under the 20% threshold) are covered by a weekly full-benchmark baseline timer (wrk full run, weekly report).
Takeaways
VPS parameter tuning is fundamentally a continuously drifting black-box optimization problem, and human on-call is the highest-cost, lowest-frequency solution available. Turn “sample → optimize → apply → rollback → audit” into a 0MB-resident systemd pipeline, with a local LLM as the human-machine translation layer, and parameters finally chase the load curve instead of freezing as a three-month-old snapshot.
One week on 1C1G: +19% throughput, −19% P99, zero false-positive rollbacks, 100% hash-chain audit. The core isn’t how big the LLM is — it’s that every stage of the pipeline is designed to be switchable, rollback-able, and auditable. That’s what makes AI ops safe to run in production.
