Featured image of post AI + VPS: Auto-Generate Ops Docs & System Knowledge Base with LLM

AI + VPS: Auto-Generate Ops Docs & System Knowledge Base with LLM

Stop spending hours manually documenting your VPS infrastructure. This article shows you how to use a local LLM (Ollama + Qwen) to automatically discover system configurations, generate readable documentation, and build a living knowledge base — all from a single command.

Introduction: Inspection Shouldn’t Be Manual Labor

You manage several VPS instances. Every time you onboard a new service or run a periodic security audit, you need to perform configuration inspection:

Is SSH running on a non-standard port?
Is root login disabled?
Are firewall rules too permissive?
Which services are exposed to the public internet?
Is SSH key authentication enforced?
Do kernel parameters match security baselines?

The traditional approach is to SSH into each server and manually check sshd_config, review iptables rules, scan sysctl settings… a dozen checkpoints per server, across 10 machines, and half a day disappears. Humans miss things, especially when the checklist exceeds 30 items.

AI-powered configuration inspection changes this. Describe your inspection requirements in natural language, and the LLM automatically collects configuration data from all servers, evaluates each setting against security baselines, generates structured inspection reports, and even suggests remediation commands — you only need to review and confirm.

This article walks you through building a complete AI Configuration Inspection System, covering data collection, LLM analysis, report generation, and remediation suggestions.


1. System Architecture

┌──────────────────────────────────────────────────────────────────────┐
│                  AI Configuration Inspector                          │
│                                                                      │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────────────┐   │
│  │  Config       │ →→ │  LLM          │ →→ │  Report & Remediation│   │
│  │  Collector    │    │  Analyzer     │    │  (Markdown/Telegram) │   │
│  │              │    │  (Ollama/     │    │                      │   │
│  └──────┬───────┘    │   Qwen2.5)    │    └──────────┬───────────┘   │
│         │             └──────┬───────┘               │                │
│  ┌──────▼───────┐    ┌──────▼───────┐    ┌──────────▼───────────┐   │
│  │  Multi-Source │    │  Safety      │    │  Historical Trends   │   │
│  │  Config Data  │    │  Rule Engine │    │  (JSON storage +     │   │
│  │  (SSH/kernel/ │    │  (CIS +      │    │   change visualization)│
│  │   network/   │    │   custom)     │    └──────────────────────┘   │
│  │   Docker)    │    └──────────────┘                              │
│  └──────────────┘                                                  │
└──────────────────────────────────────────────────────────────────────┘
         │             │                │
         ▼             ▼                ▼
   ┌───────────┐   ┌───────────┐   ┌───────────┐
   │ Server A   │   │ Server B   │   │ Server C   │
   │ (Prod)     │   │ (Staging)  │   │ (Dev)      │
   └───────────┘   └───────────┘   └───────────┘

Core Workflow

PhaseContentOutput
CollectSSH/Python batch login, collect config snapshotsRaw config JSON
AnalyzeLLM evaluates against safety baseline rulesRisk清单 + remediation suggestions
ReportGenerate Markdown inspection report, grouped by serverStructured report file
NotifyPush summary via Telegram/email, support interactive remediationNotification message

2. Environment Setup

2.1 Install Ollama and Local Model

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull models (7B is sufficient for inspection tasks — fast response)
ollama pull qwen2.5:7b
ollama pull nomic-embed-text   # for historical report similarity search

2.2 Install Dependencies

pip install ollama psutil python-dotenv paramiko

2.3 Project Structure

ai-config-inspector/
├── config_collector.py    # Multi-source config collector
├── safety_rules.py        # Safety rule library (CIS Benchmark + custom)
├── llm_analyzer.py        # LLM analysis engine
├── report_generator.py    # Report generator
├── notifier.py            # Notification (Telegram/email)
├── history.py             # Historical trend storage
├── inspector.py           # Main entry point
└── config.json            # Server list and inspection profile config

3. Configuration Data Collection Layer

Configuration collection is the foundation. We need to collect configuration data from multiple dimensions on each node:

3.1 Multi-Source Collector

# config_collector.py
import subprocess
import json
import psutil
from datetime import datetime
from pathlib import Path
import paramiko

class ConfigCollector:
    """Multi-source configuration collector — supports local and remote SSH collection"""

    # Collection item definitions
    COLLECTION_ITEMS = {
        "ssh_config": {
            "cmd": "sudo cat /etc/ssh/sshd_config 2>/dev/null | grep -v '^#' | grep -v '^$'",
            "parser": "text"
        },
        "firewall_rules": {
            "cmd": "sudo iptables -L -n --line-numbers 2>/dev/null; sudo ip6tables -L -n 2>/dev/null",
            "parser": "text"
        },
        "sysctl_params": {
            "cmd": "sudo sysctl -a 2>/dev/null | grep -E '(net\\.ipv4|net\\.ipv6|kernel\\.)'",
            "parser": "keyvalue"
        },
        "listening_ports": {
            "cmd": "sudo ss -tlnp 2>/dev/null; sudo ss -ulnp 2>/dev/null",
            "parser": "text"
        },
        "user_accounts": {
            "cmd": "cat /etc/passwd | grep -v '/nologin' | grep -v '/false'",
            "parser": "text"
        },
        "docker_config": {
            "cmd": "docker info 2>/dev/null; docker ps --format '{{.Names}}\\t{{.Image}}\\t{{.Status}}\\t{{.Ports}}'",
            "parser": "text"
        },
        "nginx_config": {
            "cmd": "sudo nginx -T 2>/dev/null | head -200",
            "parser": "text"
        },
        "installed_packages": {
            "cmd": "dpkg -l 2>/dev/null | grep '^ii' | awk '{print $2}' | sort",
            "parser": "list"
        },
        "cron_jobs": {
            "cmd": "crontab -l 2>/dev/null; ls /etc/cron.d/",
            "parser": "text"
        },
        "kernel_modules": {
            "cmd": "lsmod | awk 'NR==1{print} NF>=3{print $1}'",
            "parser": "list"
        }
    }

    def collect_local(self) -> dict:
        """Collect local server configuration"""
        snapshot = {
            "hostname": subprocess.getoutput("hostname"),
            "os": subprocess.getoutput("cat /etc/os-release | grep PRETTY_NAME"),
            "kernel": subprocess.getoutput("uname -r"),
            "uptime": psutil.boot_time(),
            "collected_at": datetime.now().isoformat(),
            "items": {}
        }

        for name, spec in self.COLLECTION_ITEMS.items():
            try:
                result = subprocess.run(
                    spec["cmd"], shell=True, capture_output=True, text=True, timeout=15
                )
                snapshot["items"][name] = {
                    "raw_output": result.stdout.strip(),
                    "exit_code": result.returncode,
                    "error": result.stderr.strip() if result.stderr else None
                }
            except subprocess.TimeoutExpired:
                snapshot["items"][name] = {"error": "Collection timeout", "raw_output": ""}
            except Exception as e:
                snapshot["items"][name] = {"error": str(e), "raw_output": ""}

        return snapshot

    def collect_remote(self, host: str, port: int = 22,
                       username: str = "root", key_path: str = None) -> dict:
        """Collect remote server configuration via SSH"""
        client = paramiko.SSHClient()
        client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

        connect_kwargs = {
            "hostname": host, "port": port, "username": username,
            "timeout": 30, "allow_agent": True, "look_for_keys": True
        }
        if key_path:
            connect_kwargs["key_filename"] = key_path

        client.connect(**connect_kwargs)
        snapshot = {"hostname": host, "collected_at": datetime.now().isoformat(), "items": {}}

        for name, spec in self.COLLECTION_ITEMS.items():
            try:
                stdin, stdout, stderr = client.exec_command(spec["cmd"], timeout=20)
                snapshot["items"][name] = {
                    "raw_output": stdout.read().decode().strip(),
                    "error": stderr.read().decode().strip() if stderr.read() else None
                }
            except Exception as e:
                snapshot["items"][name] = {"error": str(e), "raw_output": ""}

        client.close()
        return snapshot

3.2 Server Configuration

// config.json
{
  "servers": [
    {"name": "prod-web-01", "host": "10.0.1.10", "role": "Production Web", "port": 22},
    {"name": "prod-db-01",  "host": "10.0.1.20", "role": "Production DB",  "port": 22},
    {"name": "staging-01",  "host": "10.0.2.10", "role": "Staging",        "port": 22},
    {"name": "dev-01",      "host": "10.0.3.10", "role": "Development",    "port": 22}
  ],
  "ssh_key_path": "~/.ssh/id_ed25519",
  "inspection_profile": "strict"
}

4. Safety Rule Library

Raw data alone isn’t enough — we need to tell the LLM what’s safe and what isn’t. The rule library uses a layered design:

4.1 CIS Benchmark Base Rules

# safety_rules.py
"""
Safety Rule Library — Based on CIS Linux Benchmark + Custom Extensions
Each rule contains: ID, check item, expected value, severity, description, fix command
"""
import json

SAFETY_RULES = [
    {
        "id": "SSH-001",
        "category": "ssh",
        "check": "PermitRootLogin",
        "expected": "no",
        "severity": "HIGH",
        "description": "Disable direct root SSH login; use normal user + sudo instead",
        "fix_command": "sed -i 's/^#*PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config && systemctl reload sshd",
        "cis_ref": "CIS 5.2.6"
    },
    {
        "id": "SSH-002",
        "category": "ssh",
        "check": "PasswordAuthentication",
        "expected": "no",
        "severity": "HIGH",
        "description": "Disable password login, allow only key-based authentication",
        "fix_command": "sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config && systemctl reload sshd",
        "cis_ref": "CIS 5.2.8"
    },
    {
        "id": "SSH-003",
        "category": "ssh",
        "check": "Port",
        "expected_pattern": "^(?!22$)",
        "severity": "MEDIUM",
        "description": "SSH port should use a non-standard port to reduce brute-force risk",
        "fix_command": "# Change Port in /etc/ssh/sshd_config to a non-default port",
        "cis_ref": "CIS 5.2.1"
    },
    {
        "id": "NET-001",
        "category": "network",
        "check": "net.ipv4.ip_forward",
        "expected": "0",
        "severity": "MEDIUM",
        "description": "Non-router VPS should disable IP forwarding to prevent being used as a pivot",
        "fix_command": "sysctl -w net.ipv4.ip_forward=0 && echo 'net.ipv4.ip_forward=0' >> /etc/sysctl.conf",
        "cis_ref": "CIS 3.3.1"
    },
    {
        "id": "NET-002",
        "category": "network",
        "check": "net.ipv4.conf.all.accept_redirects",
        "expected": "0",
        "severity": "LOW",
        "description": "Disable ICMP redirect acceptance to prevent route hijacking",
        "fix_command": "sysctl -w net.ipv4.conf.all.accept_redirects=0",
        "cis_ref": "CIS 3.3.2"
    },
    {
        "id": "NET-003",
        "category": "network",
        "check": "net.ipv4.conf.all.send_redirects",
        "expected": "0",
        "severity": "LOW",
        "description": "Disable sending ICMP redirects",
        "fix_command": "sysctl -w net.ipv4.conf.all.send_redirects=0",
        "cis_ref": "CIS 3.3.3"
    },
    {
        "id": "NET-004",
        "category": "network",
        "check": "net.ipv4.tcp_syncookies",
        "expected": "1",
        "severity": "MEDIUM",
        "description": "Enable SYN cookies to defend against SYN Flood attacks",
        "fix_command": "sysctl -w net.ipv4.tcp_syncookies=1",
        "cis_ref": "CIS 3.3.5"
    },
    {
        "id": "FILE-001",
        "category": "filesystem",
        "check": "world_writable_tmp",
        "expected": "no",
        "severity": "LOW",
        "description": "/tmp should have sticky bit (1777) to prevent users from deleting each other's files",
        "fix_command": "chmod 1777 /tmp && chmod 1777 /var/tmp",
        "cis_ref": "CIS 1.5.1"
    },
    {
        "id": "CONTAINER-001",
        "category": "docker",
        "check": "privileged_containers",
        "expected": "0",
        "severity": "HIGH",
        "description": "No privileged-mode containers should run (equivalent to root on the host)",
        "fix_command": "# Stop and redeploy non-privileged containers",
        "cis_ref": "CIS 5.14"
    },
    {
        "id": "USER-001",
        "category": "users",
        "check": "empty_password_accounts",
        "expected": "0",
        "severity": "CRITICAL",
        "description": "No accounts with empty passwords should exist on the system",
        "fix_command": "# Lock empty-password accounts: passwd -l <user>",
        "cis_ref": "CIS 5.1.3"
    },
    {
        "id": "FIREWALL-001",
        "category": "firewall",
        "check": "default_deny_inbound",
        "expected": "yes",
        "severity": "HIGH",
        "description": "Default inbound policy should be DROP/DENY, only necessary ports open",
        "fix_command": "# Configure iptables default policy: iptables -P INPUT DROP",
        "cis_ref": "CIS 3.5.1"
    },
    {
        "id": "LOG-001",
        "category": "logging",
        "check": "auditd_enabled",
        "expected": "yes",
        "severity": "MEDIUM",
        "description": "auditd audit daemon should be enabled to record system calls",
        "fix_command": "systemctl enable --now auditd",
        "cis_ref": "CIS 4.1"
    }
]

# Convert rules to LLM-readable prompt format
def build_rule_prompt(rules: list = None) -> str:
    """Convert safety rule library to LLM-understandable prompt"""
    if rules is None:
        rules = SAFETY_RULES

    lines = ["## Safety Baseline Rules", ""]
    for rule in rules:
        lines.append(f"### {rule['id']} [{rule['severity']}]")
        lines.append(f"- Category: {rule['category']}")
        lines.append(f"- Check: {rule['check']}")
        lines.append(f"- Expected: `{rule['expected']}`")
        lines.append(f"- Description: {rule['description']}")
        lines.append(f"- Fix Command: `{rule['fix_command']}`")
        if rule.get("cis_ref"):
            lines.append(f"- CIS Reference: {rule['cis_ref']}")
        lines.append("")

    return "\n".join(lines)

5. LLM Analysis Engine

This is the core of the system — letting the LLM understand configuration data and provide risk assessment:

5.1 Analyzer Implementation

# llm_analyzer.py
import json
import ollama
from datetime import datetime
from safety_rules import SAFETY_RULES, build_rule_prompt

class ConfigAnalyzer:
    """LLM Configuration Analysis Engine"""

    def __init__(self, model: str = "qwen2.5:7b", base_url: str = "http://localhost:11434"):
        self.model = model
        self.base_url = base_url
        self.rule_prompt = build_rule_prompt()

    def analyze_snapshot(self, snapshot: dict) -> dict:
        """Analyze a single server's configuration snapshot"""
        hostname = snapshot.get("hostname", "unknown")
        collected_at = snapshot.get("collected_at", "")

        # Build configuration context
        config_context = self._build_config_context(snapshot)

        # LLM analysis prompt
        analysis_prompt = f"""You are a senior Linux system security engineer performing a configuration security inspection on a VPS.

## Server Information
- Hostname: {hostname}
- Collected at: {collected_at}

## Safety Baseline Rules
{self.rule_prompt}

## Current Configuration Data
{config_context}

## Your Task
Please check the above configuration data against each safety rule and provide for each:
1. **Status**: PASS (compliant) / FAIL (violation) / SKIP (cannot determine)
2. **Severity**: CRITICAL / HIGH / MEDIUM / LOW / INFO
3. **Description**: Brief explanation of what's wrong
4. **Fix Suggestion**: Specific remediation command or steps

Return results as a JSON array in the following format:
[
  {{
    "rule_id": "SSH-001",
    "status": "FAIL",
    "severity": "HIGH",
    "description": "PermitRootLogin is set to yes, which is a risk",
    "fix_command": "sed -i 's/^#*PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config && systemctl reload sshd",
    "cis_ref": "CIS 5.2.6"
  }}
]

Return only the JSON array, nothing else."""

        try:
            response = ollama.chat(
                model=self.model,
                messages=[{"role": "user", "content": analysis_prompt}],
                options={
                    "temperature": 0.1,
                    "num_predict": 2048
                }
            )
            result_text = response["message"]["content"]

            # Extract JSON
            start = result_text.find("[")
            end = result_text.rfind("]") + 1
            if start >= 0 and end > start:
                results = json.loads(result_text[start:end])
            else:
                results = [{"rule_id": "ERROR", "status": "SKIP", "description": f"Parse failed: {result_text[:200]}"}]

            return {
                "hostname": hostname,
                "collected_at": collected_at,
                "analyzed_at": datetime.now().isoformat(),
                "results": results,
                "summary": self._compute_summary(results)
            }

        except Exception as e:
            return {
                "hostname": hostname,
                "error": str(e),
                "collected_at": collected_at
            }

    def _build_config_context(self, snapshot: dict) -> str:
        """Convert raw configuration data to LLM-readable context"""
        lines = []
        items = snapshot.get("items", {})

        # SSH config
        ssh_raw = items.get("ssh_config", {}).get("raw_output", "")
        if ssh_raw:
            lines.append("### SSH Configuration (/etc/ssh/sshd_config)")
            lines.append(ssh_raw)
            lines.append("")

        # Kernel parameters
        sysctl_raw = items.get("sysctl_params", {}).get("raw_output", "")
        if sysctl_raw:
            lines.append("### Kernel Network Parameters (sysctl)")
            lines.append(sysctl_raw)
            lines.append("")

        # Listening ports
        ports_raw = items.get("listening_ports", {}).get("raw_output", "")
        if ports_raw:
            lines.append("### Listening Ports")
            lines.append(ports_raw)
            lines.append("")

        # User accounts
        users_raw = items.get("user_accounts", {}).get("raw_output", "")
        if users_raw:
            lines.append("### Active User Accounts")
            lines.append(users_raw)
            lines.append("")

        # Docker config
        docker_raw = items.get("docker_config", {}).get("raw_output", "")
        if docker_raw:
            lines.append("### Docker Status")
            lines.append(docker_raw)
            lines.append("")

        # Firewall
        fw_raw = items.get("firewall_rules", {}).get("raw_output", "")
        if fw_raw:
            lines.append("### Firewall Rules (iptables)")
            lines.append(fw_raw)
            lines.append("")

        return "\n".join(lines)

    def _compute_summary(self, results: list) -> dict:
        """Compute summary statistics"""
        severity_counts = {"CRITICAL": 0, "HIGH": 0, "MEDIUM": 0, "LOW": 0, "INFO": 0}
        status_counts = {"PASS": 0, "FAIL": 0, "SKIP": 0}
        failed_rules = []

        for r in results:
            sev = r.get("severity", "INFO")
            status = r.get("status", "SKIP")
            severity_counts[sev] = severity_counts.get(sev, 0) + 1
            status_counts[status] = status_counts.get(status, 0) + 1
            if status == "FAIL":
                failed_rules.append(r["rule_id"])

        # Calculate security score (100-point scale)
        critical_penalty = severity_counts.get("CRITICAL", 0) * 15
        high_penalty = severity_counts.get("HIGH", 0) * 10
        medium_penalty = severity_counts.get("MEDIUM", 0) * 5
        low_penalty = severity_counts.get("LOW", 0) * 2

        score = max(0, 100 - critical_penalty - high_penalty - medium_penalty - low_penalty)

        return {
            "total_checks": len(results),
            "passed": status_counts.get("PASS", 0),
            "failed": status_counts.get("FAIL", 0),
            "skipped": status_counts.get("SKIP", 0),
            "severity_counts": severity_counts,
            "failed_rule_ids": failed_rules,
            "security_score": score,
            "grade": self._score_to_grade(score)
        }

    def _score_to_grade(self, score: int) -> str:
        if score >= 90:
            return "A"
        elif score >= 80:
            return "B"
        elif score >= 70:
            return "C"
        elif score >= 60:
            return "D"
        else:
            return "F"

6. Report Generator

After analysis, generate a human-readable inspection report:

6.1 Markdown Report

# report_generator.py
from datetime import datetime
from typing import List, Dict

class ReportGenerator:
    """Generate Markdown-formatted inspection reports"""

    SEVERITY_EMOJI = {
        "CRITICAL": "🔴",
        "HIGH": "🟠",
        "MEDIUM": "🟡",
        "LOW": "🔵",
        "INFO": "⚪"
    }

    def generate_report(self, analyses: List[Dict]) -> str:
        """Generate full inspection report"""
        lines = [
            f"# 🔍 VPS Configuration Inspection Report",
            f"",
            f"**Generated**: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
            f"**Profile**: Security Baseline Check (CIS Benchmark + Custom Rules)",
            f"",
            f"---",
            f""
        ]

        # Global summary
        total_score = sum(a["summary"]["security_score"] for a in analyses if "summary" in a)
        avg_score = total_score / len(analyses) if analyses else 0
        total_fail = sum(a["summary"]["failed"] for a in analyses if "summary" in a)

        lines.append("## 📊 Global Summary")
        lines.append("")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        lines.append(f"| Servers Inspected | {len(analyses)} |")
        lines.append(f"| Average Security Score | {avg_score:.0f}/100 |")
        lines.append(f"| Total Violations | {total_fail} |")
        lines.append(f"| Global Grade | {self._score_to_grade(avg_score)} |")
        lines.append("")

        # Detailed report per server
        for analysis in analyses:
            hostname = analysis.get("hostname", "unknown")
            summary = analysis.get("summary", {})
            results = analysis.get("results", [])
            error = analysis.get("error")

            lines.append(f"---")
            lines.append(f"")
            lines.append(f"## 🖥️ {hostname}")
            lines.append(f"**Collected at**: {analysis.get('collected_at', 'N/A')}")
            lines.append(f"")

            if error:
                lines.append(f"⚠️ **Collection or analysis error**: {error}")
                lines.append("")
                continue

            # Server score card
            score = summary.get("security_score", 0)
            grade = summary.get("grade", "F")
            lines.append(f"> **Security Score**: {score}/100  **Grade**: {grade}")
            lines.append(f">")
            lines.append(f"> Passed: {summary.get('passed', 0)} | Failed: {summary.get('failed', 0)} | "
                        f"Skipped: {summary.get('skipped', 0)} | Total: {summary.get('total_checks', 0)}")
            lines.append(f"")

            # Severity distribution
            sev_counts = summary.get("severity_counts", {})
            if any(v > 0 for v in sev_counts.values()):
                lines.append("**Severity Distribution**: ")
                parts = []
                if sev_counts.get("CRITICAL", 0) > 0:
                    parts.append(f"🔴 Critical {sev_counts['CRITICAL']}")
                if sev_counts.get("HIGH", 0) > 0:
                    parts.append(f"🟠 High {sev_counts['HIGH']}")
                if sev_counts.get("MEDIUM", 0) > 0:
                    parts.append(f"🟡 Medium {sev_counts['MEDIUM']}")
                if sev_counts.get("LOW", 0) > 0:
                    parts.append(f"🔵 Low {sev_counts['LOW']}")
                lines.append(" | ".join(parts))
                lines.append("")

            # Failed items
            failed = [r for r in results if r.get("status") == "FAIL"]
            if failed:
                lines.append(f"### ❌ Violations ({len(failed)})")
                lines.append("")
                for r in failed:
                    emoji = self.SEVERITY_EMOJI.get(r.get("severity", "INFO"), "⚪")
                    lines.append(f"#### {emoji} {r['rule_id']} — {r.get('severity', 'INFO')}")
                    lines.append(f"- **Issue**: {r.get('description', 'N/A')}")
                    if r.get("cis_ref"):
                        lines.append(f"- **CIS Reference**: {r['cis_ref']}")
                    fix = r.get("fix_command", "")
                    if fix:
                        lines.append(f"- **Fix Command**:")
                        lines.append(f"  ```bash")
                        lines.append(f"  {fix}")
                        lines.append(f"  ```")
                    lines.append("")

            # Passed items (collapsible)
            passed = [r for r in results if r.get("status") == "PASS"]
            if passed:
                lines.append(f"<details>")
                lines.append(f"<summary>✅ Compliant Items ({len(passed)}) — Click to expand</summary>")
                lines.append(f"")
                for r in passed[:10]:
                    lines.append(f"- ✅ {r['rule_id']}: {r.get('description', 'Compliant')}")
                if len(passed) > 10:
                    lines.append(f"- ... and {len(passed) - 10} more compliant items")
                lines.append(f"</details>")
                lines.append("")

        # Remediation priority list
        lines.append("---")
        lines.append("")
        lines.append("## 🎯 Remediation Priority Recommendations")
        lines.append("")
        lines.append("Handle violations in the following order (highest to lowest severity):")
        lines.append("")

        all_failures = []
        for analysis in analyses:
            for r in analysis.get("results", []):
                if r.get("status") == "FAIL":
                    all_failures.append({
                        **r,
                        "hostname": analysis.get("hostname")
                    })

        severity_order = {"CRITICAL": 0, "HIGH": 1, "MEDIUM": 2, "LOW": 3, "INFO": 4}
        all_failures.sort(key=lambda x: severity_order.get(x.get("severity", "INFO"), 4))

        for i, f in enumerate(all_failures, 1):
            emoji = self.SEVERITY_EMOJI.get(f.get("severity", "INFO"), "⚪")
            lines.append(f"{i}. **[{f['severity']}]** {emoji} **{f['hostname']}** — {f['rule_id']}: {f.get('description', '')}")
            if f.get("fix_command"):
                lines.append(f"   ```bash")
                lines.append(f"   # {f['hostname']}")
                lines.append(f"   {f['fix_command']}")
                lines.append(f"   ```")

        lines.append("")
        lines.append("---")
        lines.append("")
        lines.append(f"*Report generated by AI Configuration Inspector | Model: Ollama Qwen2.5:7b*")

        return "\n".join(lines)

    def _score_to_grade(self, score: float) -> str:
        if score >= 90: return "A"
        elif score >= 80: return "B"
        elif score >= 70: return "C"
        elif score >= 60: return "D"
        else: return "F"

7. Notification Delivery

Push report summaries automatically after inspection completes:

# notifier.py
import os
import json
import requests
from pathlib import Path
from datetime import datetime

class Notifier:
    """Multi-channel notification delivery"""

    def __init__(self, telegram_token: str = None, telegram_chat_id: str = None):
        self.tg_token = telegram_token or os.environ.get("TELEGRAM_BOT_TOKEN")
        self.tg_chat_id = telegram_chat_id or os.environ.get("TELEGRAM_CHAT_ID")

    def send_telegram(self, report_md: str, summary: dict):
        """Push inspection summary via Telegram Bot"""
        if not self.tg_token or not self.tg_chat_id:
            print("⚠️ Telegram not configured, skipping notification")
            return

        servers = summary.get("servers", [])
        avg_score = summary.get("average_score", 0)
        total_fail = summary.get("total_failures", 0)

        msg = f"🔍 **VPS Config Inspection Complete**\n\n"
        msg += f"📅 {summary.get('generated_at', '')}\n"
        msg += f"🖥️  Servers inspected: {len(servers)}\n"
        msg += f"⭐ Average security score: **{avg_score:.0f}/100**\n"
        msg += f"❌ Total violations: {total_fail}\n\n"

        for s in servers:
            emoji = "✅" if s["score"] >= 80 else "⚠️" if s["score"] >= 60 else "🔴"
            msg += f"{emoji} **{s['hostname']}** — Score {s['score']} ({s['grade']}), "
            msg += f"{s['failed']} violations\n"

        msg += "\n📄 Full report saved locally."

        if len(msg) > 4000:
            msg = msg[:3997] + "..."

        url = f"https://api.telegram.org/bot{self.tg_token}/sendMessage"
        requests.post(url, json={
            "chat_id": self.tg_chat_id,
            "text": msg,
            "parse_mode": "Markdown"
        }, timeout=10)
        print("✅ Telegram notification sent")

    def save_report(self, report_md: str, filename: str = None):
        """Save report to local file"""
        output_dir = Path("reports")
        output_dir.mkdir(exist_ok=True)

        if not filename:
            filename = f"inspection-report-{datetime.now().strftime('%Y%m%d-%H%M%S')}.md"

        filepath = output_dir / filename
        filepath.write_text(report_md, encoding="utf-8")
        print(f"📄 Report saved: {filepath}")
        return filepath

8. Main Entry & Scheduled Tasks

# inspector.py
#!/usr/bin/env python3
"""
AI VPS Configuration Inspection System
Usage: python inspector.py [--remote] [--profile strict|basic]
"""
import json
import argparse
from pathlib import Path
from datetime import datetime

from config_collector import ConfigCollector
from llm_analyzer import ConfigAnalyzer
from report_generator import ReportGenerator
from notifier import Notifier

def main():
    parser = argparse.ArgumentParser(description="AI VPS Configuration Inspection System")
    parser.add_argument("--remote", action="store_true", help="Collect from remote servers (requires SSH key)")
    parser.add_argument("--profile", choices=["strict", "basic"], default="strict",
                        help="Inspection strictness (default: strict)")
    parser.add_argument("--config", default="config.json", help="Config file path")
    parser.add_argument("--save-report", action="store_true", help="Save report file")
    parser.add_argument("--notify", action="store_true", help="Send Telegram notification")
    args = parser.parse_args()

    config_path = Path(args.config)
    if not config_path.exists():
        print(f"❌ Config file not found: {args.config}")
        return 1

    with open(config_path) as f:
        config = json.load(f)

    collector = ConfigCollector()
    analyzer = ConfigAnalyzer()
    reporter = ReportGenerator()
    notifier = Notifier()

    print(f"🔍 Starting AI Configuration Inspection ({datetime.now().strftime('%Y-%m-%d %H:%M:%S')})")
    print(f"📋 Profile: {args.profile}")
    print()

    analyses = []
    servers = config.get("servers", [])

    for server in servers:
        name = server.get("name", "unknown")
        print(f"  🖥️  Inspecting: {name} ({server.get('host', '')}) ... ", end="", flush=True)

        if args.remote:
            snapshot = collector.collect_remote(
                host=server["host"],
                port=server.get("port", 22),
                key_path=config.get("ssh_key_path")
            )
        else:
            snapshot = collector.collect_local()
            snapshot["hostname"] = name

        result = analyzer.analyze_snapshot(snapshot)
        analyses.append(result)
        score = result.get("summary", {}).get("security_score", 0)
        grade = result.get("summary", {}).get("grade", "?")
        print(f"✅ Score {score}/100 ({grade})")

    # Generate report
    report_md = reporter.generate_report(analyses)

    if args.save_report:
        notifier.save_report(report_md)

    # Build summary for notification
    summary = {
        "generated_at": datetime.now().isoformat(),
        "servers": [
            {
                "hostname": a.get("hostname", "unknown"),
                "score": a.get("summary", {}).get("security_score", 0),
                "grade": a.get("summary", {}).get("grade", "?"),
                "failed": a.get("summary", {}).get("failed", 0)
            }
            for a in analyses
        ],
        "average_score": sum(a.get("summary", {}).get("security_score", 0) for a in analyses) / len(analyses) if analyses else 0,
        "total_failures": sum(a.get("summary", {}).get("failed", 0) for a in analyses)
    }

    if args.notify:
        notifier.send_telegram(report_md, summary)

    # Print summary to terminal
    print()
    print("=" * 60)
    print("📊 Inspection Summary")
    print("=" * 60)
    for a in analyses:
        s = a.get("summary", {})
        print(f"  {a.get('hostname', '?'):20s} | Score: {s.get('security_score', 0):3d}/100 | "
              f"Grade: {s.get('grade', '?')} | Violations: {s.get('failed', 0)}")
    print("=" * 60)

    return 0

if __name__ == "__main__":
    exit(main())

Set Up Scheduled Inspection

# Run inspection daily at 2 AM, push results to Telegram
crontab -e

# Add the following line:
0 2 * * * cd ~/ai-config-inspector && source venv/bin/activate && \
  python inspector.py --remote --save-report --notify >> /var/log/config-inspector.log 2>&1

# Run strict-mode deep inspection every Monday at 9 AM
0 9 * * 1 cd ~/ai-config-inspector && source venv/bin/activate && \
  python inspector.py --remote --profile strict --save-report --notify \
  >> /var/log/config-inspector-weekly.log 2>&1

9. Live Demo

Scenario 1: Daily Inspection

cd ~/ai-config-inspector
source venv/bin/activate
python inspector.py --remote --save-report --notify

Output Example:

🔍 Starting AI Configuration Inspection (2026-09-21 02:00:01)
📋 Profile: strict

  🖥️  Inspecting: prod-web-01 (10.0.1.10) ... ✅ Score 72/100 (C)
  🖥️  Inspecting: prod-db-01 (10.0.1.20) ... ✅ Score 85/100 (B)
  🖥️  Inspecting: staging-01 (10.0.2.10) ... ✅ Score 68/100 (C)
  🖥️  Inspecting: dev-01 (10.0.3.10) ...    ✅ Score 91/100 (A)

============================================================
📊 Inspection Summary
============================================================
  prod-web-01          | Score:  72/100 | Grade: C | Violations: 5
  prod-db-01           | Score:  85/100 | Grade: B | Violations: 2
  staging-01           | Score:  68/100 | Grade: C | Violations: 7
  dev-01               | Score:  91/100 | Grade: A | Violations: 0
============================================================
📄 Report saved: reports/inspection-report-20260921-020001.md
✅ Telegram notification sent

Scenario 2: Historical Trend Comparison

The system automatically saves each inspection’s history (JSON format), supporting trend analysis:

# history.py — Historical trend tracking
import json
from pathlib import Path
from datetime import datetime, timedelta

HISTORY_FILE = Path("history/inspection_history.json")

def load_history():
    if HISTORY_FILE.exists():
        return json.loads(HISTORY_FILE.read_text())
    return {"records": []}

def append_record(hostname: str, score: int, failed: int, grade: str):
    history = load_history()
    record = {
        "hostname": hostname,
        "timestamp": datetime.now().isoformat(),
        "score": score,
        "failed": failed,
        "grade": grade
    }
    history["records"].append(record)
    # Keep only the last 90 days
    cutoff = datetime.now() - timedelta(days=90)
    history["records"] = [
        r for r in history["records"]
        if datetime.fromisoformat(r["timestamp"]) > cutoff
    ]
    HISTORY_FILE.parent.mkdir(parents=True, exist_ok=True)
    HISTORY_FILE.write_text(json.dumps(history, ensure_ascii=False, indent=2))
    return record

def get_trend(hostname: str, days: int = 30) -> list:
    """Get score trend for a specified server"""
    history = load_history()
    cutoff = datetime.now() - timedelta(days=days)
    records = [
        r for r in history["records"]
        if r["hostname"] == hostname
        and datetime.fromisoformat(r["timestamp"]) > cutoff
    ]
    return sorted(records, key=lambda x: x["timestamp"])

Trend data can be used to generate score trend charts (with Grafana or direct ASCII output):

prod-web-01 Security Score Trend (Last 30 Days)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Sep  1  ████████████████████░░░░  78
Sep  3  ████████████████████░░░░  78
Sep  5  █████████████████████░░░  82  ← Fixed SSH root login
Sep  8  ██████████████████████░░  85  ← Enabled fail2ban
Sep 10  ██████████████████████░░  85
Sep 12  ███████████████████████░  88  ← Disabled IP forwarding
Sep 15  ███████████████████████░  88
Sep 18  ███████████████████████░  88
Sep 21  ████████████████████████  91  ← New safety rule passed

10. Extension: Custom Rules & Dynamic Baselines

10.1 Adding Custom Rules

Append your business rules to safety_rules.py:

CUSTOM_RULES = [
    {
        "id": "CUSTOM-001",
        "category": "application",
        "check": "app_health_check",
        "expected": "healthy",
        "severity": "HIGH",
        "description": "Core service health endpoint should return 200",
        "fix_command": "# Check /healthz endpoint response",
        "cis_ref": None
    }
]

10.2 Dynamic Baselines (Adaptive Rules)

For servers in different environments, set different baseline standards:

ENVIRONMENTS = {
    "production": {"profile": "strict", "min_score": 85},
    "staging":    {"profile": "standard", "min_score": 70},
    "development":{"profile": "basic",    "min_score": 50}
}

The system automatically selects the appropriate baseline standard based on server role during inspection, and highlights servers below the environment baseline in the report.


Summary

This article covered how to build an AI Configuration Inspection System with the core ideas:

  1. Automated Collection — Collect config snapshots from SSH/kernel/network/Docker multi-sources, eliminating manual per-item checks
  2. LLM Smart Analysis — Use local Qwen model to evaluate risk against CIS Benchmark rule library, far more accurate than manual review
  3. Structured Reporting — Generate Markdown reports with security scores, graded remediation suggestions, and fix commands
  4. Closed-Loop Delivery — Push summaries via Telegram, support scheduled tasks for unattended inspection

Compared to traditional inspection, this system offers:

  • 10x+ efficiency gain: 10-server inspection shrinks from 2 hours to 5 minutes
  • Unified standards: All servers use the same safety rule library, eliminating human judgment differences
  • Continuously evolving: Rule library可扩展随时, adapting to new security requirements
  • Near-zero cost: Local Ollama runs with no API call fees

Run a weekly intelligent inspection, and keep your entire VPS fleet at optimal security posture.

📺 看视频版教程 → DuckDB Lab YouTube

Subscribe for more DuckDB & AI automation tutorials