Featured image of post AI + VPS: Building an Intelligent Change Management and Release Safety System with Local LLMs

AI + VPS: Building an Intelligent Change Management and Release Safety System with Local LLMs

Traditional VPS changes rely on manual approval and experience, prone to errors and inefficiency. This article shows how to deploy an AI-driven change management system on your VPS using local LLMs, achieving automatic risk assessment, rollback plan generation, post-deployment health validation, and security gates — making every change safe and reliable.

Introduction: When Changes Become the Server’s Biggest Fear

As a VPS operator, you’ve certainly experienced this scenario:

It’s 2 AM, and you receive an urgent change notification for production — a service needs a configuration update. You hastily log into the server, manually modify the Nginx config, and restart the service. Ten minutes later, monitoring alerts fire: CPU spikes to 98%, and the service is completely unreachable.

You start troubleshooting and discover a typo in the new configuration parameter. You urgently roll back the config, and the service recovers. But deep down you know: if there had been a system to identify this risk beforehand, none of this would have happened.

The pain points of traditional VPS change management are clear:

  • Risk assessment relies on experience: Senior engineers spot issues immediately; newcomers have no idea what’s risky
  • Rollback plans are written ad-hoc: Each change requires last-minute rollback scripting under pressure
  • Post-deployment verification is manual: After changes, you must manually check every metric — easy to miss something
  • Change records are fragmented: Git commits, ticketing systems, chat logs — information is scattered
  • Approval processes are ceremonial: Urgent changes often skip approval and get retroactively documented

These issues might not be fatal on a single VPS, but when you manage 10, 50, or more servers, change risks grow exponentially.

AI-driven change management systems exist to solve exactly these problems. By deploying large language models locally on your VPS, we can achieve:

  1. Automatic change risk assessment: Analyze change content and predict potential risks
  2. Intelligent rollback plan generation: Automatically generate executable rollback scripts based on change type
  3. Post-deployment health validation: Automatically detect service status after changes and determine success
  4. Intelligent change log correlation: Link git commits, deployment records, and monitoring alerts together
  5. Automated security gates: High-risk changes automatically trigger approval workflows

Let’s build this system step by step.


1. System Architecture Design

┌─────────────────────────────────────────────────────────────────────┐
│                        User Interface Layer                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              │
│  │  Web Console │  │  Slack/Feishu│  │  CLI Tool    │              │
│  │  (React)     │  │  Bot         │  │  (Python)    │              │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘              │
│         └──────────────────┼──────────────────┘                     │
│                            ▼                                         │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │                   Change Management API Service              │    │
│  │              (FastAPI + PostgreSQL + Redis)                  │    │
│  └──────────────────┬──────────────────────────────────────────┘    │
│                     │                                                 │
│         ┌───────────┼───────────┬───────────┬───────────┐          │
│         ▼           ▼           ▼           ▼           ▼          │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│  │ Risk     │ │ Rollback │ │ Health   │ │ Log      │ │ Security │ │
│  │ Engine   │ │ Generator│ │ Validator│ │ Correlator││ Gateway  │ │
│  └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│       │            │            │            │            │        │
│  ┌────▼────────────▼────────────▼────────────▼────────────▼────┐  │
│  │                  Local LLM Service (Ollama)                   │  │
│  │              Qwen2.5-7B / Llama-3.2-3B / Gemma-3             │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                                                                       │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                    Infrastructure Layer                       │   │
│  │  Git Repo │ Prometheus │ Docker │ Kubernetes │ Config Mgmt    │   │
│  └──────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

The system consists of five layers:

  1. User Interface Layer: Web console, IM bot, and CLI — three entry points for different scenarios
  2. API Service Layer: Core service based on FastAPI, handling change requests and coordinating engines
  3. Engine Layer: Five core engines, each responsible for a different change management function
  4. LLM Service Layer: Locally deployed Ollama providing AI inference capabilities
  5. Infrastructure Layer: Existing infrastructure — Git, monitoring, containers

2. Risk Assessment Engine

Risk assessment is the core of the entire system. Its task: analyze change content and predict potential risk levels.

2.1 Risk Data Collection

The risk assessment needs to collect data across these dimensions:

# risk_analyzer.py - Risk data collection
import subprocess
import json
from datetime import datetime
from typing import Dict, List, Any

class RiskDataCollector:
    """Collect risk data related to changes"""
    
    def collect_context(self, change_id: str) -> Dict[str, Any]:
        return {
            "change_id": change_id,
            "timestamp": datetime.now().isoformat(),
            "server_info": self._collect_server_info(),
            "service_status": self._collect_service_status(),
            "recent_changes": self._collect_recent_changes(),
            "monitoring_alerts": self._collect_alerts(),
            "resource_usage": self._collect_resource_usage(),
            "dependency_map": self._collect_dependency_map(),
        }
    
    def _collect_server_info(self) -> Dict:
        """Collect basic server information"""
        return {
            "hostname": subprocess.getoutput("hostname"),
            "os": subprocess.getoutput("cat /etc/os-release | grep PRETTY_NAME"),
            "cpu_cores": subprocess.getoutput("nproc"),
            "memory_total": subprocess.getoutput("free -h | awk '/^Mem:/{print $2}'"),
            "disk_total": subprocess.getoutput("df -h / | awk 'NR==2{print $2}'"),
        }
    
    def _collect_service_status(self) -> List[Dict]:
        """Collect running service status"""
        services = {}
        containers = subprocess.getoutput(
            "docker ps --format '{{.Names}}|{{.Status}}|{{.Ports}}'"
        ).split("\n")
        for c in containers:
            if c:
                parts = c.split("|")
                services[parts[0]] = {"status": parts[1], "ports": parts[2]}
        return services
    
    def _collect_recent_changes(self, days: int = 7) -> List[Dict]:
        """Collect recent change records"""
        changes = []
        git_log = subprocess.getoutput(
            f"cd /etc/services && git log --oneline --since={days}days"
        )
        for line in git_log.split("\n")[:20]:
            if line:
                changes.append({"type": "git", "content": line})
        
        journal = subprocess.getoutput(
            f"journalctl --since '{days} days ago' "
            "-u nginx -u postgresql -u docker --no-pager"
        )
        if journal.strip():
            changes.append({"type": "journal", "content": journal[:2000]})
        return changes
    
    def _collect_alerts(self, hours: int = 24) -> List[Dict]:
        """Collect recent alert records"""
        alerts = subprocess.getoutput(
            f"curl -s 'http://localhost:9090/api/v1/alerts?"
            f"start={int(datetime.now().timestamp()) - hours*3600}'"
        )
        try:
            return json.loads(alerts).get("data", {}).get("alerts", [])
        except:
            return []
    
    def _collect_resource_usage(self) -> Dict:
        """Collect current resource usage"""
        return {
            "cpu": subprocess.getoutput("top -bn1 | grep 'Cpu(s)' | awk '{print $2}'"),
            "memory": subprocess.getoutput(
                "free | awk '/^Mem:/ {printf \"%.1f%%\", $3/$2 * 100}'"
            ),
            "disk": subprocess.getoutput(
                "df -h / | awk 'NR==2{printf \"%.1f%%\", $5}'"
            ),
            "load_avg": subprocess.getoutput(
                "cat /proc/loadavg | awk '{print $1, $2, $3}'"
            ),
        }
    
    def _collect_dependency_map(self) -> Dict:
        """Collect service dependency relationships"""
        deps = {}
        compose_files = subprocess.getoutput(
            "find /etc/services -name docker-compose.yml -o -name docker-compose.yaml"
        )
        for f in compose_files.strip().split("\n"):
            if f and os.path.exists(f):
                deps[f] = "loaded"
        return deps

2.2 LLM-Powered Risk Assessment

After collecting data, we call the local LLM for risk analysis:

# risk_assessment.py - LLM risk assessment
import json
import requests
from typing import Dict, List, Optional
from dataclasses import dataclass, field
from enum import Enum

class RiskLevel(Enum):
    LOW = "Low"
    MEDIUM = "Medium"
    HIGH = "High"
    CRITICAL = "Critical"

@dataclass
class RiskAssessment:
    risk_level: RiskLevel
    risk_score: int  # 0-100
    risk_factors: List[Dict[str, str]] = field(default_factory=list)
    recommendations: List[str] = field(default_factory=list)
    rollback_needed: bool = True
    approval_required: bool = False
    detailed_analysis: str = ""

class RiskAssessmentEngine:
    """LLM-powered risk assessment engine"""
    
    def __init__(self, ollama_url: str = "http://localhost:11434"):
        self.ollama_url = ollama_url
        self.model = "qwen2.5:7b"
    
    def assess_risk(self, context: Dict, change_description: str) -> RiskAssessment:
        """Perform risk assessment"""
        prompt = self._build_risk_prompt(context, change_description)
        llm_response = self._call_llm(prompt)
        assessment = self._parse_assessment(llm_response, change_description)
        return assessment
    
    def _build_risk_prompt(self, context: Dict, change_description: str) -> str:
        """Build risk assessment prompt"""
        risk_indicators = self._extract_risk_indicators(context)
        
        prompt = f"""You are an experienced operations security expert. Please analyze the risk of the following VPS change request.

## Change Information
{change_description}

## Current System State
- Server: {context.get('server_info', {}).get('hostname', 'unknown')}
- CPU Cores: {context.get('server_info', {}).get('cpu_cores', 'unknown')}
- Memory Total: {context.get('server_info', {}).get('memory_total', 'unknown')}
- Disk Total: {context.get('server_info', {}).get('disk_total', 'unknown')}
- Current CPU Usage: {context.get('resource_usage', {}).get('cpu', 'unknown')}
- Current Memory Usage: {context.get('resource_usage', {}).get('memory', 'unknown')}
- Current Disk Usage: {context.get('resource_usage', {}).get('disk', 'unknown')}
- Load Average: {context.get('resource_usage', {}).get('load_avg', 'unknown')}

## Running Services
{json.dumps(context.get('service_status', {}), indent=2, ensure_ascii=False)}

## Recent Alerts ({len(context.get('monitoring_alerts', []))} items)
{json.dumps(context.get('monitoring_alerts', [])[:5], indent=2, ensure_ascii=False)}

## Recent Change Records ({len(context.get('recent_changes', []))} items)
{json.dumps(context.get('recent_changes', [])[:5], indent=2, ensure_ascii=False)}

## Risk Indicators
{json.dumps(risk_indicators, indent=2, ensure_ascii=False)}

Please output the risk assessment in the following JSON format (do not include any other content):
{{
  "risk_level": "Low|Medium|High|Critical",
  "risk_score": integer 0-100,
  "risk_factors": [
    {{"factor": "Risk factor description", "severity": "high|medium|low"}}
  ],
  "recommendations": ["Recommendation 1", "Recommendation 2"],
  "rollback_needed": true/false,
  "approval_required": true/false,
  "detailed_analysis": "Detailed analysis explanation"
}}
"""
        return prompt
    
    def _extract_risk_indicators(self, context: Dict) -> Dict:
        """Extract risk indicators"""
        indicators = {}
        
        mem_usage = context.get('resource_usage', {}).get('memory', '0%')
        if '%' in mem_usage:
            try:
                mem_pct = float(mem_usage.replace('%', ''))
                indicators['memory_pressure'] = 'high' if mem_pct > 85 else ('medium' if mem_pct > 70 else 'low')
            except:
                indicators['memory_pressure'] = 'unknown'
        
        alert_count = len(context.get('monitoring_alerts', []))
        indicators['active_alerts'] = alert_count
        indicators['alert_severity'] = 'high' if alert_count > 5 else ('medium' if alert_count > 2 else 'low')
        
        recent_changes = len(context.get('recent_changes', []))
        indicators['change_frequency'] = recent_changes
        indicators['recent_change_risk'] = 'high' if recent_changes > 10 else ('medium' if recent_changes > 5 else 'low')
        
        services = context.get('service_status', {})
        degraded = [s for s, info in services.items() 
                   if 'unhealthy' in info.get('status', '').lower() 
                   or 'starting' in info.get('status', '').lower()]
        indicators['degraded_services'] = len(degraded)
        
        return indicators
    
    def _call_llm(self, prompt: str) -> str:
        """Call Ollama API"""
        try:
            response = requests.post(
                f"{self.ollama_url}/api/generate",
                json={"model": self.model, "prompt": prompt, "stream": False},
                timeout=120
            )
            return response.json().get("response", "")
        except Exception as e:
            return f"LLM call failed: {e}"
    
    def _parse_assessment(self, response: str, change_desc: str) -> RiskAssessment:
        """Parse LLM risk assessment response"""
        try:
            start = response.find('{')
            end = response.rfind('}')
            if start != -1 and end != -1:
                data = json.loads(response[start:end+1])
            else:
                return RiskAssessment(
                    risk_level=RiskLevel.LOW, risk_score=10,
                    risk_factors=[{"factor": "LLM parse error, using default low-risk strategy", "severity": "low"}],
                    recommendations=["Please manually review the change content"],
                    rollback_needed=True, approval_required=False,
                    detailed_analysis=f"LLM response parse error: {response[:500]}"
                )
            
            level_map = {
                "Low": RiskLevel.LOW, "Medium": RiskLevel.MEDIUM,
                "High": RiskLevel.HIGH, "Critical": RiskLevel.CRITICAL,
            }
            
            return RiskAssessment(
                risk_level=level_map.get(data.get("risk_level", "Low"), RiskLevel.LOW),
                risk_score=data.get("risk_score", 10),
                risk_factors=data.get("risk_factors", []),
                recommendations=data.get("recommendations", []),
                rollback_needed=data.get("rollback_needed", True),
                approval_required=data.get("approval_required", False),
                detailed_analysis=data.get("detailed_analysis", "")
            )
        except Exception as e:
            return RiskAssessment(
                risk_level=RiskLevel.MEDIUM, risk_score=30,
                risk_factors=[{"factor": f"Parse error: {e}", "severity": "medium"}],
                recommendations=["Please manually review the change content"],
                rollback_needed=True, approval_required=True,
                detailed_analysis=str(e)
            )

2.3 Risk Assessment Example

Suppose we plan an Nginx configuration update:

# Example: Assessing an Nginx config change
analyzer = RiskAssessmentEngine()
context = RiskDataCollector().collect_context("change-20260915-001")

assessment = analyzer.assess_risk(
    context=context,
    change_description="""
Change Type: Nginx Configuration Update
Change Details:
- Modify /etc/nginx/nginx.conf
- Change worker_processes from auto to 4
- Add new upstream backend configuration
- Restart nginx service
Affected Services: nginx (ports 80/443)
Scheduled Window: 2:00-3:00 AM
Owner: zhangsan
"""
)

print(f"Risk Level: {assessment.risk_level.value}")
print(f"Risk Score: {assessment.risk_score}/100")
print(f"Rollback Required: {'Yes' if assessment.rollback_needed else 'No'}")
print(f"Approval Required: {'Yes' if assessment.approval_required else 'No'}")
print(f"\nRisk Factors:")
for f in assessment.risk_factors:
    print(f"  - [{f['severity']}] {f['factor']}")
print(f"\nRecommendations:")
for r in assessment.recommendations:
    print(f"  - {r}")

Sample output:

Risk Level: Medium
Risk Score: 45/100
Rollback Required: Yes
Approval Required: No

Risk Factors:
  - [medium] worker_processes modification may affect existing connections
  - [medium] New upstream config requires backend services to be ready
  - [low] Current memory usage is normal (62%)

Recommendations:
  - Execute during the change window
  - Back up current config before making changes
  - Prepare rollback script
  - Gradually verify HTTP response codes and service availability after changes

3. Intelligent Rollback Plan Generation

Rollback is the key guarantee for change safety. Traditionally, operators must write rollback scripts on the fly during changes, often leading to omissions due to time pressure.

3.1 Rollback Plan Generator

# rollback_generator.py - Intelligent rollback plan generation
import json
import re
from typing import Dict, List, Optional
from datetime import datetime
from dataclasses import dataclass

@dataclass
class RollbackPlan:
    plan_id: str
    change_id: str
    generated_at: str
    steps: List[Dict[str, str]]
    estimated_duration: str
    risk_notes: List[str]
    validation_commands: List[str]
    full_script: str

class RollbackGenerator:
    """LLM-powered rollback plan generator"""
    
    def __init__(self, ollama_url: str = "http://localhost:11434"):
        self.ollama_url = ollama_url
        self.model = "qwen2.5:7b"
    
    def generate_rollback_plan(self, change_id: str, change_details: Dict, system_snapshot: Dict) -> RollbackPlan:
        """Generate a rollback plan"""
        prompt = self._build_rollback_prompt(change_id, change_details, system_snapshot)
        llm_response = self._call_llm(prompt)
        plan = self._parse_rollback_plan(llm_response, change_id)
        return plan
    
    def _build_rollback_prompt(self, change_id: str, change_details: Dict, system_snapshot: Dict) -> str:
        """Build rollback prompt"""
        prompt = f"""You are a senior operations engineer. Please generate a detailed rollback plan based on the following change information.

## Change Information
- Change ID: {change_id}
- Change Type: {change_details.get('type', 'unknown')}
- Change Details: {change_details.get('description', 'N/A')}
- Affected Services: {', '.join(change_details.get('services', []))}
- Execution Time: {change_details.get('executed_at', 'unknown')}

## Current System Snapshot
{json.dumps(system_snapshot, indent=2, ensure_ascii=False)}

## Change Operation Records
{json.dumps(change_details.get('operations', []), indent=2, ensure_ascii=False)}

Please output the rollback plan in the following JSON format:
{{
  "steps": [
    {{
      "order": 1,
      "action": "Rollback action description",
      "command": "Specific command to execute",
      "expected_result": "Expected result",
      "timeout_seconds": 30
    }}
  ],
  "estimated_duration": "Estimated rollback duration",
  "risk_notes": ["Rollback process risk warnings"],
  "validation_commands": ["Post-rollback validation commands"],
  "full_script": "#!/bin/bash\n# Complete rollback script\n..."
}}
"""
        return prompt
    
    def _call_llm(self, prompt: str) -> str:
        """Call Ollama"""
        try:
            response = requests.post(
                f"{self.ollama_url}/api/generate",
                json={"model": self.model, "prompt": prompt, "stream": False},
                timeout=120
            )
            return response.json().get("response", "")
        except Exception as e:
            return f"LLM call failed: {e}"
    
    def _parse_rollback_plan(self, response: str, change_id: str) -> RollbackPlan:
        """Parse rollback plan"""
        try:
            start = response.find('{')
            end = response.rfind('}')
            if start != -1 and end != -1:
                data = json.loads(response[start:end+1])
            else:
                data = self._generate_fallback_plan(change_id)
            
            return RollbackPlan(
                plan_id=f"rollback-{change_id}",
                change_id=change_id,
                generated_at=datetime.now().isoformat(),
                steps=data.get("steps", []),
                estimated_duration=data.get("estimated_duration", "Unknown"),
                risk_notes=data.get("risk_notes", []),
                validation_commands=data.get("validation_commands", []),
                full_script=data.get("full_script", "# Please review the rollback plan")
            )
        except Exception as e:
            return RollbackPlan(
                plan_id=f"rollback-{change_id}",
                change_id=change_id,
                generated_at=datetime.now().isoformat(),
                steps=[{"order": 1, "action": "Manual rollback", "command": "Please execute rollback manually", "expected_result": "Service recovery", "timeout_seconds": 0}],
                estimated_duration="Unknown",
                risk_notes=["Automatic rollback plan generation failed, please execute manually"],
                validation_commands=["systemctl status nginx"],
                full_script="# Rollback plan generation failed, please execute manually"
            )
    
    def _generate_fallback_plan(self, change_id: str) -> Dict:
        """Generate fallback rollback plan"""
        return {
            "steps": [
                {"order": 1, "action": "Restore config backup", "command": "cp /backup/config/pre-change-* /etc/", "expected_result": "Config restored", "timeout_seconds": 10},
                {"order": 2, "action": "Restart affected services", "command": "systemctl restart nginx", "expected_result": "Service restarted", "timeout_seconds": 30},
                {"order": 3, "action": "Verify service status", "command": "curl -s -o /dev/null -w '%{http_code}' http://localhost/ && systemctl is-active nginx", "expected_result": "HTTP 200, active", "timeout_seconds": 10},
            ],
            "estimated_duration": "Approximately 2 minutes",
            "risk_notes": ["This is a fallback plan, adjust according to actual situation"],
            "validation_commands": ["systemctl is-active nginx", "curl -s http://localhost/ | head -5"],
            "full_script": "#!/bin/bash\nset -euo pipefail\n\necho '=== Starting rollback for {change_id} ==='\n\necho 'Step 1: Restoring configuration...'\ncp /backup/config/pre-change-*/etc/nginx/nginx.conf /etc/nginx/nginx.conf\n\necho 'Step 2: Restarting Nginx...'\nsystemctl restart nginx\n\necho 'Step 3: Verifying service status...'\nsleep 5\nhttp_code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost/)\nif [ \"$http_code\" = \"200\" ]; then\n    echo \"Rollback successful! HTTP status: $http_code\"\nelse\n    echo \"Rollback may have issues, HTTP status: $http_code\"\nfi\n\necho '=== Rollback completed ==='"
        }

3.2 Snapshot Mechanism

To ensure rollback feasibility, the system automatically creates system snapshots before changes:

# snapshot_manager.py - Change snapshot management
import subprocess
import shutil
import os
from datetime import datetime
from pathlib import Path

class SnapshotManager:
    """Manage system snapshots before changes"""
    
    SNAPSHOT_DIR = "/var/lib/change-mgmt/snapshots"
    
    def __init__(self):
        os.makedirs(self.SNAPSHOT_DIR, exist_ok=True)
    
    def create_snapshot(self, change_id: str) -> Dict:
        """Create a pre-change snapshot"""
        snapshot_time = datetime.now().strftime("%Y%m%d_%H%M%S")
        snapshot_path = f"{self.SNAPSHOT_DIR}/{change_id}_{snapshot_time}"
        os.makedirs(snapshot_path, exist_ok=True)
        
        snapshot_data = {
            "change_id": change_id,
            "created_at": datetime.now().isoformat(),
            "path": snapshot_path,
            "files_backed_up": [],
            "configs_exported": {},
            "services_stopped": [],
        }
        
        config_paths = [
            "/etc/nginx/nginx.conf",
            "/etc/nginx/conf.d/",
            "/etc/postgresql/",
            "/etc/mysql/",
            "/etc/docker/daemon.json",
            "/etc/sysctl.conf",
            "/etc/crontab",
        ]
        
        for cfg in config_paths:
            if os.path.exists(cfg):
                dest = f"{snapshot_path}/{cfg.replace('/', '_')}"
                if os.path.isdir(cfg):
                    shutil.copytree(cfg, dest)
                else:
                    shutil.copy2(cfg, dest)
                snapshot_data["files_backed_up"].append(cfg)
        
        snapshot_data["configs_exported"]["docker_ps"] = subprocess.getoutput("docker ps -a --format '{{.Names}}|{{.Status}}'")
        snapshot_data["configs_exported"]["systemd_services"] = subprocess.getoutput("systemctl list-units --type=service --state=running | head -50")
        snapshot_data["configs_exported"]["crontab"] = subprocess.getoutput("crontab -l 2>/dev/null || echo 'no crontab'")
        snapshot_data["configs_exported"]["mounts"] = subprocess.getoutput("df -h")
        
        with open(f"{snapshot_path}/snapshot.json", "w") as f:
            json.dump(snapshot_data, f, indent=2, ensure_ascii=False)
        
        return snapshot_data

4. Post-Deployment Health Validation Engine

After a change is executed, the system needs to automatically validate whether the change succeeded. The health validation engine:

  1. Service reachability checks: Confirm key ports and services are available
  2. Metric comparison: Compare key metrics before and after changes
  3. Error rate monitoring: Monitor error rate changes post-change
  4. Automatic judgment:综合评价多项指标,判定变更是否成功
# health_validator.py - Post-deployment health validation
import subprocess
import json
import time
import re
from typing import Dict, List, Optional
from dataclasses import dataclass
from datetime import datetime, timedelta

@dataclass
class HealthCheckResult:
    check_name: str
    passed: bool
    value: str
    threshold: str
    before: Optional[float] = None
    after: Optional[float] = None

class HealthValidator:
    """Post-deployment health validation engine"""
    
    def __init__(self, ollama_url: str = "http://localhost:11434"):
        self.ollama_url = ollama_url
        self.model = "qwen2.5:7b"
    
    def validate_deployment(self, change_id: str, target_services: List[str], baseline_metrics: Dict, monitoring_window: int = 300) -> Dict:
        """Execute post-deployment health validation"""
        results = {
            "change_id": change_id,
            "validated_at": datetime.now().isoformat(),
            "checks": [],
            "overall_status": "pending",
            "llm_analysis": "",
            "recommendation": "",
        }
        
        # 1. Immediate health checks
        immediate_checks = self._immediate_checks(target_services)
        results["checks"].extend(immediate_checks)
        
        # 2. Continued monitoring
        time.sleep(min(monitoring_window, 60))
        continued_checks = self._continued_monitoring(target_services, baseline_metrics)
        results["checks"].extend(continued_checks)
        
        # 3. LLM comprehensive analysis
        llm_analysis = self._llm_health_analysis(results["checks"], baseline_metrics, target_services)
        results["llm_analysis"] = llm_analysis
        
        # 4. Comprehensive judgment
        passed_count = sum(1 for c in results["checks"] if c.passed)
        total_count = len(results["checks"])
        pass_rate = passed_count / total_count if total_count > 0 else 0
        
        if pass_rate >= 0.9:
            results["overall_status"] = "success"
            results["recommendation"] = "Change succeeded, services running normally"
        elif pass_rate >= 0.7:
            results["overall_status"] = "warning"
            results["recommendation"] = "Some checks failed, manual review recommended"
        else:
            results["overall_status"] = "failure"
            results["recommendation"] = "Multiple checks failed, immediate rollback recommended"
        
        return results
    
    def _llm_health_analysis(self, checks: List[HealthCheckResult], baseline: Dict, services: List[str]) -> str:
        """LLM comprehensive health analysis"""
        checks_summary = json.dumps([
            {"name": c.check_name, "passed": c.passed, "value": c.value}
            for c in checks
        ], indent=2, ensure_ascii=False)
        
        prompt = f"""You are an operations expert. Please analyze the following post-deployment health check results and provide professional judgment.

## Health Check Items
{checks_summary}

## Current Services
{', '.join(services)}

## Baseline Metrics
- CPU baseline: {baseline.get('cpu', 'N/A')}%
- Memory baseline: {baseline.get('memory', 'N/A')}%
- Disk baseline: {baseline.get('disk', 'N/A')}%

Please briefly analyze:
1. Did the change succeed?
2. Are there any anomalies that need attention?
3. What are the next steps?

Answer in 3-5 sentences, providing the analysis directly.
"""
        
        try:
            response = requests.post(
                f"{self.ollama_url}/api/generate",
                json={"model": self.model, "prompt": prompt, "stream": False},
                timeout=60
            )
            return response.json().get("response", "Analysis unavailable")
        except:
            return "LLM analysis unavailable"

5. Intelligent Change Log Correlation

Change management needs complete audit trails. The intelligent log correlation engine unifies scattered information:

# change_logger.py - Intelligent change log correlation
import subprocess
import json
from datetime import datetime, timedelta
from typing import Dict, List
from pathlib import Path

class ChangeLogger:
    """Intelligent change log correlation engine"""
    
    def __init__(self, db_path: str = "/var/lib/change-mgmt/changes.db"):
        self.db_path = db_path
        self.log_dir = Path("/var/lib/change-mgmt/logs")
        self.log_dir.mkdir(parents=True, exist_ok=True)
    
    def log_change(self, change_id: str, change_type: str, description: str, executor: str,
                   risk_assessment: Dict, rollback_plan: Dict, health_check: Dict) -> Dict:
        """Log a complete change record"""
        log_entry = {
            "change_id": change_id,
            "type": change_type,
            "description": description,
            "executor": executor,
            "created_at": datetime.now().isoformat(),
            "risk_assessment": risk_assessment,
            "rollback_plan": rollback_plan,
            "health_check": health_check,
            "status": "completed",
            "timeline": [{"time": datetime.now().isoformat(), "event": "Change completed", "details": "Change executed and health validation passed"}]
        }
        
        # Correlate git commits
        git_related = self._find_related_git_commits(change_id, description)
        if git_related:
            log_entry["related_commits"] = git_related
        
        # Correlate monitoring alerts
        alert_related = self._find_related_alerts(change_id)
        if alert_related:
            log_entry["related_alerts"] = alert_related
        
        log_file = self.log_dir / f"{change_id}.json"
        with open(log_file, "w") as f:
            json.dump(log_entry, f, indent=2, ensure_ascii=False)
        
        return log_entry
    
    def generate_report(self, change_id: str) -> str:
        """Generate change report"""
        log_file = self.log_dir / f"{change_id}.json"
        if not log_file.exists():
            return f"Change log not found: {change_id}"
        
        log = json.load(open(log_file))
        
        report = f"""
## Change Report: {log['change_id']}

| Item | Content |
|------|---------|
| Change Type | {log['type']} |
| Executor | {log['executor']} |
| Execution Time | {log['created_at']} |
| Status | {log['status']} |

### Change Description
{log['description']}

### Risk Assessment
- Risk Level: {log.get('risk_assessment', {}).get('risk_level', 'Unknown')}
- Risk Score: {log.get('risk_assessment', {}).get('risk_score', 'N/A')}/100
- Rollback Needed: {'Yes' if log.get('rollback_plan', {}).get('rollback_needed', True) else 'No'}
- Approval Needed: {'Yes' if log.get('risk_assessment', {}).get('approval_required', False) else 'No'}

### Health Validation
- Validation Status: {log.get('health_check', {}).get('overall_status', 'Unknown')}
- Recommendation: {log.get('health_check', {}).get('recommendation', 'N/A')}

### LLM Analysis
{log.get('health_check', {}).get('llm_analysis', 'N/A')}
"""
        return report

6. Security Gates and Approval Workflows

For high-risk changes, the system automatically triggers security gates:

# security_gateway.py - Security gates
import json
from typing import Dict
from datetime import datetime

class SecurityGateway:
    """Change security gate engine"""
    
    def __init__(self, config: Dict = None):
        self.config = config or {
            "high_risk_threshold": 70,
            "critical_risk_threshold": 85,
            "notification_channels": ["email", "slack"],
        }
    
    def check_gateway(self, risk_assessment: Dict, change_details: Dict) -> Dict:
        """Execute security gate checks"""
        gates = {
            "risk_score_gate": self._check_risk_score(risk_assessment),
            "alert_gate": self._check_active_alerts(change_details),
            "resource_gate": self._check_resource_safety(change_details),
            "time_window_gate": self._check_time_window(change_details),
            "approval_gate": self._check_approval_required(risk_assessment),
        }
        
        all_passed = all(g["passed"] for g in gates.values())
        blocked_gates = [name for name, g in gates.items() if not g["passed"]]
        
        result = {
            "timestamp": datetime.now().isoformat(),
            "all_passed": all_passed,
            "gates": gates,
            "blocked_gates": blocked_gates,
            "action": "allow" if all_passed else "block",
        }
        
        if not all_passed:
            self._send_notification(result, change_details)
        
        return result
    
    def _check_risk_score(self, risk_assessment: Dict) -> Dict:
        score = risk_assessment.get("risk_score", 0)
        threshold = self.config.get("high_risk_threshold", 70)
        if score >= threshold:
            return {"passed": False, "reason": f"Risk score {score} exceeds threshold {threshold}",
                    "severity": "critical" if score >= self.config.get("critical_risk_threshold", 85) else "high"}
        return {"passed": True, "reason": "Risk score within safe range"}
    
    def _check_active_alerts(self, change_details: Dict) -> Dict:
        alerts = change_details.get("active_alerts", [])
        critical_alerts = [a for a in alerts if a.get("severity") == "critical"]
        if critical_alerts:
            return {"passed": False, "reason": f"{len(critical_alerts)} critical alerts active, change blocked", "severity": "high"}
        return {"passed": True, "reason": "No critical active alerts"}
    
    def _check_resource_safety(self, change_details: Dict) -> Dict:
        resources = change_details.get("resource_usage", {})
        issues = []
        mem_usage = float(resources.get("memory", "0").replace("%", ""))
        if mem_usage > 85:
            issues.append(f"Memory usage {mem_usage}% too high")
        disk_usage = float(resources.get("disk", "0").replace("%", ""))
        if disk_usage > 90:
            issues.append(f"Disk usage {disk_usage}% too high")
        if issues:
            return {"passed": False, "reason": "; ".join(issues), "severity": "medium"}
        return {"passed": True, "reason": "Resource usage within safe range"}
    
    def _check_time_window(self, change_details: Dict) -> Dict:
        exec_time = change_details.get("scheduled_time", "")
        if not exec_time:
            return {"passed": True, "reason": "No execution time specified"}
        try:
            hour = int(exec_time.split(" ")[1].split(":")[0]) if " " in exec_time else 0
            peak_hours = list(range(9, 12)) + list(range(14, 18))
            if hour in peak_hours:
                return {"passed": False, "reason": f"Change time {exec_time} falls within business hours", "severity": "medium"}
        except:
            pass
        return {"passed": True, "reason": "Change time within allowed window"}
    
    def _check_approval_required(self, risk_assessment: Dict) -> Dict:
        if risk_assessment.get("approval_required", False):
            return {"passed": False, "reason": "High-risk change requires approval", "severity": "high"}
        return {"passed": True, "reason": "No additional approval needed"}
    
    def _send_notification(self, result: Dict, change_details: Dict):
        blocked_gates = result.get("blocked_gates", [])
        message = f"🚨 Change Security Gate Blocked\n\nChange ID: {change_details.get('change_id', 'N/A')}\nBlocked by: {', '.join(blocked_gates)}\n\nPlease resolve the issues and resubmit the change."
        # Send via configured channels (email, Slack, etc.)
        print(message)

7. Complete Change Workflow Integration

Integrating all components into a complete change workflow:

# change_workflow.py - Complete change workflow
import json
import asyncio
from typing import Dict, Optional
from dataclasses import dataclass
from enum import Enum

class ChangeStatus(Enum):
    PENDING = "Pending"
    RISK_ANALYSIS = "Risk Assessment"
    ROLLBACK_PREPARED = "Rollback Ready"
    GATE_CHECKED = "Gate Passed"
    EXECUTING = "Executing"
    HEALTH_CHECKING = "Health Checking"
    COMPLETED = "Completed"
    ROLLED_BACK = "Rolled Back"
    BLOCKED = "Blocked"

@dataclass
class ChangeRequest:
    change_id: str
    type: str
    description: str
    executor: str
    target_services: List[str]
    scheduled_time: str
    config_diff: str

class ChangeWorkflow:
    """Complete change management workflow"""
    
    def __init__(self):
        self.risk_analyzer = RiskAssessmentEngine()
        self.rollback_generator = RollbackGenerator()
        self.snapshot_manager = SnapshotManager()
        self.health_validator = HealthValidator()
        self.change_logger = ChangeLogger()
        self.security_gateway = SecurityGateway()
    
    async def execute_change(self, request: ChangeRequest) -> Dict:
        """Execute the complete change workflow"""
        
        workflow_result = {
            "change_id": request.change_id,
            "status": ChangeStatus.PENDING.value,
            "stages": [],
        }
        
        # Stage 1: Pre-change snapshot
        print(f"[{request.change_id}] Stage 1: Creating pre-change snapshot...")
        snapshot = self.snapshot_manager.create_snapshot(request.change_id)
        workflow_result["stages"].append({"stage": "snapshot", "status": "completed", "result": snapshot})
        
        # Stage 2: Risk assessment
        print(f"[{request.change_id}] Stage 2: Performing risk assessment...")
        context = RiskDataCollector().collect_context(request.change_id)
        risk_assessment = self.risk_analyzer.assess_risk(context=context, change_description=request.description)
        workflow_result["stages"].append({
            "stage": "risk_assessment", "status": "completed",
            "result": {"risk_level": risk_assessment.risk_level.value, "risk_score": risk_assessment.risk_score}
        })
        
        # Stage 3: Security gate check
        print(f"[{request.change_id}] Stage 3: Security gate check...")
        gate_result = self.security_gateway.check_gateway(
            risk_assessment=vars(risk_assessment),
            change_details={
                "change_id": request.change_id,
                "active_alerts": context.get("monitoring_alerts", []),
                "resource_usage": context.get("resource_usage", {}),
                "scheduled_time": request.scheduled_time,
            }
        )
        
        if not gate_result["all_passed"]:
            workflow_result["status"] = ChangeStatus.BLOCKED.value
            workflow_result["stages"].append({"stage": "security_gateway", "status": "blocked", "result": gate_result})
            print(f"🚫 Change blocked: {gate_result['blocked_gates']}")
            return workflow_result
        
        workflow_result["stages"].append({"stage": "security_gateway", "status": "passed", "result": gate_result})
        
        # Stage 4: Generate rollback plan
        print(f"[{request.change_id}] Stage 4: Generating rollback plan...")
        rollback_plan = self.rollback_generator.generate_rollback_plan(
            change_id=request.change_id,
            change_details={"type": request.type, "description": request.description, "services": request.target_services},
            system_snapshot=snapshot
        )
        workflow_result["stages"].append({
            "stage": "rollback_plan", "status": "completed",
            "result": {"steps_count": len(rollback_plan.steps), "estimated_duration": rollback_plan.estimated_duration}
        })
        
        # Stage 5: Execute change (simulated)
        print(f"[{request.change_id}] Stage 5: Executing change...")
        workflow_result["stages"].append({"stage": "execution", "status": "completed", "result": {"message": "Change operation executed"}})
        
        # Stage 6: Health validation
        print(f"[{request.change_id}] Stage 6: Performing health validation...")
        baseline = context.get("resource_usage", {})
        health_result = self.health_validator.validate_deployment(
            change_id=request.change_id, target_services=request.target_services, baseline_metrics=baseline
        )
        workflow_result["stages"].append({
            "stage": "health_check", "status": "completed",
            "result": {"overall_status": health_result["overall_status"], "recommendation": health_result["recommendation"]}
        })
        
        # Stage 7: Log change
        print(f"[{request.change_id}] Stage 7: Recording change log...")
        self.change_logger.log_change(
            change_id=request.change_id, change_type=request.type, description=request.description,
            executor=request.executor,
            risk_assessment={"risk_level": risk_assessment.risk_level.value, "risk_score": risk_assessment.risk_score},
            rollback_plan={"rollback_needed": rollback_plan.rollback_needed, "steps_count": len(rollback_plan.steps)},
            health_check=health_result,
        )
        
        workflow_result["status"] = ChangeStatus.COMPLETED.value
        print(f"✅ Change completed: {request.change_id}")
        
        return workflow_result

8. Web Console Implementation

A lightweight web console for submitting change requests and viewing results:

# web_console.py - Change management web console
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from typing import List, Optional
import asyncio

app = FastAPI(title="VPS Intelligent Change Management System", version="1.0.0")

app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])

workflow = ChangeWorkflow()

class ChangeRequestModel(BaseModel):
    change_id: str
    type: str
    description: str
    executor: str
    target_services: List[str]
    scheduled_time: str
    config_diff: str

@app.get("/")
async def root():
    return {
        "service": "VPS Intelligent Change Management System",
        "version": "1.0.0",
        "endpoints": {
            "POST /api/v1/changes": "Submit change request",
            "GET /api/v1/changes/{change_id}": "Query change status",
            "GET /api/v1/changes/{change_id}/report": "Generate change report",
            "POST /api/v1/changes/{change_id}/rollback": "Execute rollback",
        }
    }

@app.post("/api/v1/changes")
async def submit_change(request: ChangeRequestModel):
    try:
        result = await workflow.execute_change(request)
        return {"status": "success", "change_id": request.change_id, "result": result}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.get("/api/v1/changes/{change_id}")
async def get_change_status(change_id: str):
    log_file = Path(f"/var/lib/change-mgmt/logs/{change_id}.json")
    if not log_file.exists():
        raise HTTPException(status_code=404, detail="Change not found")
    log = json.load(open(log_file))
    return log

@app.get("/api/v1/changes/{change_id}/report")
async def get_change_report(change_id: str):
    logger = ChangeLogger()
    report = logger.generate_report(change_id)
    return {"change_id": change_id, "report": report}

@app.post("/api/v1/changes/{change_id}/rollback")
async def trigger_rollback(change_id: str):
    snapshot_dir = Path("/var/lib/change-mgmt/snapshots")
    snapshots = list(snapshot_dir.glob(f"{change_id}_*"))
    if not snapshots:
        raise HTTPException(status_code=404, detail="No rollback snapshot found")
    latest_snapshot = max(snapshots, key=lambda p: p.stat().st_mtime)
    manager = SnapshotManager()
    result = manager.restore_snapshot(str(latest_snapshot))
    return {"status": "rollback_initiated", "change_id": change_id, "snapshot": str(latest_snapshot), "result": result}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8080)

9. Deployment and Configuration

9.1 System Requirements

  • CPU: 2 cores minimum (4 cores recommended for 7B model)
  • Memory: 8GB minimum (16GB recommended for 7B model)
  • Disk: 50GB minimum (for models and snapshots)
  • OS: Ubuntu 22.04+ / Debian 12+

9.2 Installation Steps

# 1. Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# 2. Pull the model
ollama pull qwen2.5:7b

# 3. Install Python dependencies
pip install fastapi uvicorn pillow requests python-dotenv

# 4. Create data directories
mkdir -p /var/lib/change-mgmt/{snapshots,logs}

# 5. Deploy the application
cp change_workflow.py /opt/change-mgmt/
cp web_console.py /opt/change-mgmt/
cd /opt/change-mgmt

# 6. Manage with systemd
cat > /etc/systemd/system/change-mgmt.service << 'EOF'
[Unit]
Description=VPS Intelligent Change Management System
After=network.target ollama.service

[Service]
Type=simple
User=root
WorkingDirectory=/opt/change-mgmt
ExecStart=/usr/bin/python3 web_console.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

systemctl enable change-mgmt
systemctl start change-mgmt

# 7. Verify the service
curl http://localhost:8080/

9.3 Nginx Reverse Proxy Configuration

server {
    listen 443 ssl;
    server_name change.selfvps.net;

    ssl_certificate /etc/letsencrypt/live/selfvps.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/selfvps.net/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

10. Practical Usage Examples

10.1 Submitting an Nginx Configuration Change

curl -X POST http://localhost:8080/api/v1/changes \
  -H "Content-Type: application/json" \
  -d '{
    "change_id": "change-20260915-001",
    "type": "nginx_config_update",
    "description": "Update Nginx worker_processes parameter and add new upstream config",
    "executor": "zhangsan",
    "target_services": ["nginx"],
    "scheduled_time": "2026-09-16 02:00",
    "config_diff": "worker_processes 4;\nupstream backend { server 10.0.0.1:8080; }"
  }'

10.2 Viewing the Change Report

curl http://localhost:8080/api/v1/changes/change-20260915-001/report

10.3 Triggering Emergency Rollback

curl -X POST http://localhost:8080/api/v1/changes/change-20260915-001/rollback

11. Best Practices and Recommendations

11.1 Golden Rules of Change Management

  1. Always create a snapshot before changing — Never skip the snapshot step
  2. Define clear rollback criteria — Know exactly when to roll back before you change
  3. Test in staging first — Validate changes in a non-production environment
  4. Execute during maintenance windows — Avoid peak business hours
  5. Document everything — Complete change records are essential for audits

11.2 System Optimization Suggestions

Optimization DirectionSpecific Recommendation
Model selectionUse qwen2.5:3b for limited resources, qwen2.5:7b when available
Snapshot strategyWeekly full snapshots + incremental pre-change snapshots
Monitoring integrationConnect with Prometheus + Alertmanager
Notification channelsSupport Slack/Feishu/DingTalk/Email
Permission controlRBAC-based change approval workflows

11.3 Integration with Existing Tools

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   GitLab/Gitea  │────▶│  Change Mgmt    │────▶│   Slack/Feishu  │
│   (Code Repo)   │     │  System         │     │   (Notifications)│
└─────────────────┘     └─────────────────┘     └─────────────────┘
                              │
                    ┌─────────▼─────────┐
                    │   Prometheus      │
                    │   (Monitoring)     │
                    └───────────────────┘

Conclusion

An AI-driven change management system transforms traditionally experience-dependent change processes into automated workflows assisted by LLMs. Through four core capabilities — risk assessment, rollback plan generation, health validation, and security gates — it significantly reduces VPS change risks.

Key benefits:

  • Proactive risk identification: Catch potential issues before changes, not after
  • Automated rollback: Every change has a pre-generated rollback plan
  • Verifiable health: Automatic post-change validation instead of manual checks
  • Full traceability: Complete change logs and audit trails

When you manage more and more VPS instances, this system transitions from “nice to have” to “essential.” After all, who wants to wake up at 2 AM again to fix a typo in a config file?


References:

📺 看视频版教程 → DuckDB Lab YouTube

Subscribe for more DuckDB & AI automation tutorials