Featured image of post AI VPS 智能混沌工程:自动化故障注入与系统韧性测试

AI VPS 智能混沌工程:自动化故障注入与系统韧性测试

用 AI Agent 驱动混沌工程——自动注入故障、监测系统行为、智能恢复,让 VPS 服务在真实故障面前游刃有余,从被动救火走向主动韧性建设。

引言

你的 VPS 服务真的"扛得住"吗?

  • 数据库突然慢查询,API 响应时间从 50ms 飙到 5s;
  • 磁盘空间被日志占满,服务全部宕机;
  • 某个依赖的微服务超时,级联故障拖垮整个系统;
  • 内存泄漏缓慢累积,一周后网站彻底无法访问;
  • SSL 证书过期、 Cron 任务失败、进程死锁……

这些故障不是会不会发生的问题,而是什么时候发生的问题。传统运维的应对方式是"出了问题再修"——但用户不会等你排查。而 混沌工程(Chaos Engineering) 的理念是:在故障发生前主动制造故障,验证系统的韧性,提前发现脆弱点。

2026 年,大语言模型(LLM)和 AI Agent 技术让混沌工程变得前所未有地简单。你不再需要手动编写故障脚本、手动观察结果、手动分析根因——AI Agent 可以全自动完成整个混沌工程闭环

本文将带你构建一套 AI 驱动的 VPS 混沌工程系统,实现从故障注入、行为监测、根因分析到自动恢复的全流程自动化。


一、什么是混沌工程?为什么 VPS 需要它?

1.1 混沌工程的核心理念

混沌工程起源于 Netflix 的 Chaos Monkey 项目,核心理念是:

在受控环境中主动注入故障,验证系统是否能在真实故障中保持可用。

传统测试关注"系统是否能正常工作",而混沌工程关注**“系统在不正常时能否保持可用”**。

传统测试混沌工程
验证系统正常工作验证系统故障时仍可用
关注功能正确性关注韧性(Resilience)
一次性测试持续性验证
人工执行可自动化持续执行

1.2 VPS 场景下的混沌工程价值

对于 VPS 用户来说,混沌工程有独特的价值:

  1. 单点故障风险高:大多数 VPS 用户只有一台服务器,任何故障都意味着全量中断
  2. 资源受限:VPS 资源有限,过载时容易产生级联故障
  3. 缺乏冗余:没有多副本、没有自动故障转移,一次故障可能就是永久宕机
  4. 运维人力有限:个人运维者或小团队难以 24 小时盯守

AI 驱动的混沌工程可以帮助你在故障真正发生之前发现系统的脆弱点,并自动修复。


二、系统架构:AI 驱动的混沌工程平台

┌─────────────────────────────────────────────────────────────┐
│                    AI Chaos Engineering Platform              │
├─────────────┬─────────────┬─────────────┬───────────────────┤
│  Chaos      │  Monitor    │  AI         │  Remediation     │
│  Engine     │  Layer      │  Analysis   │  Engine          │
│             │             │  Layer      │                  │
│  • 故障注入  │  • 实时指标  │  • 根因分析  │  • 自动修复      │
│  • 场景管理  │  • 日志采集  │  • 模式识别  │  • 配置回滚      │
│  • 实验调度  │  • 告警生成  │  • 决策建议  │  • 验证恢复      │
└──────┬──────┴──────┬──────┴──────┬──────┴────────┬──────────┘
       │             │             │               │
       └─────────────┴─────────────┴───────────────┘
                          │
                          ▼
               ┌─────────────────────┐
               │     Target VPS      │
               │  (被测试的生产环境)   │
               └─────────────────────┘

核心组件

  1. Chaos Engine(混沌引擎):负责注入各种类型的故障
  2. Monitor Layer(监控层):实时采集指标、日志、链路追踪数据
  3. AI Analysis Layer(AI 分析层):用 LLM 分析故障影响、定位根因、生成决策
  4. Remediation Engine(修复引擎):自动执行修复操作并验证恢复效果

三、故障注入:AI Agent 自动设计实验场景

3.1 常见故障类型

混沌工程需要注入的故障类型很多,AI Agent 可以根据系统架构自动选择最相关的故障场景:

# 故障类型定义
CHAOS_SCENARIOS = {
    "cpu_stress": {
        "name": "CPU 压力测试",
        "description": "占用 80%+ CPU 资源,模拟计算密集型负载",
        "methods": ["stress-ng --cpu 4 --timeout 60s", "yes > /dev/null &"],
        "risk_level": "low",
        "duration": "60s"
    },
    "memory_pressure": {
        "name": "内存压力测试",
        "description": "分配大量内存直到触发 OOM,模拟内存泄漏",
        "methods": ["stress-ng --vm 2 --vm-bytes 80%", "dd if=/dev/zero of=/tmp/bigfile bs=1M count=2000"],
        "risk_level": "medium",
        "duration": "120s"
    },
    "disk_full": {
        "name": "磁盘空间耗尽",
        "description": "填满磁盘到 95%+,模拟磁盘满导致的写入失败",
        "methods": ["dd if=/dev/zero of=/tmp/filldisk bs=1M count=10000"],
        "risk_level": "high",
        "duration": "30s"
    },
    "network_partition": {
        "name": "网络分区",
        "description": "阻断特定端口或 IP,模拟网络中断",
        "methods": ["iptables -A OUTPUT -d 10.0.0.5 -j DROP", "tc qdisc add dev eth0 root netem delay 5000ms"],
        "risk_level": "high",
        "duration": "30s"
    },
    "process_kill": {
        "name": "进程终止",
        "description": "随机终止关键服务进程,模拟进程崩溃",
        "methods": ["pkill -f nginx", "systemctl stop postgresql"],
        "risk_level": "high",
        "duration": "immediate"
    },
    "dependency_failure": {
        "name": "依赖服务故障",
        "description": "模拟 Redis/MySQL 等依赖服务不可用",
        "methods": ["docker kill redis", "systemctl stop mysql"],
        "risk_level": "medium",
        "duration": "30s"
    },
    "clock_skew": {
        "name": "时钟偏移",
        "description": "修改系统时间,模拟时钟漂移导致的认证失败",
        "methods": ["date -s '2027-01-01 00:00:00'", "chronyd -s"],
        "risk_level": "medium",
        "duration": "10s"
    }
}

3.2 AI Agent 智能选择故障场景

AI Agent 不会随机注入故障——它会根据当前系统状态和架构选择最合适的故障场景

import asyncio
from langchain.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

class ChaosAgent:
    def __init__(self, llm):
        self.llm = llm
        self.system_state = {}
    
    async def select_chaos_scenario(self, system_info: dict) -> dict:
        """AI 根据系统状态选择最合适的混沌实验场景"""
        prompt = ChatPromptTemplate.from_messages([
            ("system", """你是一个混沌工程专家。
            根据当前系统状态,选择最合适的故障注入场景。
            优先选择可能暴露系统脆弱点的场景,而非显而易见的问题。
            返回 JSON 格式:{{"scenario": "场景名", "reason": "选择理由", "expected_impact": "预期影响", "safety_check": "安全检查项"}}"""),
            ("human", """当前系统状态:
            - 架构: {architecture}
            - 关键服务: {services}
            - 当前负载: {load}
            - 最近故障: {recent_incidents}
            - 备份状态: {backup_status}
            - 可用资源: {available_resources}
            
            请分析并选择最合适的混沌实验场景。""")
        ])
        
        chain = prompt | self.llm
        response = await chain.ainvoke({
            "architecture": system_info.get("architecture", ""),
            "services": ", ".join(system_info.get("services", [])),
            "load": system_info.get("load", ""),
            "recent_incidents": system_info.get("recent_incidents", "无"),
            "backup_status": system_info.get("backup_status", ""),
            "available_resources": system_info.get("available_resources", "")
        })
        
        return response.content
    
    async def validate_safety(self, scenario: dict, system_info: dict) -> bool:
        """AI 验证故障注入是否安全"""
        safety_checks = [
            "确认有最近的备份",
            "确认关键服务有重启机制",
            "确认注入时间不在业务高峰期",
            "确认有回滚方案",
            "确认监控告警已启用"
        ]
        
        prompt = ChatPromptTemplate.from_messages([
            ("system", "你是混沌工程安全专家。检查以下故障注入是否安全。"),
            ("human", """场景: {scenario}
            系统状态: {system_info}
            安全检查项: {safety_checks}
            
            请逐项检查,返回安全/不安全及理由。""")
        ])
        
        chain = prompt | self.llm
        response = await chain.ainvoke({
            "scenario": scenario,
            "system_info": system_info,
            "safety_checks": "\n".join(safety_checks)
        })
        
        return "安全" in response.content

3.3 自动化故障注入执行

import subprocess
import asyncio
from dataclasses import dataclass
from enum import Enum

class RiskLevel(Enum):
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"

@dataclass
class ChaosResult:
    scenario: str
    status: str  # success, failed, interrupted
    duration: float
    impact: dict
    recovery_time: float
    errors: list

class ChaosEngine:
    def __init__(self):
        self.executed_scenarios = []
        self.safety_lock = asyncio.Lock()
    
    async def execute_scenario(self, scenario_name: str, params: dict = None) -> ChaosResult:
        """执行混沌实验"""
        import time
        start_time = time.time()
        
        try:
            # 获取故障注入命令
            commands = self._get_commands(scenario_name)
            
            # 执行故障注入(异步)
            tasks = [self._run_command(cmd) for cmd in commands]
            await asyncio.gather(*tasks, return_exceptions=True)
            
            # 等待预期持续时间
            await asyncio.sleep(scenario.get("duration", 30))
            
            # 清理故障注入
            await self._cleanup(scenario_name)
            
            duration = time.time() - start_time
            
            return ChaosResult(
                scenario=scenario_name,
                status="success",
                duration=duration,
                impact=await self._measure_impact(),
                recovery_time=duration,
                errors=[]
            )
            
        except Exception as e:
            return ChaosResult(
                scenario=scenario_name,
                status="failed",
                duration=time.time() - start_time,
                impact={},
                recovery_time=0,
                errors=[str(e)]
            )
    
    def _get_commands(self, scenario: str) -> list:
        """根据场景返回故障注入命令"""
        commands_map = {
            "cpu_stress": ["stress-ng --cpu 4 --timeout 60s"],
            "memory_pressure": ["stress-ng --vm 2 --vm-bytes 80% --timeout 120s"],
            "disk_full": ["dd if=/dev/zero of=/tmp/filldisk bs=1M count=10000"],
            "network_partition": ["iptables -A OUTPUT -d 10.0.0.5 -j DROP"],
            "process_kill": ["pkill -f nginx"],
            "dependency_failure": ["docker kill redis"],
            "clock_skew": ["date -s '2027-01-01 00:00:00'"]
        }
        return commands_map.get(scenario, [])
    
    async def _run_command(self, cmd: str):
        """异步执行命令"""
        proc = await asyncio.create_subprocess_shell(
            cmd,
            stdout=asyncio.subprocess.PIPE,
            stderr=asyncio.subprocess.PIPE
        )
        stdout, stderr = await proc.communicate()
        return stdout, stderr
    
    async def _cleanup(self, scenario: str):
        """清理故障注入,恢复系统"""
        cleanup_commands = {
            "cpu_stress": ["pkill stress-ng"],
            "memory_pressure": ["pkill stress-ng", "swapoff -a && swapon -a"],
            "disk_full": ["rm -f /tmp/filldisk"],
            "network_partition": ["iptables -D OUTPUT -d 10.0.0.5 -j DROP"],
            "process_kill": ["systemctl start nginx"],
            "dependency_failure": ["docker start redis"],
            "clock_skew": ["systemctl restart chronyd"]
        }
        for cmd in cleanup_commands.get(scenario, []):
            await self._run_command(cmd)
    
    async def _measure_impact(self) -> dict:
        """测量故障注入的影响"""
        return {
            "cpu_usage": await self._get_metric("cpu"),
            "memory_usage": await self._get_metric("memory"),
            "disk_usage": await self._get_metric("disk"),
            "response_time": await self._get_metric("response_time"),
            "error_rate": await self._get_metric("error_rate"),
            "service_status": await self._check_services()
        }

四、监控层:实时数据采集与异常检测

4.1 多维度数据采集

混沌工程需要全面的数据采集来评估故障影响:

import psutil
import asyncio
import aiohttp
from datetime import datetime, timedelta

class MonitorLayer:
    def __init__(self):
        self.metrics_history = []
        self.baseline = {}
    
    async def collect_all_metrics(self) -> dict:
        """采集所有维度的监控数据"""
        metrics = {
            "timestamp": datetime.now().isoformat(),
            "system": await self._collect_system_metrics(),
            "application": await self._collect_app_metrics(),
            "network": await self._collect_network_metrics(),
            "logs": await self._collect_recent_logs(),
            "services": await self._check_service_status()
        }
        
        self.metrics_history.append(metrics)
        return metrics
    
    async def _collect_system_metrics(self) -> dict:
        """系统级指标"""
        return {
            "cpu_percent": psutil.cpu_percent(interval=1),
            "cpu_count": psutil.cpu_count(),
            "memory_total": psutil.virtual_memory().total,
            "memory_available": psutil.virtual_memory().available,
            "memory_percent": psutil.virtual_memory().percent,
            "disk_total": psutil.disk_usage('/').total,
            "disk_used": psutil.disk_usage('/').used,
            "disk_percent": psutil.disk_usage('/').percent,
            "load_avg": psutil.getloadavg(),
            "processes": psutil.pids()
        }
    
    async def _collect_app_metrics(self) -> dict:
        """应用级指标"""
        # 采集应用健康检查
        health_checks = {}
        services = ["nginx", "postgresql", "redis", "app"]
        
        for service in services:
            health_checks[service] = await self._check_service_health(service)
        
        return health_checks
    
    async def _collect_network_metrics(self) -> dict:
        """网络指标"""
        net_io = psutil.net_io_counters()
        return {
            "bytes_sent": net_io.bytes_sent,
            "bytes_recv": net_io.bytes_recv,
            "packets_sent": net_io.packets_sent,
            "packets_recv": net_io.packets_recv,
            "connections": len(psutil.net_connections())
        }
    
    async def _collect_recent_logs(self, minutes: int = 5) -> list:
        """采集近期日志"""
        cmd = f"journalctl -u nginx --since '{(datetime.now() - timedelta(minutes=minutes)).isoformat()}' --no-pager -n 50"
        try:
            proc = await asyncio.create_subprocess_shell(
                cmd,
                stdout=asyncio.subprocess.PIPE,
                stderr=asyncio.subprocess.PIPE
            )
            stdout, _ = await proc.communicate()
            return stdout.decode().strip().split('\n')[:20]
        except:
            return []
    
    async def _check_service_health(self, service: str) -> dict:
        """检查服务健康状态"""
        import aiohttp
        
        # 检查服务是否运行
        running = await self._is_service_running(service)
        
        # 如果是 Web 服务,检查 HTTP 响应
        http_status = None
        if service in ["nginx", "app"] and running:
            http_status = await self._check_http_health(service)
        
        return {
            "running": running,
            "http_status": http_status,
            "uptime": await self._get_uptime(service)
        }
    
    async def _is_service_running(self, service: str) -> bool:
        """检查服务是否运行"""
        import aiohttp
        proc = await asyncio.create_subprocess_shell(
            f"systemctl is-active {service}",
            stdout=asyncio.subprocess.PIPE,
            stderr=asyncio.subprocess.PIPE
        )
        stdout, _ = await proc.communicate()
        return stdout.decode().strip() == "active"

4.2 基线对比与异常检测

class AnomalyDetector:
    def __init__(self, window_size: int = 100):
        self.window_size = window_size
        self.baselines = {}
    
    def update_baseline(self, metrics: dict):
        """更新基线"""
        for key, value in metrics.items():
            if isinstance(value, (int, float)):
                if key not in self.baselines:
                    self.baselines[key] = []
                self.baselines[key].append(value)
                # 保持滑动窗口
                if len(self.baselines[key]) > self.window_size:
                    self.baselines[key] = self.baselines[key][-self.window_size:]
    
    def detect_anomaly(self, current_metrics: dict) -> dict:
        """检测异常"""
        anomalies = {}
        
        for key, value in current_metrics.items():
            if key not in self.baselines or len(self.baselines[key]) < 10:
                continue
            
            baseline = self.baselines[key]
            mean = sum(baseline) / len(baseline)
            std = (sum((x - mean) ** 2 for x in baseline) / len(baseline)) ** 0.5
            
            if std == 0:
                std = 1
            
            # Z-score 异常检测
            z_score = abs(value - mean) / std
            
            if z_score > 3:  # 超过 3 个标准差视为异常
                anomalies[key] = {
                    "current": value,
                    "baseline_mean": mean,
                    "z_score": z_score,
                    "severity": "high" if z_score > 5 else "medium"
                }
        
        return anomalies
    
    def get_alert_message(self, anomalies: dict) -> str:
        """生成告警消息"""
        if not anomalies:
            return "系统运行正常,未检测到异常"
        
        messages = []
        for key, info in anomalies.items():
            severity = "⚠️" if info["severity"] == "medium" else "🚨"
            messages.append(
                f"{severity} {key}: 当前值 {info['current']:.2f}, "
                f"基线均值 {info['baseline_mean']:.2f}, "
                f"Z-score {info['z_score']:.2f}"
            )
        
        return "\n".join(messages)

五、AI 分析层:根因分析与智能决策

5.1 LLM 驱动的根因分析

from langchain.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
import json

class AIAnalysisLayer:
    def __init__(self, llm: ChatOpenAI):
        self.llm = llm
    
    async def analyze_chaos_result(self, chaos_result: ChaosResult, 
                                    metrics_before: dict, 
                                    metrics_after: dict,
                                    anomalies: dict) -> dict:
        """AI 分析混沌实验结果"""
        
        # 准备分析上下文
        context = f"""
        混沌实验: {chaos_result.scenario}
        实验结果: {chaos_result.status}
        实验时长: {chaos_result.duration:.1f}s
        恢复时长: {chaos_result.recovery_time:.1f}s
        注入的错误: {', '.join(chaos_result.errors) if chaos_result.errors else '无'}
        
        实验前指标: {json.dumps(metrics_before, default=str, indent=2)}
        实验后指标: {json.dumps(metrics_after, default=str, indent=2)}
        
        检测到的异常:
        {json.dumps(anomalies, indent=2) if anomalies else '无异常'}
        """
        
        prompt = ChatPromptTemplate.from_messages([
            ("system", """你是一个 SRE 专家,擅长混沌工程和根因分析。
            分析以下混沌实验结果,回答:
            1. 系统是否出现了预期的故障行为?
            2. 故障的影响范围和严重程度?
            3. 根因是什么?
            4. 系统表现出了哪些韧性特征(如自动重启、降级、熔断)?
            5. 建议的改进措施?
            
            返回 JSON 格式。"""),
            ("human", context)
        ])
        
        chain = prompt | self.llm
        response = await chain.ainvoke({})
        
        # 解析响应
        try:
            analysis = json.loads(response.content)
        except:
            analysis = {"raw_analysis": response.content}
        
        return analysis
    
    async def generate_resilience_report(self, all_results: list) -> str:
        """生成韧性报告"""
        
        prompt = ChatPromptTemplate.from_messages([
            ("system", """你是 SRE 专家,根据混沌工程实验结果生成韧性评估报告。
            报告应包含:
            1. 实验概览
            2. 发现的脆弱点
            3. 系统的韧性表现
            4. 改进建议
            5. 优先级排序"""),
            ("human", """以下是混沌工程实验结果汇总:
            {results}
            
            请生成完整的韧性评估报告。""")
        ])
        
        chain = prompt | self.llm
        response = await chain.ainvoke({"results": all_results})
        
        return response.content

5.2 智能决策:何时注入、何时中止

class ChaosDecisionEngine:
    def __init__(self, llm: ChatOpenAI):
        self.llm = llm
        self.experiment_log = []
    
    async def should_inject(self, system_state: dict, pending_scenarios: list) -> dict:
        """决策是否应该注入下一个故障"""
        
        prompt = ChatPromptTemplate.from_messages([
            ("system", """你是混沌工程调度专家。
            根据系统当前状态,决定是否注入下一个故障。
            考虑因素:
            1. 系统当前负载是否过高?
            2. 是否正在进行关键业务操作?
            3. 之前的实验是否发现了未修复的问题?
            4. 是否有足够的监控覆盖?
            
            返回 JSON:{{"should_inject": true/false, "reason": "理由", "next_scenario": "场景名", "priority": "high/medium/low"}}"""),
            ("human", """当前系统状态:
            - CPU: {cpu}%
            - 内存: {memory}%
            - 磁盘: {disk}%
            - 当前时间: {time}
            - 最近实验结果: {last_results}
            - 待注入场景: {pending}""")
        ])
        
        chain = prompt | self.llm
        response = await chain.ainvoke({
            "cpu": system_state.get("cpu", 0),
            "memory": system_state.get("memory", 0),
            "disk": system_state.get("disk", 0),
            "time": system_state.get("time", ""),
            "last_results": str(self.experiment_log[-3:]) if self.experiment_log else "无",
            "pending": str(pending_scenarios)
        })
        
        try:
            decision = json.loads(response.content)
        except:
            decision = {"should_inject": False, "reason": "解析失败", "next_scenario": "", "priority": "low"}
        
        return decision
    
    async def should_abort(self, current_metrics: dict, anomalies: dict) -> bool:
        """决策是否应该中止当前实验"""
        
        # 硬性中止条件
        hard_abort_conditions = [
            ("disk_percent", 95),
            ("memory_percent", 95),
            ("cpu_percent", 98)
        ]
        
        for metric, threshold in hard_abort_conditions:
            if metric in current_metrics and current_metrics[metric] > threshold:
                return True
        
        # AI 辅助决策
        prompt = ChatPromptTemplate.from_messages([
            ("system", "你是混沌工程安全专家。判断是否应该中止当前实验。"),
            ("human", """当前指标: {metrics}
            异常检测: {anomalies}
            
            如果风险过高,返回"ABORT",否则返回"CONTINUE"。""")
        ])
        
        chain = prompt | self.llm
        response = await chain.ainvoke({
            "metrics": current_metrics,
            "anomalies": anomalies
        })
        
        return "ABORT" in response.content

六、自动修复:从故障注入到自愈闭环

6.1 自动修复引擎

class RemediationEngine:
    def __init__(self):
        self.rem mediation_history = []
    
    async def auto_remediate(self, chaos_result: ChaosResult, analysis: dict) -> dict:
        """自动修复混沌实验引入的故障"""
        
        remediation_plan = analysis.get("remediation_plan", [])
        results = {}
        
        for step in remediation_plan:
            action = step.get("action", "")
            command = step.get("command", "")
            
            try:
                # 执行修复命令
                proc = await asyncio.create_subprocess_shell(
                    command,
                    stdout=asyncio.subprocess.PIPE,
                    stderr=asyncio.subprocess.PIPE
                )
                stdout, stderr = await proc.communicate()
                
                # 验证修复效果
                post_fix_metrics = await self._collect_metrics()
                is_fixed = self._verify_remediation(chaos_result, post_fix_metrics)
                
                results[action] = {
                    "command": command,
                    "success": proc.returncode == 0,
                    "verified": is_fixed,
                    "stdout": stdout.decode()[:500],
                    "stderr": stderr.decode()[:500]
                }
                
                self.rem mediation_history.append({
                    "action": action,
                    "command": command,
                    "success": proc.returncode == 0,
                    "verified": is_fixed,
                    "timestamp": datetime.now().isoformat()
                })
                
            except Exception as e:
                results[action] = {
                    "success": False,
                    "error": str(e)
                }
        
        return results
    
    def _verify_remediation(self, chaos_result: ChaosResult, metrics: dict) -> bool:
        """验证修复是否成功"""
        # 检查关键指标是否恢复
        critical_metrics = ["cpu_percent", "memory_percent", "disk_percent"]
        
        for metric in critical_metrics:
            if metric in metrics:
                if metrics[metric] > 80:  # 恢复阈值
                    return False
        
        return True
    
    async def _collect_metrics(self) -> dict:
        """采集修复后的指标"""
        return {
            "cpu_percent": psutil.cpu_percent(interval=1),
            "memory_percent": psutil.virtual_memory().percent,
            "disk_percent": psutil.disk_usage('/').percent
        }

6.2 完整闭环流程

import asyncio
from datetime import datetime

class ChaosEngineeringLoop:
    """混沌工程完整闭环"""
    
    def __init__(self):
        self.chaos_agent = ChaosAgent(llm=ChatOpenAI(model="gpt-4o"))
        self.chaos_engine = ChaosEngine()
        self.monitor = MonitorLayer()
        self.anomaly_detector = AnomalyDetector()
        self.ai_analyzer = AIAnalysisLayer(llm=ChatOpenAI(model="gpt-4o"))
        self.decision_engine = ChaosDecisionEngine(llm=ChatOpenAI(model="gpt-4o"))
        self.remediation = RemediationEngine()
    
    async def run_experiment(self, scenario: str, params: dict = None) -> dict:
        """运行一次完整的混沌实验"""
        
        print(f"🔬 开始混沌实验: {scenario}")
        
        # 1. 收集基线指标
        print("📊 收集基线指标...")
        baseline_metrics = await self.monitor.collect_all_metrics()
        self.anomaly_detector.update_baseline(baseline_metrics["system"])
        
        # 2. 执行故障注入
        print(f"💥 注入故障: {scenario}")
        chaos_result = await self.chaos_engine.execute_scenario(scenario, params)
        
        # 3. 采集故障期间指标
        print("📈 采集故障期间指标...")
        during_metrics = await self.monitor.collect_all_metrics()
        anomalies = self.anomaly_detector.detect_anomaly(during_metrics["system"])
        
        # 4. AI 分析
        print("🧠 AI 分析中...")
        analysis = await self.ai_analyzer.analyze_chaos_result(
            chaos_result, baseline_metrics["system"],
            during_metrics["system"], anomalies
        )
        
        # 5. 自动修复
        print("🔧 自动修复中...")
        remediation_results = await self.remediation.auto_remediate(chaos_result, analysis)
        
        # 6. 验证恢复
        print("✅ 验证恢复状态...")
        post_remediation = await self.monitor.collect_all_metrics()
        is_resilient = self._check_resilience(baseline_metrics, post_remediation)
        
        # 7. 生成报告
        report = {
            "scenario": scenario,
            "timestamp": datetime.now().isoformat(),
            "chaos_result": chaos_result.__dict__,
            "analysis": analysis,
            "remediation": remediation_results,
            "is_resilient": is_resilient,
            "recommendations": analysis.get("recommendations", [])
        }
        
        print(f"📋 实验完成: {'✅ 通过' if is_resilient else '❌ 未通过'}")
        return report
    
    def _check_resilience(self, before: dict, after: dict) -> bool:
        """检查系统韧性"""
        # 简单判断:关键指标是否恢复到基线水平
        resilience_thresholds = {
            "cpu_percent": 30,
            "memory_percent": 50,
            "disk_percent": 80
        }
        
        for metric, threshold in resilience_thresholds.items():
            before_val = before.get(metric, 0)
            after_val = after.get(metric, 0)
            
            # 恢复后不应超过基线的 120%
            if after_val > before_val * 1.2 and after_val > threshold:
                return False
        
        return True
    
    async def run_battery_test(self, scenarios: list, interval: int = 60):
        """运行混沌实验电池测试"""
        results = []
        
        for scenario in scenarios:
            # AI 决策是否继续
            system_state = await self.monitor.collect_all_metrics()
            decision = await self.decision_engine.should_inject(
                system_state["system"], scenarios[len(results):]
            )
            
            if not decision.get("should_inject", False):
                print(f"⏸️ 跳过 {scenario}: {decision.get('reason', '未知原因')}")
                continue
            
            # 执行实验
            report = await self.run_experiment(scenario)
            results.append(report)
            
            # 间隔等待
            if interval > 0:
                await asyncio.sleep(interval)
        
        # 生成总体报告
        summary = await self.ai_analyzer.generate_resilience_report(results)
        
        return {
            "experiments": results,
            "summary": summary,
            "total": len(results),
            "passed": sum(1 for r in results if r["is_resilient"]),
            "failed": sum(1 for r in results if not r["is_resilient"])
        }

七、实战部署:在 VPS 上搭建混沌工程平台

7.1 项目结构

chaos-vps/
├── chaos_engine/
│   ├── __init__.py
│   ├── agent.py          # AI Agent
│   ├── engine.py         # 故障注入引擎
│   ├── monitor.py        # 监控层
│   ├── analyzer.py       # AI 分析层
│   ├── remediation.py    # 自动修复
│   └── decision.py       # 决策引擎
├── scenarios/
│   ├── cpu_stress.py
│   ├── memory_pressure.py
│   ├── disk_full.py
│   ├── network_partition.py
│   └── process_kill.py
├── config.yaml           # 配置文件
├── main.py               # 主入口
└── requirements.txt

7.2 配置文件

# config.yaml
chaos:
  # 安全设置
  safety:
    max_cpu_threshold: 90
    max_memory_threshold: 90
    max_disk_threshold: 90
    abort_on_critical: true
    require_manual_approval: false
  
  # 调度设置
  scheduling:
    default_interval: 300  # 实验间隔(秒)
    max_concurrent: 1      # 最大并发实验数
    preferred_time_range:   # 推荐执行时间
      start: "02:00"
      end: "06:00"
  
  # 场景设置
  scenarios:
    enabled:
      - cpu_stress
      - memory_pressure
      - disk_full
      - process_kill
    disabled:
      - clock_skew  # 高风险场景,默认禁用
    
  # LLM 设置
  llm:
    model: "gpt-4o"
    temperature: 0.3
    max_tokens: 2048
  
  # 监控设置
  monitoring:
    collection_interval: 10  # 秒
    baseline_window: 100     # 基线窗口大小
    anomaly_z_threshold: 3   # Z-score 异常阈值
  
  # 修复设置
  remediation:
    auto_remediate: true
    verify_after_fix: true
    max_retry: 3

7.3 主入口

# main.py
import asyncio
import yaml
from chaos_engine.agent import ChaosAgent
from chaos_engine.engine import ChaosEngine
from chaos_engine.monitor import MonitorLayer
from chaos_engine.analyzer import AIAnalysisLayer
from chaos_engine.remediation import RemediationEngine
from chaos_engine.decision import ChaosDecisionEngine

async def main():
    # 加载配置
    with open("config.yaml") as f:
        config = yaml.safe_load(f)
    
    # 初始化各组件
    monitor = MonitorLayer()
    anomaly_detector = AnomalyDetector()
    
    chaos_engine = ChaosEngine()
    chaos_agent = ChaosAgent(llm=None)  # 可选:LLM 辅助场景选择
    ai_analyzer = AIAnalysisLayer(llm=None)
    decision_engine = ChaosDecisionEngine(llm=None)
    remediation = RemediationEngine()
    
    print("🚀 VPS 混沌工程平台启动")
    print(f"📋 启用场景: {config['chaos']['scenarios']['enabled']}")
    print(f"⏰ 安全阈值: CPU={config['chaos']['safety']['max_cpu_threshold']}%")
    print(f"🔒 自动修复: {config['chaos']['remediation']['auto_remediate']}")
    
    # 运行混沌实验电池
    results = await chaos_engine.run_battery_test(
        scenarios=config['chaos']['scenarios']['enabled'],
        interval=config['chaos']['scheduling']['default_interval']
    )
    
    # 输出报告
    print("\n" + "="*50)
    print("📊 混沌工程实验报告")
    print("="*50)
    print(f"总实验数: {results['total']}")
    print(f"通过: {results['passed']}")
    print(f"未通过: {results['failed']}")
    print(f"\n韧性评估总结:\n{results['summary']}")

if __name__ == "__main__":
    asyncio.run(main())

八、实战案例:一次完整的混沌实验

8.1 场景:CPU 压力测试

🔬 开始混沌实验: cpu_stress
📊 收集基线指标...
   CPU: 12%, 内存: 45%, 磁盘: 62%
💥 注入故障: stress-ng --cpu 4 --timeout 60s
📈 采集故障期间指标...
   CPU: 95%, 内存: 48%, 磁盘: 62%
   异常检测: cpu_percent Z-score=8.2 (high)
🧠 AI 分析中...
   - 系统 CPU 使用率从 12% 飙升至 95%
   - API 响应时间从 50ms 增加到 800ms
   - 未检测到服务崩溃或数据丢失
   - 系统表现出良好的韧性:服务未崩溃,仅在压力下变慢
   - 建议:考虑增加 CPU 核心数或配置 CPU 限制
🔧 自动修复中...
   - 执行: pkill stress-ng
   - 验证: CPU 恢复正常 (15%)
✅ 验证恢复状态...
📋 实验完成: ✅ 通过

8.2 场景:磁盘空间耗尽

🔬 开始混沌实验: disk_full
📊 收集基线指标...
   CPU: 15%, 内存: 48%, 磁盘: 62%
💥 注入故障: dd if=/dev/zero of=/tmp/filldisk bs=1M count=10000
📈 采集故障期间指标...
   CPU: 25%, 内存: 50%, 磁盘: 96%
   异常检测: disk_percent Z-score=12.5 (high)
   服务状态: nginx 写入失败, PostgreSQL WAL 写入失败
🧠 AI 分析中...
   - 磁盘空间耗尽导致多个服务写入失败
   - PostgreSQL 进入只读模式
   - Nginx 无法写入访问日志
   - 系统韧性不足:关键服务因磁盘满而失效
   - 建议:
     1. 配置磁盘使用率告警(>80%)
     2. 实施日志轮转和清理策略
     3. 为 PostgreSQL 配置磁盘空间监控
🔧 自动修复中...
   - 执行: rm -f /tmp/filldisk
   - 执行: systemctl restart postgresql
   - 执行: systemctl restart nginx
   - 验证: 所有服务恢复正常
✅ 验证恢复状态...
📋 实验完成: ❌ 未通过(但已自动修复)

九、最佳实践与注意事项

9.1 混沌工程最佳实践

实践说明
从低风险开始先注入 CPU/内存压力,再尝试网络分区
建立基线在实验前采集正常状态的指标基线
逐步扩大范围单服务 → 多服务 → 全系统
保持可逆确保每次实验都能快速恢复
持续进行混沌工程不是一次性的,应定期执行
记录所有实验建立实验日志,跟踪系统韧性变化

9.2 安全注意事项

  1. 生产环境谨慎:首次运行建议在测试环境,生产环境需人工审批
  2. 备份先行:确保有最近的备份,防止数据丢失
  3. 业务低峰期:选择业务低峰期执行高风险实验
  4. 监控完备:确保有完整的监控和告警,能及时发现问题
  5. 手动中止:保留手动中止开关,AI 决策不应完全替代人工判断

9.3 与现有运维体系的集成

┌──────────────────────────────────────────────────────┐
│              混沌工程与现有运维集成                    │
├──────────────────────────────────────────────────────┤
│                                                      │
│  Prometheus ──▶ Chaos Engine(数据采集)              │
│      │                                              │
│      ▼                                              │
│  Grafana ──▶ 混沌实验结果可视化                      │
│      │                                              │
│      ▼                                              │
│  Alertmanager ──▶ AI Agent(智能告警)               │
│      │                                              │
│      ▼                                              │
│  PagerDuty/Opsgenie ──▶ 自动修复触发                 │
│      │                                              │
│      ▼                                              │
│  Ansible/Terraform ──▶ 配置回滚与修复                │
│      │                                              │
│      ▼                                              │
│  Git ──▶ 实验报告版本化存储                          │
│                                                      │
└──────────────────────────────────────────────────────┘

十、总结

AI 驱动的混沌工程让 VPS 运维从"被动救火"走向"主动防御":

  1. AI Agent 智能选择故障场景——不是随机注入,而是根据系统状态选择最有价值的实验
  2. 自动化故障注入与恢复——从注入到清理全程自动化,无需人工干预
  3. LLM 根因分析——理解故障影响,定位根本原因,生成改进建议
  4. 闭环自愈——故障注入后立即自动修复,验证系统的韧性
  5. 持续改进——每次实验都积累数据,持续优化系统架构

核心收益

  • 在真实故障发生前发现系统脆弱点
  • 减少生产环境故障率和 MTTR(平均修复时间)
  • 建立对系统韧性的量化认知
  • 培养"故障是常态,韧性是关键"的运维文化

下一步行动

  1. 在测试环境部署混沌工程平台
  2. 从 CPU 压力测试开始,逐步增加场景
  3. 将实验结果纳入运维决策
  4. 建立定期混沌工程实验机制

参考资源

📺 看视频版教程 → DuckDB Lab YouTube

Subscribe for more DuckDB & AI automation tutorials