引言
你的 VPS 服务真的"扛得住"吗?
- 数据库突然慢查询,API 响应时间从 50ms 飙到 5s;
- 磁盘空间被日志占满,服务全部宕机;
- 某个依赖的微服务超时,级联故障拖垮整个系统;
- 内存泄漏缓慢累积,一周后网站彻底无法访问;
- SSL 证书过期、 Cron 任务失败、进程死锁……
这些故障不是会不会发生的问题,而是什么时候发生的问题。传统运维的应对方式是"出了问题再修"——但用户不会等你排查。而 混沌工程(Chaos Engineering) 的理念是:在故障发生前主动制造故障,验证系统的韧性,提前发现脆弱点。
2026 年,大语言模型(LLM)和 AI Agent 技术让混沌工程变得前所未有地简单。你不再需要手动编写故障脚本、手动观察结果、手动分析根因——AI Agent 可以全自动完成整个混沌工程闭环。
本文将带你构建一套 AI 驱动的 VPS 混沌工程系统,实现从故障注入、行为监测、根因分析到自动恢复的全流程自动化。
一、什么是混沌工程?为什么 VPS 需要它?
1.1 混沌工程的核心理念
混沌工程起源于 Netflix 的 Chaos Monkey 项目,核心理念是:
在受控环境中主动注入故障,验证系统是否能在真实故障中保持可用。
传统测试关注"系统是否能正常工作",而混沌工程关注**“系统在不正常时能否保持可用”**。
| 传统测试 | 混沌工程 |
|---|---|
| 验证系统正常工作 | 验证系统故障时仍可用 |
| 关注功能正确性 | 关注韧性(Resilience) |
| 一次性测试 | 持续性验证 |
| 人工执行 | 可自动化持续执行 |
1.2 VPS 场景下的混沌工程价值
对于 VPS 用户来说,混沌工程有独特的价值:
- 单点故障风险高:大多数 VPS 用户只有一台服务器,任何故障都意味着全量中断
- 资源受限:VPS 资源有限,过载时容易产生级联故障
- 缺乏冗余:没有多副本、没有自动故障转移,一次故障可能就是永久宕机
- 运维人力有限:个人运维者或小团队难以 24 小时盯守
AI 驱动的混沌工程可以帮助你在故障真正发生之前发现系统的脆弱点,并自动修复。
二、系统架构:AI 驱动的混沌工程平台
┌─────────────────────────────────────────────────────────────┐
│ AI Chaos Engineering Platform │
├─────────────┬─────────────┬─────────────┬───────────────────┤
│ Chaos │ Monitor │ AI │ Remediation │
│ Engine │ Layer │ Analysis │ Engine │
│ │ │ Layer │ │
│ • 故障注入 │ • 实时指标 │ • 根因分析 │ • 自动修复 │
│ • 场景管理 │ • 日志采集 │ • 模式识别 │ • 配置回滚 │
│ • 实验调度 │ • 告警生成 │ • 决策建议 │ • 验证恢复 │
└──────┬──────┴──────┬──────┴──────┬──────┴────────┬──────────┘
│ │ │ │
└─────────────┴─────────────┴───────────────┘
│
▼
┌─────────────────────┐
│ Target VPS │
│ (被测试的生产环境) │
└─────────────────────┘
核心组件
- Chaos Engine(混沌引擎):负责注入各种类型的故障
- Monitor Layer(监控层):实时采集指标、日志、链路追踪数据
- AI Analysis Layer(AI 分析层):用 LLM 分析故障影响、定位根因、生成决策
- Remediation Engine(修复引擎):自动执行修复操作并验证恢复效果
三、故障注入:AI Agent 自动设计实验场景
3.1 常见故障类型
混沌工程需要注入的故障类型很多,AI Agent 可以根据系统架构自动选择最相关的故障场景:
# 故障类型定义
CHAOS_SCENARIOS = {
"cpu_stress": {
"name": "CPU 压力测试",
"description": "占用 80%+ CPU 资源,模拟计算密集型负载",
"methods": ["stress-ng --cpu 4 --timeout 60s", "yes > /dev/null &"],
"risk_level": "low",
"duration": "60s"
},
"memory_pressure": {
"name": "内存压力测试",
"description": "分配大量内存直到触发 OOM,模拟内存泄漏",
"methods": ["stress-ng --vm 2 --vm-bytes 80%", "dd if=/dev/zero of=/tmp/bigfile bs=1M count=2000"],
"risk_level": "medium",
"duration": "120s"
},
"disk_full": {
"name": "磁盘空间耗尽",
"description": "填满磁盘到 95%+,模拟磁盘满导致的写入失败",
"methods": ["dd if=/dev/zero of=/tmp/filldisk bs=1M count=10000"],
"risk_level": "high",
"duration": "30s"
},
"network_partition": {
"name": "网络分区",
"description": "阻断特定端口或 IP,模拟网络中断",
"methods": ["iptables -A OUTPUT -d 10.0.0.5 -j DROP", "tc qdisc add dev eth0 root netem delay 5000ms"],
"risk_level": "high",
"duration": "30s"
},
"process_kill": {
"name": "进程终止",
"description": "随机终止关键服务进程,模拟进程崩溃",
"methods": ["pkill -f nginx", "systemctl stop postgresql"],
"risk_level": "high",
"duration": "immediate"
},
"dependency_failure": {
"name": "依赖服务故障",
"description": "模拟 Redis/MySQL 等依赖服务不可用",
"methods": ["docker kill redis", "systemctl stop mysql"],
"risk_level": "medium",
"duration": "30s"
},
"clock_skew": {
"name": "时钟偏移",
"description": "修改系统时间,模拟时钟漂移导致的认证失败",
"methods": ["date -s '2027-01-01 00:00:00'", "chronyd -s"],
"risk_level": "medium",
"duration": "10s"
}
}
3.2 AI Agent 智能选择故障场景
AI Agent 不会随机注入故障——它会根据当前系统状态和架构选择最合适的故障场景:
import asyncio
from langchain.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
class ChaosAgent:
def __init__(self, llm):
self.llm = llm
self.system_state = {}
async def select_chaos_scenario(self, system_info: dict) -> dict:
"""AI 根据系统状态选择最合适的混沌实验场景"""
prompt = ChatPromptTemplate.from_messages([
("system", """你是一个混沌工程专家。
根据当前系统状态,选择最合适的故障注入场景。
优先选择可能暴露系统脆弱点的场景,而非显而易见的问题。
返回 JSON 格式:{{"scenario": "场景名", "reason": "选择理由", "expected_impact": "预期影响", "safety_check": "安全检查项"}}"""),
("human", """当前系统状态:
- 架构: {architecture}
- 关键服务: {services}
- 当前负载: {load}
- 最近故障: {recent_incidents}
- 备份状态: {backup_status}
- 可用资源: {available_resources}
请分析并选择最合适的混沌实验场景。""")
])
chain = prompt | self.llm
response = await chain.ainvoke({
"architecture": system_info.get("architecture", ""),
"services": ", ".join(system_info.get("services", [])),
"load": system_info.get("load", ""),
"recent_incidents": system_info.get("recent_incidents", "无"),
"backup_status": system_info.get("backup_status", ""),
"available_resources": system_info.get("available_resources", "")
})
return response.content
async def validate_safety(self, scenario: dict, system_info: dict) -> bool:
"""AI 验证故障注入是否安全"""
safety_checks = [
"确认有最近的备份",
"确认关键服务有重启机制",
"确认注入时间不在业务高峰期",
"确认有回滚方案",
"确认监控告警已启用"
]
prompt = ChatPromptTemplate.from_messages([
("system", "你是混沌工程安全专家。检查以下故障注入是否安全。"),
("human", """场景: {scenario}
系统状态: {system_info}
安全检查项: {safety_checks}
请逐项检查,返回安全/不安全及理由。""")
])
chain = prompt | self.llm
response = await chain.ainvoke({
"scenario": scenario,
"system_info": system_info,
"safety_checks": "\n".join(safety_checks)
})
return "安全" in response.content
3.3 自动化故障注入执行
import subprocess
import asyncio
from dataclasses import dataclass
from enum import Enum
class RiskLevel(Enum):
LOW = "low"
MEDIUM = "medium"
HIGH = "high"
@dataclass
class ChaosResult:
scenario: str
status: str # success, failed, interrupted
duration: float
impact: dict
recovery_time: float
errors: list
class ChaosEngine:
def __init__(self):
self.executed_scenarios = []
self.safety_lock = asyncio.Lock()
async def execute_scenario(self, scenario_name: str, params: dict = None) -> ChaosResult:
"""执行混沌实验"""
import time
start_time = time.time()
try:
# 获取故障注入命令
commands = self._get_commands(scenario_name)
# 执行故障注入(异步)
tasks = [self._run_command(cmd) for cmd in commands]
await asyncio.gather(*tasks, return_exceptions=True)
# 等待预期持续时间
await asyncio.sleep(scenario.get("duration", 30))
# 清理故障注入
await self._cleanup(scenario_name)
duration = time.time() - start_time
return ChaosResult(
scenario=scenario_name,
status="success",
duration=duration,
impact=await self._measure_impact(),
recovery_time=duration,
errors=[]
)
except Exception as e:
return ChaosResult(
scenario=scenario_name,
status="failed",
duration=time.time() - start_time,
impact={},
recovery_time=0,
errors=[str(e)]
)
def _get_commands(self, scenario: str) -> list:
"""根据场景返回故障注入命令"""
commands_map = {
"cpu_stress": ["stress-ng --cpu 4 --timeout 60s"],
"memory_pressure": ["stress-ng --vm 2 --vm-bytes 80% --timeout 120s"],
"disk_full": ["dd if=/dev/zero of=/tmp/filldisk bs=1M count=10000"],
"network_partition": ["iptables -A OUTPUT -d 10.0.0.5 -j DROP"],
"process_kill": ["pkill -f nginx"],
"dependency_failure": ["docker kill redis"],
"clock_skew": ["date -s '2027-01-01 00:00:00'"]
}
return commands_map.get(scenario, [])
async def _run_command(self, cmd: str):
"""异步执行命令"""
proc = await asyncio.create_subprocess_shell(
cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE
)
stdout, stderr = await proc.communicate()
return stdout, stderr
async def _cleanup(self, scenario: str):
"""清理故障注入,恢复系统"""
cleanup_commands = {
"cpu_stress": ["pkill stress-ng"],
"memory_pressure": ["pkill stress-ng", "swapoff -a && swapon -a"],
"disk_full": ["rm -f /tmp/filldisk"],
"network_partition": ["iptables -D OUTPUT -d 10.0.0.5 -j DROP"],
"process_kill": ["systemctl start nginx"],
"dependency_failure": ["docker start redis"],
"clock_skew": ["systemctl restart chronyd"]
}
for cmd in cleanup_commands.get(scenario, []):
await self._run_command(cmd)
async def _measure_impact(self) -> dict:
"""测量故障注入的影响"""
return {
"cpu_usage": await self._get_metric("cpu"),
"memory_usage": await self._get_metric("memory"),
"disk_usage": await self._get_metric("disk"),
"response_time": await self._get_metric("response_time"),
"error_rate": await self._get_metric("error_rate"),
"service_status": await self._check_services()
}
四、监控层:实时数据采集与异常检测
4.1 多维度数据采集
混沌工程需要全面的数据采集来评估故障影响:
import psutil
import asyncio
import aiohttp
from datetime import datetime, timedelta
class MonitorLayer:
def __init__(self):
self.metrics_history = []
self.baseline = {}
async def collect_all_metrics(self) -> dict:
"""采集所有维度的监控数据"""
metrics = {
"timestamp": datetime.now().isoformat(),
"system": await self._collect_system_metrics(),
"application": await self._collect_app_metrics(),
"network": await self._collect_network_metrics(),
"logs": await self._collect_recent_logs(),
"services": await self._check_service_status()
}
self.metrics_history.append(metrics)
return metrics
async def _collect_system_metrics(self) -> dict:
"""系统级指标"""
return {
"cpu_percent": psutil.cpu_percent(interval=1),
"cpu_count": psutil.cpu_count(),
"memory_total": psutil.virtual_memory().total,
"memory_available": psutil.virtual_memory().available,
"memory_percent": psutil.virtual_memory().percent,
"disk_total": psutil.disk_usage('/').total,
"disk_used": psutil.disk_usage('/').used,
"disk_percent": psutil.disk_usage('/').percent,
"load_avg": psutil.getloadavg(),
"processes": psutil.pids()
}
async def _collect_app_metrics(self) -> dict:
"""应用级指标"""
# 采集应用健康检查
health_checks = {}
services = ["nginx", "postgresql", "redis", "app"]
for service in services:
health_checks[service] = await self._check_service_health(service)
return health_checks
async def _collect_network_metrics(self) -> dict:
"""网络指标"""
net_io = psutil.net_io_counters()
return {
"bytes_sent": net_io.bytes_sent,
"bytes_recv": net_io.bytes_recv,
"packets_sent": net_io.packets_sent,
"packets_recv": net_io.packets_recv,
"connections": len(psutil.net_connections())
}
async def _collect_recent_logs(self, minutes: int = 5) -> list:
"""采集近期日志"""
cmd = f"journalctl -u nginx --since '{(datetime.now() - timedelta(minutes=minutes)).isoformat()}' --no-pager -n 50"
try:
proc = await asyncio.create_subprocess_shell(
cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE
)
stdout, _ = await proc.communicate()
return stdout.decode().strip().split('\n')[:20]
except:
return []
async def _check_service_health(self, service: str) -> dict:
"""检查服务健康状态"""
import aiohttp
# 检查服务是否运行
running = await self._is_service_running(service)
# 如果是 Web 服务,检查 HTTP 响应
http_status = None
if service in ["nginx", "app"] and running:
http_status = await self._check_http_health(service)
return {
"running": running,
"http_status": http_status,
"uptime": await self._get_uptime(service)
}
async def _is_service_running(self, service: str) -> bool:
"""检查服务是否运行"""
import aiohttp
proc = await asyncio.create_subprocess_shell(
f"systemctl is-active {service}",
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE
)
stdout, _ = await proc.communicate()
return stdout.decode().strip() == "active"
4.2 基线对比与异常检测
class AnomalyDetector:
def __init__(self, window_size: int = 100):
self.window_size = window_size
self.baselines = {}
def update_baseline(self, metrics: dict):
"""更新基线"""
for key, value in metrics.items():
if isinstance(value, (int, float)):
if key not in self.baselines:
self.baselines[key] = []
self.baselines[key].append(value)
# 保持滑动窗口
if len(self.baselines[key]) > self.window_size:
self.baselines[key] = self.baselines[key][-self.window_size:]
def detect_anomaly(self, current_metrics: dict) -> dict:
"""检测异常"""
anomalies = {}
for key, value in current_metrics.items():
if key not in self.baselines or len(self.baselines[key]) < 10:
continue
baseline = self.baselines[key]
mean = sum(baseline) / len(baseline)
std = (sum((x - mean) ** 2 for x in baseline) / len(baseline)) ** 0.5
if std == 0:
std = 1
# Z-score 异常检测
z_score = abs(value - mean) / std
if z_score > 3: # 超过 3 个标准差视为异常
anomalies[key] = {
"current": value,
"baseline_mean": mean,
"z_score": z_score,
"severity": "high" if z_score > 5 else "medium"
}
return anomalies
def get_alert_message(self, anomalies: dict) -> str:
"""生成告警消息"""
if not anomalies:
return "系统运行正常,未检测到异常"
messages = []
for key, info in anomalies.items():
severity = "⚠️" if info["severity"] == "medium" else "🚨"
messages.append(
f"{severity} {key}: 当前值 {info['current']:.2f}, "
f"基线均值 {info['baseline_mean']:.2f}, "
f"Z-score {info['z_score']:.2f}"
)
return "\n".join(messages)
五、AI 分析层:根因分析与智能决策
5.1 LLM 驱动的根因分析
from langchain.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
import json
class AIAnalysisLayer:
def __init__(self, llm: ChatOpenAI):
self.llm = llm
async def analyze_chaos_result(self, chaos_result: ChaosResult,
metrics_before: dict,
metrics_after: dict,
anomalies: dict) -> dict:
"""AI 分析混沌实验结果"""
# 准备分析上下文
context = f"""
混沌实验: {chaos_result.scenario}
实验结果: {chaos_result.status}
实验时长: {chaos_result.duration:.1f}s
恢复时长: {chaos_result.recovery_time:.1f}s
注入的错误: {', '.join(chaos_result.errors) if chaos_result.errors else '无'}
实验前指标: {json.dumps(metrics_before, default=str, indent=2)}
实验后指标: {json.dumps(metrics_after, default=str, indent=2)}
检测到的异常:
{json.dumps(anomalies, indent=2) if anomalies else '无异常'}
"""
prompt = ChatPromptTemplate.from_messages([
("system", """你是一个 SRE 专家,擅长混沌工程和根因分析。
分析以下混沌实验结果,回答:
1. 系统是否出现了预期的故障行为?
2. 故障的影响范围和严重程度?
3. 根因是什么?
4. 系统表现出了哪些韧性特征(如自动重启、降级、熔断)?
5. 建议的改进措施?
返回 JSON 格式。"""),
("human", context)
])
chain = prompt | self.llm
response = await chain.ainvoke({})
# 解析响应
try:
analysis = json.loads(response.content)
except:
analysis = {"raw_analysis": response.content}
return analysis
async def generate_resilience_report(self, all_results: list) -> str:
"""生成韧性报告"""
prompt = ChatPromptTemplate.from_messages([
("system", """你是 SRE 专家,根据混沌工程实验结果生成韧性评估报告。
报告应包含:
1. 实验概览
2. 发现的脆弱点
3. 系统的韧性表现
4. 改进建议
5. 优先级排序"""),
("human", """以下是混沌工程实验结果汇总:
{results}
请生成完整的韧性评估报告。""")
])
chain = prompt | self.llm
response = await chain.ainvoke({"results": all_results})
return response.content
5.2 智能决策:何时注入、何时中止
class ChaosDecisionEngine:
def __init__(self, llm: ChatOpenAI):
self.llm = llm
self.experiment_log = []
async def should_inject(self, system_state: dict, pending_scenarios: list) -> dict:
"""决策是否应该注入下一个故障"""
prompt = ChatPromptTemplate.from_messages([
("system", """你是混沌工程调度专家。
根据系统当前状态,决定是否注入下一个故障。
考虑因素:
1. 系统当前负载是否过高?
2. 是否正在进行关键业务操作?
3. 之前的实验是否发现了未修复的问题?
4. 是否有足够的监控覆盖?
返回 JSON:{{"should_inject": true/false, "reason": "理由", "next_scenario": "场景名", "priority": "high/medium/low"}}"""),
("human", """当前系统状态:
- CPU: {cpu}%
- 内存: {memory}%
- 磁盘: {disk}%
- 当前时间: {time}
- 最近实验结果: {last_results}
- 待注入场景: {pending}""")
])
chain = prompt | self.llm
response = await chain.ainvoke({
"cpu": system_state.get("cpu", 0),
"memory": system_state.get("memory", 0),
"disk": system_state.get("disk", 0),
"time": system_state.get("time", ""),
"last_results": str(self.experiment_log[-3:]) if self.experiment_log else "无",
"pending": str(pending_scenarios)
})
try:
decision = json.loads(response.content)
except:
decision = {"should_inject": False, "reason": "解析失败", "next_scenario": "", "priority": "low"}
return decision
async def should_abort(self, current_metrics: dict, anomalies: dict) -> bool:
"""决策是否应该中止当前实验"""
# 硬性中止条件
hard_abort_conditions = [
("disk_percent", 95),
("memory_percent", 95),
("cpu_percent", 98)
]
for metric, threshold in hard_abort_conditions:
if metric in current_metrics and current_metrics[metric] > threshold:
return True
# AI 辅助决策
prompt = ChatPromptTemplate.from_messages([
("system", "你是混沌工程安全专家。判断是否应该中止当前实验。"),
("human", """当前指标: {metrics}
异常检测: {anomalies}
如果风险过高,返回"ABORT",否则返回"CONTINUE"。""")
])
chain = prompt | self.llm
response = await chain.ainvoke({
"metrics": current_metrics,
"anomalies": anomalies
})
return "ABORT" in response.content
六、自动修复:从故障注入到自愈闭环
6.1 自动修复引擎
class RemediationEngine:
def __init__(self):
self.rem mediation_history = []
async def auto_remediate(self, chaos_result: ChaosResult, analysis: dict) -> dict:
"""自动修复混沌实验引入的故障"""
remediation_plan = analysis.get("remediation_plan", [])
results = {}
for step in remediation_plan:
action = step.get("action", "")
command = step.get("command", "")
try:
# 执行修复命令
proc = await asyncio.create_subprocess_shell(
command,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE
)
stdout, stderr = await proc.communicate()
# 验证修复效果
post_fix_metrics = await self._collect_metrics()
is_fixed = self._verify_remediation(chaos_result, post_fix_metrics)
results[action] = {
"command": command,
"success": proc.returncode == 0,
"verified": is_fixed,
"stdout": stdout.decode()[:500],
"stderr": stderr.decode()[:500]
}
self.rem mediation_history.append({
"action": action,
"command": command,
"success": proc.returncode == 0,
"verified": is_fixed,
"timestamp": datetime.now().isoformat()
})
except Exception as e:
results[action] = {
"success": False,
"error": str(e)
}
return results
def _verify_remediation(self, chaos_result: ChaosResult, metrics: dict) -> bool:
"""验证修复是否成功"""
# 检查关键指标是否恢复
critical_metrics = ["cpu_percent", "memory_percent", "disk_percent"]
for metric in critical_metrics:
if metric in metrics:
if metrics[metric] > 80: # 恢复阈值
return False
return True
async def _collect_metrics(self) -> dict:
"""采集修复后的指标"""
return {
"cpu_percent": psutil.cpu_percent(interval=1),
"memory_percent": psutil.virtual_memory().percent,
"disk_percent": psutil.disk_usage('/').percent
}
6.2 完整闭环流程
import asyncio
from datetime import datetime
class ChaosEngineeringLoop:
"""混沌工程完整闭环"""
def __init__(self):
self.chaos_agent = ChaosAgent(llm=ChatOpenAI(model="gpt-4o"))
self.chaos_engine = ChaosEngine()
self.monitor = MonitorLayer()
self.anomaly_detector = AnomalyDetector()
self.ai_analyzer = AIAnalysisLayer(llm=ChatOpenAI(model="gpt-4o"))
self.decision_engine = ChaosDecisionEngine(llm=ChatOpenAI(model="gpt-4o"))
self.remediation = RemediationEngine()
async def run_experiment(self, scenario: str, params: dict = None) -> dict:
"""运行一次完整的混沌实验"""
print(f"🔬 开始混沌实验: {scenario}")
# 1. 收集基线指标
print("📊 收集基线指标...")
baseline_metrics = await self.monitor.collect_all_metrics()
self.anomaly_detector.update_baseline(baseline_metrics["system"])
# 2. 执行故障注入
print(f"💥 注入故障: {scenario}")
chaos_result = await self.chaos_engine.execute_scenario(scenario, params)
# 3. 采集故障期间指标
print("📈 采集故障期间指标...")
during_metrics = await self.monitor.collect_all_metrics()
anomalies = self.anomaly_detector.detect_anomaly(during_metrics["system"])
# 4. AI 分析
print("🧠 AI 分析中...")
analysis = await self.ai_analyzer.analyze_chaos_result(
chaos_result, baseline_metrics["system"],
during_metrics["system"], anomalies
)
# 5. 自动修复
print("🔧 自动修复中...")
remediation_results = await self.remediation.auto_remediate(chaos_result, analysis)
# 6. 验证恢复
print("✅ 验证恢复状态...")
post_remediation = await self.monitor.collect_all_metrics()
is_resilient = self._check_resilience(baseline_metrics, post_remediation)
# 7. 生成报告
report = {
"scenario": scenario,
"timestamp": datetime.now().isoformat(),
"chaos_result": chaos_result.__dict__,
"analysis": analysis,
"remediation": remediation_results,
"is_resilient": is_resilient,
"recommendations": analysis.get("recommendations", [])
}
print(f"📋 实验完成: {'✅ 通过' if is_resilient else '❌ 未通过'}")
return report
def _check_resilience(self, before: dict, after: dict) -> bool:
"""检查系统韧性"""
# 简单判断:关键指标是否恢复到基线水平
resilience_thresholds = {
"cpu_percent": 30,
"memory_percent": 50,
"disk_percent": 80
}
for metric, threshold in resilience_thresholds.items():
before_val = before.get(metric, 0)
after_val = after.get(metric, 0)
# 恢复后不应超过基线的 120%
if after_val > before_val * 1.2 and after_val > threshold:
return False
return True
async def run_battery_test(self, scenarios: list, interval: int = 60):
"""运行混沌实验电池测试"""
results = []
for scenario in scenarios:
# AI 决策是否继续
system_state = await self.monitor.collect_all_metrics()
decision = await self.decision_engine.should_inject(
system_state["system"], scenarios[len(results):]
)
if not decision.get("should_inject", False):
print(f"⏸️ 跳过 {scenario}: {decision.get('reason', '未知原因')}")
continue
# 执行实验
report = await self.run_experiment(scenario)
results.append(report)
# 间隔等待
if interval > 0:
await asyncio.sleep(interval)
# 生成总体报告
summary = await self.ai_analyzer.generate_resilience_report(results)
return {
"experiments": results,
"summary": summary,
"total": len(results),
"passed": sum(1 for r in results if r["is_resilient"]),
"failed": sum(1 for r in results if not r["is_resilient"])
}
七、实战部署:在 VPS 上搭建混沌工程平台
7.1 项目结构
chaos-vps/
├── chaos_engine/
│ ├── __init__.py
│ ├── agent.py # AI Agent
│ ├── engine.py # 故障注入引擎
│ ├── monitor.py # 监控层
│ ├── analyzer.py # AI 分析层
│ ├── remediation.py # 自动修复
│ └── decision.py # 决策引擎
├── scenarios/
│ ├── cpu_stress.py
│ ├── memory_pressure.py
│ ├── disk_full.py
│ ├── network_partition.py
│ └── process_kill.py
├── config.yaml # 配置文件
├── main.py # 主入口
└── requirements.txt
7.2 配置文件
# config.yaml
chaos:
# 安全设置
safety:
max_cpu_threshold: 90
max_memory_threshold: 90
max_disk_threshold: 90
abort_on_critical: true
require_manual_approval: false
# 调度设置
scheduling:
default_interval: 300 # 实验间隔(秒)
max_concurrent: 1 # 最大并发实验数
preferred_time_range: # 推荐执行时间
start: "02:00"
end: "06:00"
# 场景设置
scenarios:
enabled:
- cpu_stress
- memory_pressure
- disk_full
- process_kill
disabled:
- clock_skew # 高风险场景,默认禁用
# LLM 设置
llm:
model: "gpt-4o"
temperature: 0.3
max_tokens: 2048
# 监控设置
monitoring:
collection_interval: 10 # 秒
baseline_window: 100 # 基线窗口大小
anomaly_z_threshold: 3 # Z-score 异常阈值
# 修复设置
remediation:
auto_remediate: true
verify_after_fix: true
max_retry: 3
7.3 主入口
# main.py
import asyncio
import yaml
from chaos_engine.agent import ChaosAgent
from chaos_engine.engine import ChaosEngine
from chaos_engine.monitor import MonitorLayer
from chaos_engine.analyzer import AIAnalysisLayer
from chaos_engine.remediation import RemediationEngine
from chaos_engine.decision import ChaosDecisionEngine
async def main():
# 加载配置
with open("config.yaml") as f:
config = yaml.safe_load(f)
# 初始化各组件
monitor = MonitorLayer()
anomaly_detector = AnomalyDetector()
chaos_engine = ChaosEngine()
chaos_agent = ChaosAgent(llm=None) # 可选:LLM 辅助场景选择
ai_analyzer = AIAnalysisLayer(llm=None)
decision_engine = ChaosDecisionEngine(llm=None)
remediation = RemediationEngine()
print("🚀 VPS 混沌工程平台启动")
print(f"📋 启用场景: {config['chaos']['scenarios']['enabled']}")
print(f"⏰ 安全阈值: CPU={config['chaos']['safety']['max_cpu_threshold']}%")
print(f"🔒 自动修复: {config['chaos']['remediation']['auto_remediate']}")
# 运行混沌实验电池
results = await chaos_engine.run_battery_test(
scenarios=config['chaos']['scenarios']['enabled'],
interval=config['chaos']['scheduling']['default_interval']
)
# 输出报告
print("\n" + "="*50)
print("📊 混沌工程实验报告")
print("="*50)
print(f"总实验数: {results['total']}")
print(f"通过: {results['passed']}")
print(f"未通过: {results['failed']}")
print(f"\n韧性评估总结:\n{results['summary']}")
if __name__ == "__main__":
asyncio.run(main())
八、实战案例:一次完整的混沌实验
8.1 场景:CPU 压力测试
🔬 开始混沌实验: cpu_stress
📊 收集基线指标...
CPU: 12%, 内存: 45%, 磁盘: 62%
💥 注入故障: stress-ng --cpu 4 --timeout 60s
📈 采集故障期间指标...
CPU: 95%, 内存: 48%, 磁盘: 62%
异常检测: cpu_percent Z-score=8.2 (high)
🧠 AI 分析中...
- 系统 CPU 使用率从 12% 飙升至 95%
- API 响应时间从 50ms 增加到 800ms
- 未检测到服务崩溃或数据丢失
- 系统表现出良好的韧性:服务未崩溃,仅在压力下变慢
- 建议:考虑增加 CPU 核心数或配置 CPU 限制
🔧 自动修复中...
- 执行: pkill stress-ng
- 验证: CPU 恢复正常 (15%)
✅ 验证恢复状态...
📋 实验完成: ✅ 通过
8.2 场景:磁盘空间耗尽
🔬 开始混沌实验: disk_full
📊 收集基线指标...
CPU: 15%, 内存: 48%, 磁盘: 62%
💥 注入故障: dd if=/dev/zero of=/tmp/filldisk bs=1M count=10000
📈 采集故障期间指标...
CPU: 25%, 内存: 50%, 磁盘: 96%
异常检测: disk_percent Z-score=12.5 (high)
服务状态: nginx 写入失败, PostgreSQL WAL 写入失败
🧠 AI 分析中...
- 磁盘空间耗尽导致多个服务写入失败
- PostgreSQL 进入只读模式
- Nginx 无法写入访问日志
- 系统韧性不足:关键服务因磁盘满而失效
- 建议:
1. 配置磁盘使用率告警(>80%)
2. 实施日志轮转和清理策略
3. 为 PostgreSQL 配置磁盘空间监控
🔧 自动修复中...
- 执行: rm -f /tmp/filldisk
- 执行: systemctl restart postgresql
- 执行: systemctl restart nginx
- 验证: 所有服务恢复正常
✅ 验证恢复状态...
📋 实验完成: ❌ 未通过(但已自动修复)
九、最佳实践与注意事项
9.1 混沌工程最佳实践
| 实践 | 说明 |
|---|---|
| 从低风险开始 | 先注入 CPU/内存压力,再尝试网络分区 |
| 建立基线 | 在实验前采集正常状态的指标基线 |
| 逐步扩大范围 | 单服务 → 多服务 → 全系统 |
| 保持可逆 | 确保每次实验都能快速恢复 |
| 持续进行 | 混沌工程不是一次性的,应定期执行 |
| 记录所有实验 | 建立实验日志,跟踪系统韧性变化 |
9.2 安全注意事项
- 生产环境谨慎:首次运行建议在测试环境,生产环境需人工审批
- 备份先行:确保有最近的备份,防止数据丢失
- 业务低峰期:选择业务低峰期执行高风险实验
- 监控完备:确保有完整的监控和告警,能及时发现问题
- 手动中止:保留手动中止开关,AI 决策不应完全替代人工判断
9.3 与现有运维体系的集成
┌──────────────────────────────────────────────────────┐
│ 混沌工程与现有运维集成 │
├──────────────────────────────────────────────────────┤
│ │
│ Prometheus ──▶ Chaos Engine(数据采集) │
│ │ │
│ ▼ │
│ Grafana ──▶ 混沌实验结果可视化 │
│ │ │
│ ▼ │
│ Alertmanager ──▶ AI Agent(智能告警) │
│ │ │
│ ▼ │
│ PagerDuty/Opsgenie ──▶ 自动修复触发 │
│ │ │
│ ▼ │
│ Ansible/Terraform ──▶ 配置回滚与修复 │
│ │ │
│ ▼ │
│ Git ──▶ 实验报告版本化存储 │
│ │
└──────────────────────────────────────────────────────┘
十、总结
AI 驱动的混沌工程让 VPS 运维从"被动救火"走向"主动防御":
- AI Agent 智能选择故障场景——不是随机注入,而是根据系统状态选择最有价值的实验
- 自动化故障注入与恢复——从注入到清理全程自动化,无需人工干预
- LLM 根因分析——理解故障影响,定位根本原因,生成改进建议
- 闭环自愈——故障注入后立即自动修复,验证系统的韧性
- 持续改进——每次实验都积累数据,持续优化系统架构
核心收益:
- 在真实故障发生前发现系统脆弱点
- 减少生产环境故障率和 MTTR(平均修复时间)
- 建立对系统韧性的量化认知
- 培养"故障是常态,韧性是关键"的运维文化
下一步行动:
- 在测试环境部署混沌工程平台
- 从 CPU 压力测试开始,逐步增加场景
- 将实验结果纳入运维决策
- 建立定期混沌工程实验机制
参考资源:
