Introduction
In modern cloud infrastructure, VPS (Virtual Private Server) operations management has always been a persistent challenge. As service scale and complexity grow, traditional manual operations can no longer meet the demands of today’s environments. This article introduces how to combine Large Language Model (LLM) technology to build a complete AI-driven VPS operations assistant system, achieving full-process intelligence from automated inspection to fault self-healing.
Why AI-Driven Operations?
Pain Points of Traditional Operations
- Low efficiency in manual inspections: Spending significant time daily checking server status, logs, and service health
- Lagging problem discovery: Issues are often discovered only after user complaints, lacking proactive alerts
- Time-consuming troubleshooting: Locating root causes in logs requires extensive experience and time
- Slow emergency response: Manual handling often cannot keep pace with business changes during outages
Core Value of AI Operations
- 24/7 uninterrupted monitoring: AI assistants can run around the clock, monitoring system states in real-time
- Intelligent anomaly detection: Using historical data and pattern recognition to detect potential issues early
- Fast root cause analysis: LLMs can quickly understand log content and identify problem sources
- Automated repair execution: For known issue types, fixes can be executed directly
System Architecture Design
Overall Architecture
┌─────────────────────────────────────────────────────┐
│ AI Operations Assistant │
├──────────┬──────────┬──────────┬────────────────────┤
│ Data │ Analysis │ Decision │ Execution │
│ Collection│ Engine │ Center │ │
├──────────┼──────────┼──────────┼────────────────────┤
│ Prometheus│ LLM API │ Rules │ Ansible / Shell │
│ Node │ (Qwen) │ Engine │ Playbook │
│ Exporter │ │ │ Scripts │
│ Telegraf │ │ │ Terraform │
└──────────┴──────────┴──────────┴────────────────────┘
Core Components
1. Data Collection Layer
The collection layer gathers data from various dimensions of the VPS:
- System Metrics: CPU, memory, disk, network usage
- Application Logs: Web server logs, database logs, application logs
- Service Status: Docker container status, Nginx/Apache health, database connection pools
- Security Events: Login attempts, firewall rule changes, SSL certificate status
2. Analysis Engine Layer
The analysis engine is the core of the system:
- Anomaly Detection: Statistical methods combined with LLM pattern recognition
- Log Analysis: Feeding raw logs into LLMs for key information extraction and classification
- Trend Prediction: Predicting resource usage trends based on historical data
3. Decision Center
The decision center formulates response strategies based on analysis results:
- Rule Matching: Pre-defined fix plans matched against known alert patterns
- LLM Reasoning: Context-aware analysis for unknown or complex scenarios
- Risk Assessment: Evaluating operation risk levels to determine if human approval is needed
4. Execution Layer
The execution layer safely carries out repair operations:
- Automated Scripts: Pre-written Shell/Python scripts for common failures
- Ansible Playbooks: Configuration management and batch operations via Ansible
- Human Approval: High-risk operations require manual confirmation before execution
Automated Inspection System
Inspection Task Design
Automated inspection is the foundation of the AI operations assistant. A comprehensive checklist needs to be designed:
# inspection-tasks.yaml
inspection_tasks:
- name: "system_health"
description: "System Health Check"
frequency: "*/5 * * * *" # Every 5 minutes
checks:
- type: "cpu_usage"
warning_threshold: 80
critical_threshold: 95
- type: "memory_usage"
warning_threshold: 85
critical_threshold: 95
- type: "disk_usage"
warning_threshold: 80
critical_threshold: 90
- type: "load_average"
warning_threshold: 2.0
critical_threshold: 5.0
- name: "service_status"
description: "Service Status Check"
frequency: "*/10 * * * *"
services:
- nginx
- docker
- mysql
- redis
- name: "log_analysis"
description: "Log Analysis"
frequency: "*/15 * * * *"
log_sources:
- "/var/log/nginx/access.log"
- "/var/log/nginx/error.log"
- "/var/log/syslog"
- name: "security_audit"
description: "Security Audit"
frequency: "0 */4 * * *" # Every 4 hours
checks:
- failed_login_attempts
- ssl_certificate_expiry
- firewall_rules
- open_ports
Inspection Report Generation
After each inspection, the system automatically generates a report and sends it to LLM for analysis:
import subprocess
import json
from datetime import datetime
def generate_inspection_report():
"""Generate inspection report and send to LLM for analysis"""
# Collect system metrics
metrics = {
"timestamp": datetime.now().isoformat(),
"cpu": get_cpu_stats(),
"memory": get_memory_stats(),
"disk": get_disk_stats(),
"network": get_network_stats(),
"services": check_service_status()
}
# Collect recent log entries
recent_logs = collect_recent_logs(hours=1)
# Build LLM prompt
prompt = f"""You are an experienced DevOps engineer. Please analyze the following VPS inspection data:
System Metrics:
{json.dumps(metrics, indent=2)}
Recent 1-hour log summary:
{recent_logs[:2000]}
Please provide:
1. Overall system health score (0-100)
2. Identified anomalies and their severity
3. Possible root cause analysis
4. Recommended actions
5. Whether immediate human intervention is required"""
# Call LLM API for analysis
analysis = call_llm_api(prompt)
# Save analysis report
save_report(metrics, analysis)
return analysis
Intelligent Log Analysis
Log Classification and Extraction
LLMs have natural advantages in processing unstructured logs. We can use LLMs to automatically parse various log formats:
def analyze_logs_with_llm(log_content):
"""Use LLM to analyze log content"""
prompt = f"""Please analyze the following server logs and provide structured output:
{log_content}
Return analysis results in JSON format, including:
- error_type: Error type classification
- severity: Severity level (info/warning/critical)
- source_component: The component that had the issue
- affected_service: Affected business service
- suggested_action: Recommended solution
- related_logs: Related log entries (if any)"""
response = call_llm_api(prompt)
return parse_json_response(response)
Common Log Pattern Recognition
LLMs can help identify the following common log patterns:
| Pattern Type | Example | Severity | Recommended Action |
|---|---|---|---|
| OOM Killer | Out of memory: Killed process | Critical | Increase memory or optimize application |
| Disk Full | No space left on device | Critical | Clean logs or expand disk |
| Connection Timeout | Connection timed out | Warning | Check network and backend services |
| SSL Error | SSL handshake failed | Warning | Check certificate validity |
| Permission Denied | Permission denied | Info | Check file permission configuration |
| Database Lock | Lock wait timeout exceeded | Warning | Optimize queries or increase timeout |
Real-Time Log Stream Analysis
For production environments, we also need real-time log stream analysis:
import asyncio
import logging
class RealtimeLogAnalyzer:
def __init__(self, llm_client):
self.llm = llm_client
self.alert_threshold = 5 # Alert after 5 occurrences
async def tail_log(self, log_path):
"""Real-time log file tracking"""
process = await asyncio.create_subprocess_shell(
f"tail -f {log_path}",
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE
)
error_counts = {}
while True:
line = await process.stdout.readline()
if not line:
break
log_entry = line.decode('utf-8').strip()
analysis = await self.analyze_entry(log_entry)
if analysis['severity'] in ['warning', 'critical']:
error_key = analysis['error_type']
error_counts[error_key] = error_counts.get(error_key, 0) + 1
if error_counts[error_key] >= self.alert_threshold:
await self.send_alert(analysis, error_counts[error_key])
async def analyze_entry(self, log_line):
"""Analyze single log entry"""
prompt = f"Analyze the severity and type of this log:\n{log_line}"
return self.llm.complete(prompt)
Fault Self-Healing System
Self-Healing Flow Design
Fault self-healing is an advanced feature of the AI operations assistant. It must execute repair operations automatically while ensuring safety:
Fault Detection → Root Cause Analysis → Solution Formulation → Risk Assessment → Execute Fix → Verify Result
Common Auto-Fix Scenarios
1. Nginx Service Anomaly
#!/bin/bash
# fix_nginx.sh
echo "Nginx anomaly detected, starting repair..."
# Check configuration syntax
nginx -t
if [ $? -ne 0 ]; then
echo "Nginx config has syntax errors"
cp /etc/nginx/nginx.conf.backup /etc/nginx/nginx.conf
nginx -t
fi
# Restart Nginx
systemctl restart nginx
if [ $? -eq 0 ]; then
echo "Nginx restarted successfully"
else
echo "Nginx restart failed, manual intervention required"
exit 1
fi
# Verify service health
curl -s -o /dev/null -w "%{http_code}" http://localhost/
2. Docker Container Anomaly
import docker
import requests
def auto_restart_container(container_name):
"""Automatically restart unhealthy Docker containers"""
client = docker.from_env()
try:
container = client.containers.get(container_name)
# Check container health status
health = container.attrs.get('State', {}).get('Health', {})
status = health.get('Status', '')
if status != 'healthy':
print(f"Container {container_name} is unhealthy, preparing restart...")
# Save container logs for analysis
logs = container.logs(tail=100).decode('utf-8')
# Call LLM to analyze logs
analysis = call_llm_analyze(logs, container_name)
# If LLM determines auto-fix is safe
if analysis['can_auto_fix']:
container.restart()
# Wait for container to start
for _ in range(30):
container.reload()
if container.status == 'running':
print(f"Container {container_name} restarted successfully")
return True
time.sleep(1)
# Send alert
send_alert(f"Container {container_name} restart failed", analysis)
except docker.errors.NotFound:
print(f"Container {container_name} not found")
except Exception as e:
print(f"Failed to restart container: {e}")
3. Disk Space Exhaustion
def handle_disk_full(mount_point='/'):
"""Handle disk space exhaustion"""
usage = get_disk_usage(mount_point)
if usage > 90:
# 1. Clean old logs
cleanup_old_logs(days=7)
# 2. Clean unused Docker data
subprocess.run(['docker', 'system', 'prune', '-f'],
capture_output=True)
# 3. Clean temporary files
subprocess.run(['find', '/tmp', '-mtime', '+3', '-delete'],
capture_output=True)
# 4. Re-check
new_usage = get_disk_usage(mount_point)
if new_usage < 80:
return True
else:
# Still insufficient, requires manual intervention
send_critical_alert(f"Disk usage still at {new_usage}% after cleanup")
return False
Safety Boundaries and Risk Control
Strict safety boundaries must be set when implementing automated repairs:
# safety-rules.yaml
safety_rules:
max_concurrent_operations: 3 # Max 3 concurrent repair operations
cooldown_period: 300 # Minimum 300 seconds between operations
require_approval_for:
- "deletion_operations"
- "configuration_changes"
- "production_service_restarts"
rollback_strategy:
enabled: true
auto_rollback_on_failure: true
backup_before_change: true
monitoring:
track_all_actions: true
alert_on_unusual_patterns: true
daily_report: true
LLM Integration Strategy
Model Selection
For VPS operations scenarios, the following models are recommended:
| Model | Characteristics | Use Case |
|---|---|---|
| Qwen-Max | Strong Chinese understanding, cost-effective | Daily reports, log analysis |
| Qwen-Turbo | Fast response, low cost | Real-time log stream analysis |
| DeepSeek-V3 | Strong code generation | Auto-generate repair scripts |
| GPT-4o | Strong general capabilities | Complex fault diagnosis |
API Call Optimization
To reduce API call costs, the following optimization strategies can be employed:
import time
from functools import lru_cache
class OptimizedLLMClient:
def __init__(self, api_key, base_url):
self.api_key = api_key
self.base_url = base_url
self.call_count = 0
self.last_reset = time.time()
@lru_cache(maxsize=1000)
def cached_complete(self, prompt_hash, system_prompt, user_prompt):
"""Cache responses for identical prompts to avoid redundant calls"""
return self._call_api(system_prompt, user_prompt)
def smart_batch(self, prompts):
"""Intelligent batching to reduce API call frequency"""
grouped = self.group_similar_prompts(prompts)
results = []
for group in grouped:
combined_prompt = self.combine_group(group)
result = self.cached_complete(
hash(combined_prompt),
"You are a DevOps expert...",
combined_prompt
)
results.extend(self.split_result(result, len(group)))
return results
def cost_tracker(self):
"""Track API call costs"""
elapsed = time.time() - self.last_reset
if elapsed > 3600:
self.call_count = 0
self.last_reset = time.time()
self.call_count += 1
return self.call_count
Prompt Engineering
Design dedicated prompt templates for operations scenarios:
SYSTEM_PROMPTS = {
"log_analysis": """You are a senior DevOps engineer skilled at extracting key information from logs.
Analyze the following logs, identifying error types, severity, and impact scope.
Return results in JSON format.""",
"incident_response": """You are handling a production environment incident.
Based on the following information, provide emergency response suggestions and long-term solutions.
Consider impact scope, recovery time, and business priority.""",
"capacity_planning": """You are responsible for VPS capacity planning.
Based on current resource usage trends, provide scaling recommendations and cost optimization plans.
Consider growth expectations for the next 3 months."""
}
Deployment and Configuration
Environment Requirements
- Operating System: Ubuntu 22.04 LTS or higher
- Python: 3.10+
- Docker: 20.10+
- LLM API: OpenAI-compatible interface support
- Monitoring Tools: Prometheus + Grafana (optional)
Installation Steps
# 1. Clone the project
git clone https://github.com/selfvps/vps-ops-assistant.git
cd vps-ops-assistant
# 2. Create virtual environment
python3 -m venv venv
source venv/bin/activate
# 3. Install dependencies
pip install -r requirements.txt
# 4. Configure environment variables
cat > .env << EOF
LLM_API_KEY=your-api-key-here
LLM_BASE_URL=https://api.example.com/v1
EOF
# 5. Initialize database
python manage.py migrate
# 6. Start the service
python manage.py runserver --workers 4
Docker Deployment
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
# docker-compose.yml
version: '3.8'
services:
ops-assistant:
build: .
ports:
- "8000:8000"
env_file:
- .env
volumes:
- ./data:/app/data
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- prometheus
- grafana
prometheus:
image: prom/prometheus:latest
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin123
Monitoring and Alerts
Grafana Dashboards
Recommended dashboards to create:
- System Overview: Real-time CPU, memory, disk, and network status
- Service Health: Running status and response times for all monitored services
- Alert History: Past 7-day alert statistics and trends
- LLM Analysis: AI assistant analysis results and repair records
- Cost Tracking: API call costs and saved operations man-hours
Alert Notification Channels
Multiple notification channels are supported:
- Slack/Discord: Real-time team notifications
- Email: Detailed alert reports
- SMS: Critical alerts
- Webhook: Custom integrations
Best Practices
1. Progressive Implementation
Don’t attempt to implement everything at once. Follow these steps gradually:
- Phase 1: Basic monitoring + scheduled inspections
- Phase 2: Log analysis + anomaly detection
- Phase 3: Root cause analysis + automated repair
- Phase 4: Predictive maintenance + capacity planning
2. Maintain Observability
Ensure every AI decision is traceable:
- Record complete context for each LLM call
- Save logs of all automatically executed operations
- Keep pre-and-post-repair system snapshots for comparison
3. Regular Review and Optimization
Review the AI assistant’s performance weekly:
- Accuracy Rate: Proportion of correctly identified and handled faults
- False Positive Rate: Proportion of incorrectly triggered alerts
- Mean Time to Repair: Time from issue detection to completion
- Cost Efficiency: Saved operations man-hours vs. API call costs
4. Human-AI Collaboration
Always retain the ability for human intervention:
- High-risk operations require manual confirmation
- Establish feedback mechanisms for operators to flag incorrect AI judgments
- Regularly update knowledge bases and repair procedures
Conclusion
AI-driven VPS operations assistants represent the future direction of operations automation. By integrating the powerful understanding and reasoning capabilities of LLMs with traditional monitoring and automation tools, we can build a more intelligent, efficient, and reliable operations system.
While fully unattended operations may not be realistic in the short term, AI assistants can handle most repetitive and time-consuming analytical work, allowing operations engineers to focus on more strategically valuable tasks.
Key success factors include:
- Choosing the right models and toolchains
- Designing reasonable automation boundaries and security policies
- Continuously optimizing and adjusting system parameters
- Establishing effective human-AI collaboration workflows
As AI technology continues to advance, we can expect VPS operations to become increasingly intelligent and automated, making operations teams more efficient and valuable than ever.
This article covers AI + VPS operations practices, from automated inspection to fault self-healing. For more details, visit our GitHub repository or contact the team for technical support.
