Introduction

As your VPS runs, data only grows: application logs, database backups, user uploads, model inference caches, CI build artifacts… Most people’s storage strategy is “store everything in standard storage, never delete.”

The result is the classic triple loss:

• Cold data occupying hot storage: S3 objects not read in 3+ months, still billed at standard storage rates;
• Hot data trapped in archives: Backups uploaded yesterday manually demoted to Glacier, requiring 3-5 hours to retrieve;
• Rule maintenance by hand: S3 lifecycle rules hardcoded in config files — business changes, access patterns shift, rules stay stale.

AI data lifecycle management takes a different approach: let a local LLM analyze each bucket’s and file class’s access heat distribution (read frequency, write frequency, last access time, data volume), combine that with business semantics to judge data value, and auto-generate hot/cold tiering plans and S3 lifecycle rules — hot data on standard/IA, cold data to archives, expired data auto-deleted — zero data exfiltration, all local inference.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    VPS Storage Layer                         │
│  /var/backups/  /opt/app/data/  S3 buckets (std/IA/archive)│
└──────────────┬──────────────────────────────────────────────┘
               │ collect (90-day access records + metadata)
               ▼
┌─────────────────────────────────────────────────────────────┐
│            Data Heat Analyzer (Python)                       │
│  ┌──────────┐  ┌──────────┐  ┌──────────────────┐          │
│  │ read freq │  │ write    │  │ last access time │          │
│  │(accesslog)│  │(mtime)   │  │(atime)          │          │
│  └──────┬───┘  └──────┬───┘  └────────┬─────────┘          │
└─────────┼──────────────┼───────────────┼────────────────────┘
          ▼              ▼               ▼
┌─────────────────────────────────────────────────────────────┐
│          Ollama + Qwen2.5 (Local Inference)                  │
│  Input: file metadata + access stats + business tags        │
│  Output: JSON tiering plan (hot/warm/cold/expire + conf)    │
└────────────────────────────┬────────────────────────────────┘
                             ▼
┌─────────────────────────────────────────────────────────────┐
│            Strategy Generator (Python)                       │
│  → S3 lifecycle rules (transition + expiration)            │
│  → Local restic archival policy                              │
│  → Hot/cold directory split + file-move cron jobs            │
└────────────────────────────┬────────────────────────────────┘
                             ▼
┌─────────────────────────────────────────────────────────────┐
│            Alert Channel (Telegram Bot)                      │
│  Tiering change notice / cost estimate / expiry warnings    │
└─────────────────────────────────────────────────────────────┘

Implementation Steps

1. Data Collection: Access Heat Profiles

First, map the access profile for every bucket and directory:

# collect_access_profile.py
import boto3, time, json, os
from collections import defaultdict

s3 = boto3.client('s3', endpoint_url='http://minio:9000')
BUCKET = 'vps-data'
NOW = time.time()

profiles = []
resp = s3.get_paginator('list_objects_v2').paginate(Bucket=BUCKET)
for obj in resp:
    last_access = obj.get('LastModified', NOW - 90*86400)  # simplified
    age_days = (NOW - last_access) / 86400
    profiles.append({
        'key': obj['Key'],
        'size': obj['Size'],
        'age_days': round(age_days, 1),
    })

# Aggregate by directory
by_dir = defaultdict(lambda: {'count': 0, 'total_size': 0, 'min_age': float('inf')})
for p in profiles:
    d = p['key'].split('/')[0] if '/' in p['key'] else '(root)'
    by_dir[d]['count'] += 1
    by_dir[d]['total_size'] += p['size']
    by_dir[d]['min_age'] = min(by_dir[d]['min_age'], p['age_days'])

with open('data/access_profiles.json', 'w') as f:
    json.dump(by_dir, f, indent=2)
print(f"Collected {len(profiles)} objects across {len(by_dir)} directories")

Key fields:

• age_days: days since last modify/access;
• total_size: total capacity per directory — determines how much money tiering actually saves;
• count: object count — distinguishes “few large files” from “massive small files.”

2. LLM Semantic Tiering

Feed profiles to the local LLM to judge heat level and data value per directory:

# llm_tier.py
import json, requests

SYSTEM = """You are a storage lifecycle expert. Given access stats and business tags,
output for each directory:
1. tier: hot(read-freq within 7d) / warm(7-30d) / cold(30-90d) / archive(>90d)
2. expire_days: recommended retention (0=immediate expiry, null=never delete)
3. reason: one sentence (<=30 words)
Output strict JSON, no markdown."""

with open('data/access_profiles.json') as f:
    profiles = json.load(f)

payload = json.dumps(profiles, ensure_ascii=False)
r = requests.post('http://localhost:11434/api/generate', json={
    'model': 'qwen2.5:7b',
    'prompt': f"{SYSTEM}\n\nData:\n{payload}\n\nOutput:"
}, timeout=120)
tiers = r.json()['response'].strip()
tiers = json.loads(tiers[tiers.index('{'):])
print(json.dumps(tiers, ensure_ascii=False, indent=2))

The LLM outputs something like:

{
  "backups":    {"tier": "archive", "expire_days": 90,  "reason": "30 daily backup copies is plenty"},
  "uploads":    {"tier": "cold",    "expire_days": 365, "reason": "User docs unused for a year, archive them"},
  "cache":      {"tier": "hot",     "expire_days": 7,   "reason": "Inference cache, hot R/W, expires in 7d"},
  "logs":       {"tier": "warm",    "expire_days": 30,  "reason": "Troubleshooting window, 30 days is sound"}
}

Note: attach business tags to the prompt (e.g., backups labeled “daily database backups”). The LLM combines semantics, not just numbers — this is the core advantage over pure rule engines.

3. Generate S3 Lifecycle Rules

Convert LLM tiering suggestions into executable S3 lifecycle rules:

# gen_s3_lifecycle.py
import json, boto3

with open('data/tier_result.json') as f:
    tiers = json.load(f)

s3 = boto3.client('s3', endpoint_url='http://minio:9000')
RULES = []
for prefix, t in tiers.items():
    rule = {
        'ID': f"llm-{prefix}",
        'Filter': {'Prefix': prefix + '/'},
        'Status': 'Enabled',
        'Transitions': [],
    }
    # hot: no transition; warm -> IA; cold/archive -> decide by expire_days
    if t['tier'] in ('cold', 'archive'):
        rule['Transitions'].append({
            'Prefix': prefix + '/',
            'StorageClass': 'INTELLIGENT_TIERING' if t['tier'] == 'cold' else 'GLACIER_IR',
            'Days': 0
        })
    if t.get('expire_days'):
        rule['Expiration'] = {'Days': t['expire_days']}
    RULES.append(rule)

s3.put_bucket_lifecycle_configuration(
    Bucket='vps-data',
    LifecycleConfiguration={'Rules': RULES}
)
print(f"Applied {len(RULES)} lifecycle rules")

Rule logic:

• hot directories: no transition, no deletion, stay on standard storage;
• cold directories: INTELLIGENT_TIERING — AWS auto-switches between standard/IA based on actual access, no retrieval fees;
• archive directories: GLACIER_IR (low throughput archive), storage price 60%+ cheaper than standard;
• Directories with expire_days: auto-delete on expiry.

4. Local Archival Policy (restic)

Beyond S3, local disk data also needs management. Use restic for local archival + periodic cleanup:

# /etc/systemd/system/archive-clean.timer
[Unit]
Description=Weekly archive cleanup

[Timer]
OnCalendar=Sun 03:00
Persistent=true

[Install]
WantedBy=timers.target

# archive_local.py
import subprocess, json

subprocess.run(['restic', '-r', 'minio:vps-data/archive', 'forget',
                '--keep-daily', '7', '--keep-weekly', '4', '--keep-monthly', '6'],
               check=True, capture_output=True)
subprocess.run(['restic', '-r', 'minio:vps-data/archive', 'prune'],
               check=True, capture_output=True)

restic’s --keep-* parameters naturally implement “tiered retention” — dedup, encrypt, upload to MinIO, keep only the latest 7 days locally.

5. Cost Estimation + Telegram Report

Every tiering change triggers an auto cost-diff report pushed to Telegram:

# cost_report.py
PRICES = {  # USD per GB per month (MinIO self-hosted ~$0.02/GB, AWS S3 standard $0.23)
    'standard': 0.02, 'intelligent': 0.015, 'glacier': 0.005, 'local': 0.001
}

def estimate(tiers, profiles):
    old_cost = sum(p['total_size'] for p in profiles.values()) * 1e-9 * PRICES['standard']
    new_cost = 0
    for p, t in tiers.items():
        size_tb = profiles[p]['total_size'] * 1e-9
        cls = {'hot': 'standard', 'warm': 'standard',
               'cold': 'intelligent', 'archive': 'glacier'}[t['tier']]
        new_cost += size_tb * PRICES[cls]
    return old_cost, new_cost, (old_cost - new_cost) / old_cost * 100 if old_cost else 0

Telegram message template:

📊 Data Lifecycle Report 2026-10-18
────────────────────
Current monthly storage: $4.80
Post-tiering monthly:    $2.15 (↓55%)
  - archive: 120GB backups → Glacier  ($0.60 → $0.30)
  - cold:    80GB uploads → Intelligent ($1.92 → $1.20)
  - hot:     30GB cache stays on standard
Expiry: 90-day-old logs auto-cleanup
────────────────────
Zero data exfiltration ✓  Local Ollama inference ✓

Security Boundaries

The entire pipeline has zero data exfiltration: access logs never touch cloud vendor APIs, LLM inference runs on local Ollama, S3 lifecycle rules are pushed directly to self-hosted MinIO via boto3.

Real-World Results

Ran 90 days of data on an 8GB/2TB VPS:

Summary

Data lifecycle management fundamentally means letting data live in the price tier it deserves. The LLM’s value isn’t speed — simple age-based rules can do that — it’s semantic business judgment: two datasets both untouched for 90 days, backups/ database dumps and cache/ inference caches have wildly different value. The LLM distinguishes; pure rules can’t.

Deployment tips:

1. Run 30 days of data collection first so access profiles have enough samples;
2. Dry-run LLM tiering output for human review before applying;
3. Never touch hot directories — only migrate cold/archive;
4. Add buffer to expire_days (backups +30 days, logs +14 days);
5. Auto-run tiering + cost report monthly, push to Telegram.