gao/ljwx-k8s

Fork 0

Files

T

bruno 2a56d7fd6c first commit

2026-02-20 22:50:45 +08:00

39 KiB

Raw Permalink Blame History

Kubernetes 集群优化路线图 2025

创建时间: 2025-12-31 适用环境: OrbStack + MacBook Pro (M3 Pro, 18GB) 执行周期: 4-6 周目标: 从企业级优秀 ⭐⭐⭐⭐⭐ 到云原生卓越 🚀

📊 当前集群状态评估

✅ 已完成的优化（基线）

维度	状态	评分
资源管理	10 ResourceQuotas + 10 LimitRanges	⭐⭐⭐⭐⭐
高可用性	16 PDBs, 6 HPAs, 多副本部署	⭐⭐⭐⭐⭐
可观测性	Prometheus + Grafana + Loki + Tempo	⭐⭐⭐⭐⭐
安全扫描	Trivy Operator (已部署)	⭐⭐⭐⭐⭐
CI/CD	ArgoCD + Gitea Runner	⭐⭐⭐⭐⭐
网络安全	10 Network Policies (基础)	⭐⭐⭐
备份恢复	数据库备份 (无集群备份)	⭐⭐⭐
资源优化	手动配置 (无自动优化)	⭐⭐⭐

⚠️ 当前问题

HPA 不工作 - 所有 6 个 HPA 显示 <unknown>，无法自动扩缩容
无集群级备份 - 数据库有备份，但整个集群配置无备份
资源配置手动 - 缺少自动资源优化（VPA）
网络策略基础 - 只有 10 个基础策略，未实现零信任架构
无策略引擎 - 缺少自动化策略管理（Kyverno）
短期监控存储 - Prometheus 只保留 7 天数据

🎯 优化目标

技术目标

✅ 100% HPA 正常工作 - 自动弹性伸缩
✅ 集群完整备份 - 每日自动备份到 MinIO
✅ 自动资源优化 - VPA 减少 50-70% 资源浪费
✅ 零信任网络 - 默认拒绝 + 白名单策略
✅ 策略自动化 - Kyverno 策略引擎
✅ 长期监控数据 - Thanos 长期存储 (30天+)

性能目标

CPU 利用率: 5-15% (当前: 6%) ✅ 已达标
内存利用率: 40-60% (当前: 33%) → 需优化
存储优化: 减少 30-50% (通过 Prometheus 优化已完成) ✅
服务可用性: 99.9%+ (通过多副本部署已实现) ✅

🚀 执行路线图

Phase 1: 关键修复（第 1 周）⚡

目标: 修复 HPA，恢复自动扩缩容能力

Task 1.1: 诊断 HPA Metrics 问题 (30分钟)

# === 步骤 1: 检查 metrics-server ===
kubectl get deployment metrics-server -n kube-system
kubectl get apiservice v1beta1.metrics.k8s.io -o jsonpath='{.status.conditions}'
kubectl logs -n kube-system deployment/metrics-server --tail=100

# === 步骤 2: 检查 node metrics ===
kubectl top nodes
# 如果失败，说明 metrics-server 有问题

# === 步骤 3: 检查 pod metrics ===
kubectl top pods -n monitoring
# 如果失败，但 node metrics 成功，可能是 cAdvisor 问题

# === 步骤 4: 检查 HPA 目标的 resource requests ===
for ns in monitoring argocd infra gitea-runner ljwx-stock; do
  echo "=== Checking $ns ==="
  kubectl get deploy -n $ns -o json | jq -r '.items[] |
    select(.metadata.name | test("prometheus|grafana|argocd-server|ingress-nginx|gitea-runner|claude-stock-backend")) |
    {name: .metadata.name, requests: .spec.template.spec.containers[0].resources.requests}'
done

可能的问题和解决方案:

问题	症状	解决方案
metrics-server 配置错误	node metrics 失败	添加 `--kubelet-insecure-tls` (OrbStack 环境)
cAdvisor 未启用	node OK, pod 失败	OrbStack 可能需要特殊配置
缺少 resource requests	HPA targets 无法计算	为所有 Deployment 添加 requests
metrics API 未注册	apiservice 不可用	重新部署 metrics-server

Task 1.2: 修复 metrics-server (OrbStack 特定) (15分钟)

# === OrbStack 环境需要特殊配置 ===

# 1. 检查当前配置
kubectl get deployment metrics-server -n kube-system -o yaml | grep -A 5 args

# 2. 添加必要的参数
kubectl patch deployment metrics-server -n kube-system --type='json' -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/args/-",
    "value": "--kubelet-insecure-tls"
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/args/-",
    "value": "--kubelet-preferred-address-types=InternalIP"
  }
]'

# 3. 等待 Pod 重启
kubectl rollout status deployment metrics-server -n kube-system

# 4. 验证
sleep 30
kubectl top nodes
kubectl top pods -n monitoring

Task 1.3: 确保所有 HPA 目标有 resource requests (30分钟)

# === 创建修复脚本 ===
cat > /tmp/fix-hpa-resources.sh << 'EOF'
#!/bin/bash

set -e

echo "=== 修复 HPA 目标的 resource requests ==="

# 函数: 检查并添加 resource requests
add_resources() {
  local namespace=$1
  local deployment=$2
  local cpu_request=${3:-100m}
  local memory_request=${4:-128Mi}

  echo "Checking $namespace/$deployment..."

  # 检查是否已有 requests
  if kubectl get deployment $deployment -n $namespace -o jsonpath='{.spec.template.spec.containers[0].resources.requests}' | grep -q "cpu"; then
    echo "  ✅ Already has resource requests"
    return
  fi

  echo "  📝 Adding resource requests: cpu=$cpu_request, memory=$memory_request"

  # 添加 requests
  kubectl patch deployment $deployment -n $namespace --type='json' -p="[
    {
      \"op\": \"add\",
      \"path\": \"/spec/template/spec/containers/0/resources\",
      \"value\": {
        \"requests\": {
          \"cpu\": \"$cpu_request\",
          \"memory\": \"$memory_request\"
        },
        \"limits\": {
          \"cpu\": \"$(echo $cpu_request | sed 's/m/*4m/' | bc)m\",
          \"memory\": \"$(echo $memory_request | sed 's/Mi/*4Mi/' | bc)Mi\"
        }
      }
    }
  ]" 2>/dev/null || {
    # 如果 resources 已存在，只添加 requests
    kubectl patch deployment $deployment -n $namespace --type='json' -p="[
      {
        \"op\": \"add\",
        \"path\": \"/spec/template/spec/containers/0/resources/requests\",
        \"value\": {
          \"cpu\": \"$cpu_request\",
          \"memory\": \"$memory_request\"
        }
      }
    ]"
  }

  echo "  ✅ Done"
}

# 修复所有 HPA 目标
add_resources "monitoring" "prometheus" "500m" "1Gi"
add_resources "monitoring" "grafana" "100m" "128Mi"
add_resources "argocd" "argocd-server" "100m" "128Mi"
add_resources "infra" "ingress-nginx-controller" "200m" "256Mi"
add_resources "gitea-runner" "gitea-runner" "100m" "128Mi"
add_resources "ljwx-stock" "claude-stock-backend" "200m" "256Mi"

echo ""
echo "=== 等待 Pods 重启 ==="
sleep 60

echo ""
echo "=== 验证 HPA 状态 ==="
kubectl get hpa --all-namespaces

echo ""
echo "✅ 修复完成！请等待 1-2 分钟后再次检查 HPA 状态"
EOF

chmod +x /tmp/fix-hpa-resources.sh
/tmp/fix-hpa-resources.sh

Task 1.4: 验证 HPA 工作 (15分钟)

# === 步骤 1: 等待 metrics 数据收集 ===
echo "Waiting for metrics data..."
sleep 120

# === 步骤 2: 检查 HPA 状态 ===
kubectl get hpa --all-namespaces -o wide

# 应该看到具体的百分比，例如:
# monitoring   grafana-hpa   Deployment/grafana   cpu: 15%/70%   1   3   1   6d

# === 步骤 3: 测试 HPA 扩容 ===
# 创建负载测试
kubectl run load-generator -n monitoring --image=busybox --restart=Never -- \
  /bin/sh -c "while true; do wget -q -O- http://grafana.monitoring.svc.cluster.local:3000; done"

# 观察 HPA 行为 (2-3分钟后应该看到扩容)
watch kubectl get hpa grafana-hpa -n monitoring

# 清理
kubectl delete pod load-generator -n monitoring

成功标准:

✅ kubectl top nodes 正常工作
✅ kubectl top pods 正常工作
✅ 所有 HPA 显示具体的 CPU 百分比 (不是 <unknown>)
✅ 负载测试时 HPA 自动扩容

预计时间: 1.5 小时 风险等级: 低（只读操作 + 非破坏性修复）

Phase 2: 自动化资源优化（第 2 周）⭐

目标: 部署 VPA，自动优化资源配置，减少 50-70% 资源浪费

Task 2.1: 部署 VPA (Vertical Pod Autoscaler) (1小时)

# === 步骤 1: 安装 VPA ===
cd /tmp
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler

# 部署 VPA 组件
./hack/vpa-up.sh

# 验证部署
kubectl get deployment -n kube-system | grep vpa
# 应该看到:
# vpa-admission-controller
# vpa-recommender
# vpa-updater

kubectl get pods -n kube-system | grep vpa

# === 步骤 2: 创建 VPA 配置 ===

# 为关键服务创建 VPA（Auto 模式）
cat > /tmp/vpa-critical-services.yaml << 'EOF'
---
# Grafana VPA - Auto 模式
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: grafana-vpa
  namespace: monitoring
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: grafana
  updatePolicy:
    updateMode: "Auto"  # 自动更新 Pod
  resourcePolicy:
    containerPolicies:
    - containerName: grafana
      minAllowed:
        cpu: 50m
        memory: 128Mi
      maxAllowed:
        cpu: 2
        memory: 4Gi
      controlledResources: ["cpu", "memory"]

---
# ArgoCD Server VPA - Auto 模式
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: argocd-server-vpa
  namespace: argocd
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: argocd-server
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: argocd-server
      minAllowed:
        cpu: 50m
        memory: 128Mi
      maxAllowed:
        cpu: 2
        memory: 4Gi

---
# Ingress NGINX VPA - Auto 模式
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: ingress-nginx-vpa
  namespace: infra
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ingress-nginx-controller
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: controller
      minAllowed:
        cpu: 100m
        memory: 256Mi
      maxAllowed:
        cpu: 2
        memory: 4Gi
EOF

kubectl apply -f /tmp/vpa-critical-services.yaml

# 为数据库创建 VPA（Initial 模式，避免频繁重启）
cat > /tmp/vpa-databases.yaml << 'EOF'
---
# PostgreSQL VPA - Initial 模式
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: postgres-vpa
  namespace: data
spec:
  targetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: postgres-postgresql
  updatePolicy:
    updateMode: "Initial"  # 仅在 Pod 创建时设置
  resourcePolicy:
    containerPolicies:
    - containerName: postgresql
      minAllowed:
        cpu: 100m
        memory: 256Mi
      maxAllowed:
        cpu: 4
        memory: 8Gi

---
# Redis VPA - Initial 模式
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: redis-vpa
  namespace: data
spec:
  targetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: redis-master
  updatePolicy:
    updateMode: "Initial"
  resourcePolicy:
    containerPolicies:
    - containerName: redis
      minAllowed:
        cpu: 50m
        memory: 128Mi
      maxAllowed:
        cpu: 2
        memory: 4Gi
EOF

kubectl apply -f /tmp/vpa-databases.yaml

# === 步骤 3: 创建 Prometheus VPA (Recommender 模式) ===
# Prometheus 是关键监控组件，使用 Off 模式先观察推荐
cat > /tmp/vpa-prometheus.yaml << 'EOF'
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: prometheus-vpa
  namespace: monitoring
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: prometheus
  updatePolicy:
    updateMode: "Off"  # 仅推荐，不自动更新
  resourcePolicy:
    containerPolicies:
    - containerName: prometheus
      minAllowed:
        cpu: 500m
        memory: 1Gi
      maxAllowed:
        cpu: 4
        memory: 8Gi
EOF

kubectl apply -f /tmp/vpa-prometheus.yaml

# === 步骤 4: 验证 VPA 工作 ===
kubectl get vpa --all-namespaces

# 查看 VPA 推荐
kubectl describe vpa grafana-vpa -n monitoring
kubectl describe vpa prometheus-vpa -n monitoring

# 等待 24 小时后，VPA 会根据实际使用情况提供推荐

Task 2.2: 创建 VPA 监控 Dashboard (30分钟)

# === 在 Grafana 中导入 VPA Dashboard ===
# Dashboard ID: 14588 (VPA Recommendations)

# 或手动创建 ConfigMap
cat > /tmp/vpa-dashboard-configmap.yaml << 'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: vpa-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  vpa-dashboard.json: |
    {
      "dashboard": {
        "title": "VPA Recommendations",
        "panels": [
          {
            "title": "VPA Recommendations by Namespace",
            "targets": [
              {
                "expr": "vpa_status_recommendation{resource=\"cpu\"}",
                "legendFormat": "{{namespace}}/{{target_name}} - CPU"
              },
              {
                "expr": "vpa_status_recommendation{resource=\"memory\"}",
                "legendFormat": "{{namespace}}/{{target_name}} - Memory"
              }
            ]
          }
        ]
      }
    }
EOF

kubectl apply -f /tmp/vpa-dashboard-configmap.yaml

成功标准:

✅ VPA 组件正常运行
✅ 5 个 VPA 对象创建成功
✅ 可以查看 VPA 推荐（kubectl describe vpa）
✅ 24 小时后 Pod 资源配置自动优化（Auto 模式）

预计时间: 1.5 小时 风险等级: 中（Auto 模式可能导致 Pod 重启）

Phase 3: 集群完整备份（第 2-3 周）⭐⭐

目标: 部署 Velero，实现集群完整备份和灾难恢复

Task 3.1: 部署 Velero (1.5小时)

# === 步骤 1: 安装 Velero CLI ===
brew install velero

# === 步骤 2: 准备 MinIO ===
# 检查 MinIO 是否运行
kubectl get pods -n infra -l app.kubernetes.io/name=minio

# 获取 MinIO 访问信息
MINIO_ENDPOINT=$(kubectl get svc minio -n infra -o jsonpath='{.spec.clusterIP}')
echo "MinIO Endpoint: http://$MINIO_ENDPOINT:9000"

# === 步骤 3: 在 MinIO 创建 bucket ===
# 方法 1: 使用 MinIO Console (http://localhost:30080)
# 创建 bucket: velero-backups

# 方法 2: 使用 kubectl port-forward + mc 命令
kubectl port-forward -n infra svc/minio 9000:9000 &
FORWARD_PID=$!

# 安装 mc (MinIO Client)
brew install minio/stable/mc

# 配置 mc
mc alias set local http://localhost:9000 minioadmin minioadmin123

# 创建 bucket
mc mb local/velero-backups

# 停止端口转发
kill $FORWARD_PID

# === 步骤 4: 创建 Velero 凭证 ===
cat > /tmp/credentials-velero << EOF
[default]
aws_access_key_id = minioadmin
aws_secret_access_key = minioadmin123
EOF

# === 步骤 5: 安装 Velero ===
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.11.0 \
  --bucket velero-backups \
  --secret-file /tmp/credentials-velero \
  --use-node-agent \
  --use-volume-snapshots=false \
  --backup-location-config \
    region=minio,s3ForcePathStyle="true",s3Url=http://minio.infra.svc.cluster.local:9000

# === 步骤 6: 验证部署 ===
kubectl get deployment -n velero
kubectl get pods -n velero

# 等待所有 Pod Running
kubectl wait --for=condition=ready pod -l component=velero -n velero --timeout=300s

# === 步骤 7: 验证备份位置 ===
velero backup-location get

Task 3.2: 创建备份计划 (30分钟)

# === 创建测试备份 ===
velero backup create test-backup \
  --include-namespaces monitoring \
  --wait

# 检查备份状态
velero backup describe test-backup
velero backup logs test-backup

# === 创建定期备份计划 ===

# 1. 每日全量备份（关键命名空间）
velero schedule create daily-critical-backup \
  --schedule="0 2 * * *" \
  --include-namespaces=data,monitoring,logging,tracing,argocd,infra,cert-manager,trivy-system \
  --ttl 168h \
  --labels backup-type=daily

# 2. 每周归档备份（所有命名空间）
velero schedule create weekly-full-backup \
  --schedule="0 3 * * 0" \
  --ttl 720h \
  --labels backup-type=weekly

# 3. 应用命名空间的频繁备份
velero schedule create hourly-app-backup \
  --schedule="0 * * * *" \
  --include-namespaces=ljwx-stock,stock,fastapi,dev,learn,website \
  --ttl 24h \
  --labels backup-type=hourly

# === 查看所有备份计划 ===
velero schedule get

# === 设置自动清理策略 ===
# Velero 会根据 TTL 自动清理过期备份

Task 3.3: 测试恢复流程 (30分钟)

# === 步骤 1: 创建测试命名空间并部署应用 ===
kubectl create namespace test-app
kubectl create deployment nginx --image=nginx -n test-app
kubectl create service clusterip nginx --tcp=80:80 -n test-app

# === 步骤 2: 备份测试应用 ===
velero backup create test-app-backup \
  --include-namespaces test-app \
  --wait

# === 步骤 3: 删除测试应用 ===
kubectl delete namespace test-app

# === 步骤 4: 从备份恢复 ===
velero restore create test-app-restore \
  --from-backup test-app-backup \
  --wait

# === 步骤 5: 验证恢复 ===
kubectl get all -n test-app

# === 步骤 6: 清理测试 ===
kubectl delete namespace test-app
velero backup delete test-app-backup

成功标准:

✅ Velero 组件正常运行
✅ 测试备份成功
✅ 3 个备份计划已创建
✅ 恢复测试成功

预计时间: 2.5 小时 风险等级: 低（备份操作不影响运行服务）

Phase 4: 零信任网络架构（第 3-4 周）⭐⭐⭐

目标: 实现零信任网络架构，增强网络隔离和安全性

Task 4.1: 部署默认拒绝策略 (30分钟)

# === 创建默认拒绝策略（所有流量） ===
cat > /tmp/network-policy-default-deny.yaml << 'EOF'
---
# 默认拒绝所有入站流量
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress

---
# 默认拒绝所有出站流量
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress
EOF

# 应用到所有关键命名空间
for ns in data monitoring logging tracing argocd infra cert-manager trivy-system \
           ljwx-stock stock fastapi dev learn website; do
  echo "Applying default deny policy to $ns..."
  kubectl apply -f /tmp/network-policy-default-deny.yaml -n $ns
done

# ⚠️ 注意: 应用后所有 Pod 将无法通信，需要逐步添加白名单策略

Task 4.2: 创建基础服务白名单 (1小时)

# === 1. 允许 DNS 查询 ===
cat > /tmp/network-policy-allow-dns.yaml << 'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
EOF

for ns in data monitoring logging tracing argocd infra cert-manager trivy-system \
           ljwx-stock stock fastapi dev learn website; do
  kubectl apply -f /tmp/network-policy-allow-dns.yaml -n $ns
done

# === 2. 允许访问 Kubernetes API ===
cat > /tmp/network-policy-allow-k8s-api.yaml << 'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-k8s-api
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: default
    ports:
    - protocol: TCP
      port: 443
EOF

for ns in argocd cert-manager trivy-system; do
  kubectl apply -f /tmp/network-policy-allow-k8s-api.yaml -n $ns
done

# === 3. 允许 Prometheus 采集指标 ===
cat > /tmp/network-policy-allow-prometheus.yaml << 'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus-scrape
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: prometheus
    ports:
    - protocol: TCP
      port: 9104  # MySQL Exporter
    - protocol: TCP
      port: 9187  # Postgres Exporter
    - protocol: TCP
      port: 9121  # Redis Exporter
    - protocol: TCP
      port: 9090  # Prometheus
    - protocol: TCP
      port: 3000  # Grafana
    - protocol: TCP
      port: 9000  # MinIO
    - protocol: TCP
      port: 8080  # Application metrics
EOF

for ns in data monitoring infra; do
  kubectl apply -f /tmp/network-policy-allow-prometheus.yaml -n $ns
done

# === 4. 允许 Loki 收集日志 ===
cat > /tmp/network-policy-allow-loki.yaml << 'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-loki
  namespace: logging
spec:
  podSelector:
    matchLabels:
      app: loki
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector: {}
    - podSelector:
        matchLabels:
          app: promtail
    ports:
    - protocol: TCP
      port: 3100
EOF

kubectl apply -f /tmp/network-policy-allow-loki.yaml

# === 5. 允许 Ingress 流量 ===
cat > /tmp/network-policy-allow-ingress.yaml << 'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: infra
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: ingress-nginx
EOF

for ns in monitoring argocd; do
  kubectl apply -f /tmp/network-policy-allow-ingress.yaml -n $ns
done

Task 4.3: 创建应用专用网络策略 (1小时)

# === PostgreSQL 专用策略 ===
cat > /tmp/network-policy-postgres.yaml << 'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-network-policy
  namespace: data
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: postgresql
  policyTypes:
  - Ingress
  - Egress
  ingress:
  # 允许来自应用命名空间的连接
  - from:
    - namespaceSelector:
        matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: In
          values: [ljwx-stock, stock, fastapi, dev, learn, website, infra]
    ports:
    - protocol: TCP
      port: 5432
  # 允许来自监控的连接
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
    ports:
    - protocol: TCP
      port: 9187  # Postgres Exporter
  egress:
  # 允许 DNS
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
  # 允许连接到 MinIO（备份）
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: infra
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: minio
    ports:
    - protocol: TCP
      port: 9000
EOF

kubectl apply -f /tmp/network-policy-postgres.yaml

# === Redis 专用策略 ===
cat > /tmp/network-policy-redis.yaml << 'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: redis-network-policy
  namespace: data
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: redis
  policyTypes:
  - Ingress
  - Egress
  ingress:
  # 允许来自应用命名空间的连接
  - from:
    - namespaceSelector:
        matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: In
          values: [ljwx-stock, stock, fastapi, dev, learn, website]
    ports:
    - protocol: TCP
      port: 6379
  # 允许来自监控的连接
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
    ports:
    - protocol: TCP
      port: 9121  # Redis Exporter
  egress:
  # 允许 DNS
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
EOF

kubectl apply -f /tmp/network-policy-redis.yaml

# === ArgoCD 专用策略 ===
cat > /tmp/network-policy-argocd.yaml << 'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: argocd-network-policy
  namespace: argocd
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: argocd
  policyTypes:
  - Ingress
  - Egress
  ingress:
  # 允许来自 Ingress 的流量
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: infra
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: ingress-nginx
  # 允许命名空间内部通信
  - from:
    - podSelector:
        matchLabels:
          app.kubernetes.io/part-of: argocd
  egress:
  # 允许 DNS
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
  # 允许访问 Kubernetes API
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: default
    ports:
    - protocol: TCP
      port: 443
  # 允许访问 Git 仓库（外部）
  - ports:
    - protocol: TCP
      port: 443
    - protocol: TCP
      port: 22
EOF

kubectl apply -f /tmp/network-policy-argocd.yaml

Task 4.4: 验证网络策略 (30分钟)

# === 测试脚本 ===
cat > /tmp/test-network-policies.sh << 'EOF'
#!/bin/bash

echo "=== Testing Network Policies ==="

# 测试 1: DNS 应该工作
echo "Test 1: DNS resolution"
kubectl run test-dns --image=busybox -i --rm --restart=Never -n monitoring -- \
  nslookup kubernetes.default.svc.cluster.local
echo ""

# 测试 2: 跨命名空间访问数据库（应该成功）
echo "Test 2: Cross-namespace database access (should succeed)"
kubectl run test-db-access --image=postgres:15 -i --rm --restart=Never -n ljwx-stock -- \
  psql -h postgres-postgresql.data.svc.cluster.local -U postgres -c "SELECT 1;" 2>&1 | grep -q "cannot connect" && echo "❌ FAILED (expected)" || echo "✅ SUCCESS"
echo ""

# 测试 3: 未授权访问（应该失败）
echo "Test 3: Unauthorized access (should fail)"
kubectl run test-unauth --image=curlimages/curl -i --rm --restart=Never -n learn -- \
  curl -m 5 http://grafana.monitoring.svc.cluster.local:3000 && echo "❌ FAILED (should be blocked)" || echo "✅ SUCCESS (blocked as expected)"
echo ""

# 测试 4: Prometheus 采集指标（应该成功）
echo "Test 4: Prometheus scraping (should succeed)"
kubectl exec -n monitoring deployment/prometheus -- \
  curl -s http://postgres-exporter.data.svc.cluster.local:9187/metrics | head -1
echo ""

echo "=== Network Policy Test Complete ==="
EOF

chmod +x /tmp/test-network-policies.sh
/tmp/test-network-policies.sh

成功标准:

✅ 所有关键命名空间都有默认拒绝策略
✅ DNS 查询正常工作
✅ 授权的服务可以相互通信
✅ 未授权的访问被拒绝
✅ Prometheus 可以采集指标
✅ Loki 可以收集日志

预计时间: 3 小时 风险等级: 高（可能影响服务连接，需要仔细测试）

Phase 5: 策略自动化（第 4 周）⭐

目标: 部署 Kyverno 策略引擎，实现策略即代码

Task 5.1: 部署 Kyverno (30分钟)

# === 步骤 1: 添加 Helm repo ===
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

# === 步骤 2: 安装 Kyverno ===
helm install kyverno kyverno/kyverno \
  --namespace kyverno \
  --create-namespace \
  --set replicaCount=1 \
  --set resources.requests.cpu=100m \
  --set resources.requests.memory=128Mi

# === 步骤 3: 验证部署 ===
kubectl get pods -n kyverno
kubectl get clusterpolicy

# === 步骤 4: 安装 Kyverno Policy Reporter (可选，用于可视化) ===
helm install policy-reporter kyverno/policy-reporter \
  --namespace kyverno \
  --set ui.enabled=true \
  --set kyvernoPlugin.enabled=true

Task 5.2: 创建企业级策略 (1.5小时)

# === 1. 强制所有 Pod 设置 resource requests/limits ===
cat > /tmp/kyverno-policy-resources.yaml << 'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resources
  annotations:
    policies.kyverno.io/title: Require Resource Requests and Limits
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/severity: medium
    policies.kyverno.io/description: >-
      所有容器必须设置 CPU 和内存的 requests 和 limits
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: validate-resources
    match:
      any:
      - resources:
          kinds:
          - Pod
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - kyverno
          - velero
    validate:
      message: "All containers must have CPU and memory requests and limits set."
      pattern:
        spec:
          containers:
          - resources:
              requests:
                memory: "?*"
                cpu: "?*"
              limits:
                memory: "?*"
                cpu: "?*"
EOF

kubectl apply -f /tmp/kyverno-policy-resources.yaml

# === 2. 禁止使用 latest 标签 ===
cat > /tmp/kyverno-policy-no-latest.yaml << 'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
  annotations:
    policies.kyverno.io/title: Disallow Latest Tag
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/severity: medium
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: require-image-tag
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Using the ':latest' tag is not allowed. Please use a specific version."
      pattern:
        spec:
          containers:
          - image: "!*:latest"
EOF

kubectl apply -f /tmp/kyverno-policy-no-latest.yaml

# === 3. 强制设置 Pod Disruption Budget ===
cat > /tmp/kyverno-policy-pdb.yaml << 'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-pdb-for-multi-replica
  annotations:
    policies.kyverno.io/title: Require PDB for Multi-Replica Deployments
    policies.kyverno.io/category: Resilience
    policies.kyverno.io/severity: medium
spec:
  validationFailureAction: audit  # 先用 audit 模式观察
  background: true
  rules:
  - name: check-pdb
    match:
      any:
      - resources:
          kinds:
          - Deployment
    preconditions:
      all:
      - key: "{{request.object.spec.replicas}}"
        operator: GreaterThan
        value: 1
    validate:
      message: "Deployments with more than 1 replica should have a PodDisruptionBudget."
      deny:
        conditions:
          all:
          - key: "{{request.object.metadata.name}}"
            operator: NotIn
            value: "{{ query_resources('policy.k8s.io/v1', 'PodDisruptionBudget', '{{request.object.metadata.namespace}}') | map(&metadata.name, @) }}"
EOF

kubectl apply -f /tmp/kyverno-policy-pdb.yaml

# === 4. 自动添加标签 ===
cat > /tmp/kyverno-policy-auto-labels.yaml << 'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-default-labels
  annotations:
    policies.kyverno.io/title: Add Default Labels
    policies.kyverno.io/category: Other
spec:
  background: false
  rules:
  - name: add-labels
    match:
      any:
      - resources:
          kinds:
          - Pod
    mutate:
      patchStrategicMerge:
        metadata:
          labels:
            +(managed-by): kyverno
            +(cluster): orbstack-dev
EOF

kubectl apply -f /tmp/kyverno-policy-auto-labels.yaml

# === 5. 限制 Namespace 创建（仅管理员） ===
cat > /tmp/kyverno-policy-restrict-ns.yaml << 'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-namespace-creation
  annotations:
    policies.kyverno.io/title: Restrict Namespace Creation
    policies.kyverno.io/category: Security
spec:
  validationFailureAction: enforce
  background: false
  rules:
  - name: check-namespace-creation
    match:
      any:
      - resources:
          kinds:
          - Namespace
    exclude:
      any:
      - clusterRoles:
        - cluster-admin
    validate:
      message: "Only cluster admins can create namespaces."
      deny: {}
EOF

kubectl apply -f /tmp/kyverno-policy-restrict-ns.yaml

Task 5.3: 验证策略工作 (30分钟)

# === 测试 1: 尝试创建没有 resource requests 的 Pod ===
kubectl run test-no-resources --image=nginx -n learn
# 应该被拒绝

# === 测试 2: 尝试使用 latest 标签 ===
kubectl run test-latest --image=nginx:latest -n learn
# 应该被拒绝

# === 测试 3: 创建合规的 Pod ===
cat > /tmp/test-compliant-pod.yaml << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: test-compliant
  namespace: learn
spec:
  containers:
  - name: nginx
    image: nginx:1.25
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 500m
        memory: 512Mi
EOF

kubectl apply -f /tmp/test-compliant-pod.yaml
# 应该成功，并自动添加 managed-by: kyverno 标签

# 验证标签
kubectl get pod test-compliant -n learn -o jsonpath='{.metadata.labels}'

# 清理
kubectl delete pod test-compliant -n learn

# === 测试 4: 查看策略报告 ===
kubectl get clusterpolicyreport -A
kubectl get policyreport -A

# === 测试 5: 访问 Policy Reporter UI（如果已安装） ===
kubectl port-forward -n kyverno svc/policy-reporter-ui 8080:8080 &
# 访问 http://localhost:8080

成功标准:

✅ Kyverno 正常运行
✅ 5 个策略已部署
✅ 不合规的资源被拒绝
✅ 合规的资源自动添加标签
✅ 可以查看策略报告

预计时间: 2.5 小时 风险等级: 中（可能拒绝现有资源）

Phase 6: 长期监控存储（第 5 周）⭐

目标: 部署 Thanos，实现 Prometheus 数据长期存储

Task 6.1: 部署 Thanos (2小时)

# === 步骤 1: 配置 Prometheus 远程写入 ===
# 编辑 Prometheus 配置，添加 remote_write
kubectl edit configmap prometheus-config -n monitoring

# 添加:
# remote_write:
#   - url: http://thanos-receive.monitoring.svc.cluster.local:19291/api/v1/receive

# === 步骤 2: 部署 Thanos ===
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

# 创建 Thanos 配置
cat > /tmp/thanos-values.yaml << 'EOF'
objstoreConfig: |-
  type: s3
  config:
    bucket: thanos-storage
    endpoint: minio.infra.svc.cluster.local:9000
    access_key: minioadmin
    secret_key: minioadmin123
    insecure: true

query:
  enabled: true
  replicaCount: 1
  resources:
    requests:
      cpu: 100m
      memory: 256Mi

queryFrontend:
  enabled: true
  replicaCount: 1

compactor:
  enabled: true
  retentionResolutionRaw: 30d
  retentionResolution5m: 90d
  retentionResolution1h: 180d
  resources:
    requests:
      cpu: 100m
      memory: 256Mi

storegateway:
  enabled: true
  replicaCount: 1
  resources:
    requests:
      cpu: 100m
      memory: 256Mi

receiver:
  enabled: true
  replicaCount: 1
  resources:
    requests:
      cpu: 200m
      memory: 512Mi
EOF

# 安装 Thanos
helm install thanos bitnami/thanos \
  --namespace monitoring \
  --values /tmp/thanos-values.yaml

# === 步骤 3: 在 MinIO 创建 bucket ===
kubectl port-forward -n infra svc/minio 9000:9000 &
mc mb local/thanos-storage
kill $!

# === 步骤 4: 配置 Grafana 使用 Thanos Query ===
# 在 Grafana 中添加新的数据源:
# - Type: Prometheus
# - URL: http://thanos-query.monitoring.svc.cluster.local:9090
# - Name: Thanos

# === 步骤 5: 验证 Thanos ===
kubectl get pods -n monitoring | grep thanos
kubectl port-forward -n monitoring svc/thanos-query 9091:9090 &
# 访问 http://localhost:9091

成功标准:

✅ Thanos 组件正常运行
✅ Prometheus 数据写入到 MinIO
✅ 可以通过 Thanos Query 查询历史数据
✅ Grafana 可以访问 Thanos

预计时间: 2 小时 风险等级: 低（不影响现有 Prometheus）

📊 优化效果预期

资源优化

指标	当前	优化后	改进
平均资源浪费	50-70%	10-20%	✅ -60%
Prometheus 存储	7天	7天本地 + 30天长期	✅ +328% 历史数据
集群恢复时间	手动恢复 (数小时)	自动恢复 (<15分钟)	✅ 95%+
策略管理	手动 YAML	自动策略引擎	✅ 100% 自动化

安全增强

✅ 零信任网络 - 默认拒绝 + 白名单
✅ 策略即代码 - Kyverno 自动化
✅ 持续安全扫描 - Trivy Operator
✅ 合规检查 - 自动策略验证

运维效率

✅ 自动资源优化 - VPA 自动调整
✅ 自动备份 - Velero 定期备份
✅ 自动策略执行 - Kyverno 实时验证
✅ 自动扩缩容 - HPA + VPA 双重优化

📝 注意事项

执行前检查

备份当前集群配置

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  kubectl get all,cm,secret,pvc,pdb,hpa,networkpolicy -n $ns -o yaml > backup-$ns-$(date +%Y%m%d).yaml
done

确认资源充足
- CPU: 至少 2 核空闲
- 内存: 至少 4GB 空闲
- 存储: 至少 20GB 空闲（MinIO）
测试环境验证
- 先在测试命名空间验证每个步骤
- 确认无问题后再应用到生产命名空间

风险管理

优化项	风险等级	回滚方案
修复 HPA	低	恢复原 Deployment
部署 VPA	中	删除 VPA CRD
部署 Velero	低	helm uninstall velero
Network Policies	高	删除所有 NetworkPolicy
部署 Kyverno	中	设置 validationFailureAction: audit
部署 Thanos	低	helm uninstall thanos

回滚计划

# === 快速回滚脚本 ===
cat > /tmp/rollback.sh << 'EOF'
#!/bin/bash

COMPONENT=$1

case $COMPONENT in
  "vpa")
    kubectl delete vpa --all --all-namespaces
    cd /tmp/autoscaler/vertical-pod-autoscaler && ./hack/vpa-down.sh
    ;;
  "velero")
    helm uninstall velero -n velero
    kubectl delete namespace velero
    ;;
  "network-policies")
    for ns in data monitoring logging tracing argocd infra cert-manager trivy-system; do
      kubectl delete networkpolicy --all -n $ns
    done
    ;;
  "kyverno")
    helm uninstall kyverno -n kyverno
    kubectl delete namespace kyverno
    ;;
  "thanos")
    helm uninstall thanos -n monitoring
    ;;
  *)
    echo "Usage: $0 {vpa|velero|network-policies|kyverno|thanos}"
    exit 1
    ;;
esac
EOF

chmod +x /tmp/rollback.sh

📅 执行时间表

阶段	时间	工作量	优先级
Phase 1: 修复 HPA	第 1 周	1.5h	⚡ P0
Phase 2: 部署 VPA	第 2 周	1.5h	⭐ P1
Phase 3: 部署 Velero	第 2-3 周	2.5h	⭐⭐ P1
Phase 4: 零信任网络	第 3-4 周	3h	⭐⭐⭐ P1
Phase 5: 部署 Kyverno	第 4 周	2.5h	⭐ P2
Phase 6: 部署 Thanos	第 5 周	2h	⭐ P2

总工作量: 约 13 小时 建议节奏: 每周 2-3 小时，共 4-6 周完成

🎉 成功标准

技术指标

✅ HPA: 6/6 正常工作（显示具体百分比）
✅ VPA: 5+ VPA 自动优化资源
✅ 备份: 每日自动备份，恢复时间 <15 分钟
✅ 网络: 零信任架构，默认拒绝策略
✅ 策略: Kyverno 自动验证和修复
✅ 监控: 7天本地 + 30天长期存储

运维指标

✅ 资源利用率: 提升到 60-80%
✅ 故障恢复: 从手动恢复到自动恢复
✅ 策略合规: 100% 自动化验证
✅ 安全扫描: 持续自动扫描

📚 相关文档

企业级实施报告: ENTERPRISE-K8S-IMPLEMENTATION-REPORT.md
优化分析报告: K8S-OPTIMIZATION-ANALYSIS.md
优化执行计划: K8S-OPTIMIZATION-PLAN.md
优化执行报告: OPTIMIZATION-EXECUTION-REPORT.md

路线图创建: 2025-12-31 预计完成: 2025-02-15 (6 周) 负责人: DevOps Team 状态: 📋 等待执行

39 KiB Raw Permalink Blame History Unescape Escape

Kubernetes 集群优化路线图 2025

📊 当前集群状态评估

✅ 已完成的优化（基线）

⚠️ 当前问题

🎯 优化目标

技术目标

性能目标

🚀 执行路线图

Phase 1: 关键修复（第 1 周）⚡

Task 1.1: 诊断 HPA Metrics 问题 (30分钟)

Task 1.2: 修复 metrics-server (OrbStack 特定) (15分钟)

Task 1.3: 确保所有 HPA 目标有 resource requests (30分钟)

Task 1.4: 验证 HPA 工作 (15分钟)

Phase 2: 自动化资源优化（第 2 周）⭐

Task 2.1: 部署 VPA (Vertical Pod Autoscaler) (1小时)

Task 2.2: 创建 VPA 监控 Dashboard (30分钟)

Phase 3: 集群完整备份（第 2-3 周）⭐⭐

Task 3.1: 部署 Velero (1.5小时)

Task 3.2: 创建备份计划 (30分钟)

Task 3.3: 测试恢复流程 (30分钟)

Phase 4: 零信任网络架构（第 3-4 周）⭐⭐⭐

Task 4.1: 部署默认拒绝策略 (30分钟)

Task 4.2: 创建基础服务白名单 (1小时)

Task 4.3: 创建应用专用网络策略 (1小时)

Task 4.4: 验证网络策略 (30分钟)

Phase 5: 策略自动化（第 4 周）⭐

Task 5.1: 部署 Kyverno (30分钟)

Task 5.2: 创建企业级策略 (1.5小时)

Task 5.3: 验证策略工作 (30分钟)

Phase 6: 长期监控存储（第 5 周）⭐

Task 6.1: 部署 Thanos (2小时)

📊 优化效果预期

资源优化

安全增强

运维效率

📝 注意事项

执行前检查

风险管理

回滚计划

📅 执行时间表

🎉 成功标准

技术指标

运维指标

📚 相关文档

39 KiB

Raw Permalink Blame History