ljwx-k8s/OBSERVABILITY-CHEATSHEET.md

可观测性快速参考卡 (Cheat Sheet)

更新时间: 2025-12-21

---

🚀 快速访问

Grafana

# 本地
http://localhost:3000

# 远程（需配置 hosts: 192.168.1.83 grafana.omniverseai.net）
http://grafana.omniverseai.net:31080

# 登录
用户名: admin
密码: admin123

Prometheus

kubectl port-forward -n monitoring svc/prometheus 9090:9090
http://localhost:9090

关键 Dashboards

PostgreSQL: http://grafana.omniverseai.net:31080/d/000000039/postgresql-database
Redis:      http://grafana.omniverseai.net:31080/d/xDLNRKUWz/4b82829
Kubernetes: http://grafana.omniverseai.net:31080/d/k8s_views_pods/kubernetes-views-pods

---

📊 关键 PromQL 查询

PostgreSQL

# 可用性
pg_up

# 连接数 / 最大连接数 (%)
(pg_stat_database_numbackends / pg_settings_max_connections) * 100

# 缓存命中率 (%)
(pg_stat_database_blks_hit / (pg_stat_database_blks_hit + pg_stat_database_blks_read)) * 100

# TPS (每秒事务数)
rate(pg_stat_database_xact_commit[5m])

# 锁等待
pg_locks_count{mode="ExclusiveLock"}

Redis

# 可用性
redis_up

# 内存使用率 (%)
(redis_memory_used_bytes / redis_memory_max_bytes) * 100

# 缓存命中率 (%)
(redis_keyspace_hits_total / (redis_keyspace_hits_total + redis_keyspace_misses_total)) * 100

# 每秒操作数
rate(redis_commands_processed_total[5m])

# 连接数
redis_connected_clients

# 键淘汰
rate(redis_evicted_keys_total[5m])

Kubernetes

# Pod 重启次数
rate(kube_pod_container_status_restarts_total[15m])

# Pod 内存使用率
container_memory_working_set_bytes / container_spec_memory_limit_bytes * 100

# Pod CPU 使用率
rate(container_cpu_usage_seconds_total[5m])

# CrashLoopBackOff Pods
kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}

业务应用

# RPS (Requests Per Second)
rate(http_server_requests_total{service="your-service"}[5m])

# 错误率 (%)
rate(http_server_requests_total{service="your-service",status=~"5.."}[5m])
  / rate(http_server_requests_total{service="your-service"}[5m]) * 100

# P95 延迟
histogram_quantile(0.95,
  rate(http_server_request_duration_seconds_bucket{service="your-service"}[5m]))

# P99 延迟
histogram_quantile(0.99,
  rate(http_server_request_duration_seconds_bucket{service="your-service"}[5m]))

---

🔧 常用 Kubectl 命令

检查组件状态

# Prometheus
kubectl get pods -n monitoring -l app=prometheus
kubectl logs -n monitoring -l app=prometheus --tail=50

# Grafana
kubectl get pods -n monitoring -l app=grafana
kubectl logs -n monitoring -l app=grafana --tail=50

# Exporters
kubectl get pods -n data | grep metrics
kubectl get svc -n data | grep metrics

检查配置

# Prometheus ConfigMap
kubectl get configmap prometheus-config -n monitoring -o yaml

# Prometheus Rules
kubectl get configmap prometheus-rules -n monitoring -o yaml

# Grafana Datasources
kubectl get configmap grafana-datasources -n monitoring -o yaml

重启服务

# Prometheus
kubectl rollout restart deployment prometheus -n monitoring

# Grafana
kubectl rollout restart deployment grafana -n monitoring

---

🎯 业务应用集成

Spring Boot - Maven 依赖

<!-- Metrics -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

<!-- Tracing -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

Spring Boot - application.yml

spring:
  application:
    name: your-service

management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus
  metrics:
    export:
      prometheus:
        enabled: true
    tags:
      application: ${spring.application.name}

Kubernetes Service Annotation

apiVersion: v1
kind: Service
metadata:
  name: your-service
  annotations:
    prometheus.io/scrape: "true"      # 必须
    prometheus.io/port: "8080"        # 应用端口
    prometheus.io/path: "/actuator/prometheus"  # Metrics 路径

OpenTelemetry Java Agent 环境变量

env:
- name: JAVA_TOOL_OPTIONS
  value: "-javaagent:/app/opentelemetry-javaagent.jar"
- name: OTEL_SERVICE_NAME
  value: "your-service"
- name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: "http://tempo.tracing.svc.cluster.local:4317"
- name: OTEL_INSTRUMENTATION_JDBC_ENABLED
  value: "true"    # SQL spans
- name: OTEL_INSTRUMENTATION_LETTUCE_ENABLED
  value: "true"    # Redis spans

Python Flask - 最小示例

# requirements.txt
prometheus-client
opentelemetry-api
opentelemetry-sdk
opentelemetry-instrumentation-flask
opentelemetry-instrumentation-psycopg2
opentelemetry-instrumentation-redis
opentelemetry-exporter-otlp-proto-grpc

# app.py
from flask import Flask
from prometheus_client import Counter, Histogram, make_wsgi_app
from werkzeug.middleware.dispatcher import DispatcherMiddleware
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.psycopg2 import Psycopg2Instrumentor
from opentelemetry.instrumentation.redis import RedisInstrumentor

app = Flask(__name__)

# Metrics
app.wsgi_app = DispatcherMiddleware(app.wsgi_app, {
    '/metrics': make_wsgi_app()
})

# Tracing
FlaskInstrumentor().instrument_app(app)
Psycopg2Instrumentor().instrument()  # SQL spans
RedisInstrumentor().instrument()      # Redis spans

---

📝 日志格式要求

JSON 格式（推荐）

{
  "timestamp": "2025-12-21 10:30:45.123",
  "level": "INFO",
  "logger": "com.example.MyService",
  "message": "User logged in successfully",
  "trace_id": "abc123def456789",
  "span_id": "xyz789"
}

KV 格式

2025-12-21 10:30:45.123 INFO [com.example.MyService] User logged in trace_id=abc123def456789 span_id=xyz789

Logback 配置（Spring Boot）

<configuration>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <includeMdcKeyName>trace_id</includeMdcKeyName>
            <includeMdcKeyName>span_id</includeMdcKeyName>
        </encoder>
    </appender>
</configuration>

---

🔍 Loki 查询语法

基础查询

# 查询特定 namespace
{namespace="your-app"}

# 查询包含 ERROR
{namespace="your-app"} |= "ERROR"

# JSON 解析
{namespace="your-app"} | json

# 提取 trace_id
{namespace="your-app"} | json | trace_id != ""

# 过滤特定 level
{namespace="your-app"} | json | level="ERROR"

聚合查询

# 错误日志计数
sum(count_over_time({namespace="your-app"} |= "ERROR" [5m]))

# 按服务统计
sum by (service) (count_over_time({namespace="your-app"}[5m]))

---

🎯 Tempo Trace 查询

TraceQL 查询

# 查询特定服务
{service.name="your-service"}

# 查询慢请求 (>1s)
{service.name="your-service" && duration > 1s}

# 查询包含错误
{service.name="your-service" && status=error}

# 查询特定 endpoint
{service.name="your-service" && http.route="/api/users"}

---

🛠️ 故障排查

Prometheus Targets DOWN

# 检查 target 状态
kubectl port-forward -n monitoring svc/prometheus 9090:9090
# 访问 http://localhost:9090/targets

# 检查 exporter pod
kubectl get pods -n data | grep metrics
kubectl logs -n data <exporter-pod>

# 检查 service
kubectl get svc -n data | grep metrics
kubectl describe svc <exporter-service> -n data

# 测试连通性
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl http://postgres-postgresql-metrics.data:9187/metrics

Grafana Dashboard 无数据

# 检查数据源
Grafana → Configuration → Data Sources → Prometheus → Test

# 检查查询
Grafana → Explore → Prometheus
# 执行: pg_up

# 检查时间范围
Dashboard 右上角 → 调整为 "Last 5 minutes"

Trace 未上报

# 检查 Tempo 可达性
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl -v tempo.tracing.svc.cluster.local:4317

# 检查应用日志
kubectl logs -n your-namespace <your-pod> | grep -i otel

# 检查环境变量
kubectl exec -n your-namespace <your-pod> -- env | grep OTEL

日志 → Trace 跳转失败

# 检查日志格式
kubectl logs -n your-namespace <your-pod> | head -10

# 确认包含 trace_id
kubectl logs -n your-namespace <your-pod> | grep trace_id

# 检查 Loki Derived Fields
Grafana → Configuration → Data Sources → Loki
# 验证正则表达式: "trace_id":\s*"(\w+)"

---

📚 关键文档

# P0 完成报告
OBSERVABILITY-P0-COMPLETE.md

# 快速开始
QUICK-START.md

# 远程访问
GRAFANA-REMOTE-ACCESS.md

# Dashboard 访问
GRAFANA-DASHBOARDS-ACCESS.md

# 业务集成
docs/observability/business-instrumentation-spring-boot.md
docs/observability/business-instrumentation-python.md

# pg_stat_statements
docs/observability/pg-stat-statements-setup.md

---

🚦 验收脚本

# 完整验收
./verify-p0-complete.sh

# 启用告警规则
./enable-prometheus-alerts.sh

# 快速验证
kubectl get pods -n monitoring
kubectl get pods -n data | grep metrics

---

📊 监控指标总览

组件	实例数	Exporter	Metrics 示例
PostgreSQL	2	postgres-exporter:9187	pg_up, pg_stat_database_*
Redis	2	redis-exporter:9121	redis_up, redis_memory_*
Kubernetes	54 Pods	kube-state-metrics	kube_pod_info, kube_pod_status_*
Prometheus	1	-	prometheus_target_*
Grafana	1	-	grafana_*
Loki	1	-	loki_*
Tempo	1	-	tempo_*

---

🔐 访问凭据

# Grafana
用户名: admin
密码: admin123

# Prometheus
无需认证

# PostgreSQL (exporter 连接)
查看: kubectl get secret -n data postgres-postgresql -o yaml

# Redis (exporter 连接)
查看: kubectl get secret -n data redis -o yaml

---

保存此文档以便快速查阅！ 📖