ljwx-stock/backend/docs/OPENTELEMETRY_GUIDE.md

OpenTelemetry 集成指南

概述

本系统已集成 OpenTelemetry，提供统一的可观测性框架，支持：

• Traces: 分布式追踪
• Metrics: 性能指标
• Logs: 结构化日志（已与 Trace ID 关联）

安装依赖

OpenTelemetry 是可选依赖，如需启用请安装：

# 安装 OpenTelemetry 核心包
pip install -r requirements.txt

# 或手动安装关键包
pip install opentelemetry-api==1.22.0 \
            opentelemetry-sdk==1.22.0 \
            opentelemetry-instrumentation-fastapi==0.43b0 \
            opentelemetry-exporter-otlp==1.22.0

如果不安装，系统会自动降级到 Prometheus + Structured Logging 模式。

配置方式

1. Console 模式（开发环境）

适合本地开发调试，所有 traces 和 metrics 输出到控制台。

# main.py
from app.observability import configure_opentelemetry

otel_config = configure_opentelemetry(
    service_name="stock-recommendation-system",
    service_version="1.3.0",
    exporter_type="console"
)

# 自动埋点 FastAPI
if otel_config:
    otel_config.instrument_app(app)

2. OTLP 模式（生产推荐）

使用 OpenTelemetry Collector 统一收集所有遥测数据。

otel_config = configure_opentelemetry(
    service_name="stock-recommendation-system",
    service_version="1.3.0",
    exporter_type="otlp",
    otlp_endpoint="localhost:4317"  # OpenTelemetry Collector 地址
)

部署 OpenTelemetry Collector:

# docker-compose.yml
services:
  otel-collector:
    image: otel/opentelemetry-collector:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"  # OTLP gRPC receiver
      - "4318:4318"  # OTLP HTTP receiver
      - "8888:8888"  # Prometheus metrics

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"

  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger, logging]

    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus, logging]

3. Prometheus + Jaeger 模式

独立配置 Prometheus 和 Jaeger。

otel_config = configure_opentelemetry(
    service_name="stock-recommendation-system",
    service_version="1.3.0",
    exporter_type="prometheus",
    prometheus_port=8001,
    jaeger_endpoint="localhost:6831"
)

Prometheus 配置:

# prometheus.yml
scrape_configs:
  - job_name: 'stock-backend-otel'
    static_configs:
      - targets: ['localhost:8001']

Jaeger 部署:

docker run -d --name jaeger \
  -p 6831:6831/udp \
  -p 16686:16686 \
  jaegertracing/all-in-one:latest

访问 Jaeger UI: http://localhost:16686

使用示例

1. 自动埋点（推荐）

FastAPI 自动埋点会追踪所有 HTTP 请求：

from app.observability import configure_opentelemetry

# 配置 OpenTelemetry
otel_config = configure_opentelemetry(exporter_type="console")

# 自动埋点
if otel_config:
    otel_config.instrument_app(app)

2. 手动创建 Span

from app.observability import trace_span

async def fetch_stock_data(symbol: str):
    # 创建自定义 span
    with trace_span("fetch_stock_data", {"symbol": symbol, "market": "A_SHARE"}):
        # 你的业务逻辑
        data = await some_data_source.get_data(symbol)
        return data

3. 添加 Span 属性

from opentelemetry import trace

@router.get("/api/stocks/{symbol}")
async def get_stock(symbol: str):
    # 获取当前 span
    span = trace.get_current_span()

    # 添加属性
    span.set_attribute("stock.symbol", symbol)
    span.set_attribute("user.id", "12345")

    # 业务逻辑
    data = await fetch_data(symbol)

    # 记录事件
    span.add_event("data_fetched", {"records": len(data)})

    return data

4. 创建自定义 Metrics

from opentelemetry import metrics

# 获取 meter
meter = metrics.get_meter(__name__)

# 创建 Counter
request_counter = meter.create_counter(
    "stock_requests_total",
    description="Total stock data requests",
    unit="1"
)

# 使用 Counter
request_counter.add(1, {"symbol": "AAPL", "status": "success"})

Kubernetes 部署

环境变量配置

# k8s/deployment.yaml
env:
  - name: OTEL_EXPORTER_TYPE
    value: "otlp"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://otel-collector.monitoring:4317"
  - name: OTEL_SERVICE_NAME
    value: "stock-backend"
  - name: OTEL_SERVICE_VERSION
    valueFrom:
      fieldRef:
        fieldPath: metadata.labels['version']

Annotations for Prometheus

# k8s/deployment.yaml
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8001"
    prometheus.io/path: "/metrics"

监控查询示例

Prometheus Queries

# HTTP 请求成功率
sum(rate(http_server_duration_count{http_status_code=~"2.."}[5m])) /
sum(rate(http_server_duration_count[5m]))

# P95 响应时间
histogram_quantile(0.95,
  sum(rate(http_server_duration_bucket[5m])) by (le, http_route)
)

# 数据源错误率
sum(rate(data_source_requests_total{status="failure"}[5m])) by (source_id)

Jaeger Trace 搜索

1. 打开 Jaeger UI: http://localhost:16686
2. 选择 Service: stock-recommendation-system
3. 选择 Operation: GET /api/collab/recommend
4. 添加 Tags: stock.symbol=AAPL
5. 查看完整调用链

Trace ID 关联

所有日志自动包含 trace_id，可以与 traces 关联：

{
  "timestamp": "2025-12-26T10:00:00.000Z",
  "level": "INFO",
  "trace_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "span_id": "abcdef123456",
  "message": "Fetching stock data",
  "extra": {
    "symbol": "AAPL",
    "market": "US"
  }
}

在 Jaeger 中复制 trace_id，在日志系统（如 Grafana Loki）中搜索：

{trace_id="a1b2c3d4-e5f6-7890-abcd-ef1234567890"}

最佳实践

1. Span 命名

使用清晰的命名约定：

# ✅ Good
with trace_span("database.query.select_stock"):
    ...

with trace_span("external_api.sina_finance.get_quote"):
    ...

# ❌ Bad
with trace_span("do_stuff"):
    ...

2. 属性 (Attributes)

遵循 OpenTelemetry 语义约定：

span.set_attribute("db.system", "postgresql")
span.set_attribute("db.statement", "SELECT * FROM stocks")
span.set_attribute("http.method", "GET")
span.set_attribute("http.url", "/api/stocks/AAPL")

3. 采样 (Sampling)

生产环境建议配置采样率以控制成本：

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

# 10% 采样率
sampler = TraceIdRatioBased(0.1)

tracer_provider = TracerProvider(
    resource=resource,
    sampler=sampler
)

4. 性能考虑

• 批量导出（默认配置）降低网络开销
• 避免在高频路径添加过多手动 span
• 使用自动埋点覆盖 80% 场景

故障排查

问题1: OpenTelemetry 未生效

检查是否正确安装依赖：

python -c "from opentelemetry import trace; print('OpenTelemetry available')"

问题2: Traces 未显示在 Jaeger

1. 检查 Jaeger Collector 是否运行：

   curl http://localhost:14269/
   

2. 检查 Jaeger endpoint 配置是否正确

3. 查看应用日志中的 OpenTelemetry 配置信息

问题3: Prometheus Metrics 未抓取

1. 验证 metrics endpoint:

   curl http://localhost:8001/metrics
   

2. 检查 Prometheus target 状态

相关文档

• OpenTelemetry Python 文档
• OpenTelemetry Collector
• Jaeger 文档
• 内部文档: STRUCTURED_LOGGING_GUIDE.md

更新日志

• 2025-12-26: 初始集成 OpenTelemetry，支持 OTLP/Prometheus/Jaeger exporters