docs(agents): add hardened AGENTS instructions for root and submodules
This commit is contained in:
@@ -1,34 +1,174 @@
|
||||
# agent-platform/AGENTS.md - 项目根主指令(硬化版)
|
||||
# Agent Platform — Project Root Instructions
|
||||
|
||||
## 目标
|
||||
企业智能体平台项目以可靠性、可审计性、可观测性和低风险变更为首要目标。
|
||||
## Project Context
|
||||
Enterprise AI agent platform powering three production scenarios:
|
||||
1. Presales — proposal documents (Word/Excel) in ~30 min from days
|
||||
2. Security — incident reports from hours to minutes
|
||||
3. Knowledge QA — citation-backed answers, ≥90 % reference coverage
|
||||
|
||||
## 全局约束
|
||||
- Python 版本必须为 3.11+。
|
||||
- 所有新增函数必须有完整类型注解;禁止裸 `Any`。
|
||||
- Pydantic 的 `list`/`dict` 字段必须使用 `Field(default_factory=...)`。
|
||||
- 涉及货币、成本、比率的字段统一使用 `Decimal`,禁止 `float`。
|
||||
- 配置加载统一通过 `get_settings()` 惰性单例,禁止模块级实例化 `Settings()`。
|
||||
- 依赖安装与执行统一使用 `uv`,并保持 `uv.lock` 与代码同步。
|
||||
## Tech Stack (all versions locked in uv.lock)
|
||||
- Orchestration : langgraph ≥ 0.3.0
|
||||
StateGraph + interrupt/resume for human-in-the-loop
|
||||
- LLM cloud : anthropic ≥ 0.50.0 → claude-sonnet-4-6-20260217
|
||||
- LLM local : Qwen2.5-14B / 7B via OpenAI-compatible HTTP endpoint
|
||||
- Vector DB : pymilvus ≥ 2.4.0 + BGE-M3 embeddings + BGE-reranker
|
||||
- API : fastapi ≥ 0.115.0 + uvicorn[standard]
|
||||
- ORM : sqlalchemy[asyncio] ≥ 2.0.30 + asyncpg ≥ 0.29.0
|
||||
- Migrations : alembic ≥ 1.13.0
|
||||
- Observability : opentelemetry-sdk ≥ 1.27.0
|
||||
+ prometheus exporter + grafana dashboards
|
||||
- Auth : python-jose[cryptography] + passlib[bcrypt]
|
||||
- Encryption : cryptography ≥ 42.0.0 (AES-256-GCM, audit payloads)
|
||||
- Docs : python-docx ≥ 1.1.2
|
||||
|
||||
## 代码组织
|
||||
- `core/` 放置领域逻辑,不依赖 `api/`。
|
||||
- `api/` 仅做协议层、鉴权、中间件与路由编排。
|
||||
- `db/` 负责仓储和迁移,不承载业务流程。
|
||||
- `profiles/` 仅存 SOP/prompt 资产,不写运行时代码。
|
||||
- `tests/` 按 `unit/integration/contracts` 分层。
|
||||
## Repository Layout
|
||||
```
|
||||
agent-platform/
|
||||
├── core/
|
||||
│ ├── engine/ # LangGraph: state, 5 nodes, graph builder
|
||||
│ ├── profile/ # SOP YAML schema (Pydantic), loader, registry
|
||||
│ ├── tools/ # BaseTool, ToolResult, ToolExecutor, registry
|
||||
│ ├── llm/ # ClaudeClient, PromptBuilder
|
||||
│ ├── verify/ # EvidenceScorer
|
||||
│ ├── audit/ # AuditLogger (3-tier), Sanitizer, Encryptor
|
||||
│ ├── observability/ # OTel setup, decorators, CostTracker, dashboards/
|
||||
│ └── router/ # ModelRouter
|
||||
├── api/
|
||||
│ ├── routes/ # health, chat, approve, trace
|
||||
│ ├── contracts/ # Pydantic req/resp models, permissions matrix
|
||||
│ └── middleware/ # tracing, auth (JWT), rbac, rate_limit
|
||||
├── profiles/
|
||||
│ ├── presales/prompts/
|
||||
│ ├── security_analysis/prompts/
|
||||
│ └── knowledge_qa/prompts/
|
||||
├── db/
|
||||
│ ├── base.py
|
||||
│ ├── models.py
|
||||
│ ├── repositories/
|
||||
│ └── migrations/versions/
|
||||
├── tests/
|
||||
│ ├── unit/
|
||||
│ ├── integration/
|
||||
│ ├── contracts/ # real Claude API; @pytest.mark.claude_api
|
||||
│ └── fixtures/
|
||||
├── scripts/
|
||||
│ ├── setup_env.sh
|
||||
│ ├── health_check.sh
|
||||
│ ├── seed_data.py
|
||||
│ └── gen_audit_key.py
|
||||
├── docker/
|
||||
│ ├── Dockerfile.api
|
||||
│ ├── otel-collector-config.yml
|
||||
│ └── prometheus.yml
|
||||
├── .env.example
|
||||
├── tests/.env.test # committed; no real secrets
|
||||
├── pyproject.toml
|
||||
├── uv.lock # MUST be committed; MUST pass uv sync --frozen
|
||||
├── alembic.ini
|
||||
├── Makefile
|
||||
└── docker-compose.yml
|
||||
```
|
||||
|
||||
## 质量门禁
|
||||
- 提交前必须通过:`ruff check`、`mypy`、`pytest --collect-only`。
|
||||
- 任何 schema、鉴权、审计、加密相关变更必须补充测试。
|
||||
- 禁止绕过失败门禁直接合并。
|
||||
## Critical Architecture Rules (NEVER Violate)
|
||||
|
||||
## 安全与审计
|
||||
- 严禁提交真实密钥、令牌、证书。
|
||||
- 审计 payload 必须加密存储,且支持 keyring 轮换。
|
||||
- 生产环境日志禁止输出敏感字段原文。
|
||||
### R1 — Tool Whitelist Enforcement
|
||||
`ToolExecutor.execute_tool()` checks `profile.allowed_tools` BEFORE
|
||||
calling any tool.
|
||||
Violation → `raise ToolNotAllowedError(tool_name, profile_name)`
|
||||
+ write AuditEvent(event_type="whitelist_violation").
|
||||
Tool implementations NEVER perform whitelist checks themselves.
|
||||
|
||||
## 变更原则
|
||||
- 小步提交,保持可回滚。
|
||||
- 不在无关文件中引入格式化噪音。
|
||||
- 新增依赖必须说明必要性与替代方案不可行性。
|
||||
### R2 — Exception Boundary
|
||||
| Layer | Raises? | On failure |
|
||||
|------------------------|---------------|------------------------|
|
||||
| ToolExecutor / Act node| YES — policy | ToolNotAllowedError |
|
||||
| BaseTool.execute() | NEVER | ToolResult(success=False) |
|
||||
| LangGraph nodes | NEVER escape | set state.error → END |
|
||||
|
||||
### R3 — State Serialization
|
||||
`AgentState` is a `BaseModel` (fully JSON-serializable at all times).
|
||||
Large objects (documents, embeddings, raw HTML) → stored in DB /
|
||||
object-store; only UUID / string references live inside state.
|
||||
Validate with `AgentState.validate_serializable(state)` after every
|
||||
node that mutates state.
|
||||
|
||||
### R4 — Approval Gates
|
||||
Any tool whose `ToolDeclaration.requires_approval` is `True`
|
||||
MUST call `interrupt(ApprovalPayload(...))` before execution.
|
||||
Resume via `Command(resume=ApprovalDecision(...))`.
|
||||
There are NO exceptions to this rule.
|
||||
|
||||
### R5 — Audit Completeness
|
||||
Every tool execution — success, failure, AND skipped — MUST produce
|
||||
an `AuditEvent` written by `AuditLogger`.
|
||||
A missing audit entry is a P0 production bug.
|
||||
`audit_event` rows are append-only; enforced by DB trigger
|
||||
`trg_audit_event_no_mutation`.
|
||||
|
||||
### R6 — Sensitive Data
|
||||
`Sanitizer.mask()` MUST be applied to all user inputs before they
|
||||
appear in logs, audit payloads, or API responses.
|
||||
Patterns:
|
||||
- Phone `1[3-9]\d{9}` → `138****8888`
|
||||
- ID card 18-digit → `****19700101****`
|
||||
- password / key fields → `[REDACTED]`
|
||||
|
||||
### R7 — Async Discipline
|
||||
No synchronous blocking inside any `async def`.
|
||||
CPU-bound operations → `asyncio.to_thread()`.
|
||||
Every external I/O call carries an explicit `timeout=` parameter.
|
||||
No `asyncio.create_task()` fire-and-forget without error collection.
|
||||
|
||||
### R8 — Decimal for Money
|
||||
All cost / fee / score calculations use `Decimal` with explicit
|
||||
precision and `ROUND_HALF_UP`.
|
||||
DB columns: `Numeric(12,6)` for USD, `Numeric(12,4)` for CNY,
|
||||
`Numeric(5,4)` for evidence scores.
|
||||
NEVER store or compute these values as `float`.
|
||||
|
||||
## Settings Access Pattern
|
||||
```python
|
||||
# CORRECT — lazy singleton, test-safe
|
||||
from core.config import get_settings
|
||||
s = get_settings()
|
||||
|
||||
# WRONG — instantiated at import time, breaks test monkeypatching
|
||||
from core.config import settings # this symbol does not exist
|
||||
```
|
||||
|
||||
## Gate Definitions
|
||||
Every task MUST pass its gate before being declared complete.
|
||||
|
||||
### Static Gate (no Docker required)
|
||||
```bash
|
||||
uv sync --frozen
|
||||
uv run ruff check . --select E,F,I --quiet
|
||||
uv run mypy core/ api/ --ignore-missing-imports --no-error-summary
|
||||
uv run pytest tests/unit/ -x -q --tb=short
|
||||
```
|
||||
|
||||
### Enhanced Gate (Docker required)
|
||||
```bash
|
||||
docker compose up -d --wait
|
||||
uv run alembic upgrade head
|
||||
uv run pytest tests/integration/ -x -q --tb=short
|
||||
bash scripts/health_check.sh
|
||||
```
|
||||
|
||||
## Testing Standards
|
||||
- Unit tests : zero external I/O; mock DB, HTTP, Claude, Milvus, Redis
|
||||
- Integration : real services via testcontainers or running Docker stack
|
||||
- Contract : real Claude API; `@pytest.mark.claude_api`;
|
||||
skipped automatically if `ANTHROPIC_API_KEY` is unset
|
||||
- Coverage gate: `--cov-fail-under=80` across `core/` and `api/`
|
||||
- Secret injection: via `tests/conftest.py` `settings_override` fixture;
|
||||
NEVER `os.environ` assignment in test files directly
|
||||
|
||||
## Commit Convention
|
||||
```
|
||||
feat(engine): implement interrupt/resume approval in act_node
|
||||
fix(verify): correct EWMA weight formula in EvidenceScorer
|
||||
test(api): add integration test for /approve expired-token path
|
||||
chore(deps): add alembic, cryptography to pyproject.toml
|
||||
refactor(tools): extract _resolve_params to standalone module
|
||||
```
|
||||
|
||||
+160
-10
@@ -1,13 +1,163 @@
|
||||
# api/AGENTS.md
|
||||
# API Layer — Submodule Instructions
|
||||
|
||||
## 范围
|
||||
`api/` 负责 FastAPI 路由、请求响应契约、中间件和权限控制。
|
||||
## FastAPI Patterns
|
||||
- All route handlers: `async def`
|
||||
- Return types: always a declared Pydantic response model — NEVER
|
||||
raw `dict`, `None`, or untyped `Any`
|
||||
- Error format (ALL error responses, without exception):
|
||||
```python
|
||||
class ErrorResponse(BaseModel):
|
||||
code: str # machine-readable, e.g. "TOOL_NOT_ALLOWED"
|
||||
message: str # human-readable Chinese message
|
||||
trace_id: str
|
||||
details: dict = Field(default_factory=dict)
|
||||
```
|
||||
- HTTP status codes:
|
||||
| Situation | Code |
|
||||
|--------------------------------------|------|
|
||||
| Success | 200 |
|
||||
| Async task accepted | 202 |
|
||||
| Client validation error | 400 |
|
||||
| Missing / invalid JWT | 401 |
|
||||
| Insufficient permission | 403 |
|
||||
| Resource not found | 404 |
|
||||
| State conflict (e.g. already decided)| 409 |
|
||||
| Pydantic validation failure | 422 |
|
||||
| Dependency unhealthy | 503 |
|
||||
| Unhandled server error | 500 |
|
||||
|
||||
## 约束
|
||||
- 路由层不承载业务规则,仅做参数校验与编排。
|
||||
- 权限、限流、鉴权失败返回明确 HTTP 状态码与错误码。
|
||||
- 合同模型字段禁止可变默认值,统一 `Field(default_factory=...)`。
|
||||
## Settings Access
|
||||
```python
|
||||
# CORRECT
|
||||
from core.config import get_settings
|
||||
s = get_settings()
|
||||
|
||||
## 工程要求
|
||||
- 新增路由必须包含至少一个契约测试。
|
||||
- 中间件顺序变更需说明兼容性影响。
|
||||
# WRONG — do not import a module-level `settings` variable
|
||||
from core.config import settings # symbol does not exist; will ImportError
|
||||
```
|
||||
|
||||
## AgentEngine Lifecycle
|
||||
- Initialised ONCE in `api/main.py` `lifespan` startup
|
||||
- Stored at `app.state.engine`
|
||||
- NEVER instantiate `AgentEngine` inside a route handler or
|
||||
per-request dependency
|
||||
|
||||
## Middleware Stack (outermost → innermost, registration order matters)
|
||||
1. `TracingMiddleware`
|
||||
- Read `X-Trace-ID` header; generate `uuid4()` if absent
|
||||
- Store in `request.state.trace_id`
|
||||
- Add `X-Trace-ID` to response headers
|
||||
- Create OTel HTTP span (method, path, status_code, duration_ms)
|
||||
- Bind `trace_id` to `structlog` context for the request lifetime
|
||||
|
||||
2. `LoggingMiddleware`
|
||||
- Log request: method, path, content-length, trace_id
|
||||
- Log response: status_code, duration_ms, trace_id
|
||||
- Sanitize `Authorization` header (log only first 12 chars + `…`)
|
||||
- NEVER log request body (may contain secrets)
|
||||
|
||||
3. `RateLimitMiddleware`
|
||||
- Redis token-bucket keyed on `tenant_id` (extracted from JWT claim
|
||||
without full JWT validation — just base64-decode the payload)
|
||||
- Limit: `settings.REDIS_RATE_LIMIT_PER_MINUTE` per tenant per minute
|
||||
- Exceeded → 429 `ErrorResponse(code="RATE_LIMIT_EXCEEDED", …)`
|
||||
|
||||
4. CORS
|
||||
- `ENVIRONMENT=development` → `allow_origins=["*"]`
|
||||
- `ENVIRONMENT=production` → `allow_origins=settings.ALLOWED_ORIGINS`
|
||||
|
||||
Authentication is handled by `require_auth` FastAPI `Depends`, NOT a
|
||||
global middleware, so that `/health`, `/docs`, `/redoc`, `/openapi.json`
|
||||
remain public.
|
||||
|
||||
## RBAC Permission Matrix
|
||||
```python
|
||||
ROLE_PERMISSIONS: dict[str, set[str]] = {
|
||||
"admin": {"chat:create","chat:read","approve:write",
|
||||
"trace:read","user:manage"},
|
||||
"manager": {"chat:create","chat:read","approve:write","trace:read"},
|
||||
"analyst": {"chat:create","chat:read","trace:read"},
|
||||
"viewer": {"chat:read","trace:read"},
|
||||
}
|
||||
```
|
||||
Usage pattern:
|
||||
```python
|
||||
@router.post("/approve/{trace_id}")
|
||||
async def approve(
|
||||
trace_id: str,
|
||||
body: ApprovalRequest,
|
||||
user: UserContext = Depends(require_auth),
|
||||
_: None = Depends(require_permission("approve:write")),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
): ...
|
||||
```
|
||||
|
||||
## SSE Event Contract (/api/v1/chat)
|
||||
All server-sent events MUST conform to this schema:
|
||||
|
||||
```
|
||||
event: node_complete
|
||||
data: {"node":"ask","status":"success","duration_ms":234,"trace_id":"…"}
|
||||
|
||||
event: approval_required
|
||||
data: {
|
||||
"trace_id":"…","step_id":"…",
|
||||
"action_description":"…","risk_level":"high",
|
||||
"approver_role":"manager","params_preview":{…}
|
||||
}
|
||||
|
||||
event: final_result
|
||||
data: {
|
||||
"output_type":"document|text|insufficient_data|clarification_needed",
|
||||
"content":"…",
|
||||
"citations":[…],
|
||||
"evidence_score":"0.8700", ← Decimal serialised as string
|
||||
"evidence_level":"sufficient",
|
||||
"trace_id":"…"
|
||||
}
|
||||
|
||||
event: error
|
||||
data: {"code":"TOOL_FAILED","message":"…","trace_id":"…"}
|
||||
|
||||
data: [DONE]
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `evidence_score` MUST be serialised as a string to preserve
|
||||
`Decimal` precision in JSON (avoids float rounding)
|
||||
- The `[DONE]` terminator MUST always be sent, even after an error
|
||||
- Do NOT send any events after `[DONE]`
|
||||
|
||||
## Approval Endpoint Rules (/api/v1/approve/{trace_id})
|
||||
Exact sequence — do not reorder:
|
||||
1. Verify caller has `approve:write` permission
|
||||
2. Load `PendingApproval` where `trace_id = :trace_id AND status = 'pending'`
|
||||
→ 404 if not found
|
||||
3. Check `expired_at > now()`
|
||||
→ 409 `{"code":"APPROVAL_EXPIRED"}` if expired
|
||||
4. Atomic DB transaction:
|
||||
a. `PendingApproval.status` ← decision
|
||||
b. `PendingApproval.decider_id` ← `user.user_id`
|
||||
c. `PendingApproval.decided_at` ← `now()`
|
||||
d. `TraceSession.status` ← `"running"` (if approved) or `"failed"`
|
||||
5. Fire-and-forget resume (do NOT await completion in the request):
|
||||
`asyncio.create_task(engine.resume(session_id, Command(resume=…)))`
|
||||
with structured error logging on task failure
|
||||
6. Return 202: `{"status":"resuming","trace_id":"…"}`
|
||||
|
||||
## Health Endpoint Contract (GET /health)
|
||||
```python
|
||||
class HealthResponse(BaseModel):
|
||||
status: Literal["ok", "degraded"]
|
||||
version: str
|
||||
environment: str
|
||||
services: dict[str, bool] # {"db": True, "redis": True, "milvus": False, …}
|
||||
checked_at: datetime
|
||||
|
||||
# Any service False → HTTP 503 (body still JSON, allows LB to see detail)
|
||||
# All services True → HTTP 200
|
||||
```
|
||||
Real connectivity checks (not just config presence):
|
||||
- `db` : `SELECT 1` via async engine
|
||||
- `redis` : `PING` via aioredis
|
||||
- `milvus`: `list_collections()` via pymilvus
|
||||
|
||||
+147
-11
@@ -1,14 +1,150 @@
|
||||
# core/AGENTS.md
|
||||
# Core Engine — Submodule Instructions
|
||||
|
||||
## 范围
|
||||
`core/` 负责智能体引擎、LLM 路由、工具执行、校验、审计与可观测基础能力。
|
||||
## Module Responsibilities
|
||||
```
|
||||
core/engine/state.py AgentState (BaseModel, fully serializable)
|
||||
core/engine/nodes.py 5 node functions: ask, plan, act, verify, report
|
||||
core/engine/graph.py StateGraph wiring + conditional edges
|
||||
core/profile/schema.py Pydantic Profile model (frozen, validated)
|
||||
core/profile/loader.py YAML → Profile with full contract validation
|
||||
core/profile/registry.py ProfileRegistry singleton with hot-reload
|
||||
core/tools/base.py BaseTool, ToolResult, ToolDeclaration
|
||||
core/tools/executor.py ToolExecutor — whitelist + permission + audit
|
||||
core/tools/registry.py ToolRegistry singleton
|
||||
core/llm/claude_client.py Async Claude API wrapper
|
||||
core/llm/prompt_builder.py All prompt text centralised here
|
||||
core/router/model_router.py Routing logic (cloud vs. local)
|
||||
core/verify/scorer.py EvidenceScorer (formula + threshold)
|
||||
core/audit/logger.py 3-tier AuditLogger (L1 PG / L2 OTel / L3 enc)
|
||||
core/audit/sanitizer.py PII masking
|
||||
core/audit/encryptor.py AES-256-GCM for L3 payloads
|
||||
```
|
||||
|
||||
## 约束
|
||||
- 保持纯业务内核,禁止引入 API 框架对象。
|
||||
- 所有状态对象、输入输出契约需类型稳定、可序列化。
|
||||
- 成本、预算、阈值统一 `Decimal`。
|
||||
- 配置访问必须走 `get_settings()`。
|
||||
## LangGraph Strict Patterns
|
||||
|
||||
## 工程要求
|
||||
- 复杂分支逻辑应有最小单元测试覆盖。
|
||||
- 审计与加密路径出现异常时必须返回可诊断错误,不可静默吞错。
|
||||
### State
|
||||
- `AgentState(BaseModel)` — JSON-serializable at all times
|
||||
- `frozen=False` (nodes mutate state in-place and return it)
|
||||
- Large objects stored externally; state holds only references (UUIDs / paths)
|
||||
- After every node that mutates state, call
|
||||
`AgentState.validate_serializable(state)` in debug/test mode
|
||||
|
||||
### Checkpointer
|
||||
- Production : `AsyncPostgresSaver` (from langgraph-checkpoint-postgres)
|
||||
- Tests only : `MemorySaver`
|
||||
- NEVER use `MemorySaver` in production code paths
|
||||
|
||||
### Conditional Edges
|
||||
- Use string literals for node names in `add_conditional_edges`
|
||||
- No lambda closures that capture mutable external state
|
||||
- Every node MUST have a defined exit route to `END` on error
|
||||
|
||||
### Approval Flow
|
||||
```
|
||||
act_node detects requires_approval=True
|
||||
→ build ApprovalPayload
|
||||
→ interrupt(payload) # LangGraph pauses here
|
||||
→ [ external /approve call ]
|
||||
→ Command(resume=ApprovalDecision(...))
|
||||
→ act_node resumes from saved checkpoint
|
||||
```
|
||||
|
||||
## Exception Boundary
|
||||
|
||||
### Node Error Boundary
|
||||
Each node wraps its body in `try/except Exception`:
|
||||
- Unrecoverable error → `state.error = {"code":…, "message":…, "node":…}`
|
||||
return state (graph routes to END via conditional edge)
|
||||
- NEVER let an unhandled exception escape a node function
|
||||
|
||||
## Claude API Contract
|
||||
- Use the `anthropic` SDK version locked in `uv.lock`
|
||||
- Follow SDK type hints exactly; do NOT hardcode beta-header string
|
||||
literals; import SDK constants where available
|
||||
- Structured output: use `tool_choice={"type": "any"}` to force
|
||||
tool calling in Ask / Plan / Verify nodes
|
||||
- Parse tool results: `content_block.input` (dict) directly —
|
||||
NEVER regex-parse the text representation of tool calls
|
||||
- Prompt Caching: add `cache_control={"type": "ephemeral"}` to
|
||||
system message blocks; if the SDK version does not support this
|
||||
field, omit it gracefully (do NOT crash)
|
||||
- Token tracking: record `usage.input_tokens`, `usage.output_tokens`,
|
||||
`usage.cache_read_input_tokens` after EVERY API call
|
||||
- Timeout: pass `httpx.Timeout(connect=10, read=60, write=10, pool=5)`
|
||||
to `AsyncAnthropic` constructor — NOT per-call
|
||||
|
||||
## Tool Contract
|
||||
Every class in `core/tools/` that implements a tool MUST:
|
||||
1. Subclass `BaseTool`
|
||||
2. Declare a `@property declaration(self) -> ToolDeclaration`
|
||||
3. Declare a `@property input_schema(self) -> type[BaseModel]`
|
||||
4. Implement `async _execute_impl(params, user_id, tenant_id,
|
||||
data_scope) -> ToolResult`
|
||||
5. Rely on the base `execute()` for: param validation → retry
|
||||
(tenacity, max=2, wait_fixed=1 s) → timeout
|
||||
(asyncio.wait_for, limit = declaration.max_duration_ms / 1000 s)
|
||||
→ catch-all → ToolResult(success=False, error=sanitized)
|
||||
6. NEVER check the tool whitelist inside a tool
|
||||
7. Honour Mock Policy: check `get_settings().ENVIRONMENT` before
|
||||
falling back to mock data; always set `meta["mock"] = True`
|
||||
|
||||
## Evidence Scoring Formula
|
||||
DO NOT modify this formula without also updating
|
||||
`tests/unit/test_scorer.py` and the Profile schema defaults.
|
||||
|
||||
```python
|
||||
# All operands and result are Decimal, rounded to 4 decimal places
|
||||
score = (
|
||||
Decimal("0.4") * retrieval_score
|
||||
+ Decimal("0.3") * completeness_score
|
||||
+ Decimal("0.3") * historical_accuracy
|
||||
)
|
||||
|
||||
# Default thresholds (overridable per Profile):
|
||||
# sufficient : score >= Decimal("0.85") → normal output
|
||||
# limited : score >= Decimal("0.60") → output with ⚠️ warning
|
||||
# insufficient : score < Decimal("0.60") → reject, explain gaps
|
||||
```
|
||||
|
||||
## Audit Layer Contracts
|
||||
|
||||
### L1 — PostgreSQL (AuditEvent)
|
||||
- Append-only: DB trigger `trg_audit_event_no_mutation` enforces
|
||||
no UPDATE or DELETE at the database level
|
||||
- ORM `AuditEventRepository` exposes NO `update`, `delete`,
|
||||
`remove`, `modify`, or `patch` methods
|
||||
- `mock_flag` column MUST be set to `True` when `ToolResult.meta`
|
||||
contains `"mock": True`
|
||||
|
||||
### L2 — OpenTelemetry spans
|
||||
- Auto-collected by `@tool_span`, `@llm_span`, `@agent_span` decorators
|
||||
- Span attributes follow OpenTelemetry GenAI semantic conventions where
|
||||
applicable
|
||||
|
||||
### L3 — Encrypted payloads (AuditPayload)
|
||||
- Cipher: AES-256-GCM
|
||||
- Key source: `get_settings().AUDIT_KEYRING_JSON` parsed as
|
||||
`dict[key_id, base64_key]`; active key from `AUDIT_ACTIVE_KEY_ID`
|
||||
- IV: `os.urandom(12)` — fresh per encryption, prepended to ciphertext
|
||||
- Stored layout: `IV (12 B) | tag (16 B) | ciphertext`
|
||||
- Key rotation: use a new `key_id`; old payloads remain decryptable
|
||||
as long as old key is still in keyring
|
||||
|
||||
## _resolve_params Template Syntax
|
||||
`act_node` resolves parameter templates before every tool call.
|
||||
Supported patterns (all `{{…}}` delimited):
|
||||
|
||||
| Template | Resolves to |
|
||||
|---------------------------------------|------------------------------------------------|
|
||||
| `{{user_input}}` | `state.user_input` (raw string) |
|
||||
| `{{user_input.field_name}}` | `state.extracted_params.get("field_name")` |
|
||||
| `{{steps.step_id.summary}}` | `state.step_outputs["step_id"]["summary"]` |
|
||||
| `{{steps.step_id.data_ref}}` | `state.step_outputs["step_id"]["data_ref"]` |
|
||||
| `{{profile.output_template}}` | `active_profile.output_template` |
|
||||
| `{{profile.terminology}}` | `active_profile.terminology` |
|
||||
|
||||
Rules:
|
||||
- Recurse into dict values and list items
|
||||
- Unresolvable key → keep original `{{…}}` string; log a WARNING;
|
||||
record in `meta["unresolved_templates"]` of the step result
|
||||
- NEVER raise an exception for a missing template key
|
||||
|
||||
+123
-9
@@ -1,11 +1,125 @@
|
||||
# tests/AGENTS.md
|
||||
# Tests — Submodule Instructions
|
||||
|
||||
## 测试分层
|
||||
- `unit/`: 纯函数与小范围组件测试,禁止外部依赖。
|
||||
- `integration/`: 数据库、缓存、向量库、第三方 API 集成测试。
|
||||
- `contracts/`: API 请求响应与权限矩阵契约测试。
|
||||
## Test Isolation Contract
|
||||
| Layer | External I/O allowed? | Services needed |
|
||||
|-------------|----------------------|--------------------------|
|
||||
| unit/ | NONE | none |
|
||||
| integration/| YES (mocked or real) | testcontainers or Docker |
|
||||
| contracts/ | Real Claude API only | ANTHROPIC_API_KEY set |
|
||||
|
||||
## 约束
|
||||
- 测试中密钥统一由 fixture 注入,不从真实环境读取。
|
||||
- 测试数据必须可重复、可清理。
|
||||
- 对审计、鉴权、预算阈值相关逻辑优先补全边界测试。
|
||||
## Secret Injection (MANDATORY pattern)
|
||||
```python
|
||||
# tests/conftest.py — root fixture, autouse=True
|
||||
@pytest.fixture(autouse=True)
|
||||
def settings_override(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-test-not-real")
|
||||
monkeypatch.setenv("DATABASE_URL",
|
||||
"postgresql+asyncpg://test:test@localhost/test_db")
|
||||
monkeypatch.setenv("JWT_SECRET_KEY",
|
||||
"test_jwt_secret_key_32chars_padded_x")
|
||||
monkeypatch.setenv("AUDIT_KEYRING_JSON",
|
||||
'{"test_key_1": "<base64-of-32-A-bytes>"}')
|
||||
monkeypatch.setenv("AUDIT_ACTIVE_KEY_ID", "test_key_1")
|
||||
get_settings.cache_clear()
|
||||
yield
|
||||
get_settings.cache_clear()
|
||||
```
|
||||
NEVER assign `os.environ[…]` directly in test files.
|
||||
NEVER import a module-level `settings` object in test files.
|
||||
|
||||
## Fixture Catalogue (tests/conftest.py)
|
||||
| Fixture name | Scope | Description |
|
||||
|------------------------|----------|------------------------------------------|
|
||||
| `settings_override` | function | autouse; injects all required secrets |
|
||||
| `async_engine` | session | in-memory SQLite or testcontainers PG |
|
||||
| `db_session` | function | isolated session; rolls back after test |
|
||||
| `async_client` | function | `httpx.AsyncClient` against test app |
|
||||
| `mock_claude` | function | respx router for api.anthropic.com |
|
||||
| `sample_presales_profile` | session | minimal Profile object (no file I/O) |
|
||||
| `tool_registry` | function | registry with mock `_execute_impl` |
|
||||
| `agent_graph_memory` | function | graph with MemorySaver (tests only) |
|
||||
|
||||
## Test Naming Convention
|
||||
```
|
||||
test_<unit_under_test>_<scenario>_<expected_outcome>
|
||||
|
||||
Examples:
|
||||
test_ask_node_missing_required_fields_returns_clarification
|
||||
test_act_node_high_risk_tool_triggers_interrupt
|
||||
test_tool_executor_unlisted_tool_raises_and_audits
|
||||
test_evidence_scorer_below_reject_threshold_returns_insufficient
|
||||
test_cost_calculation_batch_mode_halves_cost
|
||||
test_audit_event_repo_has_no_update_method
|
||||
```
|
||||
|
||||
## Coverage Requirements (enforced by gate)
|
||||
```
|
||||
core/engine/ ≥ 90 %
|
||||
core/verify/ ≥ 95 % ← formula is critical path
|
||||
core/audit/ ≥ 85 %
|
||||
core/tools/ ≥ 80 %
|
||||
api/routes/ ≥ 85 %
|
||||
api/middleware/ ≥ 80 %
|
||||
─────────────────────────────
|
||||
core/ + api/ overall ≥ 80 % ← enforced by --cov-fail-under=80
|
||||
```
|
||||
|
||||
## Mock Claude API Pattern
|
||||
```python
|
||||
# tests/fixtures/mock_claude.py
|
||||
import respx, httpx, json
|
||||
|
||||
def make_tool_use_response(tool_name: str, input_dict: dict,
|
||||
input_tokens: int = 800,
|
||||
output_tokens: int = 150,
|
||||
cache_read_tokens: int = 600) -> dict:
|
||||
return {
|
||||
"id": "msg_mock_001",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"content": [{
|
||||
"type": "tool_use",
|
||||
"id": "toolu_mock_001",
|
||||
"name": tool_name,
|
||||
"input": input_dict,
|
||||
}],
|
||||
"model": "claude-sonnet-4-6-20260217",
|
||||
"stop_reason": "tool_use",
|
||||
"usage": {
|
||||
"input_tokens": input_tokens,
|
||||
"output_tokens": output_tokens,
|
||||
"cache_read_input_tokens": cache_read_tokens,
|
||||
"cache_creation_input_tokens": 0,
|
||||
},
|
||||
}
|
||||
|
||||
def make_text_response(text: str, input_tokens: int = 500,
|
||||
output_tokens: int = 800) -> dict:
|
||||
return {
|
||||
"id": "msg_mock_002",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"content": [{"type": "text", "text": text}],
|
||||
"model": "claude-sonnet-4-6-20260217",
|
||||
"stop_reason": "end_turn",
|
||||
"usage": {
|
||||
"input_tokens": input_tokens,
|
||||
"output_tokens": output_tokens,
|
||||
"cache_read_input_tokens": 0,
|
||||
"cache_creation_input_tokens": 0,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Acceptance Thresholds (contract tests with real Claude API)
|
||||
These thresholds are checked by `tests/contracts/` and must pass
|
||||
before any production deployment:
|
||||
|
||||
| Metric | Gate |
|
||||
|--------------------------------|------------|
|
||||
| Task success rate (presales) | ≥ 90 % |
|
||||
| Citation coverage | ≥ 90 % |
|
||||
| Hallucination rate (sampled) | ≤ 5 % |
|
||||
| RAG Recall@5 | ≥ 95 % |
|
||||
| P95 end-to-end latency | ≤ 15 s |
|
||||
| Evidence score (sufficient) | ≥ 0.85 |
|
||||
|
||||
Reference in New Issue
Block a user