ljwx-stock/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

AI-powered Chinese stock (A-share) recommendation system combining ML models (XGBoost/LightGBM) with Claude AI. Features multi-source data fetching from free Chinese financial APIs (Sina, Tencent, EastMoney), advanced caching, and real-time stock analysis with natural language explanations.

Architecture: Vue 3 SPA + FastAPI + PostgreSQL + Redis + Claude AI Data Sources: Sina Finance, Tencent Finance, EastMoney (all free, no API keys) Production Status: Production Ready

Common Development Commands

Quick Start (Recommended)

One-command startup - Start everything with a single script:

# Start all services (PostgreSQL, Redis, Backend, Frontend)
./start_app.sh

# Stop services
./stop_app.sh

# Stop all including databases
./stop_app.sh --all

What it does:

• ✅ Starts PostgreSQL (port 5432) and Redis (port 6379)
• ✅ Creates and activates Python virtual environment
• ✅ Installs dependencies (cached, only updates when changed)
• ✅ Initializes database schema
• ✅ Starts backend (http://localhost:8000)
• ✅ Starts frontend (http://localhost:5173)
• ✅ Handles port conflicts automatically
• ✅ Reuses existing environment (fast restarts)

Prerequisites:

# macOS
brew install postgresql@16 redis [email protected] node
brew services start postgresql@16
brew services start redis

# Linux
sudo apt install postgresql-16 redis-server python3.11 nodejs npm
sudo systemctl start postgresql redis-server

See DEV_SETUP.md for detailed instructions.

Manual Development

# Backend (FastAPI)
cd backend
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --reload --host 0.0.0.0 --port 8000

# Frontend (Vue 3)
cd frontend
npm install
npm run dev       # Dev server on port 5173
npm run build     # Production build
npm run preview   # Preview production build

# Run Tests
cd backend
pytest tests/ -v --asyncio-mode=auto --cov=app
pytest test_smart_cache.py                    # Specific test suite
pytest test_real_data_integration.py -k live  # Live data tests

# Test data sources
python test_data_sources.py                   # Basic data source test
python test_all_data_sources.py               # Comprehensive source test

# Test dashboard APIs
bash test_dashboard_apis.sh                   # Quick API health check
bash test_data_pipeline_apis.sh               # Data pipeline endpoints

Docker Compose (Full Stack)

# Start all services (Backend, Frontend, PostgreSQL, Redis, Adminer)
docker-compose up -d

# View logs
docker-compose logs -f backend
docker-compose logs -f frontend

# Rebuild after changes
docker-compose build backend && docker-compose restart backend

# Clean slate
docker-compose down -v

Kubernetes Deployment

# Quick deploy
bash deploy-to-k8s.sh

# Manual deployment
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/configmap.yaml
kubectl apply -f k8s/postgres-redis.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml

# Check status
kubectl get all -n stock
kubectl logs -f deployment/stock-backend -n stock

# Port forwarding
kubectl port-forward svc/stock-backend 8000:80 -n stock
kubectl port-forward svc/stock-frontend 8090:8090 -n stock

# Rollback
kubectl rollout undo deployment/stock-backend -n stock

CI/CD (Gitea Actions)

# Trigger automated deployment
git add .
git commit -m "feat: your feature description"
git push origin main

# Monitor pipeline in Gitea Actions UI

# Verify CICD environment
bash verify-cicd.sh

# Check deployment
kubectl get pods -n stock -w

High-Level Architecture

System Components

┌─────────────────────────────────────────────────────────┐
│  Frontend Layer (Vue 3 + Vite)                         │
│  - views/ (DashboardV2.vue, AIDock, MarketPulse)      │
│  - components/ (reusable UI components)                 │
│  - services/api.js (Axios HTTP client)                  │
│  - store.js (Vuex/Pinia state management)               │
├─────────────────────────────────────────────────────────┤
│  API Layer (backend/app/routes/)                       │
│  - recommendations.py (ML-based stock picks)           │
│  - ai_recommendations.py (Claude-enhanced analysis)    │
│  - ai_assistant.py (Chatbot endpoints)                 │
│  - claude_deepseek_collab.py (Hybrid AI)               │
│  - data_pipeline.py (data fetch APIs)                  │
│  - risk_radar.py, backtest.py, strategies.py           │
├─────────────────────────────────────────────────────────┤
│  Business Logic (backend/app/modules/)                 │
│  - data_fetcher_v2.py (Multi-source adapter)           │
│  - feature_engineer.py (Technical indicators)          │
│  - ml_trainer.py (XGBoost/LightGBM training)           │
│  - cache_optimizer.py (Smart caching)                  │
│  - strategy_engine.py (Trading strategies)             │
│  - sentiment_fetcher.py (News sentiment)               │
├─────────────────────────────────────────────────────────┤
│  Data Adapters (backend/app/adapters/)                 │
│  - sina_adapter.py (Sina Finance)                      │
│  - tencent_adapter.py (Tencent Finance)                │
│  - eastmoney_adapter.py (EastMoney)                    │
├─────────────────────────────────────────────────────────┤
│  Data Layer                                            │
│  - PostgreSQL (Persistent storage)                     │
│  - Redis (Multi-level caching)                         │
│  - External APIs (Sina, Tencent, EastMoney, Claude)    │
└─────────────────────────────────────────────────────────┘

AI Pipeline Flow

Data Collection → Feature Engineering → ML Prediction → Claude Explanation → Cache → Frontend
     ↓                    ↓                  ↓                  ↓
Sina/Tencent/      Technical Indicators   XGBoost          Natural Language
EastMoney          PE, PB, ROE            LightGBM         Reasoning & Analysis
                   Sentiment Scores                        Risk Assessment

Multi-Level Caching Strategy

Critical for cost optimization and performance:

• L1 Cache: Python @lru_cache() (function-level, per-request)
• L2 Cache: Redis (distributed, TTL-based)
  • Quotes: 60s TTL
  • Features: 300s TTL
  • Recommendations: 300s TTL
  • Claude responses: 3600s TTL (cost optimization)
• L3 Cache: PostgreSQL (persistent)
• L4 Cache: External APIs (fallback)

Why this matters: Claude API calls cost ~$0.006-$0.015 per call. The 3600s TTL on Claude responses reduces duplicate API calls by 80%, saving ~$400/month.

Microservices Architecture

Docker Compose (Local):

• postgres (port 5434) - PostgreSQL 16
• redis (port 6381) - Redis 7 with AOF persistence
• backend (port 8001) - FastAPI with 4 workers
• frontend (port 8090) - Nginx serving Vue SPA
• adminer (port 8091) - Database UI

Kubernetes (Production):

• Namespace: stock
• StatefulSets: PostgreSQL (10Gi PVC), Redis (5Gi PVC)
• Deployments: Backend (1-3 replicas with HPA)
• Frontend: Sidecar container with Nginx
• Services: LoadBalancer for external access
• HPA: Auto-scaling at CPU 70%, Memory 80%

Key Pattern: Backend and frontend can run as separate containers in the same pod (sidecar pattern) or as separate deployments.

Tech Stack

Backend

• Framework: FastAPI 0.104.1 + Uvicorn (4 workers)
• Database: PostgreSQL 16 + SQLAlchemy 2.0.23 + Alembic
• Cache: Redis 7 + aioredis
• ML/AI:
  • XGBoost 2.0.3 (gradient boosting)
  • LightGBM 4.1.0 (ensemble models)
  • scikit-learn 1.3.2
  • Claude 4.5 Sonnet via OpenRouter (anthropic 0.39.0)
  • DeepSeek (local, via Ollama on port 11434)
• Data: pandas 2.1.3, numpy 1.26.2, tushare 1.2.80, yfinance 0.2.32
• Testing: pytest 7.4.3, pytest-asyncio, pytest-cov

Frontend

• Framework: Vue 3.3.4 (Composition API) + Vite 5.0.2
• State: Vuex 4.1.0 + Pinia 2.1.6
• Charts: ECharts 5.4.3
• Styling: Sass 1.69.5

Infrastructure

• Container: Docker + Docker Compose 3.9
• Orchestration: Kubernetes
• CI/CD: Gitea Actions
• Registry: Harbor (localhost:8443)

Critical Implementation Details

1. Claude API Integration

Location: /backend/services/claude_service.py Prompt Templates: /backend/prompts/claude_stock_analysis.md

The system uses structured prompt engineering to get consistent Claude responses:

# Cost optimization pattern
@cached(ttl=3600)  # 1-hour cache
async def analyze_stock_with_claude(symbol: str, features: dict):
    # Only send essential fields to minimize tokens
    compact_data = {
        "symbol": symbol,
        "price": features["current_price"],
        "indicators": {...}  # Filtered
    }
    # Use Sonnet 4.5 for complex analysis
    return await claude_client.analyze(compact_data)

Model Selection:

• Claude 4.5 Sonnet: Complex stock analysis, multi-factor reasoning
• Claude 3.5 Haiku: Simple Q&A, quick responses (60% cheaper)

2. Hybrid AI Architecture (Claude + DeepSeek/Qwen)

Location: /backend/app/routes/claude_deepseek_collab.py

Combines Claude's reasoning with local LLM inference (via Ollama):

1. Local LLM (DeepSeek/Qwen) generates initial technical analysis (free, local)
2. Claude reviews and enhances with market context (paid, OpenRouter API)
3. System merges both perspectives with confidence scoring

Why hybrid?: Reduces Claude API costs by 40% while maintaining quality. Requires local Ollama server running (default: port 11434).

3. Feature Engineering Pipeline

Location: /backend/app/modules/feature_engineer.py

Multi-source feature aggregation:

• Technical indicators: MA5, MA10, MA20, RSI, MACD, ATR, Bollinger Bands
• Fundamental metrics: PE, PB, ROE, industry percentiles
• Sentiment scores: News sentiment, social media sentiment
• ML features: Normalized and scaled for XGBoost/LightGBM

Critical: Features must be cached (300s TTL) because calculation is expensive (50+ indicators per stock).

4. Smart Cache Prewarming

Location: /backend/app/modules/smart_prewarmer.py

Background task that pre-fetches popular stocks before market open:

# Runs at 9:00 AM daily
async def prewarm_top_stocks():
    top_100 = await get_top_volume_stocks()
    for symbol in top_100:
        await fetch_and_cache_features(symbol)  # 5-min TTL

Impact: First user request is 95% faster (cache hit vs API call).

5. Database Schema

Location: /backend/database/schema_v2.sql (main), schema_extended.sql (full data)

Key tables:

• stocks - Stock master data (symbol, name, industry)
• fundamentals - Financial metrics (PE, PB, ROE)
• sentiment_data - News and social sentiment
• recommendations - ML predictions with timestamps
• cache_metadata - Cache hit/miss tracking
• Extended schema: t_company_info, t_sectors, t_financials, t_money_flow, t_top_list (11 additional tables for comprehensive data)

Migration: Use Alembic for schema changes (cd backend && alembic revision --autogenerate -m "description").

6. CI/CD Pipeline

Location: .gitea/workflows/cicd.yml

Multi-stage pipeline:

1. Build: Docker images for backend and frontend
2. Deploy: K8s deployment with kubectl
3. Test: Health check validation

Key Feature: Triggered on push to main/develop branches. Uses Gitea Actions runner with Kubernetes executor.

7. Environment Configuration

Location: .env.example (68 lines)

Critical environment variables:

• OPENROUTER_API_KEY - Claude API via OpenRouter (optional, for AI explanations)
• TUSHARE_TOKEN - TuShare token (legacy, optional)
• DATABASE_URL - PostgreSQL connection string
• REDIS_URL - Redis connection string
• LLM_MODEL - anthropic/claude-4.5-sonnet
• OLLAMA_BASE_URL - http://host.docker.internal:11434 (for hybrid AI)
• DEEPSEEK_MODEL - qwen3-coder:30b-a3b-q8_0 (local LLM model)
• CACHE_TTL_LLM=3600 - Claude response cache TTL (1 hour)
• CACHE_TTL_QUOTE=60 - Real-time quote cache (1 minute)

K8s Deployment: Environment variables are injected via ConfigMap.

Development Workflow

Adding a New Feature

1. Backend API: Add route in /backend/app/routes/

   from fastapi import APIRouter
   router = APIRouter()
   
   @router.get("/api/new-feature")
   async def new_feature():
       return {"status": "ok"}
   

2. Business Logic: Add module in /backend/app/modules/

   class NewFeatureService:
       async def process(self, data):
           # Add caching for expensive operations
           @cached(ttl=300)
           async def _cached_process(key):
               return await expensive_operation(key)
   

3. Frontend Component: Add Vue component in /frontend/src/components/

   <script setup>
   import { ref, onMounted } from 'vue'
   import api from '@/services/api'
   
   const data = ref(null)
   onMounted(async () => {
     data.value = await api.newFeature()
   })
   </script>
   

4. Tests: Add tests in /backend/tests/

   @pytest.mark.asyncio
   async def test_new_feature():
       response = await client.get("/api/new-feature")
       assert response.status_code == 200
   

5. Documentation: Update this file and relevant .md files

Adding Data Fetchers

The system uses a multi-source adapter pattern for data fetching:

1. Create Adapter: Add new adapter in /backend/app/adapters/
   • Implement standard interface: fetch_quote(), fetch_kline(), fetch_fundamentals()
2. Register in DataFetcherV2: Update /backend/app/modules/data_fetcher_v2.py
   • Add to priority list (e.g., Sina → Tencent → EastMoney → New Source)
3. Add Tests: Create tests in /backend/tests/test_<adapter_name>.py

Key Pattern: Adapters must return standardized data format for seamless fallback.

Database Changes

1. Edit models in /backend/app/models.py
2. Generate migration: cd backend && alembic revision --autogenerate -m "description"
3. Review migration in /backend/alembic/versions/
4. Apply: alembic upgrade head
5. Update seed data in /backend/database/

Access Points & Ports

Local Development (Docker Compose)

• Frontend: http://localhost:8090
• Backend API: http://localhost:8001
• API Docs: http://localhost:8001/docs (Swagger UI)
• PostgreSQL: localhost:5434 (user: stockai, pass: stockai2025)
• Redis: localhost:6381 (pass: redis2025)
• Adminer: http://localhost:8091

Kubernetes (Production)

• Backend: http://:80
• Frontend: http://:8090
• Health Check: http://:8000/health

External Services

• Ollama: http://localhost:11434 (for hybrid AI, optional)
• Gitea: (configured for CI/CD)

Key Documentation Files

Read in this order:

1. README.md - Project overview and quick start
2. DEV_SETUP.md - Local development without Docker
3. AGENTS.md - Repository guidelines and commit conventions
4. DATA_SOURCE_ANALYSIS.md - Data source architecture
5. DATA_SOURCE_MIGRATION_V2.md - Migration to free data sources
6. SMART_CACHE_ARCHITECTURE.md - Multi-level caching strategy
7. AI_PRODUCTION_ARCHITECTURE.md - AI integration details
8. CICD_DEPLOYMENT_GUIDE.md - CI/CD setup with Gitea Actions

Critical Performance Considerations

Cost Optimization

• Data Sources: 100% free (Sina, Tencent, EastMoney)
  • No API keys required
  • Multi-source fallback ensures 99.9% availability
• Claude API: Optional, for AI explanations only
  • Redis caching (3600s TTL) reduces 80% duplicate calls
  • Model tiering: Haiku for simple, Sonnet for complex
  • Token compression: Only send essential fields
• Local LLM: Hybrid mode with Ollama reduces Claude API costs by 40%

Performance Targets

• API response time: <200ms (cached), <2s (uncached)
• Claude response time: <3s (streaming enabled)
• Cache hit rate: >80% for popular stocks
• Database query time: <100ms (indexed queries)

Scaling Considerations

• HPA triggers: CPU 70%, Memory 80%
• Max replicas: 3 (cost-constrained)
• Database connection pool: 20 connections max
• Redis max memory: 512MB (LRU eviction)

Troubleshooting

Common Issues

1. Data fetch failures: Check data source adapters in backend logs, system falls back automatically
2. Claude API timeout: Check OPENROUTER_API_KEY (optional, only needed for AI explanations)
3. Redis connection failed: Check REDIS_URL and port 6381 (Docker) or 6379 (local)
4. K8s pod crash: Check logs with kubectl logs -f <pod> -n stock
5. CI/CD pipeline failure: Run bash verify-cicd.sh to diagnose
6. Ollama connection: Ensure Ollama is running on port 11434 for hybrid AI mode

Debug Commands

# Check backend health
curl http://localhost:8001/health

# Test Claude integration
curl -X POST http://localhost:8001/api/test/claude \
  -H "Content-Type: application/json" \
  -d '{"symbol": "AAPL"}'

# Check Redis cache
redis-cli -p 6381 -a redis2025 KEYS "*"

# Check PostgreSQL
docker exec -it <postgres-container> psql -U stockai -d stockai

# View K8s events
kubectl get events -n stock --sort-by='.lastTimestamp'

# Test data sources directly
cd backend
python test_data_sources.py
python test_all_data_sources.py

Repository Guidelines

Per AGENTS.md:

• Follow conventional commit format: feat:, fix:, test:, docs:
• Use black for Python formatting
• Use ESLint for JavaScript/Vue
• Add tests for all new features
• Update documentation when changing architecture

Important Implementation Notes

Chinese Stock Symbols

• A-share format: 6-digit code (e.g., 600000 for Pudong Development Bank)
• Market prefixes: Shanghai (60xxxx, 688xxx), Shenzhen (00xxxx, 300xxx)
• No need for exchange suffix when using Chinese data sources

Data Fetcher Architecture

The data_fetcher_v2.py implements a priority-based multi-source pattern:

1. Try primary source (fastest)
2. Fall back to secondary source if primary fails
3. Fall back to tertiary source if both fail
4. Return cached data if all sources fail

This ensures 99.9% availability without requiring API keys.

Full Data Pipeline

Use backend/run_all_fetchers.sh to fetch comprehensive A-share data:

• Company info (5,295 stocks)
• Sector/plate data (~400 sectors)
• Financial statements (4 quarters)
• Money flow (700 days)
• Top list (dragon-tiger list)

Supports test mode, incremental mode, and custom selection.

Last Updated

Date: 2025-12-23 Status: Production Ready Data Architecture: Multi-source (Sina/Tencent/EastMoney), 100% free