gao/ljwx-study

Fork 0

Files

T

bruno 8cfd53023a commit to ljwx-study

2026-01-29 18:29:48 +08:00

21 KiB

Raw Permalink Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

This is 学习助手 Pro (Learning Assistant Pro), an intelligent exam paper management and learning tracking system. Built with FastAPI (backend) and Vue 3 (frontend), the system specializes in PDF exam paper processing, automatic question extraction using AI/OCR, mistake management with spaced repetition, and learning analytics.

Core Features

PDF Processing: Hybrid text layer + OCR approach achieving 0.01s/page extraction with 95%+ accuracy (2900x performance improvement)
Question Extraction: AI-powered automatic question identification and extraction from exam papers using Claude AI or Google Gemini
Mistake Management: Leitner box system with Ebbinghaus forgetting curve-based review scheduling
Word Learning: SM-2 spaced repetition algorithm for English vocabulary learning
Analytics: Multi-dimensional learning statistics and progress visualization with Chart.js

Development Commands

Quick Start

# Recommended: Start both backend and frontend with production-ready script
./start_app.sh

# Simple development mode (less logging)
./start.sh

# Check system status and port assignments
./status_app.sh

# Stop all services
./stop_app.sh

The start_app.sh script automatically:

Reads configuration from .env file
Handles port conflicts (auto-assigns available ports)
Manages process PIDs in .backend.pid and .frontend.pid
Sets up proper logging to backend/backend.log and frontend/frontend.log
Ensures dependencies are installed

Backend Development (Python/FastAPI)

cd backend

# Install dependencies
pip install -r requirements.txt

# Start development server (reads port from .env BACKEND_PORT, defaults to 8888)
python main.py
# OR with auto-reload
uvicorn main:app --host 0.0.0.0 --port 8888 --reload

# View interactive API documentation
# Open http://localhost:8888/docs (Swagger UI)
# Or http://localhost:8888/redoc (ReDoc)

# Run test scripts
python test_split_api.py        # Test PDF splitting API
python test_all_fixes.py        # Test all configuration fixes
python test_frontend_fixes.py   # Test frontend integration
python test_paddleocr.py        # Test PaddleOCR integration and generate report

Frontend Development (Vue 3/Vite)

cd frontend

# Install dependencies
npm install

# Start development server (reads port from .env VITE_SERVER_PORT, auto-selects if unavailable)
npm run dev

# Build for production
npm run build

# Preview production build
npm run preview

# Lint and fix code
npm run lint

Database Operations

cd backend

# Initialize database (SQLite by default, auto-creates on first run)
python main.py

# Database file location: backend/wordmaster.db (SQLite)
# Or configure PostgreSQL in .env:
# DATABASE_URL=postgresql://user:password@localhost:5432/wordmaster

# Migrate database schema (if schema changes)
python migrate_db.py

# Access database directly
sqlite3 backend/wordmaster.db

Architecture Overview

Backend Architecture (FastAPI)

The backend is a monolithic FastAPI application (~2100 lines) in backend/main.py containing all models, routes, and business logic.

Key Characteristics:

Database: SQLAlchemy ORM with SQLite (default) or PostgreSQL support
Authentication: JWT tokens with HTTPBearer security, bcrypt password hashing
File Storage: Local filesystem in backend/uploads/{user_id}/ directory
Configuration: All settings loaded from .env file (no hardcoded values)
AI Integration: Supports multiple AI providers (Claude via claude_exam_parser.py, Google Gemini)

Key Backend Components

PDF Processing Pipeline

The system uses a multi-strategy PDF processing approach for optimal accuracy and performance:

Primary Processor: backend/pdf_splitter_hybrid.py (PDFSplitterHybrid)
- Text layer extraction for speed (0.01s/page)
- Automatic fallback to OCR for scanned PDFs (EasyOCR)
- Intelligent page type detection
PaddleOCR VL Parser ⭐ NEW - Recommended for Math/Physics/Chemistry:
- backend/pdf_paddleocr_parser.py (PaddleOCRParser)
- Vision-Language model specifically optimized for exam papers
- Superior accuracy for math formulas, chemical equations, physics symbols
- Automatic question structure recognition (questions, options, answers)
- Outputs both JSON and Markdown formats
- Reference: https://github.com/PaddlePaddle/PaddleOCR
- Test with: python backend/test_paddleocr.py
Enhanced Parsers:
- backend/pdf_exam_parser_enhanced_v2.py (PDFExamParserEnhancedV2) - Advanced question extraction
- backend/pdf_image_extractor.py (PDFImageExtractor) - Image correlation with questions
- backend/claude_exam_parser.py (ClaudeExamParser) - AI-powered parsing via Claude API
Utilities:
- backend/utils/question_formatter.py (QuestionFormatter) - Math symbol correction and formatting

OCR Strategy Selection: The system includes intelligent OCR engine selection via backend/ocr_engine_selector.py:

Math/Physics/Chemistry papers: Auto-selects PaddleOCR VL for best formula/symbol recognition
Chinese/English papers: Auto-selects hybrid approach (text layer + EasyOCR fallback)
Fast preview: Text layer extraction only (text_only engine)
Highest accuracy: Claude AI parser (claude engine)
Manual override: Users can specify preferred engine via API or UI

Performance Optimization:

OCR Result Caching (backend/ocr_result_cache.py): File hash-based caching achieving 480x speedup on repeated processing
GPU Acceleration: Enable via use_gpu=True parameter for 3-5x speed boost with PaddleOCR
Cache Statistics: Hit rate tracking, automatic expiration (30 days default)
See backend/PHASE4_OPTIMIZATION.md for async task queue options (Celery, RQ) and advanced optimizations

Database Models

All SQLAlchemy models are defined in backend/main.py (lines 54-170):

Core Models:

User - User accounts with bcrypt hashed passwords
Word - English vocabulary with phrases and examples
UserWordProgress - SM-2 spaced repetition state (EF, interval, repetitions, due date)
Paper - Exam papers with subject, grade, type, scores, and file references
Question - Extracted questions from papers
QuestionCollection - Question grouping and management
Mistake - Wrong answers with Leitner box level, review schedule, and analysis
ReviewLog - Review session history with ratings and time spent

Important Fields:

Papers: file_key, pdf_key, thumb_key for file storage paths
Mistakes: review_level (1-5 Leitner boxes), next_review_date, bbox for positioning
UserWordProgress: ef (ease factor), interval, reps, is_leech for SM-2 algorithm

API Structure

All routes are in backend/main.py:

Authentication & Users:

POST /api/auth/register - User registration
POST /api/auth/login - JWT token generation
GET /api/auth/me - Current user info

Paper Management:

POST /api/papers - Upload PDF with metadata
GET /api/papers - List papers with filtering (subject, grade, exam_type)
GET /api/papers/{id} - Paper details
PUT /api/papers/{id} - Update paper metadata
DELETE /api/papers/{id} - Delete paper and files
POST /api/papers/{id}/split - Extract questions using AI/OCR
- Query params: ocr_engine (hybrid/paddleocr/text_only/claude), use_gpu (bool), use_claude (legacy)
- Auto-selects optimal engine based on subject if not specified
- Returns cached results when available (480x faster)

Question Extraction:

GET /api/questions - List questions by paper or collection
POST /api/questions - Manually add question
GET /api/questions/{id} - Question detail
PUT /api/questions/{id} - Update question
DELETE /api/questions/{id} - Delete question

Mistake & Review:

POST /api/mistakes - Create mistake from question
GET /api/mistakes - List mistakes (filterable by subject, review status)
GET /api/mistakes/due - Mistakes due for review today
POST /api/mistakes/{id}/review - Submit review and update Leitner box
GET /api/review/stats - Review statistics

Word Learning:

GET /api/words - List vocabulary
GET /api/words/due - Words due for review (SM-2 algorithm)
POST /api/words/{id}/review - Submit review and update SM-2 state
POST /api/words/import - Bulk import from CSV/JSON

Analytics:

GET /api/dashboard - Dashboard summary (review due counts, recent activity)
GET /api/stats - Detailed statistics by subject

Frontend Architecture (Vue 3)

Stack:

Vue 3 with Composition API (<script setup>)
Element Plus UI component library
Pinia for state management
Vue Router with authentication guards
Vite as build tool and dev server
Chart.js for analytics visualization
Axios for HTTP requests

Configuration:

frontend/vite.config.js - Vite configuration with API proxy to backend
frontend/src/config/index.js - Environment-based config (reads from import.meta.env.VITE_*)
frontend/.env.development - Development environment variables

Key Frontend Components

State Management (Pinia):

frontend/src/stores/auth.js - Authentication state, JWT token storage, login/logout

Views (all in frontend/src/views/):

DashboardEnhanced.vue (MainDashboard) - Home page with stats, recent papers, review due items
PaperManagementEnhanced.vue (SubjectPapers) - Subject-specific paper listing with upload
PaperDetailEnhanced.vue (PaperDetail) - Single paper view with question extraction trigger
PaperViewerEnhanced.vue (PaperViewer) - PDF rendering with annotation overlay
ReviewEnhanced.vue (MistakeReview) - Mistake review interface with Leitner box progression
Statistics.vue - Detailed analytics with charts
Login.vue / Register.vue - Authentication forms

Router Configuration (frontend/src/router/index.js): Routes with meta: { requiresAuth: true } are protected by beforeEach navigation guard:

/ → MainDashboard (protected)
/dashboard → redirects to /
/subjects/:subject → SubjectPapers (protected)
/papers/:id → PaperDetail (protected)
/papers/:id/view → PaperViewer (protected)
/review → MistakeReview (protected)
/review/today → MistakeReview filtered to today (protected)
/statistics → Statistics (protected)
/login → Login (redirects to / if authenticated)
/register → Register

Important Implementation Details:

PaperViewer.vue constructs image URLs using VITE_BACKEND_PORT from environment
All API calls use base URL from frontend/src/config/index.js
Authentication uses Authorization: Bearer <token> header pattern
File uploads use FormData with multipart/form-data encoding

Development Guidelines

Configuration Management (CRITICAL)

All configuration MUST be in .env files - zero hardcoded values allowed.

See CONFIG_GUIDE.md for complete details. Key points:

Backend: Reads BACKEND_PORT, API_URL, DATABASE_URL, SECRET_KEY from .env at project root
Frontend: Reads VITE_* prefixed variables (e.g., VITE_BACKEND_PORT, VITE_API_BASE_URL)
Configuration Loading: Backend uses python-dotenv from Path(__file__).parent.parent / '.env'
Frontend Config: vite.config.js and frontend/src/config/index.js use import.meta.env.VITE_*

When changing ports or URLs:

Update .env in project root (e.g., BACKEND_PORT=9000)
Update frontend/.env.development if needed
Restart both services: ./stop_app.sh && ./start_app.sh

Backend Development

Adding Models:

Add SQLAlchemy model class in backend/main.py (after existing models around line 54-170)
Run python migrate_db.py to update schema (or delete wordmaster.db for fresh start in dev)
Create corresponding Pydantic schemas for request/response validation

Adding API Endpoints:

Add route in backend/main.py (group with related endpoints)
Use @app.get/post/put/delete decorators
Protected routes: Add current_user: User = Depends(get_current_user) parameter
Follow existing patterns for database session: db: Session = Depends(get_db)
Test immediately at http://localhost:8888/docs

PDF Processing:

Primary module: backend/pdf_splitter_hybrid.py (PDFSplitterHybrid class)
Import and instantiate: from pdf_splitter_hybrid import PDFSplitterHybrid
Extend for new features or create new parser in backend/pdf_*.py
AI-powered parsing: Use claude_exam_parser.py (ClaudeExamParser) as reference

Authentication Pattern:

from fastapi import Depends
security = HTTPBearer()

def get_current_user(credentials: HTTPAuthorizationCredentials = Depends(security), db: Session = Depends(get_db)) -> User:
    # Validates JWT token and returns User object
    # Raises HTTPException(401) if invalid

Frontend Development

Adding Views:

Create component in frontend/src/views/YourView.vue

{ path: '/your-path', name: 'YourView', component: YourView, meta: { requiresAuth: true } }

Import: import YourView from '../views/YourView.vue'
Protected routes need meta: { requiresAuth: true }

State Management:

Create Pinia store in frontend/src/stores/yourStore.js
Follow pattern from auth.js: export const useYourStore = defineStore('your', { ... })
Use in components: const yourStore = useYourStore()

API Calls:

Use config from frontend/src/config/index.js

Example:

import config from '@/config'
import axios from 'axios'

const response = await axios.get(`${config.API_BASE_URL}/your-endpoint`, {
  headers: { Authorization: `Bearer ${authStore.token}` }
})

Environment Variables:

Access with import.meta.env.VITE_YOUR_VAR
Must be defined in .env or frontend/.env.development with VITE_ prefix
Vite inlines these at build time

File Upload and Storage

Backend Storage Structure:

backend/uploads/
  └── {user_id}/
      ├── papers/          # Original uploaded PDFs
      ├── images/          # Extracted images from PDFs
      └── thumbnails/      # Paper preview thumbnails

Upload Flow:

Frontend: FormData with file in multipart/form-data request
Backend: Receives UploadFile, saves to uploads/{user_id}/papers/
Database: Stores relative path in file_key, pdf_key, or image_key fields
Serving: Backend serves via /api/file/upload/{file_key} route

Testing and Development

Backend Testing:

Interactive API docs: http://localhost:8888/docs (Swagger UI)
Alternative docs: http://localhost:8888/redoc (ReDoc)
Test scripts: backend/test_split_api.py, backend/test_all_fixes.py
Manual testing: Use FastAPI docs "Try it out" feature with authentication

Frontend Testing:

Hot reload: Vite dev server auto-reloads on file changes
Browser DevTools: Check Network tab for API calls
Vue DevTools: Install browser extension for component inspection

PDF Processing Tests:

Create test files in backend/test_*.py
Test hybrid approach: PDFSplitterHybrid.split_pdf()
Verify OCR fallback by using scanned PDF samples

Deployment

Development Environment (Local)

# Recommended approach
./start_app.sh    # Starts both services with full logging
./status_app.sh   # Check running status
./stop_app.sh     # Clean shutdown

# Configuration
- Backend: Reads port from .env BACKEND_PORT (default 8888)
- Frontend: Reads port from .env VITE_SERVER_PORT, auto-selects if occupied
- Database: SQLite at backend/wordmaster.db
- File storage: backend/uploads/

Kubernetes Deployment (Production)

The project includes complete K8s deployment configuration for production:

Quick Deploy:

./deploy.sh    # One-command deployment to K8s cluster

What it does:

Builds Docker images (pager-backend:latest, pager-frontend:latest)
Creates namespace learn
Deploys PostgreSQL, backend, frontend
Sets up Ingress for learn.omniverseai.net

K8s Resources (defined in k8s/learn-deployment.yaml):

PostgreSQL StatefulSet with persistent volume
Backend Deployment (2 replicas) with health checks
Frontend Deployment (2 replicas) with Nginx
Services for internal communication
Ingress with TLS for external access

CI/CD Pipeline (.gitea/workflows/ci-cd.yaml):

Triggered on push to main/master branch
Stages: Build → Test → Deploy → Verify
Uses Gitea Actions (GitHub Actions compatible)
Requires KUBECONFIG secret in Gitea

Kubernetes Commands:

# Check deployment status
kubectl get pods -n learn
kubectl get svc -n learn
kubectl get ingress -n learn

# View logs
kubectl logs -n learn -l app=pager-backend -f
kubectl logs -n learn -l app=pager-frontend -f

# Restart deployment
kubectl rollout restart deployment/pager-backend -n learn

# Scale replicas
kubectl scale deployment/pager-backend -n learn --replicas=3

# Access database
kubectl exec -it -n learn <postgres-pod> -- psql -U pager_user -d pager_db

See DEPLOYMENT.md for comprehensive K8s deployment guide.

Production Considerations

Security:

Change SECRET_KEY in .env to a strong random value
Configure CORS: Set allow_origins to specific domains (not ["*"])
Use HTTPS/TLS: Ingress configured with cert-manager or manual TLS certificates
Environment variables: Store sensitive values in K8s Secrets, not in code

Database:

Development: SQLite (backend/wordmaster.db)
Production: PostgreSQL (configured via DATABASE_URL in .env)
K8s: PostgreSQL StatefulSet with PersistentVolumeClaim for data persistence

File Storage:

Development: Local backend/uploads/ directory
Production K8s: Consider PersistentVolume or object storage (S3, MinIO)
Backup: Regular backups of database and upload directory

Configuration:

Backend: Set ENVIRONMENT=production in .env
Frontend: Build with npm run build, serve static files with Nginx
Reverse Proxy: Nginx or K8s Ingress handles routing and TLS

Key Files and Documentation

Core Application Files

Backend (backend/main.py - 2114 lines):

Lines 54-170: SQLAlchemy database models
Lines 40-46: Environment variable configuration
Routes grouped by feature (auth, papers, questions, mistakes, words, analytics)

Frontend Router (frontend/src/router/index.js):

Line 76-86: Authentication guard logic
Route definitions with requiresAuth meta