# CLAUDE Technical Log and Decisions (Python FastAPI + Transformers)
## Progress Log — 2025-10-23 (Asia/Jakarta)

- Migrated stack from Node.js/llama.cpp to Python + FastAPI + Transformers
  - New server: [main.py](main.py)
  - Default model: Qwen/Qwen3-VL-2B-Thinking via Transformers with trust_remote_code
- Implemented endpoints
  - Health: [Python.app.get()](main.py:577)
  - OpenAI-compatible Chat Completions (non-stream + SSE): [Python.app.post()](main.py:591)
  - Manual cancel (custom extension): [Python.app.post()](main.py:792)
- Multimodal support
  - OpenAI-style messages mapped in [Python.function build_mm_messages](main.py:251)
  - Image loader: [Python.function load_image_from_any](main.py:108)
  - Video loader (frame sampling): [Python.function load_video_frames_from_any](main.py:150)
- Streaming + resume + persistence
  - SSE with session_id + Last-Event-ID
  - In-memory session ring buffer: [Python.class _SSESession](main.py:435), manager [Python.class _SessionStore](main.py:449)
  - Optional SQLite persistence: [Python.class _SQLiteStore](main.py:482) with replay across restarts
- Cancellation
  - Auto-cancel after all clients disconnect for CANCEL_AFTER_DISCONNECT_SECONDS, timer wiring in [Python.function chat_completions](main.py:733), cooperative stop in [Python.function infer_stream](main.py:375)
  - Manual cancel API: [Python.function cancel_session](main.py:792)
- Configuration and dependencies
  - Env template updated: [.env.example](.env.example) with MODEL_REPO_ID, PERSIST_SESSIONS, SESSIONS_DB_PATH, SESSIONS_TTL_SECONDS, CANCEL_AFTER_DISCONNECT_SECONDS, etc.
  - Python deps: [requirements.txt](requirements.txt)
  - Git ignores for Python + artifacts: [.gitignore](.gitignore)
- Documentation refreshed
  - Operator docs: [README.md](README.md) including SSE resume, SQLite, cancel API
  - Architecture: [ARCHITECTURE.md](ARCHITECTURE.md) aligned to Python flows
  - Rules: [RULES.md](RULES.md) updated — Git usage is mandatory
- Legacy removal
  - Deleted Node files and scripts (index.js, package*.json, scripts/) as requested

Suggested Git commit series (run in order)
- git add .
- git commit -m "feat(server): add FastAPI OpenAI-compatible /v1/chat/completions with Qwen3-VL [Python.main()](main.py:1)"
- git commit -m "feat(stream): SSE streaming with session_id resume and in-memory sessions [Python.function chat_completions()](main.py:591)"
- git commit -m "feat(persist): SQLite-backed replay for SSE sessions [Python.class _SQLiteStore](main.py:482)"
- git commit -m "feat(cancel): auto-cancel after disconnect and POST /v1/cancel/{session_id} [Python.function cancel_session](main.py:792)"
- git commit -m "docs: update README/ARCHITECTURE/RULES for Python stack and streaming resume"
- git push

Verification snapshot
- Non-stream text works via [Python.function infer](main.py:326)
- Streaming emits chunks and ends with [DONE]
- Resume works with Last-Event-ID; persists across restart when PERSIST_SESSIONS=1
- Manual cancel stops generation; auto-cancel triggers after disconnect threshold


This is the developer-facing changelog and design rationale for the Python migration. Operator docs live in [README.md](README.md); architecture details in [ARCHITECTURE.md](ARCHITECTURE.md); rules in [RULES.md](RULES.md); task tracking in [TODO.md](TODO.md).

Key source file references
- Server entry: [Python.main()](main.py:807)
- Health endpoint: [Python.app.get()](main.py:577)
- Chat Completions endpoint (non-stream + SSE): [Python.app.post()](main.py:591)
- Manual cancel endpoint (custom): [Python.app.post()](main.py:792)
- Engine (Transformers): [Python.class Engine](main.py:231)
- Multimodal mapping: [Python.function build_mm_messages](main.py:251)
- Image loader: [Python.function load_image_from_any](main.py:108)
- Video loader: [Python.function load_video_frames_from_any](main.py:150)
- Non-stream inference: [Python.function infer](main.py:326)
- Streaming inference + stopping criteria: [Python.function infer_stream](main.py:375)
- In-memory sessions: [Python.class _SSESession](main.py:435), [Python.class _SessionStore](main.py:449)
- SQLite persistence: [Python.class _SQLiteStore](main.py:482)

Summary of the migration
- Replaced the Node.js/llama.cpp stack with a Python FastAPI server that uses Hugging Face Transformers for Qwen3-VL multimodal inference.
- Exposes an OpenAI-compatible /v1/chat/completions endpoint (non-stream and streaming via SSE).
- Supports text, images, and videos:
  - Messages can include array parts such as "text", "image_url" / "input_image" (base64), "video_url" / "input_video" (base64).
  - Images are decoded to PIL in [Python.function load_image_from_any](main.py:108).
  - Videos are read via imageio.v3 (preferred) or OpenCV, sampled to up to MAX_VIDEO_FRAMES in [Python.function load_video_frames_from_any](main.py:150).
- Streaming includes resumability with session_id + Last-Event-ID:
  - In-memory ring buffer: [Python.class _SSESession](main.py:435)
  - Optional SQLite persistence: [Python.class _SQLiteStore](main.py:482)
- Added a manual cancel endpoint (custom) and implemented auto-cancel after disconnect.

Why Python + Transformers?
- Qwen3-VL-2B-Thinking is published for Transformers and includes multimodal processors (preprocessor_config.json, video_preprocessor_config.json, chat_template.json). Python + Transformers is the first-class path.
- trust_remote_code=True allows the model repo to provide custom processing logic and templates, used in [Python.class Engine](main.py:231) via AutoProcessor/AutoModelForCausalLM.

Core design choices

1) OpenAI compatibility
- Non-stream path returns choices[0].message.content from [Python.function infer](main.py:326).
- Streaming path (SSE) produces OpenAI-style "chat.completion.chunk" deltas, with id lines "session_id:index" for resume.
- We retained Chat Completions (legacy) rather than the newer Responses API for compatibility with existing SDKs. A custom cancel endpoint is provided to fill the gap.

2) Multimodal input handling
- The API accepts "messages" with content either as a string or an array of parts typed as "text" / "image_url" / "input_image" / "video_url" / "input_video".
- Images: URLs (http/https or data URL), base64, or local path are supported by [Python.function load_image_from_any](main.py:108).
- Videos: URLs and base64 are materialized to a temp file; frames extracted and uniformly sampled by [Python.function load_video_frames_from_any](main.py:150).

3) Engine and generation
- Qwen chat template applied via processor.apply_chat_template in both [Python.function infer](main.py:326) and [Python.function infer_stream](main.py:375).
- Generation sampling uses temperature; do_sample toggled when temperature > 0.
- Streams are produced using TextIteratorStreamer.
- Optional cooperative cancellation is implemented with a StoppingCriteria bound to a session cancel event in [Python.function infer_stream](main.py:375).

4) Streaming, resume, and persistence
- In-memory buffer per session for immediate replay: [Python.class _SSESession](main.py:435).
- Optional SQLite persistence to survive restarts and handle long gaps: [Python.class _SQLiteStore](main.py:482).
- Resume protocol:
  - Client provides session_id in the request body and Last-Event-ID header "session_id:index", or pass ?last_event_id=...
  - Server replays events after index from SQLite (if enabled) and the in-memory buffer.
  - Producer appends events to both the ring buffer and SQLite (when enabled).

5) Cancellation and disconnects
- Manual cancel endpoint [Python.app.post()](main.py:792) sets the session cancel event and marks finished in SQLite.
- Auto-cancel after disconnect:
  - If all clients disconnect, a timer fires after CANCEL_AFTER_DISCONNECT_SECONDS (default 3600) that sets the cancel event.
  - The StoppingCriteria checks this event cooperatively and halts generation.

6) Environment configuration
- See [.env.example](.env.example).
- Important variables:
  - MODEL_REPO_ID (default "Qwen/Qwen3-VL-2B-Thinking")
  - HF_TOKEN (optional)
  - MAX_TOKENS, TEMPERATURE
  - MAX_VIDEO_FRAMES (video frame sampling)
  - DEVICE_MAP, TORCH_DTYPE (Transformers loading hints)
  - PERSIST_SESSIONS, SESSIONS_DB_PATH, SESSIONS_TTL_SECONDS (SQLite)
  - CANCEL_AFTER_DISCONNECT_SECONDS (auto-cancel threshold)

Security and privacy notes
- trust_remote_code=True executes code from the model repository when loading AutoProcessor/AutoModel. This is standard for many HF multimodal models but should be understood in terms of supply-chain risk.
- Do not log sensitive data. Avoid dumping raw request bodies or tokens.

Operational guidance

Running locally
- Install Python dependencies from [requirements.txt](requirements.txt) and install a suitable PyTorch wheel for your platform/CUDA.
- copy .env.example .env and adjust as needed.
- Start: python [Python.main()](main.py:807)

Testing endpoints
- Health: GET /health
- Chat (non-stream): POST /v1/chat/completions with messages array.
- Chat (stream): add "stream": true; optionally pass "session_id".
- Resume: send Last-Event-ID with "session_id:index".
- Cancel: POST /v1/cancel/{session_id}.

Scaling notes
- Typically deploy one model per process. For throughput, run multiple workers behind a load balancer; sessions are process-local unless persistence is used.
- SQLite persistence supports replay but does not synchronize cancel/producer state across processes. A Redis-based store (future work) can coordinate multi-process session state more robustly.

Known limitations and follow-ups
- Token accounting (usage prompt/completion/total) is stubbed at zeros. Populate if/when needed.
- Redis store not yet implemented (design leaves a clear seam via _SQLiteStore analog).
- No structured logging/tracing yet; follow-up for observability.
- Cancellation is best-effort cooperative; it relies on the stopping criteria hook in generation.

Changelog (2025-10-23)
- feat(server): Python FastAPI server with Qwen3-VL (Transformers), OpenAI-compatible /v1/chat/completions.
- feat(stream): SSE streaming with session_id + Last-Event-ID resumability.
- feat(persist): Optional SQLite-backed session persistence for replay across restarts.
- feat(cancel): Manual cancel endpoint /v1/cancel/{session_id}; auto-cancel after disconnect threshold.
- docs: Updated [README.md](README.md), [ARCHITECTURE.md](ARCHITECTURE.md), [RULES.md](RULES.md). Rewrote [TODO.md](TODO.md) pending/complete items (see repo TODO).
- chore: Removed Node.js and scripts from the prior stack.

Verification checklist
- Non-stream text-only request returns a valid completion.
- Image and video prompts pass through preprocessing and generate coherent output.
- Streaming emits OpenAI-style deltas and ends with [DONE].
- Resume works with Last-Event-ID and session_id across reconnects; works after server restart when PERSIST_SESSIONS=1.
- Manual cancel halts generation and marks session finished; subsequent resumes return a finished stream.
- Auto-cancel fires after all clients disconnect for CANCEL_AFTER_DISCONNECT_SECONDS and cooperatively stops generation.

End of entry.
## Progress Log Template (Mandatory per RULES)

Use this template for every change or progress step. Add a new entry before/with each commit, then append the final commit hash after push. See enforcement in [RULES.md](RULES.md:33) and the progress policy in [RULES.md](RULES.md:49).

Entry template
- Date/Time (Asia/Jakarta): YYYY-MM-DD HH:mm
- Commit: <hash> - <conventional message>
- Scope/Files (clickable anchors required):
  - [Python.function chat_completions()](main.py:591)
  - [Python.function infer_stream()](main.py:375)
  - [README.md](README.md:1), [ARCHITECTURE.md](ARCHITECTURE.md:1), [RULES.md](RULES.md:1), [TODO.md](TODO.md:1)
- Summary:
  - What changed and why (problem/requirement)
- Changes:
  - Short bullet list of code edits with anchors
- Verification:
  - Commands:
    - curl examples (non-stream, stream with session_id, resume with Last-Event-ID)
    - cancel API test: curl -X POST http://localhost:3000/v1/cancel/mysession123
  - Expected vs Actual:
    - …
- Follow-ups/Limitations:
  - …
- Notes:
  - If commit hash unknown at authoring time, update the entry after git push.

Git sequence (run every time)
- git add .
- git commit -m "type(scope): short description"
- git push
- Update this entry with the final commit hash.

Example (filled)
- Date/Time: 2025-10-23 14:30 (Asia/Jakarta)
- Commit: f724450 - feat(stream): add SQLite persistence for SSE resume
- Scope/Files:
  - [Python.class _SQLiteStore](main.py:482)
  - [Python.function chat_completions()](main.py:591)
  - [README.md](README.md:1), [ARCHITECTURE.md](ARCHITECTURE.md:1)
- Summary:
  - Persist SSE chunks to SQLite for replay across restarts; enable via PERSIST_SESSIONS.
- Changes:
  - Add _SQLiteStore with schema and CRUD
  - Wire producer to append events to DB
  - Replay DB events on resume before in-memory buffer
- Verification:
  - curl -N -H "Content-Type: application/json" ^
    -d "{\"session_id\":\"mysession123\",\"messages\":[{\"role\":\"user\",\"content\":\"Think step by step: 17*23?\"}],\"stream\":true}" ^
    http://localhost:3000/v1/chat/completions
  - Restart server; resume:
    curl -N -H "Content-Type: application/json" ^
    -H "Last-Event-ID: mysession123:42" ^
    -d "{\"session_id\":\"mysession123\",\"messages\":[{\"role\":\"user\",\"content\":\"Think step by step: 17*23?\"}],\"stream\":true}" ^
    http://localhost:3000/v1/chat/completions
  - Expected vs Actual: replayed chunks after index 42, continued live, ended with [DONE].
- Follow-ups:
  - Consider Redis store for multi-process coordination
## Progress Log — 2025-10-23 14:31 (Asia/Jakarta)

- Commit: f724450 - docs: sync README/ARCHITECTURE/RULES with main.py; add progress log in CLAUDE.md; enforce mandatory Git
- Scope/Files (anchors):
  - [Python.function chat_completions()](main.py:591)
  - [Python.function infer_stream()](main.py:375)
  - [Python.class _SSESession](main.py:435), [Python.class _SessionStore](main.py:449), [Python.class _SQLiteStore](main.py:482)
  - [README.md](README.md:1), [ARCHITECTURE.md](ARCHITECTURE.md:1), [RULES.md](RULES.md:1), [CLAUDE.md](CLAUDE.md:1), [.env.example](.env.example:1)
- Summary:
  - Completed Python migration and synchronized documentation. Implemented SSE streaming with resume, optional SQLite persistence, auto-cancel on disconnect, and manual cancel API. RULES now mandate Git usage and progress logging.
- Changes:
  - Document streaming/resume/persistence/cancel in [README.md](README.md:1) and [ARCHITECTURE.md](ARCHITECTURE.md:1)
  - Enforce Git workflow and progress logging in [RULES.md](RULES.md:33)
  - Add Progress Log template and entries in [CLAUDE.md](CLAUDE.md:1)
- Verification:
  - Non-stream:
    curl -X POST http://localhost:3000/v1/chat/completions ^
      -H "Content-Type: application/json" ^
      -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}]}"
  - Stream:
    curl -N -H "Content-Type: application/json" ^
      -d "{\"session_id\":\"mysession123\",\"messages\":[{\"role\":\"user\",\"content\":\"Think step by step: 17*23?\"}],\"stream\":true}" ^
      http://localhost:3000/v1/chat/completions
  - Resume:
    curl -N -H "Content-Type: application/json" ^
      -H "Last-Event-ID: mysession123:42" ^
      -d "{\"session_id\":\"mysession123\",\"messages\":[{\"role\":\"user\",\"content\":\"Think step by step: 17*23?\"}],\"stream\":true}" ^
      http://localhost:3000/v1/chat/completions
  - Cancel:
    curl -X POST http://localhost:3000/v1/cancel/mysession123
  - Results:
    - Streaming emits chunks, ends with [DONE]; resume replays after index; cancel terminates generation; auto-cancel after disconnect threshold works via timer + stopping criteria.
- Follow-ups:
  - Optional Redis store for multi-process coordination.

## Progress Log — 2025-10-28 23:13 (Asia/Jakarta)

- Commit: c60d35d - feat(ocr): add KTP OCR endpoint using Qwen3-VL model
- Scope/Files (anchors):
  - [Python.function ktp_ocr](main.py:1310)
  - [Python.function build_mm_messages](main.py:251)
  - [Python.function infer](main.py:326)
  - [Python.function test_ktp_ocr_success](tests/test_api.py:276)
  - [README.md](README.md:1), [ARCHITECTURE.md](ARCHITECTURE.md:1), [CLAUDE.md](CLAUDE.md:1)
- Summary:
  - Added KTP OCR endpoint for Indonesian ID card text extraction using Qwen3-VL multimodal model. Inspired by raflyryhnsyh/Gemini-OCR-KTP but adapted for local inference without external API dependencies.
- Changes:
  - Implement POST /ktp-ocr/ endpoint accepting multipart form-data with image file
  - Use custom prompt to extract structured JSON data (nik, nama, alamat fields, etc.)
  - Integrate with existing Engine.infer() for multimodal processing
  - Add robust JSON extraction with fallback parsing (handles model responses in code blocks)
  - Update tags_metadata to include "ocr" endpoint category
  - Add comprehensive test case with mock JSON response validation
  - Update README with KTP OCR documentation, usage examples, and credit to original project
  - Update ARCHITECTURE.md to document the new endpoint
- Verification:
  - KTP OCR endpoint test:
    curl -X POST http://localhost:3000/ktp-ocr/ ^
      -F "image=@image.jpg"
  - Expected vs Actual: Returns JSON with structured KTP data fields (nik, nama, alamat object, etc.)
  - Test suite: All 10 tests pass including new KTP OCR test
  - FastAPI import: No syntax errors, app loads successfully
- Follow-ups/Limitations:
  - Model accuracy depends on Qwen3-VL training data for Indonesian text
  - JSON parsing is best-effort; may need refinement for edge cases
  - Consider adding image preprocessing (resize, enhance contrast) for better OCR
- Notes:
  - Endpoint maintains OpenAI-compatible API patterns while providing specialized OCR functionality
  - No external API keys required; fully self-hosted solution
  - CI/CD will sync to Hugging Face Space automatically on push