|
@@ -0,0 +1,649 @@
|
|
|
|
|
+# CLAUDE.md — Tasmanian Planning Scheme Assistant
|
|
|
|
|
+# tasplanning.report
|
|
|
|
|
+
|
|
|
|
|
+This file gives Claude Code full context on the project architecture, conventions, and
|
|
|
|
|
+deployment workflow. Read this before making any changes.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Project Overview
|
|
|
|
|
+
|
|
|
|
|
+**tasplanning.report** is an AI-powered assistant for the Tasmanian Planning Scheme (TPS). It allows planners, building designers, architects, and consultants to ask natural-language questions about SPPs and LPS documents and receive clause-cited answers.
|
|
|
|
|
+
|
|
|
|
|
+**Core capability:** Retrieval-Augmented Generation (RAG) — user queries are embedded, matched against a Qdrant vector database of official planning documents, and the retrieved context is injected into an LLM prompt. The LLM generates a cited answer using only the retrieved context — it never invents clause numbers.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Repository Layout
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+/home/modulos_llm/
|
|
|
|
|
+├── pdfs/
|
|
|
|
|
+│ ├── as/ # Australian Standards in PDF
|
|
|
|
|
+│ ├── lps/ # Local Planning Scheme in PDF
|
|
|
|
|
+│ ├── ncc/ # National Construction Code in PDF
|
|
|
|
|
+│ ├── tps/ # Tasmanian Planning Scheme in PDF
|
|
|
|
|
+├── backend/ # FastAPI Python backend
|
|
|
|
|
+│ ├── app.py # Main API — /ask, /feedback, /councils, /readyz, admin endpoints
|
|
|
|
|
+│ ├── telemetry.py # SQLite telemetry DB, /telemetry POST endpoint, schema init
|
|
|
|
|
+│ ├── ingest.py # PDF ingestion into Qdrant (run manually)
|
|
|
|
|
+│ ├── ingest_ollama.py # Ollama-based ingestion variant
|
|
|
|
|
+│ └── requirements.txt # Python dependencies
|
|
|
|
|
+├── public/ # PHP/Apache web frontend (bind-mounted into web container)
|
|
|
|
|
+│ ├── index.php # Landing page with demo modal and waitlist
|
|
|
|
|
+│ ├── local_state-planning-scheme.php # Main chat assistant UI
|
|
|
|
|
+│ ├── site-report.php # Property lookup (LIST/ArcGIS + Google Maps)
|
|
|
|
|
+│ ├── section-builder.php # Planning report section builder
|
|
|
|
|
+│ ├── byok-settings.php # API key management (BYOK feature)
|
|
|
|
|
+│ ├── dashboard.php # Internal query monitoring dashboard
|
|
|
|
|
+│ ├── waitlist.php # PHPMailer waitlist signup endpoint
|
|
|
|
|
+│ ├── gmaps-key.php # Google Maps API key proxy
|
|
|
|
|
+│ ├── faq.php # FAQ page
|
|
|
|
|
+│ ├── privacy.php # Privacy policy
|
|
|
|
|
+│ ├── terms.php # Terms of use
|
|
|
|
|
+│ ├── js/
|
|
|
|
|
+│ │ └── api-status.js # Shared live API health indicator
|
|
|
|
|
+│ ├── css/
|
|
|
|
|
+│ │ └── report.css # Shared stylesheet for index.php
|
|
|
|
|
+│ ├── vendor/ # Composer PHP dependencies (PHPMailer, phpword, google/apiclient)
|
|
|
|
|
+│ └── composer.json # PHP dependency manifest
|
|
|
|
|
+├── web/
|
|
|
|
|
+│ └── Dockerfile # Custom PHP 8.3 + Apache image
|
|
|
|
|
+├── cache/
|
|
|
|
|
+├── creds/
|
|
|
|
|
+│ ├── oauth-client.json
|
|
|
|
|
+│ ├── service-account.json
|
|
|
|
|
+├── images/
|
|
|
|
|
+├── models/
|
|
|
|
|
+├── qdrant_storage/ # Qdrant vector DB persistent storage
|
|
|
|
|
+│ ├── aliases/ #
|
|
|
|
|
+│ ├── collections/ #
|
|
|
|
|
+│ ├── raft_state.json #
|
|
|
|
|
+├── telemetry_data/
|
|
|
|
|
+│ └── telemetry.db # SQLite telemetry database
|
|
|
|
|
+├── venv/
|
|
|
|
|
+│ ├── include
|
|
|
|
|
+│ │ └── python3.1.2
|
|
|
|
|
+├── docker-compose.yml # Full stack orchestration
|
|
|
|
|
+└── .env # Secrets (SMTP_PASS, GMAPS_API_KEY, TPR_IP_SECRET, etc.)
|
|
|
|
|
+└── CHANGELOG.md
|
|
|
|
|
+└── CLAUDE.md
|
|
|
|
|
+└── README.md
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Docker Services
|
|
|
|
|
+
|
|
|
|
|
+All services defined in `docker-compose.yml`. Run from `/home/modulos_llm/`.
|
|
|
|
|
+
|
|
|
|
|
+| Service | Container | Port | Purpose |
|
|
|
|
|
+|---|---|---|---|
|
|
|
|
|
+| `qdrant` | `modulos-qdrant` | 6333, 6334 | Vector database |
|
|
|
|
|
+| `backend` | `modulos-backend` | 2281→8000 | FastAPI API |
|
|
|
|
|
+| `web` | `modulos-web` | 2380→80 | PHP/Apache frontend |
|
|
|
|
|
+| `sqliteweb` | `modulos-sqliteweb` | 8091→8080 | SQLite Web UI (internal) |
|
|
|
|
|
+| `composer` | *(run-once)* | — | PHP dependency installer |
|
|
|
|
|
+
|
|
|
|
|
+### Common commands
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# Start everything
|
|
|
|
|
+docker compose up -d
|
|
|
|
|
+
|
|
|
|
|
+# Restart a single service (backend changes need restart, web changes do not)
|
|
|
|
|
+docker compose restart backend
|
|
|
|
|
+
|
|
|
|
|
+# View logs
|
|
|
|
|
+docker logs modulos-backend --tail 50
|
|
|
|
|
+docker logs modulos-web --tail 50
|
|
|
|
|
+
|
|
|
|
|
+# Force recreate (picks up docker-compose.yml changes)
|
|
|
|
|
+docker compose up -d --force-recreate web
|
|
|
|
|
+
|
|
|
|
|
+# Run composer to install/update PHP dependencies
|
|
|
|
|
+docker compose run --rm composer
|
|
|
|
|
+# Or directly:
|
|
|
|
|
+docker run --rm -v /home/modulos_llm/public:/app composer:2 \
|
|
|
|
|
+ require phpmailer/phpmailer --no-interaction --no-progress --ignore-platform-reqs
|
|
|
|
|
+
|
|
|
|
|
+# One-off Python command in backend container
|
|
|
|
|
+docker exec modulos-backend python3 -c "..."
|
|
|
|
|
+
|
|
|
|
|
+# Check API health
|
|
|
|
|
+curl -s https://api.modulos.com.au/readyz
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Important: bind mounts
|
|
|
|
|
+
|
|
|
|
|
+- `./backend:/app` — backend Python files are live. Edit files on host, **restart backend** to pick up changes.
|
|
|
|
|
+- `./public:/var/www/html` — PHP files are live. Edit files on host, **no restart needed**.
|
|
|
|
|
+- `/home/modulos_llm/telemetry_data:/data` — telemetry DB shared between backend and web containers.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Backend (`app.py`)
|
|
|
|
|
+
|
|
|
|
|
+### Key environment variables
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+OLLAMA_URL=http://192.168.8.73:11434 # Windows machine running Ollama
|
|
|
|
|
+QDRANT_URL=http://qdrant:6333 # Internal Docker network
|
|
|
|
|
+OLLAMA_KEEP_ALIVE=-1 # Keep model loaded permanently (-1 = forever)
|
|
|
|
|
+CHAT_MODEL=llama3.1:8b-instruct-q4_K_M # Current model (Q4 quantization)
|
|
|
|
|
+EMBED_MODEL=nomic-embed-text # Embedding model
|
|
|
|
|
+QDRANT_COLLECTION=planning_docs # Vector collection name
|
|
|
|
|
+CORS_ORIGINS=https://tasplanning.report,http://localhost:3000,...
|
|
|
|
|
+TPR_DB=/data/telemetry.db
|
|
|
|
|
+TPR_IP_SECRET=<secret> # HMAC secret for IP hashing
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### API endpoints
|
|
|
|
|
+
|
|
|
|
|
+| Method | Path | Purpose |
|
|
|
|
|
+|---|---|---|
|
|
|
|
|
+| GET | `/readyz` | Health check — returns `{"ok":true}` |
|
|
|
|
|
+| GET | `/councils` | List indexed council names |
|
|
|
|
|
+| POST | `/ask` | Main RAG query endpoint |
|
|
|
|
|
+| GET | `/ask` | Same, query string params |
|
|
|
|
|
+| POST | `/feedback` | Store thumbs up/down with query+answer |
|
|
|
|
|
+| POST | `/telemetry` | Generic event logging |
|
|
|
|
|
+| GET | `/admin/stats` | Collection statistics |
|
|
|
|
|
+| GET | `/admin/files` | List indexed documents |
|
|
|
|
|
+| GET | `/admin/sample` | Sample random chunks |
|
|
|
|
|
+| GET | `/admin/export` | NDJSON export of all chunks |
|
|
|
|
|
+
|
|
|
|
|
+### `/ask` request body (`AskBody`)
|
|
|
|
|
+
|
|
|
|
|
+```python
|
|
|
|
|
+{
|
|
|
|
|
+ "query": "What are the setbacks in the Village Zone?",
|
|
|
|
|
+ "council": "launceston", # optional — filters to that council's LPS
|
|
|
|
|
+ "top_k": 8, # number of chunks to retrieve
|
|
|
|
|
+ "scope": "state_plus_local", # state_plus_local | state_only | local_only | any
|
|
|
|
|
+ "section_id": null, # optional — triggers specific output format guide
|
|
|
|
|
+ "include_ncc": false,
|
|
|
|
|
+ "include_standards": false,
|
|
|
|
|
+ "context_only": false # BYOK mode — returns context+prompt without calling Ollama
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### `/ask` response
|
|
|
|
|
+
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "answer": "...",
|
|
|
|
|
+ "sources": [
|
|
|
|
|
+ {"source_file": "tasmanian-planning-scheme.pdf", "page": 42, "chunk_index": 3, "score": 0.87}
|
|
|
|
|
+ ]
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+When `context_only: true` (BYOK mode):
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "context_only": true,
|
|
|
|
|
+ "context": "...",
|
|
|
|
|
+ "prompt": "...", // full prompt ready to send to external LLM
|
|
|
|
|
+ "sources": [...],
|
|
|
|
|
+ "sections": [...]
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Ollama settings (in `ollama_chat()`)
|
|
|
|
|
+
|
|
|
|
|
+```python
|
|
|
|
|
+"keep_alive": -1, # top-level param, NOT inside options
|
|
|
|
|
+"options": {
|
|
|
|
|
+ "num_ctx": 6144, # context window — don't change between requests or model reloads
|
|
|
|
|
+ "num_predict": 1024, # max output tokens
|
|
|
|
|
+ "temperature": 0.15, # low = more deterministic
|
|
|
|
|
+ "top_p": 0.85,
|
|
|
|
|
+ "top_k": 40,
|
|
|
|
|
+ "repeat_penalty": 1.15,
|
|
|
|
|
+ "stop": ["QUESTION:", "---END---"],
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Important:** `keep_alive` must be a top-level JSON key, not inside `options`. Putting it
|
|
|
|
|
+inside `options` causes Ollama to silently ignore it.
|
|
|
|
|
+
|
|
|
|
|
+### Prompt structure
|
|
|
|
|
+
|
|
|
|
|
+The system prompt uses `## AUTHORITY ORDER`, `## STRICT RULES`, `## OUTPUT FORMAT` headers.
|
|
|
|
|
+llama3.1 responds well to Markdown section headers as instruction anchors. The fallback
|
|
|
|
|
+phrase when context is insufficient: *"The provided context does not cover this — check the
|
|
|
|
|
+TPSO viewer directly at tpso.planning.tas.gov.au"*
|
|
|
|
|
+
|
|
|
|
|
+### Qdrant document schema
|
|
|
|
|
+
|
|
|
|
|
+Each chunk in the vector DB has this payload:
|
|
|
|
|
+
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "corpus": "tps", // "tps" | "lps" | "ncc" | "as"
|
|
|
|
|
+ "council": "launceston", // only for LPS chunks
|
|
|
|
|
+ "source_file": "launceston-lps.pdf",
|
|
|
|
|
+ "page": 42,
|
|
|
|
|
+ "chunk_index": 3,
|
|
|
|
|
+ "text": "..."
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Scope filtering uses `corpus` and `council` exact-match filters in Qdrant.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Frontend Pages
|
|
|
|
|
+
|
|
|
|
|
+### Design system
|
|
|
|
|
+
|
|
|
|
|
+All pages use a consistent dark design system. Never use Bootstrap on redesigned pages.
|
|
|
|
|
+
|
|
|
|
|
+```css
|
|
|
|
|
+--bg: #0b0f0e; /* page background */
|
|
|
|
|
+--bg-1: #111614; /* slightly lighter panels */
|
|
|
|
|
+--bg-2: #181e1b; /* card interiors */
|
|
|
|
|
+--bg-card: #141a17;
|
|
|
|
|
+--border: rgba(255,255,255,0.07);
|
|
|
|
|
+--accent: #2ddc8a; /* green accent */
|
|
|
|
|
+--accent-dim: rgba(45,220,138,0.10);
|
|
|
|
|
+--text-primary: #eaf0ec;
|
|
|
|
|
+--text-secondary:#8fa899;
|
|
|
|
|
+--text-muted: #4f6459;
|
|
|
|
|
+--danger: #f08080;
|
|
|
|
|
+--serif: 'DM Serif Display', Georgia, serif;
|
|
|
|
|
+--sans: 'DM Sans', system-ui, sans-serif;
|
|
|
|
|
+--mono: ui-monospace, 'Cascadia Code', Menlo, monospace;
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Fonts loaded from Google Fonts: `DM Serif Display` (headings/display) + `DM Sans` (body).
|
|
|
|
|
+Bootstrap Icons CDN used for icons: `https://cdn.jsdelivr.net/npm/bootstrap-icons@1.11.3/font/bootstrap-icons.css`
|
|
|
|
|
+
|
|
|
|
|
+### Page inventory
|
|
|
|
|
+
|
|
|
|
|
+| File | Purpose | Notes |
|
|
|
|
|
+|---|---|---|
|
|
|
|
|
+| `index.php` | Landing page | Has demo modal, waitlist form, hero search |
|
|
|
|
|
+| `local_state-planning-scheme.php` | Main chat assistant | Two-panel: sidebar + chat thread |
|
|
|
|
|
+| `site-report.php` | Property lookup | Google Maps PlaceAutocomplete, LIST/ArcGIS API |
|
|
|
|
|
+| `section-builder.php` | Report builder | 3-panel: context / sections / preview |
|
|
|
|
|
+| `byok-settings.php` | API key management | Keys in localStorage only, never server |
|
|
|
|
|
+| `dashboard.php` | Query monitoring | Restricted by IP via .htaccess |
|
|
|
|
|
+| `waitlist.php` | Waitlist endpoint | PHPMailer, reads SMTP_* from environment |
|
|
|
|
|
+| `gmaps-key.php` | Maps key proxy | Serves GMAPS_API_KEY from env, checks HTTP_HOST |
|
|
|
|
|
+| `faq.php` | FAQ | Accordion, sticky sidebar, FAQPage schema |
|
|
|
|
|
+| `privacy.php` | Privacy policy | |
|
|
|
|
|
+| `terms.php` | Terms of use | |
|
|
|
|
|
+
|
|
|
|
|
+### Cross-tab data passing
|
|
|
|
|
+
|
|
|
|
|
+Property data passes from `site-report.php` → `section-builder.php` via `localStorage`:
|
|
|
|
|
+
|
|
|
|
|
+```javascript
|
|
|
|
|
+// site-report.php writes:
|
|
|
|
|
+localStorage.setItem('tpr_builder_ctx', JSON.stringify({ ctx, written_at: Date.now() }));
|
|
|
|
|
+
|
|
|
|
|
+// section-builder.php reads (30-minute TTL):
|
|
|
|
|
+const { ctx, written_at } = JSON.parse(localStorage.getItem('tpr_builder_ctx'));
|
|
|
|
|
+if (Date.now() - written_at > 30 * 60 * 1000) { /* expired */ }
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Never use `sessionStorage` for cross-tab data** — each tab gets its own isolated sessionStorage.
|
|
|
|
|
+
|
|
|
|
|
+### BYOK (Bring Your Own Key)
|
|
|
|
|
+
|
|
|
|
|
+localStorage keys:
|
|
|
|
|
+- `tpr_byok_active` — active provider: `'internal'` | `'anthropic'` | `'openai'` | `'grok'` | `'ollama'`
|
|
|
|
|
+- `tpr_byok_key_{provider}` — API key for each provider
|
|
|
|
|
+- `tpr_byok_model_{provider}` — selected model for each provider
|
|
|
|
|
+
|
|
|
|
|
+BYOK flow in `local_state-planning-scheme.php`:
|
|
|
|
|
+1. POST to `/ask` with `context_only: true` → get RAG context + pre-built prompt
|
|
|
|
|
+2. Call provider API directly from browser with the prompt
|
|
|
|
|
+3. Render answer with provider badge (purple for external, green for internal)
|
|
|
|
|
+
|
|
|
|
|
+Supported providers: Anthropic (`claude-sonnet-4-5`), OpenAI (`gpt-4o-mini`), xAI Grok (`grok-3-mini`), local Ollama.
|
|
|
|
|
+
|
|
|
|
|
+### API status indicator
|
|
|
|
|
+
|
|
|
|
|
+`/js/api-status.js` is included on every page:
|
|
|
|
|
+```html
|
|
|
|
|
+<script src="/js/api-status.js" data-api="https://api.modulos.com.au"></script>
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Polls `/readyz` every 30s (10s when down). Updates `.status-dot` and `.nav-status-text`.
|
|
|
|
|
+Nav HTML must include:
|
|
|
|
|
+```html
|
|
|
|
|
+<div class="nav-status">
|
|
|
|
|
+ <span class="status-dot"></span>
|
|
|
|
|
+ <span class="nav-status-text">API live</span>
|
|
|
|
|
+</div>
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Feedback system
|
|
|
|
|
+
|
|
|
|
|
+Each assistant message stores context as `data-*` attributes for scope-safe feedback:
|
|
|
|
|
+
|
|
|
|
|
+```javascript
|
|
|
|
|
+div.dataset.query = query;
|
|
|
|
|
+div.dataset.scope = scope;
|
|
|
|
|
+div.dataset.provider = provider;
|
|
|
|
|
+div.dataset.answer = answer.replace(/<[^>]*>/g, '').substring(0, 4000);
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+`window.feedback(msgId, verdict, btn)` reads these attributes and POSTs to `/feedback`.
|
|
|
|
|
+**Never rely on closure variables** — use `data-*` attributes or `localStorage` reads.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Telemetry Database
|
|
|
|
|
+
|
|
|
|
|
+SQLite at `/home/modulos_llm/telemetry_data/telemetry.db`
|
|
|
|
|
+
|
|
|
|
|
+### Tables
|
|
|
|
|
+
|
|
|
|
|
+**`ask_logs`** — every query:
|
|
|
|
|
+```sql
|
|
|
|
|
+id, ts, sid, ip_hash, query, normalized, scope, allow_tps,
|
|
|
|
|
+latency_ms, model, ok, topk_json, tokens_in, tokens_out, answer
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**`feedback`** — thumbs up/down:
|
|
|
|
|
+```sql
|
|
|
|
|
+id, ts, sid, ip_hash, verdict, query, answer, note, model, scope, sources_json
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**`events`** — generic telemetry events:
|
|
|
|
|
+```sql
|
|
|
|
|
+id, ts, type, sid, ua, ip_hash, data_json
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Adding missing columns (migration pattern)
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+docker exec modulos-backend python3 - << 'EOF'
|
|
|
|
|
+import sqlite3, os
|
|
|
|
|
+db = sqlite3.connect(os.getenv('TPR_DB', '/data/telemetry.db'))
|
|
|
|
|
+cols = [r[1] for r in db.execute("PRAGMA table_info(ask_logs)").fetchall()]
|
|
|
|
|
+if 'answer' not in cols:
|
|
|
|
|
+ db.execute("ALTER TABLE ask_logs ADD COLUMN answer TEXT DEFAULT ''")
|
|
|
|
|
+ db.commit()
|
|
|
|
|
+ print("✓ added")
|
|
|
|
|
+db.close()
|
|
|
|
|
+EOF
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Useful queries
|
|
|
|
|
+
|
|
|
|
|
+```sql
|
|
|
|
|
+-- Today's queries with answers
|
|
|
|
|
+SELECT ts, query, answer, latency_ms, scope, model
|
|
|
|
|
+FROM ask_logs WHERE ts LIKE '2026-03-26%' ORDER BY ts DESC;
|
|
|
|
|
+
|
|
|
|
|
+-- Thumbs-down for prompt improvement
|
|
|
|
|
+SELECT query, answer, note, model, scope
|
|
|
|
|
+FROM feedback WHERE verdict='down' ORDER BY ts DESC;
|
|
|
|
|
+
|
|
|
|
|
+-- Satisfaction rate
|
|
|
|
|
+SELECT
|
|
|
|
|
+ SUM(CASE WHEN verdict='up' THEN 1 ELSE 0 END) as up,
|
|
|
|
|
+ SUM(CASE WHEN verdict='down' THEN 1 ELSE 0 END) as down,
|
|
|
|
|
+ ROUND(SUM(CASE WHEN verdict='up' THEN 1.0 ELSE 0 END) / COUNT(*) * 100) as pct
|
|
|
|
|
+FROM feedback;
|
|
|
|
|
+
|
|
|
|
|
+-- Repeated queries (cache candidates)
|
|
|
|
|
+SELECT normalized, COUNT(*) as hits, ROUND(AVG(latency_ms)) as avg_ms
|
|
|
|
|
+FROM ask_logs WHERE ok=1 GROUP BY normalized ORDER BY hits DESC LIMIT 20;
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## PHP/Composer
|
|
|
|
|
+
|
|
|
|
|
+### Installed packages (composer.json)
|
|
|
|
|
+
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "require": {
|
|
|
|
|
+ "phpoffice/phpword": "^1.3",
|
|
|
|
|
+ "phpmailer/phpmailer": "^6.9",
|
|
|
|
|
+ "erusev/parsedown": "^1.7",
|
|
|
|
|
+ "google/apiclient": "^2.15"
|
|
|
|
|
+ }
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Installing/updating dependencies
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# Add a new package (use docker run to avoid platform issues)
|
|
|
|
|
+docker run --rm -v /home/modulos_llm/public:/app composer:2 \
|
|
|
|
|
+ require vendor/package --no-interaction --no-progress --ignore-platform-reqs
|
|
|
|
|
+
|
|
|
|
|
+# Update all (run from project root)
|
|
|
|
|
+docker compose run --rm composer update --no-interaction --ignore-platform-req=ext-gd
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Known issue:** `phpoffice/phpword 1.4.0` requires `ext-gd` which isn't in the Composer
|
|
|
|
|
+container — always use `--ignore-platform-req=ext-gd` or `--ignore-platform-reqs`.
|
|
|
|
|
+
|
|
|
|
|
+### PHPMailer environment variables
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+SMTP_HOST=mail.modulos.com.au
|
|
|
|
|
+SMTP_PORT=465 # 465=SSL, 587=TLS
|
|
|
|
|
+SMTP_USER=no-reply@modulos.com.au
|
|
|
|
|
+SMTP_PASS=<from .env>
|
|
|
|
|
+SMTP_FROM=no-reply@modulos.com.au
|
|
|
|
|
+SMTP_FROM_NAME=Tas Planning Assistant
|
|
|
|
|
+NOTIFY_EMAIL=ben@modulos.com.au
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Infrastructure
|
|
|
|
|
+
|
|
|
|
|
+### Reverse proxy
|
|
|
|
|
+
|
|
|
|
|
+Nginx/Cloudflare sits in front. The web container is on port 2380, backend on 2281.
|
|
|
|
|
+Public URLs:
|
|
|
|
|
+- `https://tasplanning.report` → web container (port 2380)
|
|
|
|
|
+- `https://api.modulos.com.au` → backend container (port 2281)
|
|
|
|
|
+
|
|
|
|
|
+### Google Maps
|
|
|
|
|
+
|
|
|
|
|
+`gmaps-key.php` serves the API key from env — never put it in page source.
|
|
|
|
|
+`site-report.php` loads Maps via:
|
|
|
|
|
+```javascript
|
|
|
|
|
+const keyRes = await fetch('/gmaps-key.php');
|
|
|
|
|
+const { key } = await keyRes.json();
|
|
|
|
|
+await google.maps.importLibrary('places');
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+`PassEnv GMAPS_API_KEY` is written to Apache config in the web container startup command.
|
|
|
|
|
+
|
|
|
|
|
+### Google Docs export
|
|
|
|
|
+
|
|
|
|
|
+OAuth2 credentials at `/var/www/secure/service-account.json` inside the web container.
|
|
|
|
|
+`GDOC_PARENT_ID` and `GDOC_SHARE_EMAIL` set in docker-compose environment.
|
|
|
|
|
+
|
|
|
|
|
+### Dashboard access control
|
|
|
|
|
+
|
|
|
|
|
+`/home/modulos_llm/public/.htaccess` restricts `dashboard.php` to specific IPs:
|
|
|
|
|
+
|
|
|
|
|
+```apache
|
|
|
|
|
+<IfModule mod_rewrite.c>
|
|
|
|
|
+ RewriteEngine On
|
|
|
|
|
+ RewriteCond %{REMOTE_ADDR} ^YOUR\.IP\.HERE$
|
|
|
|
|
+ RewriteRule ^dashboard\.php$ - [L]
|
|
|
|
|
+ RewriteCond %{HTTP:X-Forwarded-For} ^YOUR\.IP\.HERE
|
|
|
|
|
+ RewriteRule ^dashboard\.php$ - [L]
|
|
|
|
|
+ RewriteRule ^dashboard\.php$ https://tasplanning.report/ [R=302,L]
|
|
|
|
|
+</IfModule>
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Key Conventions
|
|
|
|
|
+
|
|
|
|
|
+### Never do these
|
|
|
|
|
+
|
|
|
|
|
+- **Never use `sessionStorage` for cross-tab data** — use `localStorage` with a TTL
|
|
|
|
|
+- **Never put API keys in PHP page output** — use env vars + proxy endpoints
|
|
|
|
|
+- **Never add Bootstrap to redesigned pages** — use the custom dark design system
|
|
|
|
|
+- **Never change `num_ctx` between requests** — Ollama reloads the model on every change
|
|
|
|
|
+- **Never put `keep_alive` inside Ollama `options{}`** — it must be a top-level key
|
|
|
|
|
+- **Never rely on closure variables in `window.*` functions** — use `data-*` attributes
|
|
|
|
|
+- **Never run `composer install` when lock file is out of sync** — run `composer update` first
|
|
|
|
|
+- **Never commit `.env`** — it contains SMTP_PASS, API keys, secrets
|
|
|
|
|
+
|
|
|
|
|
+### Always do these
|
|
|
|
|
+
|
|
|
|
|
+- **Always restart the backend** after changing `app.py` — it's not hot-reloaded
|
|
|
|
|
+- **Always use `--ignore-platform-reqs`** with composer when adding packages
|
|
|
|
|
+- **Always run `apachectl -t`** after changing Apache config or `.htaccess`
|
|
|
|
|
+- **Always escape output** in PHP with `htmlspecialchars($s, ENT_QUOTES, 'UTF-8')`
|
|
|
|
|
+- **Always use `credentials: 'omit'`** for cross-origin fetch to the API
|
|
|
|
|
+- **Always add new DB columns** via `ALTER TABLE` migration, not by dropping/recreating
|
|
|
|
|
+- **Always test `context_only: true`** after prompt changes to verify RAG retrieval before LLM issues
|
|
|
|
|
+
|
|
|
|
|
+### Scope values
|
|
|
|
|
+
|
|
|
|
|
+| Value | Searches |
|
|
|
|
|
+|---|---|
|
|
|
|
|
+| `state_plus_local` | SPPs + selected council's LPS |
|
|
|
|
|
+| `state_only` | SPPs only |
|
|
|
|
|
+| `local_only` | Selected council's LPS only |
|
|
|
|
|
+| `any` | All indexed documents |
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Prompt Engineering Notes
|
|
|
|
|
+
|
|
|
|
|
+The current prompt structure (in `do_ask()` in `app.py`):
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+## AUTHORITY ORDER
|
|
|
|
|
+1. SPP (baseline)
|
|
|
|
|
+2. LPS for selected council (overrides SPP)
|
|
|
|
|
+3. NCC (optional, building control only)
|
|
|
|
|
+4. Australian Standards (optional)
|
|
|
|
|
+
|
|
|
|
|
+## STRICT RULES
|
|
|
|
|
+- Use ONLY information in CONTEXT
|
|
|
|
|
+- Fallback phrase: "The provided context does not cover this..."
|
|
|
|
|
+- Every claim must cite (filename, p.N)
|
|
|
|
|
+- Distinguish A (Acceptable Solutions) vs P (Performance Criteria)
|
|
|
|
|
+
|
|
|
|
|
+## OUTPUT FORMAT
|
|
|
|
|
+- Markdown with ## headings
|
|
|
|
|
+- Tables for setbacks, parking rates, multiple standards
|
|
|
|
|
+- ## Sources section at end of every response
|
|
|
|
|
+
|
|
|
|
|
+## CONTEXT
|
|
|
|
|
+{retrieved chunks}
|
|
|
|
|
+
|
|
|
|
|
+{format_guide} # section-specific formatting instructions
|
|
|
|
|
+
|
|
|
|
|
+## QUESTION
|
|
|
|
|
+{query}
|
|
|
|
|
+
|
|
|
|
|
+## ANSWER:
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Debugging RAG vs prompt issues:**
|
|
|
|
|
+1. Use `context_only: true` to inspect what context was retrieved
|
|
|
|
|
+2. If context contains the answer but model missed it → prompt issue
|
|
|
|
|
+3. If context doesn't contain the answer → RAG/retrieval issue (increase `top_k`, check chunk boundaries)
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Development Workflow
|
|
|
|
|
+
|
|
|
|
|
+### Making backend changes
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# Edit the file
|
|
|
|
|
+nano /home/modulos_llm/backend/app.py
|
|
|
|
|
+
|
|
|
|
|
+# Restart to pick up changes
|
|
|
|
|
+docker compose restart backend
|
|
|
|
|
+
|
|
|
|
|
+# Watch logs
|
|
|
|
|
+docker logs modulos-backend -f
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Making frontend changes
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# Edit directly — changes are live immediately
|
|
|
|
|
+nano /home/modulos_llm/public/local_state-planning-scheme.php
|
|
|
|
|
+# No restart needed
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Adding a new PHP page
|
|
|
|
|
+
|
|
|
|
|
+1. Create `pagename.php` in `/home/modulos_llm/public/`
|
|
|
|
|
+2. Use the dark design system CSS variables
|
|
|
|
|
+3. Include nav with `.status-dot` and `.nav-status-text`
|
|
|
|
|
+4. Add `<script src="/js/api-status.js" data-api="https://api.modulos.com.au"></script>`
|
|
|
|
|
+5. Add link to nav on other pages
|
|
|
|
|
+
|
|
|
|
|
+### Adding a new API endpoint
|
|
|
|
|
+
|
|
|
|
|
+1. Add Pydantic model for request body if needed
|
|
|
|
|
+2. Add endpoint function with `@app.post("/path")` and `@limiter.limit("20/minute")`
|
|
|
|
|
+3. Add telemetry insert inside a `try/except` — never let logging break the endpoint
|
|
|
|
|
+4. Restart backend: `docker compose restart backend`
|
|
|
|
|
+
|
|
|
|
|
+### Ingesting new planning documents
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# Copy PDFs to backend folder
|
|
|
|
|
+cp new-lps.pdf /home/modulos_llm/backend/
|
|
|
|
|
+
|
|
|
|
|
+# Run ingestion (adjust args for corpus/council)
|
|
|
|
|
+docker exec modulos-backend python3 ingest.py \
|
|
|
|
|
+ --pdf new-lps.pdf \
|
|
|
|
|
+ --corpus lps \
|
|
|
|
|
+ --council launceston
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## External Services
|
|
|
|
|
+
|
|
|
|
|
+| Service | URL | Purpose |
|
|
|
|
|
+|---|---|---|
|
|
|
|
|
+| Ollama | `http://192.168.8.73:11434` | LLM inference (Windows machine, RTX 4070 Super) |
|
|
|
|
|
+| Qdrant | `http://qdrant:6333` (internal) | Vector search |
|
|
|
|
|
+| TPSO Viewer | `https://tpso.planning.tas.gov.au/tpso/external/planning-scheme-viewer/30` | SPP viewer (iframe) |
|
|
|
|
|
+| LIST GIS | `https://services.thelist.tas.gov.au/arcgis/rest/services` | Property/parcel data |
|
|
|
|
|
+| Google Maps | `https://maps.googleapis.com` | Address autocomplete in site-report |
|
|
|
|
|
+| Anthropic API | `https://api.anthropic.com/v1/messages` | BYOK Claude |
|
|
|
|
|
+| OpenAI API | `https://api.openai.com/v1/chat/completions` | BYOK GPT |
|
|
|
|
|
+| xAI API | `https://api.x.ai/v1/chat/completions` | BYOK Grok |
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Current Model
|
|
|
|
|
+
|
|
|
|
|
+- **Model:** `llama3.1:8b-instruct-q4_K_M`
|
|
|
|
|
+- **Hardware:** NVIDIA GeForce RTX 4070 Super (12GB VRAM), Windows host
|
|
|
|
|
+- **VRAM usage:** ~4.5GB model + ~1GB KV cache at num_ctx=6144
|
|
|
|
|
+- **Typical latency:** 4-12s depending on answer length
|
|
|
|
|
+- **Memory bandwidth:** saturated at ~95-98% during inference (bandwidth-bound, not compute-bound)
|
|
|
|
|
+- **Upgrade path:** RTX 4090 (24GB) would allow llama3.1:70B-Q4 and roughly 2× throughput
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Security Notes
|
|
|
|
|
+
|
|
|
|
|
+- IP hashes in telemetry use HMAC-SHA256 with `TPR_IP_SECRET` — not reversible
|
|
|
|
|
+- BYOK keys stored in browser `localStorage` only — never sent to our servers
|
|
|
|
|
+- `gmaps-key.php` checks `HTTP_HOST`/`HTTP_ORIGIN` before serving the Maps key
|
|
|
|
|
+- `dashboard.php` restricted by `.htaccess` IP allowlist
|
|
|
|
|
+- `DEMO_TOKEN` in docker-compose is for optional API gating (currently disabled)
|
|
|
|
|
+- `.env` file contains all secrets — never commit to version control
|