# CLAUDE.md — Tasmanian Planning Scheme Assistant
# tasplanning.report

This file gives Claude Code full context on the project architecture, conventions, and
deployment workflow. Read this before making any changes.

---

## Project Overview

**tasplanning.report** is an AI-powered assistant for the Tasmanian Planning Scheme (TPS). It allows planners, building designers, architects, and consultants to ask natural-language questions about SPPs and LPS documents and receive clause-cited answers.

**Core capability:** Retrieval-Augmented Generation (RAG) — user queries are embedded, matched against a Qdrant vector database of official planning documents, and the retrieved context is injected into an LLM prompt. The LLM generates a cited answer using only the retrieved context — it never invents clause numbers.

---

## Repository Layout

```
/home/modulos_llm/
├── pdfs/
│   ├── as/                    # Australian Standards in PDF
│   ├── lps/                   # Local Planning Scheme in PDF
│   ├── ncc/                   # National Construction Code in PDF
│   ├── tps/                   # Tasmanian Planning Scheme in PDF
├── backend/                   # FastAPI Python backend
│   ├── app.py                 # Main API — /ask, /feedback, /councils, /readyz, admin endpoints
│   ├── telemetry.py           # SQLite telemetry DB, /telemetry POST endpoint, schema init
│   ├── ingest.py              # PDF ingestion into Qdrant (run manually)
│   ├── ingest_ollama.py       # Ollama-based ingestion variant
│   └── requirements.txt       # Python dependencies
├── public/                    # PHP/Apache web frontend (bind-mounted into web container)
│   ├── index.php              # Landing page with demo modal and waitlist
│   ├── local_state-planning-scheme.php   # Main chat assistant UI
│   ├── site-report.php        # Property lookup (LIST/ArcGIS + Google Maps)
│   ├── section-builder.php    # Planning report section builder
│   ├── byok-settings.php      # API key management (BYOK feature)
│   ├── dashboard.php          # Internal query monitoring dashboard
│   ├── waitlist.php           # PHPMailer waitlist signup endpoint
│   ├── gmaps-key.php          # Google Maps API key proxy
│   ├── faq.php                # FAQ page
│   ├── privacy.php            # Privacy policy
│   ├── terms.php              # Terms of use
│   ├── js/
│   │   └── api-status.js      # Shared live API health indicator
│   ├── css/
│   │   └── report.css         # Shared stylesheet for index.php
│   ├── vendor/                # Composer PHP dependencies (PHPMailer, phpword, google/apiclient)
│   └── composer.json          # PHP dependency manifest
├── web/
│   └── Dockerfile             # Custom PHP 8.3 + Apache image
├── cache/
├── creds/
│   ├── oauth-client.json
│   ├── service-account.json
├── images/
├── models/
├── qdrant_storage/            # Qdrant vector DB persistent storage
│   ├── aliases/               #
│   ├── collections/           #
│   ├── raft_state.json        #
├── telemetry_data/
│   └── telemetry.db           # SQLite telemetry database
├── venv/
│   ├── include
│   │   └── python3.1.2
├── docker-compose.yml         # Full stack orchestration
└── .env                       # Secrets (SMTP_PASS, GMAPS_API_KEY, TPR_IP_SECRET, etc.)
└── CHANGELOG.md
└── CLAUDE.md
└── README.md

```

---

## Docker Services

All services defined in `docker-compose.yml`. Run from `/home/modulos_llm/`.

| Service | Container | Port | Purpose |
|---|---|---|---|
| `qdrant` | `modulos-qdrant` | 6333, 6334 | Vector database |
| `backend` | `modulos-backend` | 2281→8000 | FastAPI API |
| `web` | `modulos-web` | 2380→80 | PHP/Apache frontend |
| `sqliteweb` | `modulos-sqliteweb` | 8091→8080 | SQLite Web UI (internal) |
| `composer` | *(run-once)* | — | PHP dependency installer |

### Common commands

```bash
# Start everything
docker compose up -d

# Restart a single service (backend changes need restart, web changes do not)
docker compose restart backend

# View logs
docker logs modulos-backend --tail 50
docker logs modulos-web --tail 50

# Force recreate (picks up docker-compose.yml changes)
docker compose up -d --force-recreate web

# Run composer to install/update PHP dependencies
docker compose run --rm composer
# Or directly:
docker run --rm -v /home/modulos_llm/public:/app composer:2 \
  require phpmailer/phpmailer --no-interaction --no-progress --ignore-platform-reqs

# One-off Python command in backend container
docker exec modulos-backend python3 -c "..."

# Check API health
curl -s https://api.modulos.com.au/readyz
```

### Important: bind mounts

- `./backend:/app` — backend Python files are live. Edit files on host, **restart backend** to pick up changes.
- `./public:/var/www/html` — PHP files are live. Edit files on host, **no restart needed**.
- `/home/modulos_llm/telemetry_data:/data` — telemetry DB shared between backend and web containers.

---

## Backend (`app.py`)

### Key environment variables

```bash
OLLAMA_URL=http://192.168.8.73:11434      # Windows machine running Ollama
QDRANT_URL=http://qdrant:6333             # Internal Docker network
OLLAMA_KEEP_ALIVE=-1                      # Keep model loaded permanently (-1 = forever)
CHAT_MODEL=llama3.1:8b-instruct-q4_K_M   # Current model (Q4 quantization)
EMBED_MODEL=nomic-embed-text              # Embedding model
QDRANT_COLLECTION=planning_docs           # Vector collection name
CORS_ORIGINS=https://tasplanning.report,http://localhost:3000,...
TPR_DB=/data/telemetry.db
TPR_IP_SECRET=<secret>                    # HMAC secret for IP hashing
```

### API endpoints

| Method | Path | Purpose |
|---|---|---|
| GET | `/readyz` | Health check — returns `{"ok":true}` |
| GET | `/councils` | List indexed council names |
| POST | `/ask` | Main RAG query endpoint |
| GET | `/ask` | Same, query string params |
| POST | `/feedback` | Store thumbs up/down with query+answer |
| POST | `/telemetry` | Generic event logging |
| GET | `/admin/stats` | Collection statistics |
| GET | `/admin/files` | List indexed documents |
| GET | `/admin/sample` | Sample random chunks |
| GET | `/admin/export` | NDJSON export of all chunks |

### `/ask` request body (`AskBody`)

```python
{
  "query": "What are the setbacks in the Village Zone?",
  "council": "launceston",     # optional — filters to that council's LPS
  "top_k": 8,                  # number of chunks to retrieve
  "scope": "state_plus_local", # state_plus_local | state_only | local_only | any
  "section_id": null,          # optional — triggers specific output format guide
  "include_ncc": false,
  "include_standards": false,
  "context_only": false        # BYOK mode — returns context+prompt without calling Ollama
}
```

### `/ask` response

```json
{
  "answer": "...",
  "sources": [
    {"source_file": "tasmanian-planning-scheme.pdf", "page": 42, "chunk_index": 3, "score": 0.87}
  ]
}
```

When `context_only: true` (BYOK mode):
```json
{
  "context_only": true,
  "context": "...",
  "prompt": "...",   // full prompt ready to send to external LLM
  "sources": [...],
  "sections": [...]
}
```

### Ollama settings (in `ollama_chat()`)

```python
"keep_alive": -1,          # top-level param, NOT inside options
"options": {
  "num_ctx": 6144,          # context window — don't change between requests or model reloads
  "num_predict": 1024,      # max output tokens
  "temperature": 0.15,      # low = more deterministic
  "top_p": 0.85,
  "top_k": 40,
  "repeat_penalty": 1.15,
  "stop": ["QUESTION:", "---END---"],
}
```

**Important:** `keep_alive` must be a top-level JSON key, not inside `options`. Putting it
inside `options` causes Ollama to silently ignore it.

### Prompt structure

The system prompt uses `## AUTHORITY ORDER`, `## STRICT RULES`, `## OUTPUT FORMAT` headers.
llama3.1 responds well to Markdown section headers as instruction anchors. The fallback
phrase when context is insufficient: *"The provided context does not cover this — check the
TPSO viewer directly at tpso.planning.tas.gov.au"*

### Qdrant document schema

Each chunk in the vector DB has this payload:

```json
{
  "corpus": "tps",                          // "tps" | "lps" | "ncc" | "as"
  "council": "launceston",                  // only for LPS chunks
  "source_file": "launceston-lps.pdf",
  "page": 42,
  "chunk_index": 3,
  "text": "..."
}
```

Scope filtering uses `corpus` and `council` exact-match filters in Qdrant.

---

## Frontend Pages

### Design system

All pages use a consistent dark design system. Never use Bootstrap on redesigned pages.

```css
--bg:            #0b0f0e;    /* page background */
--bg-1:          #111614;    /* slightly lighter panels */
--bg-2:          #181e1b;    /* card interiors */
--bg-card:       #141a17;
--border:        rgba(255,255,255,0.07);
--accent:        #2ddc8a;    /* green accent */
--accent-dim:    rgba(45,220,138,0.10);
--text-primary:  #eaf0ec;
--text-secondary:#8fa899;
--text-muted:    #4f6459;
--danger:        #f08080;
--serif:         'DM Serif Display', Georgia, serif;
--sans:          'DM Sans', system-ui, sans-serif;
--mono:          ui-monospace, 'Cascadia Code', Menlo, monospace;
```

Fonts loaded from Google Fonts: `DM Serif Display` (headings/display) + `DM Sans` (body).
Bootstrap Icons CDN used for icons: `https://cdn.jsdelivr.net/npm/bootstrap-icons@1.11.3/font/bootstrap-icons.css`

### Page inventory

| File | Purpose | Notes |
|---|---|---|
| `index.php` | Landing page | Has demo modal, waitlist form, hero search |
| `local_state-planning-scheme.php` | Main chat assistant | Two-panel: sidebar + chat thread |
| `site-report.php` | Property lookup | Google Maps PlaceAutocomplete, LIST/ArcGIS API |
| `section-builder.php` | Report builder | 3-panel: context / sections / preview |
| `byok-settings.php` | API key management | Keys in localStorage only, never server |
| `dashboard.php` | Query monitoring | Restricted by IP via .htaccess |
| `waitlist.php` | Waitlist endpoint | PHPMailer, reads SMTP_* from environment |
| `gmaps-key.php` | Maps key proxy | Serves GMAPS_API_KEY from env, checks HTTP_HOST |
| `faq.php` | FAQ | Accordion, sticky sidebar, FAQPage schema |
| `privacy.php` | Privacy policy | |
| `terms.php` | Terms of use | |

### Cross-tab data passing

Property data passes from `site-report.php` → `section-builder.php` via `localStorage`:

```javascript
// site-report.php writes:
localStorage.setItem('tpr_builder_ctx', JSON.stringify({ ctx, written_at: Date.now() }));

// section-builder.php reads (30-minute TTL):
const { ctx, written_at } = JSON.parse(localStorage.getItem('tpr_builder_ctx'));
if (Date.now() - written_at > 30 * 60 * 1000) { /* expired */ }
```

**Never use `sessionStorage` for cross-tab data** — each tab gets its own isolated sessionStorage.

### BYOK (Bring Your Own Key)

localStorage keys:
- `tpr_byok_active` — active provider: `'internal'` | `'anthropic'` | `'openai'` | `'grok'` | `'ollama'`
- `tpr_byok_key_{provider}` — API key for each provider
- `tpr_byok_model_{provider}` — selected model for each provider

BYOK flow in `local_state-planning-scheme.php`:
1. POST to `/ask` with `context_only: true` → get RAG context + pre-built prompt
2. Call provider API directly from browser with the prompt
3. Render answer with provider badge (purple for external, green for internal)

Supported providers: Anthropic (`claude-sonnet-4-5`), OpenAI (`gpt-4o-mini`), xAI Grok (`grok-3-mini`), local Ollama.

### API status indicator

`/js/api-status.js` is included on every page:
```html
<script src="/js/api-status.js" data-api="https://api.modulos.com.au"></script>
```

Polls `/readyz` every 30s (10s when down). Updates `.status-dot` and `.nav-status-text`.
Nav HTML must include:
```html
<div class="nav-status">
  <span class="status-dot"></span>
  <span class="nav-status-text">API live</span>
</div>
```

### Feedback system

Each assistant message stores context as `data-*` attributes for scope-safe feedback:

```javascript
div.dataset.query    = query;
div.dataset.scope    = scope;
div.dataset.provider = provider;
div.dataset.answer   = answer.replace(/<[^>]*>/g, '').substring(0, 4000);
```

`window.feedback(msgId, verdict, btn)` reads these attributes and POSTs to `/feedback`.
**Never rely on closure variables** — use `data-*` attributes or `localStorage` reads.

---

## Telemetry Database

SQLite at `/home/modulos_llm/telemetry_data/telemetry.db`

### Tables

**`ask_logs`** — every query:
```sql
id, ts, sid, ip_hash, query, normalized, scope, allow_tps,
latency_ms, model, ok, topk_json, tokens_in, tokens_out, answer
```

**`feedback`** — thumbs up/down:
```sql
id, ts, sid, ip_hash, verdict, query, answer, note, model, scope, sources_json
```

**`events`** — generic telemetry events:
```sql
id, ts, type, sid, ua, ip_hash, data_json
```

### Adding missing columns (migration pattern)

```bash
docker exec modulos-backend python3 - << 'EOF'
import sqlite3, os
db = sqlite3.connect(os.getenv('TPR_DB', '/data/telemetry.db'))
cols = [r[1] for r in db.execute("PRAGMA table_info(ask_logs)").fetchall()]
if 'answer' not in cols:
    db.execute("ALTER TABLE ask_logs ADD COLUMN answer TEXT DEFAULT ''")
    db.commit()
    print("✓ added")
db.close()
EOF
```

### Useful queries

```sql
-- Today's queries with answers
SELECT ts, query, answer, latency_ms, scope, model
FROM ask_logs WHERE ts LIKE '2026-03-26%' ORDER BY ts DESC;

-- Thumbs-down for prompt improvement
SELECT query, answer, note, model, scope
FROM feedback WHERE verdict='down' ORDER BY ts DESC;

-- Satisfaction rate
SELECT
  SUM(CASE WHEN verdict='up' THEN 1 ELSE 0 END) as up,
  SUM(CASE WHEN verdict='down' THEN 1 ELSE 0 END) as down,
  ROUND(SUM(CASE WHEN verdict='up' THEN 1.0 ELSE 0 END) / COUNT(*) * 100) as pct
FROM feedback;

-- Repeated queries (cache candidates)
SELECT normalized, COUNT(*) as hits, ROUND(AVG(latency_ms)) as avg_ms
FROM ask_logs WHERE ok=1 GROUP BY normalized ORDER BY hits DESC LIMIT 20;
```

---

## PHP/Composer

### Installed packages (composer.json)

```json
{
  "require": {
    "phpoffice/phpword": "^1.3",
    "phpmailer/phpmailer": "^6.9",
    "erusev/parsedown": "^1.7",
    "google/apiclient": "^2.15"
  }
}
```

### Installing/updating dependencies

```bash
# Add a new package (use docker run to avoid platform issues)
docker run --rm -v /home/modulos_llm/public:/app composer:2 \
  require vendor/package --no-interaction --no-progress --ignore-platform-reqs

# Update all (run from project root)
docker compose run --rm composer update --no-interaction --ignore-platform-req=ext-gd
```

**Known issue:** `phpoffice/phpword 1.4.0` requires `ext-gd` which isn't in the Composer
container — always use `--ignore-platform-req=ext-gd` or `--ignore-platform-reqs`.

### PHPMailer environment variables

```bash
SMTP_HOST=mail.modulos.com.au
SMTP_PORT=465           # 465=SSL, 587=TLS
SMTP_USER=no-reply@modulos.com.au
SMTP_PASS=<from .env>
SMTP_FROM=no-reply@modulos.com.au
SMTP_FROM_NAME=Tas Planning Assistant
NOTIFY_EMAIL=ben@modulos.com.au
```

---

## Infrastructure

### Reverse proxy

Nginx/Cloudflare sits in front. The web container is on port 2380, backend on 2281.
Public URLs:
- `https://tasplanning.report` → web container (port 2380)
- `https://api.modulos.com.au` → backend container (port 2281)

### Google Maps

`gmaps-key.php` serves the API key from env — never put it in page source.
`site-report.php` loads Maps via:
```javascript
const keyRes = await fetch('/gmaps-key.php');
const { key } = await keyRes.json();
await google.maps.importLibrary('places');
```

`PassEnv GMAPS_API_KEY` is written to Apache config in the web container startup command.

### Google Docs export

OAuth2 credentials at `/var/www/secure/service-account.json` inside the web container.
`GDOC_PARENT_ID` and `GDOC_SHARE_EMAIL` set in docker-compose environment.

### Dashboard access control

`/home/modulos_llm/public/.htaccess` restricts `dashboard.php` to specific IPs:

```apache
<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{REMOTE_ADDR} ^YOUR\.IP\.HERE$
  RewriteRule ^dashboard\.php$ - [L]
  RewriteCond %{HTTP:X-Forwarded-For} ^YOUR\.IP\.HERE
  RewriteRule ^dashboard\.php$ - [L]
  RewriteRule ^dashboard\.php$ https://tasplanning.report/ [R=302,L]
</IfModule>
```

---

## Key Conventions

### Never do these

- **Never use `sessionStorage` for cross-tab data** — use `localStorage` with a TTL
- **Never put API keys in PHP page output** — use env vars + proxy endpoints
- **Never add Bootstrap to redesigned pages** — use the custom dark design system
- **Never change `num_ctx` between requests** — Ollama reloads the model on every change
- **Never put `keep_alive` inside Ollama `options{}`** — it must be a top-level key
- **Never rely on closure variables in `window.*` functions** — use `data-*` attributes
- **Never run `composer install` when lock file is out of sync** — run `composer update` first
- **Never commit `.env`** — it contains SMTP_PASS, API keys, secrets

### Always do these

- **Always restart the backend** after changing `app.py` — it's not hot-reloaded
- **Always use `--ignore-platform-reqs`** with composer when adding packages
- **Always run `apachectl -t`** after changing Apache config or `.htaccess`
- **Always escape output** in PHP with `htmlspecialchars($s, ENT_QUOTES, 'UTF-8')`
- **Always use `credentials: 'omit'`** for cross-origin fetch to the API
- **Always add new DB columns** via `ALTER TABLE` migration, not by dropping/recreating
- **Always test `context_only: true`** after prompt changes to verify RAG retrieval before LLM issues

### Scope values

| Value | Searches |
|---|---|
| `state_plus_local` | SPPs + selected council's LPS |
| `state_only` | SPPs only |
| `local_only` | Selected council's LPS only |
| `any` | All indexed documents |

---

## Prompt Engineering Notes

The current prompt structure (in `do_ask()` in `app.py`):

```
## AUTHORITY ORDER
1. SPP (baseline)
2. LPS for selected council (overrides SPP)
3. NCC (optional, building control only)
4. Australian Standards (optional)

## STRICT RULES
- Use ONLY information in CONTEXT
- Fallback phrase: "The provided context does not cover this..."
- Every claim must cite (filename, p.N)
- Distinguish A (Acceptable Solutions) vs P (Performance Criteria)

## OUTPUT FORMAT
- Markdown with ## headings
- Tables for setbacks, parking rates, multiple standards
- ## Sources section at end of every response

## CONTEXT
{retrieved chunks}

{format_guide}  # section-specific formatting instructions

## QUESTION
{query}

## ANSWER:
```

**Debugging RAG vs prompt issues:**
1. Use `context_only: true` to inspect what context was retrieved
2. If context contains the answer but model missed it → prompt issue
3. If context doesn't contain the answer → RAG/retrieval issue (increase `top_k`, check chunk boundaries)

---

## Development Workflow

### Making backend changes

```bash
# Edit the file
nano /home/modulos_llm/backend/app.py

# Restart to pick up changes
docker compose restart backend

# Watch logs
docker logs modulos-backend -f
```

### Making frontend changes

```bash
# Edit directly — changes are live immediately
nano /home/modulos_llm/public/local_state-planning-scheme.php
# No restart needed
```

### Adding a new PHP page

1. Create `pagename.php` in `/home/modulos_llm/public/`
2. Use the dark design system CSS variables
3. Include nav with `.status-dot` and `.nav-status-text`
4. Add `<script src="/js/api-status.js" data-api="https://api.modulos.com.au"></script>`
5. Add link to nav on other pages

### Adding a new API endpoint

1. Add Pydantic model for request body if needed
2. Add endpoint function with `@app.post("/path")` and `@limiter.limit("20/minute")`
3. Add telemetry insert inside a `try/except` — never let logging break the endpoint
4. Restart backend: `docker compose restart backend`

### Ingesting new planning documents

```bash
# Copy PDFs to backend folder
cp new-lps.pdf /home/modulos_llm/backend/

# Run ingestion (adjust args for corpus/council)
docker exec modulos-backend python3 ingest.py \
  --pdf new-lps.pdf \
  --corpus lps \
  --council launceston
```

---

## External Services

| Service | URL | Purpose |
|---|---|---|
| Ollama | `http://192.168.8.73:11434` | LLM inference (Windows machine, RTX 4070 Super) |
| Qdrant | `http://qdrant:6333` (internal) | Vector search |
| TPSO Viewer | `https://tpso.planning.tas.gov.au/tpso/external/planning-scheme-viewer/30` | SPP viewer (iframe) |
| LIST GIS | `https://services.thelist.tas.gov.au/arcgis/rest/services` | Property/parcel data |
| Google Maps | `https://maps.googleapis.com` | Address autocomplete in site-report |
| Anthropic API | `https://api.anthropic.com/v1/messages` | BYOK Claude |
| OpenAI API | `https://api.openai.com/v1/chat/completions` | BYOK GPT |
| xAI API | `https://api.x.ai/v1/chat/completions` | BYOK Grok |

---

## Current Model

- **Model:** `llama3.1:8b-instruct-q4_K_M`
- **Hardware:** NVIDIA GeForce RTX 4070 Super (12GB VRAM), Windows host
- **VRAM usage:** ~4.5GB model + ~1GB KV cache at num_ctx=6144
- **Typical latency:** 4-12s depending on answer length
- **Memory bandwidth:** saturated at ~95-98% during inference (bandwidth-bound, not compute-bound)
- **Upgrade path:** RTX 4090 (24GB) would allow llama3.1:70B-Q4 and roughly 2× throughput

---

## Security Notes

- IP hashes in telemetry use HMAC-SHA256 with `TPR_IP_SECRET` — not reversible
- BYOK keys stored in browser `localStorage` only — never sent to our servers
- `gmaps-key.php` checks `HTTP_HOST`/`HTTP_ORIGIN` before serving the Maps key
- `dashboard.php` restricted by `.htaccess` IP allowlist
- `DEMO_TOKEN` in docker-compose is for optional API gating (currently disabled)
- `.env` file contains all secrets — never commit to version control