Etusivu Tutki Apua
Rekisteröidy Kirjaudu sisään
benjamin.harris
/
tas_councils
1
0
Haarauta 0
Tiedostot Esitykset 0 Vetopyynnöt 0 Wiki
Puu: 64b09021af
Haarat Tunnisteet
tas_councils
/
CLAUDE.md

CLAUDE.md 8.2 KB
Historia Raaka

CLAUDE.md — Project Guide for Claude Code

What This Project Does

This is a scraping pipeline that collects Tasmanian planning development applications (DAs) from 29 council websites, geocodes them, and serves them via a PHP search portal. The code is a mix of Ruby (scrapers, enrichment), PHP (web portal), and shell (orchestration), all wired together with Docker Compose.

Key Files

File Role
lib/db.rb DB client, ensure_table!, upsert (dynamic columns, write-once semantics)
lib/http.rb HTTP client — retries, cookie jar, browser-fingerprint headers, 403/406 warmup, curl fallback
lib/geocode.rb Google Maps geocoding with SHA1 cache in geo_cache table
lib/enrich.rb enrich_after_upsert! — geocoding + property lookup after each DB write
lib/util.rb parse_aus_date, council-name/table-name mappings
lib/scraper_helpers.rb Shared helpers: abs_url, text_or, upsert_and_enrich!
run_all.sh Discovers scrapers/*.rb, filters by ONLY/SKIP, runs each with TABLE_NAME set
entrypoint.sh Docker entry; waits for DB then runs run_all.sh (looping if SCRAPE_EVERY_MINUTES is set)
scrapers/*.rb One scraper per council — parses HTML, upserts rows, calls enrich_after_upsert!
web/index.php Search portal — dynamic UNION across all da_* tables

Running Things Locally

# Full stack
docker compose up -d

# Run all scrapers once
docker compose run --rm scraper /app/run_all.sh

# Run a single scraper
TABLE_NAME=da_brighton DEBUG=1 ruby scrapers/brighton.rb

# Run a subset
ONLY=meandervalley,kent docker compose run --rm scraper /app/run_all.sh

# Geocode backfill (batch, all tables)
docker compose run --rm \
  -e GOOGLE_MAPS_API_KEY="..." \
  scraper ruby /app/tools/backfill_geocode.rb

# Geocode backfill (single table)
docker compose run --rm \
  -e GOOGLE_MAPS_API_KEY="..." \
  -e ONLY_TABLE=da_brighton \
  scraper ruby /app/tools/backfill_geocode.rb

Architecture Conventions

Each scraper follows this pattern:

  1. TABLE = ENV.fetch("TABLE_NAME") — set by run_all.sh from the filename
  2. DB.ensure_table!(TABLE) — idempotent schema setup (all columns already included)
  3. Fetch HTML via Http.get(url) (handles retries, cookies, WAF warmup, browser-fingerprint headers)
  4. Parse with Nokogiri
  5. DB.upsert(TABLE, row) — upserts on (council_reference, address), write-once for date_received
  6. enrich_after_upsert!(table:, council_reference:, address:) — geocodes and enriches

WAF / Cloudflare handling:

  • lib/http.rb sends a full browser fingerprint on every request: User-Agent, sec-ch-ua*, Sec-Fetch-*, Upgrade-Insecure-Requests. This satisfies most WAF header checks automatically.
  • For sites that also need a warm cookie state (e.g. Burnie, King Island, Latrobe, Derwent Valley), the scraper implements a proactive homepage warmup before fetching the target page — see burnie.rb as the reference implementation.
  • Some councils (Kentish, Derwent Valley via direct site) use Cloudflare JS challenge which cannot be solved without a real browser. These exit cleanly with a warning. Where a PlanBuild equivalent exists (council code in COUNCIL_MAP), data is still collected via planbuild.rb.
  • The warmup pattern (custom CookieJar + http_get with redirect handling) is self-contained in scrapers that need it and does not depend on lib/http.rb.

Write-once fields (in DB.upsert):

  • date_received — never overwritten once set
  • date_received_raw — never overwritten once non-blank
  • document_url / local_document_url — new value only replaces if existing is NULL

Table names:

  • Always derived from the scraper filename: scrapers/foo.rb → da_foo
  • run_all.sh sets TABLE_NAME=da_<basename> before invoking each scraper
  • The COUNCIL_MAP in lib/util.rb maps internal council keys to table names (used by PlanBuild integration)

Error Handling Conventions

After a refactor, the project follows these rules:

  • URI building (URI.join, URI.parse) → rescue URI::InvalidURIError
  • DB operations (prepare/execute) → rescue Mysql2::Error => e; warn "[scraper] ..."
  • Zlib decompression → rescue Zlib::Error
  • Date parsing (Date.strptime, Date.parse) → rescue ArgumentError, Date::Error
  • JSON parsing → rescue JSON::ParserError
  • Network/HTTP → rescue Net::HTTPError, Net::ReadTimeout, Net::OpenTimeout, OpenSSL::SSL::SSLError, Errno::ECONNRESET, EOFError
  • Enrichment failures always warn to stderr — do not gate them behind ENRICH_DEBUG
  • No bare rescue — always specify the exception class(es)

Adding or Modifying a Scraper

When a council changes its website markup, only that scraper needs updating. The typical failure mode is:

  • Found 0 rows — CSS selector no longer matches; inspect the live page and update the selector
  • HTTP 403/406 — Council site added WAF; check Http.get options or add a proactive warmup step (see burnie.rb)
  • Cloudflare JS challenge ("Just a moment" in body) — cannot be solved in Ruby; exit cleanly with a warning
  • date_received all nil — Date format changed; update the format string passed to Util.parse_aus_date or Date.strptime

Template choice:

  • Simple HTML list/table → copy glamorgan.rb
  • Link/PDF listing → copy centralhighlands.rb
  • WAF-protected site needing homepage warmup → copy kingisland.rb (minimal) or burnie.rb (full-featured with PDF download)
  • Multi-hop redirect to detail pages → copy derwentvalley.rb

The shared infrastructure (Http, DB, enrich_after_upsert!) handles everything else.

Database Notes

  • MariaDB 10.11, utf8mb4 encoding throughout
  • Schema is created on-the-fly — CREATE TABLE IF NOT EXISTS + ALTER TABLE ... ADD COLUMN IF NOT EXISTS
  • Schema changes go in lib/migrate.rb (new migration at end of MIGRATIONS array) or lib/db.rb (ensure_table!) for columns every new table gets
  • The geo_cache table stores geocoding results keyed by SHA1 of the normalised query string — avoids redundant Google API calls
  • The UNIQUE KEY uniq_ref_addr (council_reference, address) constraint drives the upsert behaviour

Web Portal Notes

  • web/index.php dynamically discovers all da_* tables and builds a UNION query
  • It handles missing columns gracefully (not all tables have every column)
  • web/backfill_pid_title.php is a legacy admin tool — it should not be publicly accessible; consider moving it out of the web root or placing it behind authentication

Common Gotchas

  • TABLE constant conflicts: Each scraper defines TABLE = ENV.fetch("TABLE_NAME") at the top level. If you require two scrapers in the same Ruby process you'll get a constant redefinition warning. Each scraper is designed to be run as a standalone script.
  • COUNCIL_FILTER / COUNCIL_WHITELIST: The docker-compose.yml has a COUNCIL_WHITELIST env var that is passed to the scraper container but is not wired into run_all.sh. Use ONLY / SKIP in run_all.sh instead.
  • PlanBuild scrapers: planbuild.rb handles councils on the state-run PlanBuild portal. It writes to per-council tables using Util.ref_to_table. These run alongside the council-specific scrapers.
  • PDF downloads: Only happen when DOWNLOAD_ATTACHMENTS=1. Files land in DOWNLOAD_DIR/<councilname>/. The web portal serves them from /downloads/ via an Apache alias.
  • Non-ASCII in PDF URLs: Some council sites embed Unicode characters (e.g. en-dash –) directly in PDF filenames. Always percent-encode hrefs before passing to URI.join — see burnie.rb first_pdf_on_detail for the pattern.
  • Redirect loops in Net::HTTP.start blocks: next inside a Net::HTTP.start block exits the block, not the enclosing while loop. Use a redirect_to variable set inside the block and call next on the while loop after the block returns — see burnie.rb http_get_with_cookies.
  • Cloudflare JS challenge vs IP block: A JS challenge ("Just a moment") may work from a residential IP but always block from a datacenter/Docker IP. Detect it and exit cleanly. Sites confirmed blocked in Docker: derwentvalley.tas.gov.au, latrobe.tas.gov.au.