FloorVote — Sync Architecture

How legislative data flows from LegiScan → central → tenants

Detail:
🏛 LegiScan Pull API ▼
Source of truth for 50 states + Congress. Four key endpoints.
Source of truth for all 50 states + US Congress.
Updates bill data as state legislatures act; reflects changes in change_hash.
  • getMasterList — all bills with title + description (~400 KB)
  • getMasterListRaw — all bills, hash only (~100 KB)
  • getBill — single bill, full JSON (sponsors, history, votes, texts, roll calls…)
  • getBillText — base64-encoded HTML or PDF, fetched from state legislature site
Only central calls LegiScan — tenant workers never call it directly, so all API traffic and quota sit on one account.
every hour
⏰ Central Cron — fires every hour (0 * * * *) ▼
Per-state keyword union → change detection or full sync depending on session age.
Determines tracked states = explicit non-wildcard stateCoverage + any state where an active tenant already has bills.
Builds a per-state keyword union (only keywords from tenants that cover each state — prevents cross-state keyword pollution).
For each tracked state, checks if any session is past its sync frequency:
Raw pass — default ET hours 7, 9, 11, 15, 17, 19, 21 ▼
1
getMasterListRaw per active session
~100 KB • bill_id + change_hash only
2
Compare hashes against central bills table; for new bills, insert with change_hash='' sentinel (not queued). The stored hash is advanced only for matched bills — unmatched (match_type=null) changed bills are left stale so the next full pass still detects them (advancing it here would hide the change from the full pass and swallow it).
3
Queue to ingestor only: bills where match_type ∈ {'keyword','manual'} for at least one covering tenant AND the hash changed (the ingestor re-writes the hash via getBill). No stubOnly notifications — raw pass has no titles, so stubs go quiet until the next full pass picks up the (intentionally still-stale) hash.
Full pass — default ET hours 5, 13, 23 ▼
1
refreshStateSessions → getSessionList (once daily at 5 ET only)
Corrects sine_die, auto-discovers new sessions
2
getMasterList?id=<sessionId> → full list with title + description
Same LegiScan op as getMasterList?state=, called via the getMasterListBySession wrapper. ~400 KB per session.
3
Save to R2 cache at sessions/{id}/masterlist.json
Used by keyword-resync for instant matching without an API call
4
Keyword match title + description against per-tenant union
Matching → bill_tenants.match_type='keyword'; non-matching → match_type=null stub link (still tracked metadata-wise, no AI).
5
Dispatch: writes new change_hash + masterlist fields to central bills table, then:
  • → ingestor queue (will fire getBill): newly-matched bills, OR matched bills with changed hash.
  • → tenant queue directly (no API): stubOnly messages for match_type=null links whose hash changed, so tenant refreshes denormalized status/last_action.
  • Nothing: unchanged bills, or never-matched bills that still don't match.
API gate: the cron is the only thing that controls getBill spend. A bill triggers a getBill call only when it is keyword- or manually-tracked by at least one covering tenant AND its change_hash moved (or it just newly matched). Unmatched changes update central state but never reach the ingestor.
→ ingestor queue (central-ingestor, batches of 5)
⚙️ Central Ingestor Queue — central-ingestor ▼
getBill + persist child tables + store text in R2 → notify tenant queues.
⚠ API quota: each non-skipFetch message triggers one getBill() call, drawn from the deployment's monthly API budget. The cron (Stage 2) is the API gate: it only queues bills whose change_hash moved, so unchanged bills never reach the ingestor. reprocess bypasses the ingestor entirely (zero calls); redownload-texts uses skipFetch: true.
1
skipFetch branch: if message has skipFetch: true (bulk seed, redownload-texts) → download any missing texts from state legislature, skip getBill, jump to step 4.
All other messages fall through to step 2.
2
getBill from LegiScan — always, for non-skipFetch messages.
Full JSON: bill row + sponsors, history, votes, texts, supplements, amendments, calendar, subjects, SASTSs…
3
Unified persist to D1 + R2: snapshot existing child rows → run detectChanges → write bill_change_log rows for each diff → upsert bills row → delete+reinsert bill_history / bill_sponsors / bill_sasts / bill_subjects / bill_calendar / bill_referrals → upsert bill_texts / bill_supplements / bill_amendments / roll_calls → download any new text versions to bills/legiscan-{billId}/texts/{docId}.{ext} → stamp texts_fetched_at → derive text_status.
Single path — no fast/full bifurcation. Heals missing R2 text on every ingest.
4
notifyTenants: read bill_tenants → send {tenantId, billId, text_status, match_type, changes} to each tenant queue.
Marks notifiedAt.
Note: per-legislator vote rows (roll_call_votes) are NOT written by this path — getBill returns summaries only. Bulk-seeded bills have them; live-ingested bills do not.
→ per-tenant queue (floorvote-{id}-queue)
🏢 Tenant Queue + Worker — one per org ▼
Fetch bill + text → AI (if keyword/manual match + text available) → upsert to tenant D1.
1
Fetch bill from central
GET /bills/{id}
Metadata + fullJson + match_type + text_status
2
Fetch text from central
GET /bills/{id}/text
HTML or PDF from R2 — only if text_status is 'available' or 'in_r2'
3
AI processing
One-pass: summary + tags + relevance
Only when match_type IN ('keyword', 'manual') and text confirmed available. Bills with text_status='no_texts' wait for LegiScan to publish text.
Result upserted into tenant D1 → visible in UI. Three independent concepts: match_type (keyword / manual / null), text_status (not_checked / available / no_texts / in_r2), ai_processed_at. Feed events emitted for status/text/action changes. change_hash dedup prevents redundant AI re-runs.
🖱 Manual Triggers
Add Bill (by number) ▼
  1. Tenant → POST central/admin/ingest-bill
  2. Central: getBill + store text in R2
  3. Tenant creates bill_tenants entry (match_type='manual')
  4. Tenant queue runs AI
Manual bills are never demoted or removed by keyword-resync.
Sync Bill Tracker(auto on keyword save) ▼
Cleanup pass (remove old)
  1. Find keyword bills that no longer match any keyword
  2. If engaged (votes/comments) → promote to match_type='manual'
  3. If unengaged → demote to match_type=null (lightweight stub)
Forward pass (add new)
  1. POST central/admin/keyword-resync-tenant
  2. Central scans bills table + R2 masterlist cache
  3. New matches → bill_tenants + ingestor queue → AI
R2 cache catches bills never previously fetched. Warm after first full sync (~24h of tracking a state).
Admin: Reprocess / Redownload ▼
Reprocess: re-queues every linked bill directly to the tenant queue. Zero LegiScan API calls — re-runs AI from existing data.

Redownload texts: sends linked bills to ingestor with skipFetch: true — downloads text from the legislature site, no getBill() (zero API quota). Use to heal missing bill text.
🗄 Storage ▼
Central R2 + D1 (central-bills-ls) and one D1 per tenant.
Central R2 (central-bill-texts)
 • bills/{id}/{docId}.html|pdf — bill text
 • sessions/{id}/masterlist.json — masterlist cache

Central D1 (central-bills-ls, LegiScan env)
 Core: sessions, bills, bill_tenants, tenants, keyword_registry
 Rich: people, committees, roll_calls, bill_sponsors, bill_history, bill_texts, bill_amendments, bill_change_log, session_sync_log, api_call_log

Tenant D1 (floorvote-{id}, one per org)
 • bills: match_type · text_status · ai_processed_at
 • AI results, engagement (votes, comments, notes, positions)

End-to-End Latency

▼
Event Mechanism Latency
New bill introduced in LegiScan Next full sync discovers it via getMasterList + keyword match up to 24h
Tracked bill gets status/text update Hourly change check (getMasterListRaw) detects changed hash → ingestor ≤ 1h
Bill with no text → text published by LegiScan Hash changes → ingestor fetches text → AI runs automatically ≤ 1h
Keywords updated → remove old bills keyword-resync cleanup pass runs synchronously on save immediate
Keywords updated → add bills (in central DB) keyword-resync scans central bills table immediate
Keywords updated → add bills (never fetched) keyword-resync scans R2 masterlist cache (populated by first full sync) immediate if cache warm, else up to 24h
Manual "Add bill" by number ingest-bill → ingestor → tenant queue → AI 1–3 min