Data Validation
In production
BenefitWatch
7-BATCH ATHENA VALIDATION ENGINE
Run batches of Athena SQL checks for a carrier, audience, and plan year.
Vantage's namesake. A ~4,200-line dispatch runs seven batches, 130+ checks: Processing, QA, Rate, Metadata, Rule Checker (market-wide, all-issuer), URR, and Plan Mapping (cross-carrier). Results are per-check flag tables you acknowledge, dismiss, and re-run per row; a single year-scoping call shadows the table names, so any look-back year runs with no per-query edits.
- A race-free cross-process run guard (diskcache test-and-set with PID-liveness reclaim) rejects a second concurrent run, so two heavy jobs can't OOM the 2 GB box while every other tool runs free.
Scope 600+ carriers Checks 130+ across 7 batches Engine AWS Athena Lives in Vantage
AthenaTrino SQLdiskcacheS3Plotly DashData-Driven Testing
Data Validation
In production
QHP Validation
OFFLINE CMS QHP-TEMPLATE VALIDATOR
Drag-drop CMS QHP templates, get a color-coded Excel report, ~90 checks, no Athena.
Auto-detects each file's template family (Plans & Benefits, Rates Table, Service Areas, or a non-standard carrier rate file) and runs the matching checks from a ~5,000-line openpyxl engine, enforcing ACA/IRS/CMS rules into a multi-sheet workbook.
- Benefits: metal-level vs name, deductible/MOOP parity, coinsurance over 100%, CSR actuarial-value bands, HSA IRS limits (incl. the PY2027 Bronze/Catastrophic exemption).
- Rates: 14-char HIOS, age-curve monotonicity, tobacco not below non-tobacco, and implausible-value floors and ceilings.
- A CMS county cross-check reads the workbook's own hidden reference sheet, resilient to column reordering, to catch counties that would fail SERFF/HIOS resolution.
Input Up to 20 .xlsx Checks ~90 structural + rule Engine openpyxl (offline) Output Multi-sheet Excel
openpyxlCMS QHP TemplatesACA / IRS RulesExcel Reporting
Data Validation
In production
UHC Validation & Oxford
LIVE CARRIER-API RECONCILIATION
Calls live UnitedHealthcare / Optum / Oxford APIs and reconciles their rates and benefits against our warehouse.
A ~3,200-line Dash-free service (so it runs in a background worker) answers one question: which plans are rate-quotable by the carrier but missing from our benefits data? An async rate-quote flow (submit, then poll, one quote per rating area) and a sync per-state benefits fetch both diff against warehouse data.
- Age-banded rate-vector diffs keyed by (rating area, contract code) with per-age tolerance, cross-referenced against benefits-data gaps.
- SAM parity: a ~144 MB SAM zip streams to a Flask route and a background daemon compares it against the B2B API.
- Validates past quarters by posting a non-past in-quarter effective date (the API rejects past dates) while keeping the true quarter start for scoping.
Sources UHC / Optum / Oxford APIs Compares Rates + benefits Extra SAM parity Runs Background worker
UHC / Optum / Oxford APIsboto3Background WorkersRate Reconciliation
Data Validation
In production
Rate Validation
MARKET-WIDE RATE CHECKS
Market-wide, all-issuer rate checks: rating-area rollover, tier violations, invalid age curves, and suspicious values.
BenefitWatch's Rate tab promoted to its own page with zero duplication: a thin layout mounts the real Batch-3 trigger and renders hidden stubs for every other component id the shared BenefitWatch callbacks bind to, so the callback graph resolves while runs stay market-wide.
- Reuses BenefitWatch's entire engine through stub components, with no forked logic.
Scope All issuers Source BenefitWatch Batch 3 Pattern Zero duplication Lives in Vantage
AthenaReuse over DuplicationRate Validation
Data Validation
In production
Service Area Validation
YEAR-OVER-YEAR SERVICE-AREA DIFF
Year-over-year diff of each carrier's county/ZIP service-area composition, flagging shrinkage.
The raw plan-by-area-by-ZIP join is ~37M rows nationally, so the query collapses geography with server-side array_agg (packing fips and county name together) and transfers only a few thousand rows, surviving the 2 GB box. Areas are matched across years by greedy best-first ranking on Jaccard similarity, since ids churn at renewal.
- RED / AMBER / INFO severity with config-tunable thresholds; drills into per-plan detail only when a single issuer is supplied.
Compares County / ZIP composition Flags Significant shrinkage Match Jaccard similarity Scale 37M-row join
Athenaarray_aggJaccard SimilarityYoY Diff
Data Validation
In production
ARIA
SBC VALIDATION ENGINE (NO LLM)
SBC validation, no LLM: pdfplumber plus regex end-to-end.
Discovers SBC PDFs in S3, extracts fields, caches parsed JSON, and validates against the warehouse via Athena in background callbacks with live progress. Runs the whole bucket or one carrier and audience behind a 200-plan gate, checking coverage period, SBC name vs warehouse metal/type/deductible, and per-benefit cost-share.
- Tracks each PDF's LastModified to re-parse only when a document actually changed.
- Deterministic by design: regex on parsed text is faster, cheaper, and fully auditable, with a persisted dismissed-flags set and CSV export.
Engine pdfplumber + regex LLM? No, by design Checks 3 per plan Runs Background w/ progress
pdfplumberRegexAthenaS3Background Callbacks
Data Validation
In production
WREN
PLAN-BENEFIT CHANGE DETECTION & TRIAGE
Detects and categorizes every plan-benefit change so reviewers can filter the flood down to what actually matters.
JOINs the live plan table against an admin-managed baseline to surface every changed plan (all carriers or one), scoped by a plan-year picker, reading a CSV at the exact S3 path that backs an Athena external table. With 600+ carriers moving thousands of fields a season, the hard part isn't listing the diffs — it's making them triageable.
- Anomaly detection and flag categorization score and group each change, so reviewers filter by type, severity, or carrier instead of scrolling an undifferentiated wall of diffs.
- Run-status tracking (new / persisting / cleared), per-field human-readable change descriptions, and activity-feed attribution, with access enforcement down to CSS-hiding the review controls for view-only users.
Engine Athena diff Triage Anomaly detection + flags Tracking New / persisting / cleared LLM? No
AthenaS3 External TablesAnomaly DetectionFlag CategorizationAccess Gating
Analytics
In production
Publishing Dashboard
QUARTERLY PUBLISHING COMPLETION
Quarter-tabbed live tracking of carrier publishing completion, with a 15-query Open-Enrollment view.
Only the active quarter hits Athena; the others render from per-quarter S3 snapshots with cache-age badges. The Q1 view fans 15 queries across a thread pool (audience splits, SBC attachment rate, unknowns-per-plan, backlog, embargoed counts). It fails closed by design, so missing inputs can never inflate the reported completion rate.
- A Manual Metric Override panel (S3-persisted) corrects completion for edge cases, including in-SQL market-exit overrides that drop a departing carrier from both the denominator and the weighted rate.
View Q1–Q4 tabs Engine Dash background callback Panel Manual Metric Override Source Athena + Jira
Plotly DashThreadPoolExecutorAthenaJiraS3 Snapshots
Analytics
In production
Build Season Report
JIRA-DRIVEN TEAM QUALITY SCORE
Reconstructs each ticket's lifecycle from its Jira changelog and grades the whole team.
Computes throughput, correction rate (benchmarked per ticket type so higher-effort work isn't unfairly penalized), distinct HIOS touched, and processing-time histograms. The headline is a team credit score: every published ticket scored into one volume-weighted 0–100 grade, with a per-processor Report Card (Accuracy, Rework, Speed, Precision) on a Plotly radar.
- A shared, I/O-free scoring engine normalizes each component to 0–100 via piecewise-linear anchor maps and combines processors by ticket-count-weighted average, like a credit-hour-weighted GPA.
Source Jira (JQL) Output Team credit score Grades A–F Cadence Weekly
Jira RESTJQLPlotlyScatterpolar RadarPandas
Analytics
In production
Season Benchmarks
TWO-SEASON PROCESSOR COMPARISON
Compares processor metrics across two build seasons.
The year pair auto-advances from config. Deltas render as a grouped two-season Plotly bar chart, reusing the Report Card's analysis functions and S3 data with no forked logic.
- Direction-aware coloring respects per-metric good-if-lower semantics, so a drop isn't always green.
Compares Two build seasons Coloring Direction-aware Reuses Report Card logic Source Jira
JiraYoY BenchmarkingPlotly
Analytics
In production
Metrics & Data Ops Quality
DATA-QUALITY METRICS FROM BENEFITWATCH FLAGS
The team's data-quality scoreboard: how the flags BenefitWatch raises get validated, corrected, and cleared over time.
Reads the flags that BenefitWatch's batch queries raise and tracks each one to resolution — active, validated, corrected, or dismissed — with month-over-month deltas, so the team can see whether data quality is actually trending up rather than just churning. Exact dismissal accounting keeps those deltas honest across runs.
- Turns per-run flag output into a resolution trend, so recurring data-quality problems surface instead of getting lost in the noise of any single batch.
Source BenefitWatch flags Tracks Validated / corrected / dismissed Deltas Month-over-month Lives in Vantage
Plotly DashAthenaFlag ResolutionTrend Reporting
Analytics
In production
Flag Analytics
RULE-CHECKER FLAG HISTORY
Persistent six-chart history of Rule Checker flags.
Volume over time, check leaderboard, new-vs-persistent, carrier concentration, dismissal distribution, and a batch-detail table, all from one S3 artifact, as a standalone page and an on-demand Individual + Small Group summary on the Metrics page.
- Turns raw flag runs into trend signal, so recurring defect categories stay visible all season.
Source rule_checker_metrics (S3) Charts 6 Views History + summary Lives in Vantage
S3PlotlyFlag Analytics
Analytics
In production
Acquisition Sheet
COLLABORATIVE CARRIER-ACQUISITION TRACKER
Collaborative, spreadsheet-style tracker of the carrier acquisition universe, on Dash AG Grid.
A three-layer state model: an immutable Athena universe cached per quarter to S3, a shared editable S3 overlay keyed by a stable row key, and a session-scoped pending-edit store with an explicit Save bar. Each state deep-links to its Data Ops Confluence guide by numeric page id, so renames never break links.
- Replaces a brittle shared Google Sheet with a governed, editable grid.
Grid Dash AG Grid State 3-layer model Scope Plan year / quarter Links Confluence by page id
Dash AG GridS3 OverlayAthenaConfluence
Analytics
In production
Documents
QUICK-ACCESS HUB FOR OPS REFERENCES
Quick-access hub for the references Ops reaches for every day.
A curated set of links to the runbooks and resources the operations team uses most, in one predictable place inside Vantage instead of scattered across drives.
- One home for the docs Ops actually uses.
Type Quick-access links Audience Operations team Content Reference docs Lives in Vantage
OperationsQuick LinksReference Hub