Every agency already has the data it needs to run better. It is in the case system, the permitting system, the work-order system, the call logs, the spreadsheets someone keeps because the systems do not talk. What it lacks is a place where those records line up, and an agreed meaning for the numbers that come out.
The common answer is a data platform program: a lake, a warehouse, a catalogue, a governance board, and eighteen months. The programs that succeed are usually the ones that started with one question and grew. This note describes how to start that way.
Start from a decision, not a dataset
Pick a decision someone makes every week with incomplete information. How many inspections are overdue and where. Which call types are growing. Whether the new intake form changed processing time. The first pipeline exists to answer that question, and only that question, reliably, every morning.
The three parts that matter
- Extraction that does not disturb the source. Records are copied from each system on a schedule, through its own export, database replica, or API, never by adding load to a production system during business hours. Every copy is dated so a number can be traced to the records it came from.
- One place where the numbers agree. A warehouse (a managed database is enough to start) where the copied records are cleaned, joined, and shaped into the tables the question needs. This is where "a case" means the same thing whichever system it came from.
- A definition for every number. "Overdue" has a definition: what date, which status, which exclusions. It is written down, next to the query that computes it, and changed only in writing. This is the part most projects skip and the part that makes the dashboard trusted.
Then the dashboard, last
Dashboards are the visible end of the work and the least of it. Once the numbers are right and defined, any tool can display them. The rule is that the tool shows the warehouse's numbers and never computes its own, so there is one source and no arguments about which chart is right.
Records and privacy
Operational records contain personal information. The pipeline's design says which fields are copied, which are masked, who can query the warehouse at which level, and how long derived data is kept. Access is logged. For the questions that only need counts and trends, the warehouse can hold no personal fields at all, which is the simplest answer to most privacy reviews.
Growing it
The second question reuses most of the first pipeline's work: the extraction, the warehouse, the definitions. By the fifth, the agency has a data platform, built from questions it actually asked, with a catalogue of definitions its own staff wrote. That is the program, arrived at without starting one.
What to ask for
- The first question in writing, with the decision it supports and who makes it.
- The source systems and how each will be read, with the schedule.
- The definitions document as a deliverable, owned by the program, not the contractor.
- A pipeline that runs unattended and tells someone when it fails.
- The dashboard, built on the warehouse's numbers only.
Working with us
We build pipelines, warehouses, and dashboards that turn operational records into decision-ready information, inside the agency's boundary or its government cloud region. If there is a question your data should already answer, tell us what you're working on.