Seven lessons on data misalignment in education

If you have worked in an education program for more than a year, you have lived this in some form: the data tells one story, the program tells another, and the funder wants a third. Sometimes it is a definitions issue between reporting systems. Sometimes it is a timing mismatch between fiscal years. And sometimes — most damaging of all — it is a gap between the students the program was funded to serve and the students actually in the room. Seven lessons from three decades of working with education data, starting with the one that costs the most.

1. When eligibility data is hard to access, proxies quietly replace it

This is the single most consequential misalignment we see, and it is also the one programs are least aware of. A program is funded to serve a specific population — students who scored below 22 on the math ACT, say, or students reading below grade level — and the roster of students actually served doesn't match that criterion. In programs we have evaluated, we have found that as many as 80% of the students being served did not meet the eligibility data the program was funded on.

It is almost never bad faith. It is friction. The real data lives in a PDF the school received and would need a ruler and a highlighter to extract — the names then retyped into a spreadsheet to form a roster. Or it lives in a state database that nobody has the time or access to pull from before the program starts. So the staff reach for a proxy. Both of the following are patterns we have heard directly from school staff: "students in Advanced Functions instead of Pre-Calculus probably have low ACT scores, so we'll use that course roster"; and, more memorably, "the poorer the family, the less likely the kids can read, and you can usually tell income by the mother's pocketbook."

The harder variant shows up when a program relies on referrals rather than a data-defined roster — "please refer students who are at risk of dropping out," for example. Every referrer is then using their own internal proxy, often one they have never articulated. When we dig into what those proxies actually are, we have heard answers like "they are a minority race, and we know minorities are at risk of dropping out." Referral-driven programs are where we see the most severe misalignment, because there is no single proxy to correct — there are dozens, each sitting unexamined in a different person's head, and many reduce to stereotype rather than evidence.

Proxies are intuitive and sometimes correlate with the real criterion. They also mislead in both directions. Kids who needed the program don't get in because the proxy filters them out. Kids who didn't need it take the seat of one who did. And the reported impact of the program on its actual funded population ends up unknowable, because the population wasn't the funded one. The fix is upstream: remove the friction between the eligibility criterion and the roster, so the real list lands on the site coordinator's desk at the start of the program in a usable format — and when referrals are unavoidable, give referrers an explicit, data-defined checklist rather than a description. This is unglamorous work and it is the single highest-leverage thing an evaluator can do for a program in its first six months.

2. Different measurements were built to answer different questions

District assessments were built to answer "is this student on track?" State assessments were built to answer "are schools doing what the state expects?" Federal measures were built to answer "are the federal dollars producing the intended population-level change?" Each question is legitimate. Each question requires different data. It is not a conspiracy that the measurements don't line up — they were never designed to.

3. Definitions are almost never the same

"Attendance" is a good example. Is a student counted present if they are on campus for part of the day? Part of each period? Enrolled but not physically present? Every system has a different rule. When your 21st CCLC report says 184 students attended at least 30 days and the district's system says 162, the discrepancy is almost never a bug. It is a definition mismatch, and tracking it down is a full afternoon of work the first time and a smaller one thereafter.

4. Time windows don't line up either

The fiscal year isn't the school year isn't the program year isn't the reporting year. When your program runs September through May and the funder reports on a July-to-June cycle, you will always have one month of the new program year already complete before the prior year's report is due. Plan for it.

5. The "single source of truth" rarely exists

Large organizations periodically try to unify their data into a single warehouse. The effort is almost always worthwhile and almost never fully succeeds. The data sources don't agree, the definitions don't reconcile, and new data streams keep arriving. The useful goal is not "one source" but known and documented reconciliations between the sources. That's achievable. Perfection isn't.

6. Disaggregation reveals the real story

A program average that looks fine often hides a cohort that is struggling and a cohort that is thriving. Disaggregation — by site, by grade, by demographic, by dosage — is where the actionable information lives. Funders have increasingly caught on to this; so should program leadership. If your current data system cannot produce disaggregated reports in hours instead of weeks, that is a capacity gap worth closing.

7. Most misalignment gets resolved by a spreadsheet and a patient person

This is not a glamorous lesson but it is the true one. Somebody sits down with the funder's required format, the output of the district SIS, and a blank spreadsheet, and they do the mapping. Column by column. Once. Then they document the mapping so the next person doesn't have to redo the work. This is the single highest-leverage piece of data infrastructure a small program can build, and most don't.

What to do about all this

There is no magic tool. The programs that handle data well do a few disciplined things: they pull the real eligibility data before the program starts instead of relying on proxies; they write down their definitions; they document their reconciliations; they build dashboards that answer the actual questions leadership asks; and they invest in one dedicated person (internal or contracted) who owns the plumbing. It is not exciting work. It is the work that keeps a grant renewable.

If your program is at the point where data pain is a weekly occurrence, the path out is probably not a new platform — it is a short engagement to document what you have, fix the three highest-pain reconciliations, and train the next person. That's work Edstar does often. It is also work you can do yourself if you have the time. Either way, someone has to do it. The data does not align itself.