--- title: "Measure before you optimise — Erkan Malcok" description: "Dashboards and metrics are easy to collect and hard to interpret. This piece argues for measuring the right thing before improving what is merely visible." date: "2026-09-01" updated: "2026-09-02" canonical: "https://erkanmalcok.com/articles/measure-before-you-optimise/" kind: "Opinion" series: "education-and-ai" tags: - "Educational analytics" - "Data science" --- # Measure before you optimise — Erkan Malcok Why the first question in analytics is what counts as evidence. An organisation launches a new dashboard. Engagement rises. Leaders feel progress. Teams adjust their work to improve the numbers. Months later, someone asks a simpler question: *Are students actually learning more?* Nobody is sure. The dashboard was never wrong in a technical sense. It reported what it was built to count. The failure was upstream: the team optimised a **visible proxy** before agreeing what success should mean. In education, analytics, and data-led improvement, the same pattern appears often. We measure what is easy, then treat the measurement as the goal. ## What you will take away By the end of this article, you should be able to: - distinguish an outcome from a convenient proxy; - anticipate how targets change behaviour; and - design a small set of measures around a decision rather than a dashboard. ## The temptation Data collection has never been cheaper. Learning platforms, attendance systems, formative quizzes, and survey tools can produce continuous streams of numbers. That abundance creates a false sense of clarity. It is tempting to believe that: - more data automatically means better decisions; - a rising metric means improvement; - if something cannot be measured easily, it must be less important. Each belief is sometimes true. None is reliably true without context. A metric is a compression of reality. Compression is useful when the map matches the territory. When it does not, optimisation becomes sophisticated motion without direction. This is not an argument against measurement. It is an argument for **sequencing**: decide what matters, define evidence carefully, then improve — not the reverse. ## Measure and metric A **measure** is an attempt to represent a property of interest: attendance, accuracy on a quiz, time on task, teacher feedback quality, confidence after a lesson. A **metric** is a measure elevated into routine comparison: a KPI, a league table column, a traffic-light threshold on a dashboard. The gap between them is where misuse begins. Measures are observations. Metrics are decisions about what will be watched, rewarded, or punished. Useful questions before any metric goes live: 1. **Construct.** What human capability or outcome is this supposed to reflect? 2. **Validity.** Under what conditions does movement in this number actually indicate movement in that outcome? 3. **Cost.** What behaviour might this metric accidentally incentivise? 4. **Burden.** What must teachers, students, or analysts do to produce it — and does that burden distort the activity being measured? If those questions have no satisfactory answers, the metric is not ready to drive action. It may still be exploratory. It should not yet be authoritative. ## Goodhart and education Goodhart’s argument originated in monetary-policy analysis ([Goodhart](https://link.springer.com/chapter/10.1007/978-1-349-17295-5_4)) and is often summarised as: when a measure becomes a target, it ceases to be a good measure. Education offers clear examples: - **Time on platform** rises while thinking does not. - **Quiz scores** improve through retakes without durable recall. - **Submission rates** hit one hundred per cent when the task becomes copyable [fluent work](/articles/fluent-answers-are-not-understanding/). - **Lesson completion** increases when content is simplified beyond the instructional objective. In each case, the metric can improve while the underlying aim stalls or regresses. The system learns to perform for the instrument. There is a related risk in removing [productive struggle](/articles/productive-struggle-in-learning/) in the name of efficiency. If the measured outcome is speed or completion, interventions that shorten difficulty may look successful while weakening learning. None of this requires bad faith. Most metric failures are structural: the number was available before the theory was clear. ## A small example Suppose a school tracks “weekly maths quiz average” as a headline indicator. The number is easy to compute: ```python def weekly_class_average(scores_by_student: dict[str, list[float]]) -> float: all_scores = [score for scores in scores_by_student.values() for score in scores] if not all_scores: raise ValueError("no scores to average") return sum(all_scores) / len(all_scores) ``` A rising average might mean: - teaching improved; - the quiz became easier; - weaker students stopped attempting it; - more pupils used outside help; - the cohort changed. The function is correct. The interpretation is not determined by the function. A better sequence might be: 1. State the outcome: for example, “pupils can solve linear equations under mild variation without prompts.” 2. Choose evidence: pre/post tasks, error analysis, delayed recall, teacher judgement against a rubric. 3. Use the weekly average as one **diagnostic** signal among others — not as the definition of success. Analytics should narrow uncertainty, not replace judgement with a single line on a chart. ## What to establish first Before optimising, clarify the decision the data is meant to support. **Name the decision.** “Improve maths outcomes” is not a decision. “Change how we allocate intervention time for Year 9 algebra” is closer. **Separate leading and lagging evidence.** Some measures hint early; others confirm later. Treat them differently. **Prefer triangulation.** One metric is a clue. Converging evidence from different methods is stronger. **Document what would falsify your belief.** If scores rise but pupils cannot explain their method, your improvement hypothesis is weak. **Audit incentives.** Ask what a busy teacher or student would reasonably do to make the number look better with the least genuine learning. **Start small and reversible.** Pilot changes before embedding them in policy, performance review, or public reporting. These steps slow the rush to optimise. That delay is often where quality enters. ## Implications for practice For educators and analysts, the practical posture is sceptical but constructive: - Use data to **question** assumptions, not only to confirm them. - Treat dashboards as conversation starters, not verdicts. - Invest in measurement design with the same seriousness as intervention design. - Report limitations alongside findings — especially when decisions affect pupils. For technical readers, the same principle applies outside classrooms. Feature usage, model accuracy, and conversion rates are metrics. They are not substitutes for asking whether the product does what it should for the people using it. Measure before you optimise does not mean measure everything. It means measuring **the right things**, in the right order, with enough humility to change course when the map stops matching the territory. ## What to reconsider If you work with educational data or improvement projects, it is worth asking: - Which metrics in your current reports are targets in practice, not just observations? - What outcome would make you abandon a rising number as misleading? - What evidence would you trust if the dashboard were unavailable? Optimisation is not the first move. Clarity about what counts as better is. ## Try it yourself Choose one metric your team or classroom watches. Write down the outcome it is supposed to represent, one way a reasonable person could improve the metric without improving the outcome, and one independent check that would expose the gap. ### Answer A useful answer separates three things: 1. **Outcome:** the change that matters to people. 2. **Proxy:** the observable number used to estimate that change. 3. **Counter-check:** evidence gathered differently enough to reveal gaming or measurement error. For example, quiz completion may proxy practice, but clicking quickly can raise completion without learning. A delayed explanation or unfamiliar transfer question provides a stronger counter-check than another completion count. ## Related reading - [Fluent answers are not understanding](/articles/fluent-answers-are-not-understanding/) examines another seductive proxy: polished output. - [Teaching computing beyond syntax](/articles/teaching-computing-beyond-syntax/) shows how assessment can make reasoning visible. ## References - [Charles Goodhart, "Problems of Monetary Management" in Monetary Theory and Practice](https://link.springer.com/chapter/10.1007/978-1-349-17295-5_4)