Measure accepted AI work and the effort behind it

Measure an AI workflow through accepted deliverables and the total effort required to produce them, with quality held to a consistent standard and waiting recorded separately.

A faster first draft is an interesting observation. Whether it helps the team depends on the checking, correction and coordination still required before anyone can use it. Establishing a small measurement contract before a pilot makes that difference visible and gives the decision owner a credible basis for continuing, revising or stopping.

Define accepted work before the pilot

Choose a recurring deliverable with a named reviewer: for example, an operating review brief. Describe what the reviewer must be able to do with it. They may need to trace material claims, compare options and approve a bounded next step.

Translate that purpose into acceptance criteria. The sources must support consequential statements, calculations must be correct, uncertainty must be visible and the recommendation must stay within the decision’s authority. Adapt the criteria to the work; do not assume a polished document meets them.

Google’s HEART paper from CHI 2010 offers a process for connecting goals to metrics. Here, the editorial application is to define useful work first and choose measures that can reveal whether the pilot delivers it.

Record the acceptance result for every attempted deliverable, including abandoned and rejected drafts. Excluding difficult cases after they fail can make the measured workflow look better while concealing the cost of using it.

Establish a comparable baseline

Collect recent examples produced under the usual process. Record task difficulty, source condition, reviewer and scope. A brief assembled from consistent inputs is not equivalent to one that first requires reconciling conflicting records.

The HBS team’s September 2023 research summary describes AI performance varying across knowledge tasks. A practical implication is to evaluate the selected workflow rather than borrowing an improvement rate from another activity.

Keep baseline and pilot acceptance criteria aligned. If the pilot receives simpler assignments or a lighter review, label that difference. When equivalent comparisons are unavailable, report the result as exploratory and identify what prevents a stronger conclusion.

Use a sample that the team can inspect honestly. A small pilot may reveal a broken handoff or a costly review burden. It cannot establish that the same result will hold across every function, reviewer or future model.

Count all active effort

For each attempt, capture preparation, review and rework as separate components. Preparation includes arranging authorized inputs and producing the draft. Review includes checking evidence and exercising judgment. Rework includes corrections and the further checking they trigger.

Avoid counting the same minutes twice. If one reviewer logs review with corrections included, use that combined figure consistently or separate it before comparison. When two people work simultaneously, record both people’s active effort; calendar duration alone does not capture the team’s labor.

Record waiting in a separate field. A deliverable can require less active effort yet arrive later because it waits for a source or approval. Conversely, it can reach the reviewer sooner without reducing anyone’s workload. Both observations may matter, but they answer different questions.

Measurement field	Record for baseline and pilot
Attempt and scope	Deliverable, difficulty, inputs and reviewer
Acceptance	Accepted, rejected or abandoned, with reason
Preparation	Active minutes gathering, organizing and drafting
Review	Active minutes checking and applying judgment
Rework	Active minutes correcting and checking again
Waiting	Elapsed delay and its cause, recorded separately
Quality	Material errors, omissions and required corrections
Use	Whether the accepted deliverable informed an approved decision

Interpret the result with a quality guardrail

An illustrative calculation shows why the full effort matters. Suppose a hypothetical baseline requires 90 minutes of preparation and 40 minutes of review and rework, totaling 130 minutes. A hypothetical AI attempt takes 45 minutes to prepare and 95 minutes to review and correct, totaling 140 minutes.

Preparation is faster in this illustration, while total active effort increases by 10 minutes. These sample figures show why the full workflow belongs in the scorecard.

Review quality alongside effort. A mean improvement does not excuse a consequential unsupported recommendation. Report the number of accepted attempts and material failures, and inspect why failures occurred before expanding the workflow.

Separate released capacity from cash savings

If comparable accepted work takes less total effort, describe the measured time difference with its scope and sample size. Identify whether the released time was actually available for another useful activity. Fragmented minutes may not translate into a new deliverable or a reduced staffing requirement.

A financial claim needs additional evidence about spending, additional costs and the mechanism that changes the budget or output. Multiplying minutes by a salary rate can describe an allocation assumption; it does not prove cash was saved.

At the review, decide whether the evidence supports expansion, revision or stopping. The AI transformation solution provides a starting point for framing that bounded pilot. Bring a comparable baseline, an acceptance standard and a reviewer who can explain why the work is useful.