Did the new instructions make the difference?

A training coordinator sees fewer incomplete practice packets after revising an instruction sheet. The next week also has easier orders, fewer interruptions and an extra coach. The improvement is worth noticing. The record does not isolate which change produced it.

The coordinator asks a narrower question: does the revised sheet change the time needed to complete the same paper task correctly? The case, participants and results below are invented. No customer work or production system is involved.

Compare both versions in one session

Eligibility is set before recruitment: trainees must have received the same introductory briefing and not have seen this task or either instruction sheet. Twelve volunteer and all meet those conditions. All twelve take part.

Each receives the same fictional order, three item cards and a blank dispatch note. They must enter the destination, item references and quantities, and the agreed pickup window. Both instruction versions contain the same facts, required steps and checks. The revision changes how those instructions are ordered and grouped, not the required result.

Before anyone sees either sheet, a fair random draw selects six of the twelve participant codes, without replacement. Those six receive the old sheet; the remaining six receive the revision. The assignments are then fixed. Nobody moves a quick participant to the revised group after seeing a result.

The NIST explanation of completely randomized designs describes assigning the conditions being compared randomly to experimental units. Here, the unit is a trainee, and the assigned condition is the instruction version.

Everyone works individually in the same session, with an identical set of materials ready. There is no coaching, sharing, interruption or wait for a shared resource. The old-sheet group supplies a comparison under those common conditions, rather than a record from an easier or harder week.

The coordinator does not have everyone use the old sheet and then repeat the task with the revision. That would compare first and second attempts as well as versions. Each trainee supplies one first attempt; random allocation decides which condition that attempt uses.

Define a finished packet before timing

The timer starts when each trainee opens the task envelope, including the assigned instructions. It stops at the first submission. Reading, writing and the trainee's own checking are included. The fixed accuracy key requires every destination, reference, quantity and pickup detail to match the supplied order and cards.

Accuracy is checked after submission; that verifier's work is outside the timed outcome. In this supplied record, every trainee passes on the first submission, with no missing result. The times below are exact minutes for the illustration.

All twelve first submissions, checked against the same accuracy key

Scroll across to see every column. Use the arrow keys when the table is focused.

The old group's times total 90 minutes; dividing by six gives 15. The revised group's total is 78, giving 13. Its mean is two minutes lower in this session. These are group averages, not a two-minute improvement demonstrated for every person.

If a submission were wrong or unfinished, it would remain in the record with that status. Dropping it and averaging only the convenient successful cases would change the comparison. Faster first submissions would not, by themselves, establish faster correct work.

Separate recruitment from assignment

These twelve people volunteered. Random assignment determines which sheet each receives; it does not change how they entered the session. Nothing supplied here establishes that they represent every employee, future trainee or difficult job.

Equal group sizes also do not guarantee equal experience, reading speed or patience. The draw avoids deliberately giving one version to people expected to be fast. A particular small allocation can still leave differences between the groups.

Consider a different assignment rule: experienced volunteers choose the revision while beginners use the old sheet. Even the same fifteen- and thirteen-minute means would now leave instruction version entangled with experience. Calling both groups volunteers would not repair that allocation.

NIST's discussion of nuisance factors notes that conditions such as operator and time can affect a measured response.

Read back the two-minute difference

This design is aimed at learning about the instruction version. Random allocation improves that comparison, but one small observed difference does not establish an exact, lasting two-minute effect. Different allocations, people or practice tasks could give different results. No uncertainty analysis is supplied here.

Nor has the exercise measured total operating cost. Preparation, the verifier's checking and any later maintenance have not been counted. The separate finished-process comparison includes that wider work.

Repair the claim that the new instructions save everyone two minutes. A supported summary is that, in one randomly allocated practice session, all twelve submissions passed the same key and the revised-sheet group's mean was thirteen minutes versus fifteen for the old sheet. That preserves the useful finding without turning it into a permanent staffing assumption.

Before carrying the result into another setting, name the version compared, how it was assigned and what counted as correct completion. Then identify the next decision this limited answer could change. Testing the business you actually plan to run follows what a small trial can leave untested.

For a memoir about the choices and responsibilities behind an operation, read the opening of Vijay Sheth's Infinite Upside: Building Wealth With Room to Live.