How to Measure a Fleet Driving Simulator Pilot

1. Which safety problem should the pilot measure?

Start with the driving situations that create risk in your operation, not with a simulator feature list. Review incident investigations, claims, near-miss reports, and any reliable telematics data. A route-delivery fleet might prioritize backing and low-speed parking. A highway fleet may need more attention on following distance, space management, or hazard response.

That approach follows NIOSH guidance on planning workplace driver training, which recommends using fleet crash and claims patterns to select training topics. OSHA’s guidance for employers also calls for training suited to the vehicles and hazards drivers face.

Define the baseline before training starts

Choose one primary safety question, a defined group of drivers, and a period of past data. For less frequent incidents, six to twelve months of history may give more context than one month, although a useful window depends on fleet size and exposure. Keep definitions stable across periods:

• Incident type: What counts as a backing collision, a preventable incident, or a reported near miss? Who makes that judgment?
• Exposure: How many miles were driven? For backing-specific risk, can you also count stops or backing maneuvers reliably?
• Operating conditions: Did routes, vehicle types, hiring patterns, weather, or reporting rules change?

Write down the data source, owner, timeframe, and definition for each measure. If a backing event happened in a depot rather than on a road, miles alone may be a poor measure of how often drivers faced that task. Use the most relevant dependable denominator and report it consistently.

2. What belongs on a fleet simulator pilot scorecard?

A practical scorecard follows the path from training to work: Did drivers take part? Did their decisions improve in practice? Did that improvement appear on the road? Only then should you look for changes in incident rates. This separates training activity from outcomes, consistent with the CDC Program Evaluation Framework.

Our ADS materials describe Vantage reporting for measures such as reaction time, awareness, and decision-making. Ask to see the actual report fields for the scenario and configuration you plan to pilot. Pair any simulator record with instructor notes and an on-road checklist. They measure different things.

Interpretation matters: A better simulated reaction time doesn’t establish a lower crash rate. A telematics event can also need context. For example, one hard-braking event may reflect a safe response to a sudden hazard rather than poor driving. NIOSH’s fleet safety guidance recommends looking at behavior patterns and coaching context.

3. How do you make the pilot results credible?

Compare similar drivers and conditions

Whenever practical, use a similar group receiving your existing training, then offer the simulator to that group later. A staggered rollout can help distinguish training effects from seasonal changes or new fleet policies without denying anyone required instruction. If that isn’t possible, compare each driver’s before-and-after observations, but label the findings as less certain.

Keep route types, vehicle classes, job tenure, observation checklists, and incident definitions as consistent as possible. A driver assigned to fewer high-risk stops may have fewer events without changing their behavior. The CDC’s 2024 evaluation framework report explains why comparison design limits the claims an evaluation can support.

Set a practical observation schedule

As a planning example, gather simulator measures at baseline and after practice, conduct on-road checks at about 30 and 90 days, and review incident rates over a longer period suited to the fleet’s mileage and event frequency. Choose and document your actual windows in advance. Short pilots may be adequate to test coaching and logistics but too small to establish a change in crashes.

Measure the work behind the training

Include equipment and facility costs, instructor time, employee time away from routes, travel, setup, IT needs, and data review. Record who can access driver-level assessments and how long records will be kept. The fleet training overview shows how we position simulation within a workplace safety program; your pilot should still confirm the required scenario fit, space, staffing, reporting workflow, and product terms for your organization.

4. How do you calculate a change in incident rates and costs?

Use both the incident count and the amount of driving. Research published in Safety Science in 2019 compared safety practices and collision rates across 70 U.S. companies using vehicle miles traveled as an exposure measure. It found associations between several safety practices and outcomes; it did not test our ADS product or establish a simulator-specific effect.

The decline in this invented example cannot be credited to simulator training. There are few incidents, no comparison group, and no information about route or weather changes. For a backing-heavy operation, changes in backing attempts may matter more than changes in mileage. The $400 figure is an example of a pilot cost calculation, not a product price or evidence of return on investment.

Report claims cost and downtime separately when your records support them. Avoid projecting insurance savings or an ROI percentage from an incident-rate change unless you can show that the change is attributable to the training and use defensible cost assumptions.

5. When should you expand, revise, or stop the pilot?

Agree on the decision rules with safety, operations, and finance teams before training begins. A useful readout should show the sample size, completion rate, skill and field observations, incident counts and exposure, full costs, comparison, and important limitations.

Expand if the evidence supports it

Consider a phased rollout when training is practical to deliver, drivers show a meaningful improvement on a consistent on-road checklist, and the cost fits your program. Treat changes in crash rates as supporting evidence only when the data and follow-up period make that comparison credible.

Revise when learning doesn’t reach the road

If simulator scores rise but on-road behavior does not, look at scenario choice, instructor feedback, and reinforcement after training. If participation is low, investigate scheduling and staffing before concluding that drivers or the tool are ineffective.

Pause if the evidence is unreliable

Don’t scale on the strength of a percentage from a handful of incidents, inconsistent reporting rules, or a cost estimate that excludes driver time. Gather better data, adjust the design, or use a narrower pilot to answer the unresolved question. Simulation should complement required vehicle-specific instruction and behind-the-wheel assessment, not replace them.

Key takeaways

• Define the fleet risk and baseline before choosing training scenarios.
• Measure training reach, simulator practice, coaching, and real-world driving separately.
• Use consistent incident definitions and exposure-adjusted rates, not raw crash counts alone.
• Set comparison methods, costs, observation dates, and rollout criteria in advance.
• Describe any improvement at the level the evidence actually supports.

Frequently asked questions

What is the most important KPI for a fleet simulator pilot?

No single KPI is enough. A documented improvement in the target behavior during on-road observation, paired with training completion and exposure-adjusted incident tracking, gives a stronger picture than simulator scores alone.

How long should a fleet simulator pilot run?

Long enough to complete training and observe drivers on the job. A 30- or 90-day behavior check may be practical, but crash-rate changes often require more driving exposure and a longer follow-up. Set the period based on your fleet’s incident frequency.

Can we calculate a return on investment from the pilot?

You can calculate the pilot’s total cost and cost per completed driver. A defensible financial return requires reliable evidence of benefits, such as avoided costs, plus a sound way to distinguish training effects from other changes. Don’t assume that training automatically reduces premiums or claims.