The short answer

Measure a dealership video pilot using a predefined customer group, a consistent appointment outcome, and a comparable group receiving your usual follow-up. Track replies and total staff time alongside kept appointments. A before-and-after change alone does not show that video caused the difference.

Choose one workflow and one primary outcome

Begin with a specific question: does adding a personalized confirmation video change the share of booked appointments that are kept? That is a clearer pilot than sending video across every lead source and then looking for a number that improved.

Agree on eligibility before the pilot starts. For example, use appointments set by the same BDC for the same type of visit. Define the observation period, which appointments count, and how duplicate bookings, cancellations, and reschedules will be handled. Keep those rules the same for both groups.

  • Primary outcome: kept appointments divided by eligible booked appointments.
  • Secondary outcomes: customer replies and rescheduled visits, with a fixed follow-up window.
  • Effort: staff minutes spent recording, preparing, reviewing, sending, and handling responses.
  • Guardrails: incorrect details, delivery problems, opt-outs, and customer complaints.

Make the comparison as fair as the workflow allows

If operationally practical, randomly assign eligible appointments to your usual follow-up or the same follow-up with video. Keep the cadence, salesperson assignment rules, and booking process otherwise consistent. Record the assignment even if a message fails to send, so delivery failures do not disappear from the result.

If random assignment is not practical, use a clearly described comparison group with similar lead sources, appointment types, and dates. Acknowledge that differences between groups may explain some of the result. A before-and-after comparison can be affected by a promotion, a holiday, staffing changes, or inventory availability.

Set the review date and sample target before starting. A convenient four-week calendar window is not a guarantee of enough appointments. Low-volume stores may need more time, and small differences in a small sample should be treated cautiously.

Use a scorecard with explicit denominators

The example below is illustrative arithmetic, not a VoxRefine result or a dealership benchmark. Both groups contain 200 eligible appointments. In practice, preserve the raw counts and the definitions alongside the rates.

Illustrative example only. These are not measured customer results.
MeasureUsual follow-upFollow-up with video
Eligible appointments200200
Kept appointments100110
Kept appointment rate50%55%
  • Usual follow-up rate: 100 ÷ 200 = 50%.
  • Video-group rate: 110 ÷ 200 = 55%.
  • Absolute difference: 55% − 50% = 5 percentage points.
  • Relative difference: (55% − 50%) ÷ 50% = 10%.

Separate a difference from a conclusion

In the example, ‘5 percentage points higher’ and ‘10% relative lift’ describe the same arithmetic. Neither number alone establishes that the intervention caused the change. Ten additional kept appointments in one group can still reflect random variation or differences in the people assigned to it.

Review sample size, uncertainty, delivery failures, and differences between the groups before describing a result as repeatable. If the relative-lift denominator is zero, relative lift is undefined; show the raw rates instead. Ask an analyst for a suitable uncertainty estimate before publishing a performance claim.

Keep viewing metrics separate from appointment outcomes. A message can be watched without producing a visit, and a customer can attend without watching. Do not count every sold unit in the video group as a sale caused by video.

Count the work around the video

Record the full workflow, not just the seconds spent speaking. Include finding the customer record, checking details, preparing the message, reviewing it, sending it, and handling replies. For a source-recording workflow, include setup and maintenance time, then report how those costs are spread across the pilot’s actual volume.

Time saved in recording is useful only if the next part of the process works. If video generates replies, someone must handle them. Review who owns that queue, how quickly it is worked, and whether the team can maintain the process without extra cleanup.

Decide what earns an expansion

Write down the conditions for expanding before you see the result. Those conditions might include fewer recording minutes per eligible customer, acceptable message accuracy, and an appointment outcome that justifies a larger test. Choose thresholds from your store’s priorities rather than a vendor’s headline.

End the pilot with one of three decisions: expand, revise and test again, or stop. Save the group definitions, raw counts, dates, exclusions, and workflow changes with that decision. A well-documented inconclusive result is more useful than a confident claim you cannot reproduce.