Consulting service 04
Rollout and operation
Calibrate the standard in a pilot, then run it together
A new standard isn't used for certification straight away. In a pilot we compare scores from AI and human raters and adjust the criteria. Once the assessment is live, people review borderline results and appeals.
Method
- 01
Pilot
A small group goes first so we can check the real test conditions and difficulty.
- 02
Scoring agreement check
Human raters and the AI score the same sessions separately and we compare the differences.
- 03
Calibration
We adjust difficulty, wording, weights and the passing score.
- 04
Rater review and appeals
Raters listen to the recordings to make decisions on borderline results and appeals.
- 05
Reporting and coaching
We report the results and provide a coaching guide for managers.
Deliverables
- Scoring agreement report
- Final certification criteria
- Operation report
- Manager coaching guide
The calibrated standard is locked for each round in Design, and results appear by unit in Assess.
Shall we start with the standard?
In a demo we'll show you what kind of assessment your own service standards and training materials can support.