You have a scorecard, you have several interviewers, and yet the scores vary. One interviewer gives a 4 where another sees a 2. This is rarely a matter of ‘gut feeling’; it’s usually a problem of interpretation. In 45 minutes, you can significantly reduce that variation with a short, tightly facilitated calibration session.
What exactly does ‘calibrating a scorecard’ mean?
Calibrating interviewers’ scorecards means that, as a team, you set the same assessment criteria: what constitutes a 1, 3 or 5 for your team, and what evidence supports each score. You make implicit standards explicit, so that discussions focus less on personalities and more on observable behaviour.
Calibration is not about “agreeing on the candidate”. It is about agreeing on the rubric: the same words, the same standards, the same interpretation of evidence.
When do you realise that calibration is missing?
- Scores vary systematically (rather than incidentally) between interviewers.
- The debriefing focuses mainly on ‘impressions’ and says little about specific answers.
- You can see score inflation: everyone scores around 4–5 “because otherwise there’d be no one left”.
- New interviewers perform strikingly differently from their experienced colleagues.
Why this has a direct impact on both quality and speed
Without calibration, you have to “renegotiate” what a score means after every interview. That takes time, leads to unnecessary uncertainty and makes it harder to explain your decision to hiring managers and candidates.
Calibration makes scores comparable. This allows you to make decisions more quickly, as discussions are shorter and focus on evidence.
The 45-minute scorecard calibration: agenda and preparation
This approach works best if you repeat it for each job category (for example: operators, fitters, team leads or sales staff). You don’t need much: one scorecard, one facilitator and two sample answers.
Would you like to find out more about data-driven recruitment?
- Recruitment using AI and data
- Takes an average of 3–8 weeks to complete
What to prepare in advance (10 minutes’ preparation)
- The scorecard with 4–6 key criteria and a clear scale (for example, 1–5).
- 2 sample answers per criterion (may be fictitious or anonymised). Include one “borderline case”.
- A simple decision rule: Do we award marks for each criterion and then calculate an average, or is it a knockout system?
- A note format: for each criterion, “score + 1 sentence of evidence”.
Who needs to be there?
- All regular interviewers for this type of role (ideally 3–6 people).
- The hiring manager or the person with final responsibility for the decision.
- One facilitator to oversee the process (from HR or recruitment).
Step-by-step: calibrating a scorecard in 45 minutes
0–5 mins: rules to prevent disruption
- We assess evidence, no personality (“likeable”, “a smooth talker”).
- We use the same scale definitions as in the scorecard.
- We are, in fact, noting down: what did the candidate say or do in their answer?
5–15 mins: explore one criterion in depth using key examples
Choose one criterion that often gives rise to discussion, such as problem-solving skills or safety awareness. Work together to define which indicators correspond to a 1, a 3 and a 5.
Use “anchor phrases” that anyone can use during the interview, such as:
- 1: remains general; no specific actions or measurable results.
- 3: describes a logical approach and can explain why.
- 5: works in a hypothesis-driven way, checks assumptions and identifies the impact/outcome.
15–25 mins: individual marking of the same model answer
Read (or play) an example answer aloud. Everyone marks their answers silently and independently. Afterwards, everyone writes down one sentence of evidence for each mark.
This is the crux of the matter: you eliminate peer pressure and groupthink by scoring individually first.
25–35 mins: discussing differences without compromising
Start with the highest and lowest scores. Keep asking: “What evidence in the answer led you to choose 2/4?”
If the evidence is the same but the score differs, then your rubric is too vague. In that case, adjust the anchor points (not the individual’s score).
35–42 mins: arranging appointments for the upcoming interviews
- Which 2–3 follow-up questions do we use as standard when in doubt?
- Which words do we remove from notes (such as “click”, “energy”, “dominant”) because they are difficult to assess?
- Which two criteria carry the most weight for this role?
42–45 mins: mini-check and ownership
Agree on who will update the scorecard and when you’ll recalibrate it (for example, after five interviews or when a new interviewer joins). Keep it simple, otherwise it won’t happen.
A handy table: how to quickly ensure consistent scoring
This table helps you to quickly see during the session whether differences are due to interpretation or to missing information.
| Situation | What you’ll see in the debrief | What you’re changing |
|---|---|---|
| The heading is too vague | Everyone cites the same evidence, but gives different scores | Make anchors more specific (1/3/5) and add example sentences |
| Asking follow-up questions about differences | One interviewer goes into detail, the other doesn’t | Agree on standard follow-up questions for each criterion |
| Score inflation | Almost everything is 4–5; there is little difference | Definition of “5 – Step Up”: ‘exceeding expectations’ with demonstrable results |
| Notes are interpretative | Words such as “confident” and “fits into the team” dominate | Note format: “quote/fact + context + result” |
Common mistakes when calibrating (and how to avoid them)
1) Tailoring the assessment to the candidate rather than to the scorecard
If the discussion comes down to “do we think he’s any good?”, then you’re already too late. Calibration is a skill: the same yardstick, not the same opinion.
2) Trying to address too many criteria at once
Pick one criterion where there’s the most confusion. If you try to redefine six criteria in 45 minutes, you’ll end up with half-baked agreements.
3) No consideration given to privacy and retention periods
Scorecards and interview notes are recruitment data. Make sure you only store what is necessary, are transparent with candidates, and do not keep them for longer than you need to. The Explanation from the Dutch Data Protection Authority regarding job application data provides practical guidelines.
How this ties in with a structured interview
Calibration only really works if the interview itself is already structured: the same questions, the same order and the same rubrics. If everyone asks different questions, you’re comparing apples with pears.
If you still have work to do in this area, start by laying the foundations: How to conduct a structured interview in 8 steps. Combine that with a robust scorecard, such as the one in Create a selection scorecard for hiring managers.
A simple next step you can take today
Plan a single 45-minute session for your most common job category. Use two real (anonymised) responses and ask everyone to score them silently first.
If you want to get this right more quickly for blue-collar, sales or maritime roles, it may help to work together to identify the right criteria, key indicators and a workable candidate journey. If you’d like to discuss a data-driven approach that reduces noise and speeds up decision-making, please feel free to contact FosFor.