Your job advert is attracting responses, but there aren’t enough candidates who are truly suitable. In that case, it makes sense to tweak the text. But which version works better? With an A/B test for your job advert, you don’t have to guess: you compare two versions side by side and measure which one generates more suitable responses. The short […]
The short answer: test one element at a time, run both versions simultaneously via the same channel, and measure the quality of the responses, not just the quantity. Do you receive few responses each week? In that case, a small difference often doesn’t mean much. We won’t be covering the actual writing of the text here. You can read about what the content should include in Using target audience data for your job advertisement.
Which parts of a job advertisement do you check first?
Start with the elements that a candidate often sees first or that may influence their decision to apply. The title and the opening are usually the first things they see. That is why they are logical places to start.
The job title
The title can affect how easily your job vacancy is found. According to Indeed Specific, commonly used job titles work better than those people are less likely to search for. Their survey revealed that 36% of people who use job sites search using the title of the role they are looking for. So, for example, compare a creative title with the one your target audience actually uses.
The opening
The opening is often one of the first things someone reads, and is therefore a logical part of the assessment. You can compare an opening that focuses on the company with one that states straight away what the candidate will be doing and what the benefits are.
Salary and terms of employment
Research by LinkedIn Using a heatmap of job adverts, it showed that the salary range and benefits were by far the most frequently highlighted. Specific details about the role, such as duties and location, also helped candidates. This makes the placement and format of your terms and conditions a logical area for testing: at the top or bottom, a range or a starting figure, or perhaps another condition listed first.
The application stage
The way you explain what happens after a response can also make all the difference. You can read about how to put this into words in Explaining the application process in your job advert. For a test, you could, for example, compare a brief explanation with one that also mentions the lead time.
How do you design a fair test?
A test is only fair if the only difference between the versions is the component you wish to test. Any other changes make the result less reliable.
Work in this order:
- Choose one question. For example: does a common job title generate more suitable responses than our current title?
- Make two versions. Version A is your current text; version B differs in exactly one respect.
- Define in advance what constitutes ‘appropriate’. Use your existing mandatory requirements, such as a driving licence or certificate, so that you don’t end up changing the criteria retrospectively.
- Decide on the duration and the objective. Agree on how long the trial will last and the minimum number of responses you want to see before making a decision.
- Run both versions at the same time. Through the same channel, targeting the same audience and with the same budget.
- Rate the comments without knowing which version it was. This will help you avoid letting your preference for a particular version colour your judgement.
How many responses do you need before a test is meaningful?
More than you’d probably hope for. With small sample sizes, any difference is likely to be down to chance. Ten responses to version A and seven to version B are no proof that A is better.
Exactly how much you need depends on a few key ingredients. According to Stately For a reliable A/B test, you need a sufficiently large sample size. To calculate this, you need to take into account, amongst other things, the current conversion rate, the smallest difference you wish to be able to demonstrate, the distribution between the versions, the significance level and the desired power.
What does that mean in practice?
- Are you looking for a subtle difference, Then you’ll need plenty of visitors and comments.
- Do you expect there to be a big difference, For example, if you use a completely different title, you’ll spot things more quickly.
- Know your current conversion rate. You enter this as the initial value. The size of your sample depends partly on that conversion rate and on the other input values mentioned, such as the smallest difference you wish to demonstrate.
For hard-to-find roles in engineering or the maritime sector, the volume is often low. In such cases, it is better to test using an intermediate step with more data – such as clicks on the job advert or the initiation of an application – and to use the relevant responses as a second signal. An online calculation tool helps you assess in advance whether your test is feasible.
How can you prevent a particular channel or season from affecting your test?
By distributing the versions simultaneously and at random. Google describes an A/B test as a randomised experiment in which each variant is shown simultaneously to a random group of users. This reduces the likelihood that timing or differences between the groups will account for the outcome.
Common mistakes that can disrupt your test:
| Error | Why things go wrong | Better |
|---|---|---|
| Version A in March, version B in July | A holiday, seasonal work or a competitor with a new vacancy could explain the difference | Run both versions simultaneously during the same period |
| Version A on social media, version B on a job site | You’re comparing channels, not texts | Test both versions within the same channel |
| Adjusting the budget or targeting during the test | One of the versions suddenly finds itself with a different audience | Lock settings until the test is complete |
| Stopping too early because one version is ahead | Early differences often disappear | Adhere to the agreed duration or number |
If you can’t split the data arbitrarily – for example, on a job site that only allows a single piece of text – then don’t alternate every month; instead, keep the periods short and comparable. Also make a note of any other changes that occurred during that time, so that you interpret the results with caution.
Do you measure your responses or appropriate responses?
Appropriate responses. A text that generates twice as many responses is not an improvement if you mainly attract candidates who do not meet the basic requirements. In that case, the winning version will actually cost you more time.
So, for each version, take a look at a few figures side by side:
- number of views and clicks;
- number of job applications started and completed;
- proportion of responses that pass the initial screening;
- the number of candidates who go through to an interview.
You can only compare individual figures if both versions received exactly the same reach. In practice, this is rarely the case. You should therefore calculate the same conversion rate for each version: divide the number of relevant responses by the number of views, or by the number of applications initiated if you are testing at the application stage. A hypothetical example: version A receives 1,000 impressions and 12 matching responses, which is 1.2%. Version B receives 800 views and also 12 matching responses, which is 1.5%. In terms of numbers, they appear the same; as a ratio, they do not. Whether such a difference is more than mere coincidence depends on your sample, as described above.
Before you start, decide which ratio will be your primary metric. This should relate to your test question: if you’re testing the title, the obvious choice is relevant responses per view. If you’re testing the explanation of the application step, it’s more likely to be completed applications per application started. As well as this ratio, look at the absolute numbers too. A high ratio based on just a handful of views doesn’t mean much, and ultimately you need enough suitable candidates.
You can find out how to set up and read these figures in Measuring recruitment campaign figures and measuring the quality of your pool of applicants.
When is testing more beneficial than optimising directly based on target audience data?
Testing is no substitute for knowing what matters to your target audience. Employers often think that certain benefits are important, whilst data shows that the target audience actually has different priorities. You cannot improve a text based on incorrect assumptions by testing two weak versions against each other.
A handy rule of thumb:
- Start by optimising based on target audience data If your text is still based on internal assumptions, or if your volume is low, you’ll make the most profit in a single step.
- Have a go if the fundamentals are right and you’re torn between two good options, such as two job titles or two ways of phrasing your requirements. Or if you’re recruiting for the same role on a regular basis, so that a single insight pays off multiple times.
At FosFor, we combine the two. We build the campaign based on target audience data and labour market data, and then optimise it based on performance. In our weekly strategic meetings, we discuss the results and the next steps. You can read about how this works on the page about our data-driven approach.
Preparing your first test
This week, choose one job advert with sufficient reach and one element to test – ideally the title. Make a note of what constitutes an appropriate response and how long the test will run. Then let the data speak for itself.
Are you unsure whether your vacancy is substantial enough to warrant a test, or whether the problem lies more with your target audience or the follow-up process? Arrange an introductory meeting and we’ll look at your hard-to-fill vacancy together.