Flying blind?

Every month the ONS releases lots of data on the condition of the UK labour market – including information about employment, jobs, pay, and vacancies. But for the first time (that we are aware of), today’s statistics were released without the data that comes from the ONS’s Labour Force Survey (LFS) – the source for some of the most important labour market information, from the all-important unemployment and employment rates, to breakdowns of labour market outcomes for different groups of people (such as by sex, region, disability, age, and so on).

The ONS has pushed back the publication of LFS data by a week due to their concerns about the quality of the survey. This is a big deal in statistics world and, by extension, in the real world. Policy makers (in particular the Bank of England) need to make difficult judgements about the strength of the labour market as they consider whether to raise, hold or cut interest rates. Uncertainty about key data makes that harder.

Of course, the LFS isn’t the only source of labour market data, and the ONS did still publish all the normal non-LFS data today, including data on pay, vacancies, and a count of the number of employees from HMRC’s PAYE system. Those all point, as they and the LFS have done in recent months, to a labour market which is at a turning point, with demand appearing to cool, and early signs of this starting to affect pay growth. For a closer look at today’s data, my colleague Hannah Slaughter has got the breakdown. Here, we’re going to look into the issues with the LFS, and its wider implications for economists and policy makers.

Different data from different indicators.

In their statement, the ONS pointed towards the problem of falling survey response rates complicating the production of key statistical indicators. Have those falling response rates had a tangible impact on the data?

When we look closer, there has been a significant divergence between the LFS and other sources of employment data. In the LFS, total jobs are roughly at pre-Covid levels. But tax data and the ONS’s Workforce Jobs (WFJ) data suggest job totals are approximately one million higher than before the pandemic.

It certainly isn’t new to see different estimates of employment levels in those datasets, since they come from contrasting sources and have varying coverage and definitions. But this is the first time the LFS has given such a dramatically different picture to the WFJ on longer term employment trends. In past downturns, the LFS tended to show similar trends to WFJ in terms of the size of job losses and the extent of the recovery. The pattern has broken post-pandemic.

It goes without saying that a discrepancy of a million jobs is substantial. For example, if we (speculatively) linked the UK employment rate to the WFJ series instead of the LFS, the divergence becomes even more clear – the WFJ data imply a very high employment rate, far above pre-Covid levels, whereas in the LFS the employment rate is below pre-Covid levels.

But which data is correct?

The obvious starting point is that two datasets outweigh one. PAYE and WFJ data are totally independent of each other (PAYE covers all employee jobs paying tax, while WFJ comes from a large survey of businesses) but indicate similar trends.

It’s also worth noting that there are clearly major concerns about the LFS, not least from the ONS. Falling response rates are a long-term issue, and they have consistently lowered sample sizes. The working age sample was 31,000 in Q2 2023, down from 99,000 in 1992.

Small samples are bad because you get larger variance (estimates come with larger ‘confidence intervals’ and time series bounce around more). But if a sample is representative (or, crucially, if you know what ways it isn’t so you can reweight the data) your estimates should still be centred on the right number.

Bias is the bigger problem. ‘Biased’ data means your estimates are centred in the wrong place, and it comes about when your sample becomes unrepresentative in ways you don’t know about. This happened during Covid-19.

Socially-distanced surveys made it harder to gather data.

During the pandemic, social distancing meant that the ONS had to stop face-to-face interviews and rely instead on phone interviews. This had a bigger effect on response rates among some groups (especially renters) than others. The ONS endeavoured to correct for this by tweaking its weighting.

The ONS has not clearly indicated their exact reason for delaying publication of the LFS. One possible scenario is that they’ve spotted another demographic affected by high non-response rates, which is causing bias, and they are trying to account for that with their weights again.

But it’s hard to see which demographic that could be. If we drill down into the LFS employment rate by change in their sample makeup since 2021 it’s hard to spot an obvious contender with a high employment rate whose share of the sample has fallen (or vice versa).

Is there a path out of uncertainty?

It’s clear the LFS has a levels problem (remember that one million jobs discrepancy), but less clear whether there is necessarily a problem with its estimates of employment rates. One scenario could be that the ONS are happy that their results aren’t biased – i.e. they think their estimates of employment and unemployment rates are reliable – but perhaps their assumptions about the population aren’t correct, which leads to under-weighted data, and lower estimates of employment.

The ONS will release the revised LFS data next week, and we may see them address the levels of the LFS without big changes to the overall rates.

It’s unlikely that new data will paint a significantly different picture about the state of the labour market. The LFS has been broadly consistent with other data in indicating a tight but cooling labour market – we also see this in falling vacancies, slowing pay growth, and stalling or falling employment.

We’ll have to wait to hear from the ONS next week. They’re already very aware of the discrepancies outlined above – which is partly why there’s a new LFS coming next year. Nonetheless, this extent of uncertainty around something as basic as the employment rate is far from ideal.

Update 24/10/23. The ONS has now released its full labour market overview, but with a caveat on “increased uncertainty around the Labour Force Survey (LFS) estimates” and an alternative series of adjusted headline estimates in place of the usual detailed tables. These adjusted figures don’t look vastly different to older data points, and don’t change the trend we’ve been seeing of a gradually cooling labour market. But without detailed breakdowns beneath the headline figures, the new adjusted data series doesn’t give us the complete picture of the labour market. While tax data can fill in some of the gaps for us, it is unable to provide data on important trends such as the reasons behind economic inactivity. And the size of the discrepancy between the ONS’s different data sources on labour market conditions remains of major concern to policy decision makers: we still don’t know whether employment has failed to grow compared to pre-pandemic (as the LFS suggests), or risen by one million (as other data sources show). Given that this data is central to the Bank’s decisions on interest rates, this uncertainty is a serious problem.