Automatically Hire

A few years back, while completing my graduate studies in Org Psychology, I very diligently did the work to produce a lit review and a research proposal on something called SJTs. For the uninitiated, SJTs (Situational Judgement Tests) are beloved by practitioners and hiring panels for their structural utility. You’ve probably done some before. It’s a type of behavioural assessment where normally, you’re presented with hypothetical situations, response options, and you’re asked to rank them in some way. Hiring panels love these tests. Over the past two years of job-hunting, I have seen many of them being used (badly) in the wild. But more on that later.

To have a text-based and cost-effective method to reduce a large number of job applicants is an obvious selling point. They’re accepted as a reliable and valid method that can be used to measure a broad range of constructs (i.e., abstract idea, trait, or process that can’t be observed (remember this)). And technically, yes, the tests are ‘reliable’ and ‘valid’ but it’s worth noting that in meta-analyses and reviews on SJTs, there’s an estimated average validity coefficient around 0.32 - 0.34. In the event that you didn’t have to suffer through statistics in your formal education, let me put that another way - a validity coefficient is ‘proof’ that a test is measuring what it's designed to measure. And those scores are not super high, but that’s pretty normal in psychology because human behaviour is complex - especially in real-life settings like a job interview. 

What does it matter, though - right? Companies have to hire new people, and that’s a lot of time and effort. And the hiring process takes other things into account, too. 

Yes. And no. We tend to put a lot of trust in tests, assume they are well-made and more objective than humans which, we assume, leads to fairer outcomes for everyone involved. It is rarely at the forefront of the hiring panel’s collective decision-making - nor the job applicant’s response, I wouldn’t imagine - that tests are made by humans in the first place, and have designs that are evidence-based but sometimes it's mediocre... even contested. Tests are fine, but they do not always deserve the blind authority we give them.

In some ways, my example is rapidly becoming quaint. Even though I was writing my lit review on the limitations of some hiring approaches back in 2022, it’s almost passe given the more recent additions of automation. In a recent large-scale study conducted in the US, researchers found some concerning results. Millions of applications were made by over a hundred applicants in multiple market sectors - all going through the same algorithm of a single vendor as a first point of the ‘sift’ process. The researchers suggest that this demonstrates how an algorithmic monoculture can result in systemic rejections. 

At this point, for me, the question is - how is anyone supposed to take the hiring process seriously anymore? There is a great lack of transparency around what companies are using in their job selection procedures. And even if they were forthcoming, the vendors are just as vague about their products. AI as a blob lacks transparency as a general foundational rule. And who is getting what they want here, aside from whoever makes some money from selling the CV scanning product? I’m sure the line with most HR departments is that they are looking for the ‘right people’, and the ‘right fit’ for the role. I’m less sure how that’s possible when a shitty machine-learning response is deleting potential candidates on what appears to be determination to secure gainful employment. I’m very sure that this is not working well for people who are looking for work. 

I’ve seen this process up close in all its delusional splendor. In fairness, I’m not a good example because my CV has all the things one is not supposed to have on there. I’ve worked all around the world, I’ve changed careers a few times, I’ve formally studied things that don’t seem to hang well together, I have gaps sprinkled throughout as I waited on various governments to decide whether I was work-visa-worthy. On paper, my career is a mess. Bots hate me and I like that. My work and my ongoing professional evolution does not make sense to an LLM, but it would make sense to a human. I wonder what this is showing up like for others. Given the recent record numbers of unemployment in the UK, I suspect things are not going well. 

Meritocracy was always a bit of a long shot. If you are capable of honest self-reflection, you know how swayed you are by things that shouldn’t matter. If you’re not capable of honest self-reflection, let us all engage in a small moment of humility as we recognise the possibility that we may be susceptible to any number of biases - recency, halo, affinity, gender… 

There was a time, however, when we had a better chance at giving meritocracy a shot. We made tests and anonymised CVs and came up with all sorts of training and procedures to make the hiring process more meritocratic. Whatever illusion used to exist in that social contract - I’ll work hard as an applicant if you work hard to be as meritocratic as you can manage - is gone. We’re now at the whim of machine ‘learning’, and hiring panels pretending its objective. 

Kinda makes me long for a poorly done SJT.

Have you been suffering through the nightmare that is the AI Hiring Process? Send me a short voice note about it here - https://www.speakpipe.com/Auto_Not_Hired

Illustration by Public domain vectors on Unsplash ‍