Not Hired Automatically
Opinion Piece
A few years back, while completing my graduate studies in Org Psychology, I very diligently did the work to produce a lit review and a research proposal on something called SJTs. For the uninitiated, SJTs (Situational Judgement Tests) are beloved by work psychology practitioners and, more generally by hiring panels, for their structural utility. You’ve probably done some before. It’s that type of behavioral assessment where you’re presented with hypothetical situations, response options, and are then asked to rank the options in some order. Hiring panels love these tests. Over the past two years of job-hunting, I have seen many of them being used (badly) in the wild. But more on that later.
To have a text-based and cost-effective method to reduce a large number of job applicants is an obvious selling point. They’re accepted as a reliable and valid method that can be used to measure a broad range of constructs (i.e., abstract idea, trait, or process that can’t be observed (remember this). And technically, yes, the tests are ‘reliable’ and ‘valid’ but it’s worth noting that in meta-analyses and reviews on SJTs, there’s an estimated average validity coefficient around 0.32 - 0.34. In the event that you didn’t have to suffer through statistics in your formal education, let me put that another way - a validity coefficient is ‘proof’ that a test is measuring what it's designed to measure. And those scores are not super high, but that’s pretty normal in psychology because human behaviour is complex - especially in real-life settings like a job interview.
What does it matter, though - right? Companies have to hire new people, and that’s a lot of time and effort. And the hiring process takes other things into account, too.
Yes. And no. We tend to put a lot of trust in tests, assume they are well-made and more objective than humans which, we assume, leads to fairer outcomes for everyone involved. It is rarely at the forefront of the hiring panel’s collective decision-making - nor the job applicant’s response, I wouldn’t imagine - that tests are made by humans in the first place, and have designs that are evidence-based but sometimes it's mediocre... even contested. Remember when I mentioned testing abstract ideas we can’t observe? Then, in their construction much was inferred from a collection of items that correlated. The critique here is that by engaging in this type of analysis, we end of making something concrete this actually isn’t.
Tests are fine, but they do not always deserve the blind authority we give them.
In the opening paragraph of this opinion piece, I mentioned I’d encountered SJTs in the wild as I smiled through job interviews where everyone in the room knew it was a terrible fit. It was particularly painful to watch one panel go through their eleven (yes!) questions, mixing the competency-based with the Situational with the ‘Personality’. Unsurprisingly, it took ages to get through and most of it was largely unhelpful as the questions had little to do with the expected responsibilities of the role. But three humans make the process happen, not an algorithm.
In some ways, my example of flawed job selection practices is rapidly becoming quaint. Even though I wrote my lit review on the limitations of hiring assessments back in 2022, it’s almost passé given the more recent addition of automation. To illustrate this, let’s consider a recent large-scale study conducted in the US. In it, researchers found some concerning results after looking at AI use at sifting stages. In the study sample, millions of applications were made by over a hundred applicants in multiple market sectors. They all went through the same algorithm of a single vendor as a first point of the ‘sift’ process. Countless CVs were never seen by a human, and not necessarily because it wasn’t meeting criteria. CVs were being deleted simply because they had been deleted before during a different trawl. The researchers suggest that this demonstrates how an algorithmic monoculture can result in systemic rejections. I encourage you to read the paper, particularly if you work in People.
A question that keeps coming up for me is: How is anyone supposed to take the hiring process seriously anymore? How are applicants supposed to engage in this process with any authenticity or even hope? It’s unbearably dehumanising. I wouldn’t like to suggest that pre-AI job selection procedures were flawless, but this scale of pretence, illusion, and obfuscation is impressively bad.
There is a great lack of transparency around what companies are using in their job selection procedures. And even if they are forthcoming, the vendors are just as vague about their AI-powered products. AI as a blob lacks transparency as a general foundational rule in explaining what is actually being done and how. Who is this working well for? What is the point is making the hiring process this useless? How do we radically reject this? Applications must be handwritten and sent by mail only! Companies must respond to each one!
I’ve experienced this new AI-driven hiring process up close in all its ghosting splendor. In fairness, I’m not a good example because my CV is a perfect case of what-not-to-do in a career. I’ve worked all around the world, I’ve changed careers a few times, I’ve formally studied things that don’t seem to hang well together, I have gaps sprinkled throughout (as I waited on various governments to decide whether I was work-visa-worthy). On paper, my career is a mess. Bots hate me - and I like that. My work and my ongoing professional evolution does not make sense to an LLM, but it does make sense to a human. I wonder how this is showing up for others currently going through it. Given the recent record numbers of unemployment in the UK, I suspect things are not going well.
Meritocracy in hiring was always a bit of a long shot. If you’re willing to engage in some honest self-reflection, you know how swayed you are by things that shouldn’t matter. Let us all take a minute to find a small moment of humility as we recognise the possibility that we may be susceptible to any number of biases - recency, halo, affinity, gender… and that they lead us to make less than ideal decisions more than we might want to admit. Hiring has always struggled with issues of bias in relation to gender and race. One of the hardest things to shake an organisation free from is the conviction that they are different - that they are above all that. However, ‘blinding’ the application process has decades-old evidence demonstrating how it increases the likelihood of women (and older candidates) getting through a sift. Yes, some argue that AI tools will continue to improve on this and make us fair at last. Sadly, diversity in jobs where AI-powered hiring tools are being developed still leaves a lot to be desired.
There was a time, not too long ago, when we had a better chance at giving meritocracy a shot. Psychologists made tests and companies asked for anonymised CVs, and we came up with all sorts of training and protocol to make the hiring process fairer. Did it actually result in everything being fair? No, but the issues were much more localised to industry, location, or an organisation. Now, we still have those issues plus this new circle of AI hell. Whatever shaky premise that used to hold together that section of our social contract (i.e., I’ll work hard as an applicant if you work hard to be fair in your hiring) is gone. We’re now at the whim of machine ‘learning’, panels pretending its objective, and a few vendors struggling to hear you over the ka-ching of the cash register.
Kinda makes me long for a theoretically flawed, mediocre-ly designed, and poorly executed SJT. At least I knew I was dealing with humans.
Have you been suffering through the nightmare that is the AI Hiring Process? Send me a short voice note about it here - https://www.speakpipe.com/Auto_Not_Hired
Illustration by Public domain vectors on Unsplash