← All Notes

I built an AI interviewer. It gave "Thank you." a 3.6 out of 5. Here is why a human, not a score, should decide who gets hired.

Related: another bootcamp student argues that taxing AI is the wrong fix for its effect on work, in Don't Tax the Tokens.

On 10 November, a hiring software company called Workable announced that its AI recruiting agent has read 1.2 million job applications since it launched in May. These are the company's own figures. It picked out more than 350,000 candidates to talk to and held screening conversations with them. Only 0.1% of those candidates asked to speak to a human instead. Two days earlier, TechRadar reported a survey by the recruitment company Use.AI of 6,731 job seekers, which found that 29% of people who already have a job have walked away from a hiring process because it included an AI interview.

So AI is already doing the first part of hiring, at a huge scale, and a lot of candidates don't like it. Workable says its agent supports human-led decisions, and I believe that's the intention. But I want to say clearly what I think, as someone who has actually built one of these: AI can help, but it should never be the one deciding who gets hired.

Here's why.

I answered an interview question with "Thank you." That was the whole answer. The AI gave it 3.6 out of 5.

I know this because I built the AI. It's called InterVox, a mock interviewer I made for my apprenticeship. You upload your résumé and the job description, you answer questions out loud, and at the end you get a score and a better version of each answer. It's meant for practice, so you can get ready for a real interview.

While I was testing it, I tried answering a question with small talk, something like "hi, how are you, nice to meet you." It scored 3.56. That's not a bad score, that's an average candidate. And it was completely wrong.

That's the moment I stopped trusting an AI's score on its own.

It wasn't just one mistake

The greeting wasn't the only problem. To be honest, I faced too many issues with getting the scores right:

None of these looked broken. The AI gave a number, the number looked reasonable, and the report looked professional. If I hadn't been testing my own app on purpose, I would never have seen them. And a candidate who got one of those scores would never know it was wrong.

The AI didn't learn. I added a check.

I used to think that when you catch an AI making a mistake and fix it, that's how it learns. That's not what happened.

The model InterVox uses is exactly the same today as the day I started. It didn't get smarter. What changed was my code. I asked my agent to add checks that look at the answer before the AI's score counts: is it too short, is it just small talk, is it actually about the question? If it's off-topic, the relevance score is capped. I also fixed the weights in code myself, 70% for the role and 30% for communication, so the AI can't decide that anymore.

That changed how I think about this. The AI doesn't fix its own mistakes. A person has to notice them and build something to catch them. And I only fixed the bugs I happened to notice. There are probably a lot more in there that I haven't seen yet.

Where AI belongs in hiring

I'm not against AI in hiring. I built one, so that would be strange. AI can do a lot: it can score answers, point out what a candidate missed, notice how clearly they explain things, and save an interviewer a lot of time.

But it should only score and suggest. A human should make every yes or no. That includes the no's: before any candidate gets a rejection because of an AI score, a person should look at their application first.

Cartoon: a robot hands a folder of AI notes (score 4.2 out of 5, strong on SQL, check teamwork) to a person at a desk with YES and NO stamps, who says "Thanks. I'll decide." Caption: AI suggests. A human decides.

At first I thought AI could at least handle the early rounds, like round 1 and round 2, and leave the final decision to a person. Then I thought about my own bug the other way around. My AI gave a bad answer a good score. What if it gives a good candidate a bad score in round 1? That person gets rejected, and no human ever sees them. They don't get to argue. Nobody even knows it happened.

In the final round, at least there's a person in the room. In round 1, a wrong score just disappears. And round 1 is exactly where AI is being used most: Workable's agent read 1.2 million applications to choose who to talk to. That's why I think a human has to decide at every step, not only at the end.

"But humans are biased too"

This is the strongest argument against me, and it's true. Human interviewers can judge people on their accent, how they look, or what mood they're in that day. AI is consistent. It doesn't get tired or have a bad morning.

But consistent isn't the same as right. My AI gave "Thank you." a 3.6, and it would have done that for every single candidate, the same way, every time. A consistent mistake doesn't go away. It just happens to everyone.

There's also something an AI can't do. When you talk to a person in an interview, you get a feeling for them: how they communicate, how they explain things, whether you could actually work with them. An interviewer might finish a conversation and think, "I'd like to work with this person." An AI only looks at the skills and the answers. That feeling matters in a real job, and right now only a person can have it.

So yes, people are biased. The answer is to help them with AI, not to replace them with it. Let the AI score, and let the human decide, knowing what the AI thinks.

What I'd tell a company

If you're thinking about letting an AI decide who gets hired, try what I did. Answer one of its questions with "thank you" and see what score you get. Then ask yourself how many mistakes like that you'll never find, because the person it happened to was rejected and never came back to tell you.

I built an AI interviewer, and I'm still working on it. I'll keep using it to help people practice, and I think AI can make interviews better. But it should never be the last word on someone's job. That decision should always be up to a human.

— Hibah Nafees

Related on Fire Times: Don't Tax the Tokens, another student's opinion piece, on why fencing AI out of work is the wrong answer to what it does to jobs.

Sources