AI Broke Your Interview Process
“Never stop hiring” isn’t just a mantra; it’s been my reality this year. I had to fill many team lead and developer positions, and to achieve it, I relied on a battle-tested framework:
- Check for soft skills like communication style.
- Dig deeper if you perceive common symptoms of red flags.
- Ask for previous projects and technical challenges.
- Proceed with coding challenge and sometimes live coding session.
It worked like a charm until it didn’t.
With AI (i.e., LLMs) in the form of chatbots, the game has changed. Generic questions get generic answers. Common algorithmic challenges are solved quickly and efficiently by ChatGPT. Competent candidates use AI to craft polished narratives based on their individual experiences to promote themselves. The use of AI is more apparent in written and prepared interview steps, such as coding challenges and assessment tasks.
AI usage sparked a wave of hiring-policy overhauls (see HRD, and Sparkhire). I also re-evaluated my hiring strategy. Ultimately, I had to modify the coding challenges and the format of the interviews.
So what really matters in a hire? For me, the answer is problem-solving and getting things done. Using the right tool for the job is fine, as long as the solution doesn’t suffer. I don’t expect a developer to know APIs by heart or to code everything by hand when an IDE generates boilerplate code just fine. AI is yet another tool that can help you build reliable code, but be cautious, as bad AI output can cause harm, just like poor analytics.
Problems to overcome
The initial interview mainly stayed the same. Before ChatGPT, good candidates also prepared for canned questions, but many struggled with questions like “What are your strengths and learning fields?” or “When was the last time you received feedback and what was it?”. Asking an AI to help you is the same as reaching out to a friend to help you with preparing answers. To me, that’s not cheating, just getting ready. Unfortunately, with more people having the “right” answers, it’s more complicated than ever to tell who’s genuine.
Two decades ago, the most common coding challenge was FizzBuzz on a whiteboard. Once this became common knowledge, companies moved to algorithmic challenges like array scanning and rotation. That worked because anyone with fundamentals could solve them. Knowing common data structures and algorithms was enough to achieve good results. But even a decade ago, many companies moved to more realistic challenges. My first coding challenge was building a TODO app in Rails. It took me roughly an hour to complete all the requested features. AI is now capable of writing complete applications in a couple of minutes.
The interview process loses its purpose when you cannot distinguish between fit and unfit candidates for the job.
Even when someone uses AI or asks a friend to code for them, it will all be evident once you ask them to add new features to their solution in a live coding session. Copying and pasting the solutions becomes obvious, and typing the correct solution from top to bottom into your IDE is also easily detectable.
The solution
You have to change your approach unless you’re hiring prompt jockeys instead of engineers.
Keep the canned questions to a minimum and focus on open questions with follow-ups. Ask the candidates about their projects, how they approached them, and how they solved the problems. What feedback did they receive from their peers and their lead, and how did they react?
Craft a challenge that includes dirty data and gaps in the brief, which trips up AI and mirrors real-world messiness. Commonly, engineers clean migration data and challenge user stories. AI either follows the task description mindlessly or hallucinates solutions when details are missing. The open task fosters creativity, and the messy data will highlight how the candidate approaches edge cases and errors in their solution.
For example: “Build an API for XYZ that is production-ready. [Describe the business logic without mentioning edge cases]. You can skip authentication.” Provide a CSV file with seed data that includes common problems encountered in migration projects, such as missing fields and commas in titles.
The term ‘production-ready’ is powerful, allowing candidates to focus on various areas, such as error handling, pagination, and comprehensive testing. With time constraints, it is rare to find a perfect solution. However, the focus reveals what the candidate values most and provides an easy way to initiate a constructive discussion in later interview steps.
In the coding session, I ask candidates to add new features to the solution they submitted, which shows me how well they know their codebase and how they approach problems. Candidates can use any tool they want, as long as I can see how they utilize it and how they incorporate it into their code. For me, it’s comparable to using Google and StackOverflow in 2018. Everyone did it.
If your search query is solid and you find the solution, then go for it; otherwise, start coding.
Outcome
With a few tweaks, you can still spot great hires. The key is to ask open-ended questions with follow-ups, introduce ambiguity in the task, and proceed with additional features during the coding session. You’ll spend the same time but hire a better team.