A recent experiment conducted by interviewing.io highlighted how ChatGPT could help interviewees to cheat on technical interviews.
The most notable result from the experiment was that not a single interviewer sniffed out that a candidate might be using ChatGPT to give their answers. They also observed that being tasked with a unique problem vs. a more traditional LeetCode type of challenge significantly hurt the performance of the AI “cheaters.”
The author’s takeaway was not that surprising: “The obvious conclusion from these results is that companies need to start asking custom questions immediately, or they are at serious risk of candidates cheating during interviews (and ultimately not getting useful signal from their interviews)!”
It was also the wrong conclusion.
What are you hiring for?
The risk of not using more custom, open-ended questions in your coding interviews is not that people will cheat on them. Instead, it’s that you’ll bias your hiring toward candidates who over-prepare for these “standardized” problems. Software development is no longer an assembly line job. Like any highly skilled trade, your ability to effectively use your tools is more important than your ability to regurgitate a memorized solution.
In fact, the interview process is an excellent opportunity to evaluate a candidate’s ability to analyze the output generated by LLMs critically. While these tools are powerful, they are not infallible. A skilled developer should be able to identify potential errors, inconsistencies, or areas for improvement in the generated code and make necessary adjustments. Likewise, a more junior developer with more confidence using these tools could set themself apart among many similar candidates.
Don’t be lazy.
I think part of the problem is that (ahem) some hiring managers want to be lazy. Why bother spending all this time developing a thorough hiring process that actually evaluates the qualities of the developer we want to hire if I can automate everything using content others have created? Can’t I just hire someone without ever meeting them in person?
I recognize that there are reasonable concerns about using LLMs, both practical and ethical. And, some companies may require developers to forego their use. In that case, I think you have two options.
- Fight an endless losing battle trying to stay ahead of the LLMs with ever more specific and challenging questions until you lose the ability to hire anyone because your interviews are so onerous.
- Invest in your hiring infrastructure to conduct sealed, on-site interviews, so that you can still evaluate your candidates in a real-world environment that approximates the actual job you’re hiring for.
Sometimes, every problem looks like a nail.
Imagine hiring a carpenter and inventing a nailing task no one would ever want to build. It’s just really hard to do with a nail gun so you have to use a hammer. And also, you don’t actually want to watch them hammer. You just want them to describe their hammering technique and estimate how long it might take them to hammer in all the nails in your silly problem.
That’s where interviewing software developers is headed if we try to eliminate the use of LLMs. Love it or hate it, LLMs aren’t going away anytime soon. So, let your candidates “cheat” in their coding interviews. Instead of spending a ton of time and energy trying to fool the machines, spend your time creating a hiring process that is humane.