The Tech Recruiter’s Dilemma – Technically Sound or Nerves of Steel?
Developer interviews are considered to be at the core of evaluating technical skills while recruiting software professionals. Traditionally, such interviews require the candidate to solve coding problems on a whiteboard in front of an interview panel. Although the intent is to understand the job-seekers’ skills of coding and problem-solving, a recent path-breaking study reveals that such interviews are effectively evaluating their stress levels and anxiety while performing the task – rather than technical knowledge.
The study
Conducted by researchers from North Carolina State University in collaboration with Microsoft, this study published in the form of a paper in the journal of the ESEC/FSE (ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering)is probably the first of its kind on the topic. It involved a block randomized controlled trial on a total of 48 undergraduate and graduate students of the University – divided into two groups. The aim was to analyse the real-life developer interview situation by comparing performances in two different simulated settings – public and private.
The public setting mirrored the real-life scenario where a candidate had to perform on the whiteboard standing in full glare of the critical eyes of the evaluators. In the private setting, participants solved the same technical problem on a whiteboard – but in an isolated room. The interviewer left the room after explaining the problem and the participants were aware that they will not return until the allotted time was over.While 26 participants were tested in public setting, 22 were evaluated in private.
The findings
The summary finding was an eye-opener of sorts. To quote the research paper: “… participants reported high-levels of stress and often had difficulty coping with someone else being present while they attempted to solve a cognitively-demanding task. The impact on performance was drastic: nearly twice as many participants failed to solve the problem correctly, and the median correctness score was cut more than half, when simply being watched.”
The researchers also noted: “In contrast, participants in the private setting reported feeling at ease, having time to understand the problem and reflect on their solution. …measures of cognitive load and stress were significantly lower in the private setting, and majority of participants solved the problem correctly. We also observed that no women successfully solved the problem in the public setting, whereas all women solved it correctly in the private setting.”
The explanation
So what is happening here? IT companies have traditionally relied on the developer interview to assess technical skills of a programmer. It was a time-tested litmus test to judge coding capabilities, clarity of concept as well as the ability to communicate ideas. However, the study pointed out, that despite the best of intentions such developer interviews have“an uncanny resemblance to the trier social stress test”, a procedure that“involve shaving a subject prepare and then deliver an interview-style presentation and perform mental arithmetic, all in front of an audience.Alone, none of these actions consistently induce stress in a subject;however, the unique combination of cognitive-demanding tasks with a social-evaluative threat (essentially being watched) is consistent and powerful.”
Essentially, the standard format of the technical interview induces huge performance anxiety and resultant stress on the candidate – however capable. Those who can withstand this stress and solve the problem with confidence, gets recruited. But those who get rattled by the criticising stare of the interviewers, either perform poorly on the whiteboard despite full capability or despite solving the problem projects an extremely nervous image to the interview panel – thus reducing their chances of getting selected. Thus, in effect, the developer interviews are basically weeding out candidates who are anxiety-prone and with low stress threshold, not choosing the candidate best equipped with technical knowledge. It is a selection tool alright – but this is not perhaps the true motive of the recruiters who employ it!
In-camera evaluation?
The study revealed that those who took the test in private setting did remarkably well, with no judging eye scanning them while working on the problem. As the paper suggested: “…private interview settings have considerable advantages for candidates, that both reduce their stress and allow more accurate assessment of their problem-solving abilities.” Some of the comments made by the private interview participants describing their experience, and recorded in the paper, are worth noting:
- “pretty comfortable”
- “less stressed”
- “not feel any stress”
- “better than panel”
- “difficult to code when someone watches over”
- “did not feel very rushed since I was the one who determined when I was done”
So is in-camera interview the best way forward? The researchers would not like to rush into that judgement, and rightly so. They noted that the private setting was not without its problems. The foremost difficulty was the lack of feedback, without which the participants were often “unsure” whether they are on the right track. In a face-to-face scenario, instant feedback would have aided a prompt course correction. Some got stuck when there was uncertainty regarding the problem that required clarification. One participant even quit the study – unable to proceed without explanation.
Another hurdle for the private setting participants was managing time. Many reported getting immersed in the problem and losing track of the passage of time in complete isolation– leading to a rush towards the end.
Remedial guidelines
Towards the later part of their paper, the researchers have offered four broad guidelines for making developer interviews more effective and minimize inconveniences faced by the candidates. However, they categorically mentioned that these recommendations are “hypotheses that need further evaluation rather than outright policy.”
- Encourage retrospective think-aloud rather than concurrent think-aloud to assess explanation skills: In technical interviews, candidates are usually asked to think aloud as they go on solving the problem on whiteboard so that the interviewers can follow the stream of logic being employed and how well the candidate is able to articulate them. This is called concurrent think-aloud and is known to distract many candidates from the problem at hand – thus leading to poor performance. The researchers suggested allowing candidates to first solve the problem privately, and then meet the interviewers to explain their logic. This is retrospective think-aloud
- Evaluate what kind of stress is necessary for the position being filled in: For some roles, handling stress is a part of the job. However, the interviewers must assess what kind of stress is involved in the real-life scenario. If all that is needed is completing tasks within strict timelines, then it is “time pressure” that the candidate must learn to handle – but not “performance anxiety” as is created in the interviews. Some companies are already trying to address this by providing as much detail as possible before the interview. Honeycomb, a women-lead DevOps company, even provides the questions in advance to eliminate undue stress.
- Provide accessible alternatives so that the interview is less intimidating:Simple steps like a warmup interview where candidates get an opportunity to familiarize themselves with the setting and the interview format, and clarify doubts regarding the process – but are not evaluated – can go a long way in mitigating stress. Other such measures can be “free reset” – where the candidate can ask for a different problem without getting penalised; “partial program sketches” – in which participants do not write codes from the scratch but merely fleshes out an initial skeleton with the partial solution; or “familiar affordances” – where the candidate encouraged to use familiar resources to perform the task, like allowing the use of paper-and-pencil instead of the whiteboard, if that is more comfortable for the candidate.
- Consider impacts on talent and diversity: This is one subtle issue that the study dares to raise. Being anxiety-prone or nervous by nature is no crime – an estimated 40% of the adult US population suffer from performance anxiety. And there are so many forms of neuro-diversity among the population – like dyslexia, autism, or anxiety. That does not mean that they are technically incompetent, and most of them are perfectly capable individuals with therapeutic support. Moreover, there are job seekers who come from disadvantaged backgrounds and hence can feel tense during a tough evaluative process – although they might have all requisite education to crack the test. This is where the researchers put forth the crucial question: should recruiters continue using a hiring process that inadvertently excludes such a large population of otherwise-qualified candidates, only because they happen to perform poorly due to performance anxiety? Would it not go against the grain of diversity and inclusive hiring practices that the corporate world is eagerly embracing?
The study is indeed pioneering in many respects and provokes pertinent questions that are central to the programmer recruitment process in the IT industry. The researchers admit in conclusion that the study has its limitations as it only examined one coding challenge, and within participants from just one University. However, they hoped it would inspire further studies on larger scale, and industry recruiters will actively participate by experimenting with alternative hiring procedures. Incidentally, companies like Microsoft, Google and CoderPad are already collaborating with North Carolina State University in search of better evaluation methods.
Acknowledgement:
MahnazBehroozi, Shivani Shirolkar, Titus Barik, and Chris Parnin: “Does Stress Impact Technical Interview Performance?”; ESEC/FSE ’20, November 8–13, 2020, Sacramento, CA
