Why run technical interviews?

It's tempting to think, especially early on in a startup journey, that the reason we run technical interviews for Software Developers is to determine whether or not they can "do the job". What this presupposes is that the ability to work with software is a binary skill that is either present or absent. But this is a strange perspective. After all, one assumes that the ability to perform basic plumbing tasks (like, for example, replacing a kitchen sink) is a fairly binary skill; yet how many of us would bother to give a prospective plumber a practical exam before hiring them? Indeed, how many of us would do any kind of due diligence on a plumber before hiring them? How many of us get three quotes and then hire the cheapest? How many of us don't even get three quotes...?

There are three important reasons we might attempt to technically assess someone:

1. We want to know the degree of competency they have
2. We want to know how they might be to work with
3. We want to get an idea of their relative strengths and preferences

Degrees of competency

Intuitively we know that there are degrees of competency in most fields, although the degree to which we care about that depends on the context. In the plumbing example, we might not mind particularly how competent the plumber is who replaces our kitchen sink. On the other hand, if we are having an entire luxury bathroom installed in our new extension, we might feel a little uncomfortable to know that our plumber had only been at it for four months after switching careers from high frequency trading in The City (hey, it happens!) One dimension which obviously applies here is the level of risk — usually financial risk — associated with the work. If the sink replacement goes wrong, we probably just lose the labour cost of the shoddy workman, possibly a few damaged parts. If the new bathroom goes wrong it could set us back the cost of our next holiday! First world problems!

Quotes for plumbing work tend to follow the market, rather than being strongly associated with competency. That's why it's usually more reliable to go by recommendation than by cost when hiring tradespeople[*]. Hiring Software Developers works a little bit differently, especially for permanent hires[†]. "The market" tells us that when we hire a permanent Software Developer with 1 year of commercial experience, they are worth £x; and that when we hire a permanent Software Developer with 10 years of commercial experience, they are worth £y — where y >> x.

Of course common sense, if not prior experience, tells us that years of experience isn't a reliable measure of competency [TODO: ref] and so we can therefore infer that the competency of individual Software Developers with the same years of experience must vary. If we're going to all the trouble of compensating someone £y, we'd much rather make sure that their true level of competency aligned with that level of compensation.

Early on in a startup, it's uncommon to be able to exactly specify the competency level you require relative to the cost. "As good as possible and as cheap as possible" seems to be the starting point. What this implies is that making the first technical hire or two is actually quite nuanced.

As a team grows, however, budgets tend to be better understood and the focus on competency can be more structured. An excellent tool for mapping out team members' competencies is the competency framework, of which many volumes have already been written; and my advice would be that the best time to construct such a framework is slightly before you think it's needed.

Assuming that you know at least the rough level of competency you want to hire, then, the first reason for technical assessment at interview is to gauge the competency level of your candidate relative to target (and to cost).

What they're like to work with

Software Development is by nature a collaborative exercise most of the time. If you meet a developer whose favourite way of working is to shut themselves off from you and the world — beware! Unless you happen to know of some magical process — other than talking — for communicating exactly what the business's context is, and its needs are, you are almost certainly going to end up in the wrong place with someone like this. In any case, it's unlikely that just one developer will do — you're probably going to need a few — and so all of your developers will need to communicate with each other. More than that, it would probably be a very good thing if they were happy working together too. After you've experienced the pain of making those first few technical hires, you're going to want to do everything you can to avoid unnecessary churn in your team, and there's nothing that churns a team up more than an a***hole in the ranks!

It's important, therefore, to see your prospective developer in action, and collaborating with others. Do they share a keyboard nicely? Do they treat others with basic respect and dignity? Do they listen carefully and deeply? Can they communicate abstract ideas clearly and concisely? All these and more are critically important parts of the job and simply must be assessed as part of the interview process.

Strengths and preferences

The last reason to technically assess an interviewee is to determine (or at least estimate) the relative strengths of their competencies on a more granular level.

In real life, we all have different strengths and weakness. I might have a really sharp mind, for example, instantly being able to visualise a solution in my head; but I might struggle to translate that into working code. Or I might have an almost intuitive talent building visual interfaces, layouts and human interactions; but can't quite seem to wrap my head around databases. Or maybe I can work with small software modules like I was born to it; but when it comes to joining those modules together into big systems I'm all at sea.

It's important to understand the relative strengths of the individual members of a team, so that you can build a picture of how the overall team's strengths and weaknesses overlap. The worst possible situation you can find your team in is when zero team members exhibit a critical strength, because then the team is not functional. The second worst situation is when only one team member exhibits that strength, because it means that the team can operate at a functional level, but the unexpected loss of the one key person leads to a reversion to dysfunction. This is known as a key person risk. A well-constructed team attempts to ensure that at least two members exhibit critical strengths to maintain some sensible redundancy.

The other important reason for assessing strengths is to identify where a new team member might need support from others. Support is generally in finite supply, so whilst the team may be able to sustain one new hire who needs additional support working with front-end technology, for example, it probably can't sustain repeated hires with the same specific need for support.

A slightly different type of strength is a preference — how much an individual wants to do specific things. An individual might have excellent managerial skills, for example, but that doesn't mean they necessarily enjoy managing. Preferences can sometimes be brushed over in the short term: hiring for a specific assessable skill is easy to do. But chickens eventually come home to roost, and while skills can be improved with training, inclinations are more personal and more stubborn. Don't assume that your new hire will "come round" to wanting to manage people. If they explicitly tell you they don't want to do it, they probably mean it!

Of course, assessing preference is a slightly different process from assessing a skill. Usually it's easier to do: you just have to ask! Don't forget to do it though and, again, don't leave this part to a non-technical interviewer to cover off. What someone wants to do is just as important a part of your overall skills map as what they can do now.

Outline process

We've been interviewing Software Developers for years now, and the basic process has varied surprisingly little over that time.

There are two main technical stages:

1. The technical screen
2. The main technical challenge

The focal point of the whole technical interview process is the main technical challenge, so we start there, and work backwards

Main technical challenge — live coding

If you are hiring a Software Developer into an existing team, it's vitally important that the team is able to select that individual. This is one of those situations where leaders should really take a back-seat as much as possible. Assuming the team you already have is aligned to company culture, it will naturally select candidates who meet their needs and fit, given some minimal support and guidance. In our experience, it's unusual that leadership strongly advocates hiring a candidate that the team are uncomfortable hiring. The converse is more often seen, where the team advocates hiring a candidate but leadership sees a potential problem from a company fit position. Where these discrepancies occur it is an opportunity to explore some of the reasons — is there a misalignment between leadership and the team on company values perhaps?

The best way for the team to assess a new peer is to see them in action. For Software Developers, this means writing code in something resembling a realistic scenario. For this reason we strongly favour the live coding challenge. Live coding challenges involve solving problems in real time using standard tools (IDE and terminal) and guided by the interviewers. This contrasts to other sometimes popular methods of technical assessment such as take-home challenges (where the candidate is sent exercise to complete in their own time, often submitting the solutions electronically) and whiteboard challenges (where the candidate is asked to work through problems in the abstract without the use of a computer).

Our live coding challenges have typically run between one and two hours, and have taken varied forms depending on the context of the business in question. Common forms are:

• Taking an example application, critiquing and refactoring it
• Taking a half-completed application and writing the missing functionality
• Provided a set of raw data, writing some basic applications to process or transform that data

The common elements are typically:

1. If the candidate is expected to be competent with a particular programming language, use that language for the example application (we have hired into teams which used multiple languages before and in that case several versions of the exercise were created and the candidate allowed to choose their preference).
2. The application should be correctly sized to provide a good challenge, but also small enough to allow completion of functionality within the interview timeframe. Candidates feel much more positive about the experience if they are allowed to finish with something working!
3. There should be plenty of scope for variation within the brief, to explore areas of interest or potential concern. Similarly there should be lots of opportunity for discussion (e.g. "What do you think about the use of the strategy pattern here...", etc.)
4. Although interview exercises are usually fairly contrived, it's useful if the exercise reflects somewhat on the types of problems the candidate would be solving if they got the job. For example, if the team is developing a web application, a small full-stack web application exercise would be more useful than a straight up algorithm exercise. Conversely for a purely backend role it might be distracting to introduce UI elements into the exercise.

We usually split the technical exercise into two parts.

The first part asks the candidate to simply explore and attempt to understand the project. They should be able to talk with some confidence about what they see, dependent on the level they are being hired at. This is where the communication element starts to be assessed.

The second part is where the candidate gets to put the keyboard to work and make some changes. Here we're looking for fluency with the language, degree of systematic thought processes (do they think about what they're doing or just start trying stuff), ability to explain what they are doing (either as they do it, or immediately before or immediately afterwards — this can be a personal preference, but the important point is they should be able to explain themselves at some point!), etc.

This split helps ease the candidate into the exercise — starting to write code straight away can be daunting — and also helps the interview team to start thinking about directions in which they think it might be useful to take the practical stage.

Throughout the technical exercise, the interview team should be assessing team fit as well as technical competency. Of the two, technical competency tends to be somewhat easier to assess, since it's the team's bread and butter. Team fit assessment should take into account company culture and values (if available, it's helpful to have the team remind themselves of these before the interview), as well as the more intuitive answer to the basic question "Would I be able to work with this person?" More nuanced considerations are: "Is the way this person is behaving consistent with the level they are being hired at?" (here again, if a competency / progression chart is available, it's useful having the team review it ahead of the interview) and "How much support would this person need from the rest of the team?" or "How much support could I see this person providing to the test of the team?", as appropriate.

In the interests of shaking things up, getting a more diverse view and simply giving people a break, we often like to wind down the technical interview stage by closing the laptop, switching in a couple of different developers, and having a more general chat. A good rule of thumb is not to have any more than two or three interviewing developers in the room with the candidate at any time to avoid overwhelming them. The person "leading" the interview normally stays in to provide some consistency while the other one or two people tag out.

Technical screen — live comprehension

The most significant problem with the main technical challenge is that it's very time intensive to have three or four team members taken out of the mix and locked into an interview for two hours. Allowing for context switching, prep time and debrief time, it can be a hugely disruptive process, especially when the recruitment pipeline is healthy and the team is interviewing several candidates a week.

For this reason, it's desirable to only interview candidates in whom we have a relatively high degree of confidence that they meet the minimum requirements. That's not to say that the main technical challenge necessarily has a high pass rate; but if possible we would like to discount clearly unsuitable candidates before the main technical stage.

Many years ago, we developed an effective technical screening process which could be handled by a single interviewer in about 30 minutes over the phone (no, not a Zoom call, but an actual telephone, with a little curly wire coming out of the bottom of it connected to a box on the desk with some numbered plastic buttons).

This took the form of a comprehension exercise. We would email the candidate a booklet of code snippets shortly before the call, and then over the phone we would ask them some pre-arranged questions on the snippets. Since the roles we were hiring for at the time involved use of multiple technologies and languages, the booklet had a variety of numbered snippets, each intended to assess a different aspect. This meant that we could select a technology from the candidate's CV and ask them questions about that.

We found this technical screen exercise hugely valuable in weeding out candidates who might have talked a good talk around technologies but when pressed didn't have any degree of deep knowledge, or didn't really understand what they were doing. Introducing this screening stage dramatically decreased the number of candidates making it to the main technical exercise, and saved the rest of the development team a bunch of time.

Today we still do similar screening exercises. The only thing that tends to vary is how far in advance we give the code handouts to the candidates. Lately, we've tended to do video calls and screen share in real time. Partly this allays fears that the candidate might quickly "revise" things they are purportedly strong at ahead of time, but mostly it's just more straightforward and most candidates don't seem to mind.

Two other popular screening methods worth mentioning are:

• When a non-technical person (e.g. recruiter) asks the candidate screening questions, either assessing directly or taking notes for later assessment
• Using purpose built digital assessment platforms (e.g. HackerRank, Codility, etc.) that, to a large extent, automate the technical screening process

We don't advocate for either of these approaches.

We see numerous problems with the digital assessment platforms. While they certainly save time, we think they deliver a particularly poor experience for candidates, and anecdotally candidates don't like doing them. We've spoken to candidates who will actively disengage from recruitment processes which use these platforms, and in a demand-heavy market, they certainly have that option. Also, the assessments tend to be focused around algorithmic problem solving which, while interesting and occasionally, dare we say it, fun; tend to miss the point about what constitutes good software development skills. Finally, by automating the whole process you miss out on an immense wealth of human cues like how the candidate interacts on a personal level, how well they match company culture and values, etc. that you will otherwise have to assess at some later stage. Given the eye-wateringly high pricing on these platforms, we don't think they represent great value for money and potentially harm your candidate engagement.

Regarding getting non-technical people to do technical screening, it's not a terrible idea, but it's hard to understand how it can really work reliably in practice. Very little technical screening involves binary right/wrong answers to questions. Most effective technical screening is designed to assess depth of understanding. Asking a candidate "What are the five principles in SOLID?" might identify a number of candidates who know the answer, but there'd be absolutely no guarantee that any of them would understand how to actually apply any of them. Likewise, would we could easily penalise someone who couldn't remember the name of some of the principles, but did know how to apply them. A person with deep technical understanding can very easily and efficiently employ their own "bullshit detector" to questions like this. A similar argument applies to screening questions based on code snippets. Providing a non-technical interviewer a set of right answers doesn't afford them the capacity to give benefit of the doubt to candidates who might misunderstand the question or make some simple and forgivable assumptions. The only other alternative here would be making the code samples excessively trivial so that there was no room for misinterpretation on either side; which obviously limits the scope of what they can usefully assess.

The idea of the non-technical interviewer simply recording answers for later assessment by a technical colleague seems like a reasonable compromise, although we would question how much time this ultimately solves, and it still suffers from the weakness of losing signal in the re-transmission process.

Other stages

We've only covered technical interview stages here, but it's usually important to treat technical hires holistically and broadly in line with other hires. For that reason, it's not unusual that there is an initial meeting with someone from the People Team before the technical screening stage, and ideally there should be a final non-technical stage following the main technical challenge. This bookending does have the unfortunate effect of drawing out the process somewhat in comparison to other hires (sometimes by one or two extra interview stages), but our view is that in order to set the right tone with the candidate and within and without the engineering team, it's important that candidates feel like the technical assessment is only one part of a holistic hiring process. To do otherwise can potentially lead to undesirable feelings of exceptionalism in what is already a field that suffers from the tendency to separate itself from the rest of the business world.

Footnotes

• True story: years ago I got two quotes for an emergency boiler replacement job. The first was from a plumbing company I had used before; the second was a new recommendation, and he had an almost flawless rating over hundreds of jobs on CheckATrade. The first quote came in slightly over 3x as expensive as the second. I went with the recommended, highly rated, significantly cheaper second company and I've been using them ever since. They are amazing. Sometimes you don't always get what you pay for.

† Contract hires also show some degree of variation in rates, although it's less common to find that rates correspond to years of experience / seniority.