Software Developer Technical Tests: What are they and why are they a problem?
A technical test is part of the job interview process a software developer has to complete before progressing to the next stage of the interview.
Interviewing for a software development role often consists of at least three stages. In recent years, candidates will likely have experienced an interview process such as that described below:
1. The Screening Call
The candidate will have an initial phone call or face-to-face interview with a member of the company’s HR team.
This is a general conversation to ask about your background and experience. You may be asked if you have experience in a particular programming field or with a specific Application Programming Interface (API) or Software Development Kit (SDK) but usually nothing too technical.
You will also be given a general idea of what working at the company could be like. This is a good opportunity to ask about standard working hours, flexible working options and other company operations.
This stage could take anywhere from twenty minutes to an hour, or perhaps even longer.
If both parties are happy to continue, the interview process can move on to stage two.
2. The Technical Test
The candidate is assigned a technical test to work on at home. This will be a programming task of some kind.
It is unlikely you will have any direct contact with the actual development team during this stage. Part of the purpose of this test is to filter out a large number of applicants that don’t already very closely fit what the development team are currently doing or looking for. Generally, most developers — being inclined to introverted personalities, and already feeling overworked — really do not want to waste any time talking to candidates if they can help it.
The companies you are applying to often aren’t interested in how much overall experience you already have, nor that you are willing and able to learn whatever it is they are currently working on, they just want an ‘easy fit’ candidate that they don’t have to train up and that they can manage as little as possible. The technical test is thus used as a tool to determine if a candidate will start producing meaningful work in as short a time as possible after being hired.
Once the test is completed and returned to the company, the development team will assess the task to decide if they want to continue any further with the interview process.
If the candidate is deemed to fail the test, they may only hear back via an email from the company’s HR team. The interview process can be terminated at this point.
If the candidate passes the test, HR will either email or phone the candidate to make an appointment for the next phase. The interview process can then move on to stage three.
3. The Real Interview
This meeting is arranged with the development team lead and may include some other less senior members of that team. This can be an onsite face-to-face or, as more recently has become the norm, a remote video conference call.
One or more of the development team will question the candidate about their technical knowledge. They may ask about the technical test itself, such as why you went about a problem in a particular way or to critique areas they felt you were lacking in — which you’ll have to constructively respond to. You could also be given more theoretical problems to solve and have to discuss how you would tackle them and justify your reasoning.
This part of the interview is also a personality test. The development team members you talk to will want to see if you gel well with them or not. Your response to criticism and confrontation will also be measured by team leads who prefer that management style.
This stage, usually the last of the interview process, can take at least an hour or two to complete.
What Can a Technical Test Involve?
For at least the past two decades, software developers have been asked to carry out some kind of technical test as part of an interview process.
The manner of testing can vary, it may depend on how a company prefers to assess candidates but also whatever the current industry trend is. That said, there is no single ‘industry standard’ on the correct way to write code or how to implement a structured solution or framework when building software. As such, how you are tested can come down to the personal preferences of the lead developer for the development team you are interviewing for.
A technical test can be the worst part of an interview process for a developer. Problems that would normally be solved on a day-to-day basis can completely stump a candidate when ‘put on the spot’, under pressure and with added time constraints. Anyone who has had to do an exam of any kind, for education or professionally, will have had a similar experience.
Failing at this stage will often prevent any further progress getting potential employment with a company, even when the candidate is more than skilled enough to carry out the expected work the actual role would have required.
In my experience, from attending interviews throughout my career, a technical test could have consisted of one or more of the following:
1. The Face to Face
Being asked technical questions during an onsite interview with an existing member of the development team.
This is mostly conversational only and meant to probe your general knowledge or perhaps ascertain how you might try to tackle a specific problem. You may be asked to draw some diagrams on a white board to show an application framework design for a solution you have suggested.
There is usually no code to write for this type of test.
2. The Exam
Having to fill in one or more question & answer forms, presented in an exam-like format where you are allocated a certain amount of time to write down your answers.
This may involve hand-writing some basic code, answering multiple choice questions or explaining your understanding of a programming concept. The candidate would carry out the exam on-site as part of an interview process that could take several hours of the day.
There can be a small amount of code to write for this type of test.
3. The Take-Home Development Task
Being assigned development work to carry out at home.
This could be as simple as writing a few functions to perform a task or designing and developing an entire working application, mobile app or web service.
A task requirements document (if provided) should explain all the work you are expected to carry out, what the code should do and how a working application should look and behave. Sometimes you’ll be given additional requirements such as ‘how’ to carry out the task, such as using a specific API or SDK or including unit tests (additional code you have to write that tests if other code you have already written continues to work as expected) etc.
Time allocated for such a task can vary from a few hours to a few days. Once the candidate has completed the assigned task, they are required to return all code (workspace and project files etc) to the employer via email attachment, cloud file service or online source code repository such as GitHub.
Working on this task will likely involve writing a large amount of code.
The candidate will probably have to allocate additional time for researching concepts they are not already familiar with, before they can even start coding a solution.
This third option now seems to be the most favoured method of assessing a candidate’s technical ability.
Some Negative Aspects of Take-Home Technical Tests
To a certain degree, we can appreciate the need to assess a candidate’s technical skill. The lure of higher-than-average salaries probably attracts a lot of interest from completely unqualified candidates. No company wants to waste time and resources hiring someone who really has no ability to do the expected work.
Considering technical tests from the perspective of a software engineer (and with no small amount of previous experience), there are some really quite serious problems with how these are designed and the burden they place on candidates.
These are some of the most obvious issues with take-home technical tests:
The candidate is assumed to own all the necessary hardware and software to work on the task.
The computer equipment you have access to in a work environment is often completely different to what you have at home.
At work, a developer’s PC or Mac is likely equipped with a powerful processor, 32GB+ RAM and a 1TB+ SSD. Any necessary Integrated Development Environment (IDE) software and other development tools are already installed and carefully configured. The developer could also be utilising two 24” HD (or better) monitors to work across. A powerful machine and supporting setup are usually required for intensive development work.
At home, the same developer might only own an old laptop with only 4GB RAM and a standard hard disk with only 250GB total capacity. The laptop hard drive might be filled up with MP3’s and other junk they’ve accumulated over the years. The screen could be quite small at 15” or less with lower than HD resolution support.
That developer might also prefer a complete separation of work and personal life. As such, they don’t necessarily have any development software already installed on their laptop as they wouldn’t normally do any programming in their personal time outside of work.
When a candidate is asked to do work on a technical test at home, they might not have the necessary hardware at all. Even if they do, it could be completely inadequate to practically carry out any work on. A home computer with a slow processor, hard disk and limited RAM will drastically slow down any normal pace of software development. An underpowered computer can even be more prone to lock ups, crashes and restarts as it struggles to cope with development software that is very demanding of system resources.
If we assume a candidate does have even the minimum specification computer to work on, they will still need to install all the necessary IDE software and supporting tools. Not all development software is free, and that which is can require huge downloads, taking hours over slow internet connections. Once installed, development software has to be setup and configured to operate in a fashion the developer is used to, all of which can take many hours.
Given the above, a candidate may be at a severe disadvantage when compared to other candidates, not because of their level of skill or experience but simply that they don’t own a powerful computer and supporting setup to work on at home effectively.
The task requirements document can be too vague.
The specifications for the task might consist of only a few paragraphs of text. This can result in a lot of ambiguity, particularly where a user interface is required, or complex back-end processes built.
In a normal work environment, a development task might have to be discussed by a team after an initial brief and inadequate specification is written. This discussion can ‘force out’ diagrams of what is actually needed and any specific implementation requirements.
The candidate can end up implementing a large amount of work to cover all the possible solutions the employer might be looking for as they don’t want to appear unknowledgeable in any area. The end result may even be nothing like the employer really expected to see.
There is no mention of the expected framework structure or design pattern to be used.
There are many ways to solve the same problem in software development. How you go about structuring code and implementing your solution often comes down to your existing experience and where you have worked before.
A self-taught developer is likely to have evolved their own approach to programming, eventually settling on a comfortable system that allows them to tackle most problems.
A developer who has worked within a company has probably been forced to adopt an approach to programming which existing development teams have already established. Adhering to a company’s coding conventions is done to try to get everyone developing in a similar manner, so one developer has an easier time trying to understand another developer’s code. This can be a company-wide policy, but it can also vary between different programming teams within that same company. It can just come down to the personal preferences of a development team lead.
Unfortunately, this learned or adopted behaviour can often be completely at odds with how the next company or development team want things done. For a developer trying to progress their career by moving from one team or company to another, each transition can be extremely jarring as they are forced to ditch their conditioned approach to programming and learn the new system being imposed. Think of it like having a car crash, losing the ability to walk and then having to self-rehabilitate so as to teach yourself how to walk all over again from scratch.
In trying to establish some sort of industry recognised approach to software development, we have something called design patterns. These are generally well documented methods of structuring code to create applications and services that developers from any industry can learn. A choice of design pattern, such as Model-View-Controller (MVC) or Model-View-ViewModel (MVVM), defines how your overall software solution is structured.
Though design patterns are helpful to know, there is no particular one that is better than another. Similarly, development teams can ignore the existence of any recognised design pattern and carry on with their own approach regardless.
Again, we are back to the truth of there being no single industry standard on the correct way to write code, implement solutions or choose a design pattern.
The development team you are completing the test for will have a very specific preference for how they expect you to write code. As a candidate you can’t know what that is in advance.
The consequence of all this is that the candidate can end up trying to implement a ‘bit of everything’ they know to try to show they do have an understanding about various design patterns, building application frameworks and implementation techniques. This requires a lot of extra work, resulting in an overall project that can appear disjointed or even messy and not representative at all of their usual standard of work.
The task may involve using one or more third party API/SDKs that the candidate has no prior experience of.
Regardless of how much development experience you already have from working in your past roles, when you start looking at new jobs there will likely be a list of requirements you have absolutely no experience of.
As developers, we have to learn new things all the time. Programming languages come and go whilst new devices can require learning completely different approaches to software development. Some development work can require learning something new every week, or even every day.
Learning and understanding new things requires a lot of reading, research and often experimentation. Learning how to use a new API or SDK along with all its quirks and pitfalls can take at least a few days if not weeks or even months.
If the assigned task requires any work the candidate has not already had prior experience with, they will have to invest a large amount of time to learn and understand all that before attempting to implement a solution. This is rarely factored in when an assignment has a time limit. The expectation seems to be that you should know it all already.
The unfortunate reality of technical tests is that they can be designed to eliminate candidates that don’t already know the API/SDK being used. The company or development team is specifically looking for someone with that experience already, they don’t want to spend the time or resources training a new developer hire to learn those skills if they can help it.
The scope of what is being asked is excessive.
To the untrained reader, a brief task specification document (e.g., two to three paragraphs) may give the appearance of a small amount of work being asked for.
A developer / software engineer has to read between the lines, extrapolating on what may appear as a simple requirement statement to grasp the full scope of what is really being asked for.
An experienced developer will also want to build their solution in a well-engineered and documented manner with an easy-to-understand code base. Just ‘throwing something together’ will completely go against hard learned engineering principles and conditioned programming methodology. Well implemented code requires a lot of time and effort.
The task specification document may also stipulate that the developer demonstrate use of ‘Thorough Error Checking’, ‘Unit Testing’ and ‘Use of GitHub’. These are not trivial things to do, and even for a simple minimum-viable-product (MVP) it can add as much as an extra 40% to the workload.
On a personal note: I’ve generally found that creating an MVP whilst employed within a company will take at least one to two weeks if not longer. However, I’ve recently experienced technical tests asking for an entire MVP to be developed in just a few hours or a couple of days at most.
The candidate is very likely to have to work many hours or even several days to complete all the expected work. All such work has to be carried out in the candidate’s own personal time and without pay.
The time allocated to complete and return the work is unrealistic.
This issue is strongly related to that described above, where the actual requirements of the task require far more work than the company has given you an estimate for.
This will be of no surprise to anyone with prior industry experience. How many times have you or your team been asked by management to implement a software solution for something in ‘a few days’ or ‘just a week or two’ when it actually took several months of hard work?
A candidate may only be given 1–3 hours to complete work that may normally take an entire working day or longer. If the allocated time was 2–3 days, then it’s more than likely to mean a whole week of actual work to fully implement.
If the candidate does try to stick rigidly to the stated time constraints, the work they produce is very unlikely to be representative of their skills, knowledge or ability.
Working on a technical test requires sacrificing a lot of personal time.
Given the aforementioned points, we can see many hours of work have to be put in to complete a technical task assignment for just one job application.
It is more than likely a candidate will have to apply to numerous companies before being offered a position, having to tackle a separate technical test for each job application.
A candidate may still be in full time employment or have other obligations when applying for a new job. As such, working on multiple technical tests requires careful scheduling around their existing work and personal lives. This is a process that can go on over the course of weeks or months — however long it takes to get offered a position.
Working on any technical test will require a candidate to sacrifice their evenings and weekends (assuming that even fits within the time allocated), missing out on time otherwise spent with family, friends or other personal interests such as exercise.
Your past experience, achievements and work resume have no value if you fail the test.
In your current or past development role you may have worked on some very difficult problems and implemented some amazing solutions. Most of that work was probably far more complex and challenging than the technical test you have just been assigned.
You probably had to read a few books, online articles and experimented with a lot of code before you completed any project for a company you’ve previously worked for. Whatever problem you were ever given, you solved it through self-learning and hard work. After all, your previous employers never provided training for developers to ‘learn new things’, you were just expected to get on with whatever you’ve been asked to do.
The technical test you’ve been given requires existing knowledge of some database, RESTful API or other SDK you’ve never used. You may have even stated this up front when applying for the role, after all you are used to picking things up fairly quickly so anything you don’t already know can be learnt. You end up burning a lot of the allocated time for the technical task on learning ‘that new thing’. It didn’t leave you much time to properly implement everything else as thoroughly and well documented as you usually would. There wasn’t enough time to create multiple projects in your workspace to separate out all the different front-end and back-end components. There wasn’t enough time to create the refined, polished and user-friendly UI you would normally design and create. The end result is something that ‘just about works’ but you’ve had to throw it together in a manner you would never have done while employed in a previous role.
After handing back the project to the employer, you eventually get an email telling you you’ve failed the technical test.
You may get no further information as to why you failed the test. You’re no wiser to what the company or development team were really looking for in the test so the whole thing feels like a complete waste of your time.
On the other hand, perhaps you do get some feedback. If you’re lucky you might even get some positive comments about what you’ve done. Unfortunately, it’s more than likely the reviewer has taken the opportunity to rip to shreds your work, seemingly relishing in pointing out all the flaws, omissions, wrong choice of design pattern or coding convention and so on. You may have already anticipated some of the criticism, given the limited time you had to work on it and the need to take shortcuts or makes guesses to the actual requirements as you’ve been forced to work on the task in a manner you never normally would.
Ok, so you’ve failed the test but surely the employer has seen your work history on your CV/resume and that you are an experienced, knowledgeable developer who’s worked on a lot of projects over your career?
Perhaps the company’s HR team have read some of your CV, but it’s very unlikely the development team has bothered. You didn’t pass the test so why waste time reading a candidate’s CV? After all, that’s why they put the test in place — to filter out as many candidates as possible so they are only left with the ‘ideal fit’ developers to actually interview and that they don’t have to train up if hired.
My perception is that companies do not want to spend their own time or money training developers to do the work they actually need. The preference appears to be to hire a developer from another company, where the experience and training has already been acquired (usually by being self-taught on the job) so the skills are effectively obtained for free.
What you are being tested on can become completely irrelevant within a few months after being hired.
A technical test can be specifically designed around the type of development work the company are currently doing. Due to demands, more developers are required to work on an ongoing project to speed up development, so the test is tailored to the specific skills currently required.
For example, some company — let’s call them ACME — are developing an iPhone app that requires the use of onscreen maps to show the user where the nearest electric car charging station is. We’ll call this fictional app ChargerMap. In this case, Apple’s MapKit framework would be an obvious API to use so the company’s developers will have had to start learning how to use it when they began the project and as they continue to work on it.
Let’s say a candidate applies for a job at ACME, passes all stages of the interview, teaches themselves MapKit whilst doing the technical test (or perhaps already knew it beforehand), and is then offered a position on the development team. Three months after the new hire starts work, ACME decide to abandon development on the ChargerMap app for whatever reason. The new hire is now in a position that they’ve been employed to do a job that no longer exists.
A new iPhone app development project is spun up within ACME to replace ChargerMap, this one is all about streaming videos from the internet. We’ll call this fictional app VideoAce. The project requirements have already been set, a basic design proposed, and all the developers are assigned tasks to start work. At this point any existing knowledge of MapKit is pretty useless.
ACME didn’t plan for training when switching to the new VideoAce project, after all they have lots of employed developers who can do anything they tell them to do. All developers — the new hire included — will have to start teaching themselves how to use AVKit (Apple’s API for playing video content) from scratch at the same time as starting work on the new project.
This approach to starting a new development project can result in a lot of trial and error as developers try to understand AVKit and get things to work at the same time. Taking some wrong directions in development also results in a lot of code being written, binned and then replaced as the right way to use AVKit is teased out i.e., figuring out the nuances, working around the design flaws and bugs etc. This is all a lot of hard work for developers. It would have been a lot easier if they’d been provided proper training on how to use AVKit before starting work on the new project.
Twelve months down the road, VideoAce is sort-of working with only its basic feature requirements in place. During this time some developers have left the company, they weren’t happy ChargerMap got scrapped after all the work they put in on it and didn’t enjoy working on VideoAce at all. ACME now need to hire more developers to work on VideoAce. A new technical test is drawn up by some of the current developers to try to filter down candidates to only those that are already very proficient in using AVKit.
And so, the cycle starts again.
In Conclusion
I have tried to write this article with some impartiality. However, I’ve probably been unable to hide the fact that I am not at all a fan of take-home technical tests.
If we were to apply the same type of interview testing to another industry, would we consider any of the following a reasonable thing to ask a candidate to perform as unpaid work when applying for a skilled job?
· A surgeon to demonstrate removing an appendix from a patient, without access to a proper surgery room or equipment, and within a time limit.
· A car mechanic to service and MOT a car, using their own tools and car bay which is assumed they have access to at home.
· An IT support engineer to dissemble a server they’ve never seen before, replace a broken component and then rebuild it all. The server should then connect back to the unfamiliar network without any issues, all within an arbitrary time limit.
· A pilot to fly from the US to UK and back within a set amount of time. The weather will be perfect for the duration of the flight and no control tower redirections will be necessary (all assumed to be the case when the pilot tries the flight). The pilot should, of course, provide their own plane.
Yes, it is all meant to sound quite ridiculous.
In my opinion, developer take-home technical tests should be abolished.
These are some key reasons why I think technical tests are a failure for candidates and companies:
1. They place an excessive burden on candidates, necessitating many hours of unpaid work.
2. They have a negative impact on a candidate’s personal life and knock-on effect to family.
3. They unfairly disadvantage those without adequate home computer setups.
4. They do not allow a candidate to demonstrate their true experience, knowledge and skills.
5. They do not reflect how a candidate would normally work on a day-to-day basis.
6. They indicate a very short-sighted strategy for hiring that will ultimately hinder future development projects the company may wish to start.
7. They prevent companies from hiring many skilled, knowledgeable and experienced developers simply because those people aren’t very good in ‘exam’ specific conditions.
One headline I have seen repeatedly in the news for the past decade or so is that there is a supposed shortage of skilled developers in the UK. I don’t think this is really the case at all. Companies quite probably have a shortage of employed developers because they place too many absurd obstacles in the path of those seeking employment. It has also been my experience that companies do not manage, support or look after their developers well at all, as such retention levels are typically very low.
A single better alternative to technical tests would require another whole article to discuss.
My simple suggestion would be for companies to start investing in properly training developers with the specific skills they require rather than expect them to have those skills already and then refusing to hire them when they don’t.
When I graduated from University over twenty years ago and started looking for my first software development job, the most common response I got for any submitted application was “Come back in a year or two after you’ve gained the necessary experience working in another company.” Two decades on I’m finding things haven’t changed at all.
If you are a developer, have you had similar experiences or thoughts on this subject?
If you are a project manager, part of an HR team or recruitment agency that is finding the use of technical tests obstructive to successful hiring, what ideas do you have that could provide a better solution?
I would like to see a push for changes in the industry, relegating the travesty of developer take-home technical tests as a thing of the past. If you agree, please raise this issue with your development leads, project managers and HR teams.
Thank you for taking the time to read this article.
