I’ve been interviewing, and one thing that I have found interesting are take home problems. I’ve seen them in so many different forms, and I have done most of them. I did a lot of interviewing at Google, and I grade the take home problem that my current company gives. I wanted to give an overview of the different things that I’ve seen, and then my opinion on what companies should and shouldn’t do.
The first big variable that I have seen is the build environment. There are two broad categories: A controlled environment with a web IDE, or letting the candidate do it on their own. Each type has pros and cons.
These environments lend themselves to shorter timescales and easier grading of the solutions. Because the environment is already set up and ready to go, the candidate can dive right in on the problem. There’s no need to set up the build tooling to go with a problem, or set up version control (the web client can keep an edit history). This means that the candidate needs to allocate less time to the problem. It is also easier to grade the test, as it was written in the same environment where it would be run.
A big con is the lack of a REPL, and the fixed set of libraries. Many languages allow you to interact with the code by writing snippets of code to be evaluated and printed. This is really useful for checking your code before writing a formal unit test. Web environments usually don’t allow this, and they only have a run and submit button. In some ways by fixing the library and hiding the REPL, you are hobbling a language that you are asking the candidate to use. One example problem I solved, I wanted to use NumPy, but it wasn’t available. This is completely unrealistic. There are very few professional circumstances where you would choose Python but prohibit NumPy
I had a really negative experience with Codility. Maybe it would be better if the company had supplied more example tests, but if one of your solutions fails, you don’t have the ability to go back and edit the code. I had over an hour left on my timer. This is completely unrealistic. In ACM competitions for example, if your code fails, they tell you that it fails, but not why. You can then work on it and resubmit.
This gives the candidate much more freedom in terms of libraries and using the REPL. It also introduces many more vagaries into the problem:
I will suggest my answers to these problems. They may not work in a particular case, like if you want to give a Windows or Mac specific take home problem, but in the modern world, enough of the take home problems can be run this way that this advice should work for the overwhelming majority.
Code should be submitted via a webpage as a tarball. In addition to
any other documentation or build instructions, the code should contain a
Dockerfile. This solves both the problem of how to run the code, and the
problem of whether it will run in your environment. Tell the candidate
about the size of the VM that will run their code. Upon submission,
automatically run the code by creating a VM for security, and then in
the VM
docker build -t <tag> && docker run --rm -i tag < input > output
A second best option for this problem is to rely on a language specific package manager. Haskell has cabal, NodeJS has npm, python has pip. This isn’t as repeatable as Docker however. If it’s automated, or if you have at least reasonable feedback loops on code that won’t run in your environment, this isn’t a dealbreaker.
Only one take home problem that I was given specifically asked for unit tests. Asking for unit tests work better in an uncontrolled environment. In a controlled environment, you are essentially supplying the unit tests. If you are doing this, please supply a variety rather than a single example. There is a tradeoff in asking for unit tests. It does make the problem more like real work. However, it makes the problem longer, and there is a limit to how much time it is reasonable to ask a candidate to spend on a take home problem.
In addition to asking for unit tests, I was asked that my code be well documented. I was writing Haskell for this particular problem, so I added Haddock comments on all top level definitions and all arguments. One of the problems with asking for documentation is that in many ways, the problem statement is the documentation. What’s left to document is basically comments about the code and its internals. The externals are specified by the problem. As with unit tests, asking for this works better in an unconstrained environment.
There are two kinds of feedback that a candidate needs: Their code doesn’t run in your environment, and a complete grading of their code. Ideally the running of their code in your environment would be automated, so they can resubmit quickly. Retain their original solution so you can grade them on what changed. Changing the build files is better than changing the actual code. If it’s not automated, you should ensure that the recruiter knows how to run but not grade the solutions, and that the candidate gets timely feedback about whether their solution worked in your environment.
Do you want the candidate to write the code in whatever language they are most comfortable with, or do you want them to use one of the languages that your company uses? My preference would be to allow the candidate to choose, but that can be overridden by circumstances. At Google, we said that interviewing was trying to give the candidate an opportunity to shine, not to plumb the depths of their ignorance. If you will be teaching the candidate the language (you use a niche language), then let them solve it however. If you want to verify that a candidate is comfortable with the language they will be using if they get the job, go with that. If you go this route, pin it to exactly that language.
If a candidate’s code runs successfully, and meets other criteria like performance, then you should review it. If the code passes review (review should be liberal due to time constraints), you could discuss the code in at least one interview. If the problem is smallish and being used as a standalone screen, then you don’t need to discuss it in an interview. Skipping the discussion makes more sense if you are using a constrained, automated environment.
There are several reasons to give candidates a bounded amount of time to complete the problem. For hiring efficiency, you want to give them a bounded amount of time to begin the problem. You don’t want to let a candidate string you along. Two weeks to begin the problem is probably a maximum. The constrained environments automatically take care of timing the candidate by giving them a fixed amount of time to solve the problem.
For an unconstrained environment, you should give the candidate an expected amount of time, and a bounded time window. If you think the problem should take 4 hours, then you should tell the candidate this. You should have them start when they feel they have that much time available, but you should give them a bigger window to submit their solution, like 24 hours. You can request that they submit their code as a git repository for timestamps. If you want timestamps, have them create an initial commit as soon as they start. Yes a candidate can game this, but most won’t.
The reason to give candidates a small amount of time from receiving the problem to submitting their solution is that otherwise it can be difficult to tell a candidate that is in demand and therefore busy, from a candidate that took the full week to actually solve the problem. Git timestamps may give an indication, but a candidate could have worked on it in blocks. By requesting that the candidate block out time and then giving them the problem, you set a tighter bound on how much time they can actually spend.
How do you know how long a problem is? It may take an engineer from your company, who is familiar with the problem, less time than a quality candidate. The best data would come from people that you know are quality, but don’t work at your company. Here’s an idea: use linkedin and have your employees offer to pay people they felt were good engineers at previous companies to solve the problem. Have the candidates sign an NDA. If you get a few to solve your problem and report how long it took them, you can get a feel for how long the problem takes. You can also keep data on how long it takes candidates, but note that this doesn’t track people who gave up. Even with the biases, tracking the data can still be useful.
In 2019 I was given a 3-part take home problem that I just didn’t do. Two of the parts were super simple fizz-buzz style problems. The third was implementing a program to call a REST API. The first two are redundant. If you can do the third, you can program. The third is redundant though too. Python has a library called decorest That already solves this. Even worse, in 2010 I is was fixing a different libary to do this for GWT
What you pick as a problem says something about your company. Make sure that you understand what it says and that it is a message that you want to convey.
Yes, I think that the ability to do it on your own schedule makes candidates willing to spend a little more time on a problem than they would in an interview. I was willing to spend 2 nights in a Hotel to interview at Google, so this is a much lower time commitment. My personal preference is for time boxed unconstrained problem, with a free choice of languages and a follow-up interview. One company said they did a pair programming exercise as the follow-up with the interviewer actually writing some code. I didn’t get to try that, but I was curious about how it would go. I feel that my abilities have shown best in doing this particular kind of problem.