Codility – First Thoughts

So a potential client asked me to take a test at the Codility site. This is a site that does an automated test of candidate’s coding skills. It basically asks you to write a function to perform a task, then runs unit tests against your function.

They provide a demo, this is a very easy task but has a vague requirement that the code should be ‘efficient’. This requirement is so nebulous as to be useless. The requirement was in bold, this doesn’t make it any less vague, it just makes it bold and vague

The demo task has one simple and obvious solution, and more complex solutions that would be faster on larger data, but slower on smaller data sets.

Now, experienced coders know that the right approach to optimization is, to quote Brian Kernighan,“first make it right, then make it fast”, on the grounds that it’s easier to make right code fast, than fast code right. Or, as Donald Knuth put it: “Premature optimization is the root of all evil”.

My score for the demo was 100% for ‘correctness’ and 25% for performance – giving an overall score of 62%. From the names of the test data sets (you can’t see the actual data), it looks like they tested it with worst-case data for the obvious solution.

When I took the test itself, again one task had a vague “efficient” requirement. This task involved allocation of seats on an plane, and had an argument for the number of rows of seats in the plane. The instructions stated this parameter could be up to 100,000. Again, the task had a simple brute force solution, and a more complex solution that was faster on large data sets.

Naively I assumed the rows parameter would not often be the maximum value, on the grounds that a plane with 100,000 seats would be 55 miles long, and it could take 16 hours to walk to a seat.

This solution also got a bad score for performance. It seems “efficiency” is a code word for “best worst case performance” or perhaps “lowest time complexity”.

I have to agree with what this blogger said:

Overall, I’m still impressed by Codility. If coders know what is expected of them then it can be used as a good test. But the tasks are considerably harder than a “fizzbuzz” style test – they are really lateral thinking puzzles rather than coding tasks. Being familiar with the language is necessary but not sufficient to pass, as this programmer found out.

Assume the test will use the maximum array sizes with worst case data.

Use an IDE to write your code. While Codility has a built-in IDE, it does not have much debugging ability.

Write test cases for your function, assuming it will be tested with max and min values, including empty arrays.

The Codility tasks usually have a simple brute force solution, and a more complex solution that is faster on large data sets. They want the more complex one. If you find yourself writing nested loops or multiple functions, You’re probably doing it wrong.

For a client or an employer, it’s best to use Codility as a simple “fizzbuzz” test and set a low pass threshold. When two engineers can both code a function 100% correctly according to reasonable interpretations of the requirements, and yet one get 100% and end the other 60% because they interpreted “efficient” differently, it has a high margin of error.

Codility also has a series of coding exercises, these work in a similar way to the tasks – you can test your answer for correctness and performance, except with the option to retake the task.

These are a good way to test your coding ability, and I will post a set of solutions to a sample of these exercises soon.