Nature of unit tests

Every person who heard about software development also heard about unit testing.
As usual, there are a lot of definitions of what exactly unit testing is, so let us choose one for the purpose of this article.
According to Wikipedia – unit testing is a software testing method by which individual units of source code are tested to determine whether they are fit for use.
Typically, unit tests (UTs) are automatically executed when new changes come.
Thanks to that developers know that they are not breaking anything with their changes.  

But is it true?

Unit tests are small, usually, developers are able to write them fast as there are tons of different frameworks. It is quite easy to test new changes, check corner cases, and achieve good code coverage (why it is not so important we will cover in some next articles). The issue is that UTs are checking only units. It can have different meanings as it depends on what unit means in the project. Sometimes it can be a function, sometimes it can be a class, sometimes it can be another piece of software that we would like to test, but always it is a very small one. If developers need to write tests (yes, there are projects where developers do not write tests – we will also back to this), they start with UTs, because they cover their changes without touching others, which is very convenient. 

And this is a reason why they can be useless. 

Consider the following scenario visible on the drawing: 

Every “Unit test” covers a small piece of software (Unit). In the worst-case scenario, every one of them can be written by a different person as they provide different functionalities. The system needs to ensure that the message that comes from the EXT-X interface will be properly manipulated to reach the EXT-Y interface. Here are scenarios that can happen and are hard to avoid. 

1. Different schema

In unit tests, normal practice is to mock (or stub) external interfaces (from the unit’s point of view). Based on the above picture – Unit test A will mock interface B from Unit Test B. Thanks to that, we are able to check what values are sent by A to B. Let’s consider the following example: 

No matter in what language you are writing (strong typing or not), you can easily make a mistake when writing unit tests for such API. Without looking at the code in the B unit, you need to make assumptions about what format you are sending a request. Is the timestamp format in ISO8601? Or maybe it uses Unix epoch time? 

Some of the above formats can be strongly typed, but in many cases, you will face simple types. And there is another risk… 

2. Out-of-limit

Above we mentioned the “string” type, but a similar issue might happen with “number”. Let’s consider such a situation: 

The definition is quite simple – we are passing “height” and “width”, for instance, to draw some rectangles. Unfortunately, we face a similar issue – is it in inches? Centimeters? Meters? But that is not all – let’s say Unit C can’t draw a rectangle bigger than 100 meters. How to avoid a situation where someone will assume 200 meters in Unit Test B? It is very hard. 

3. Stateful execution

This case is applicable for stateful units, but also to units that are using the database. In unit tests, it can happen much more often than in bigger-scope tests. See drawing below:

Everything is about side effects, which are produced in normal operations. We can have a lot of messages (or calls), which are exchanges between units, even generated by the single message from “EXT-X input”. It is almost impossible to test such a scenario unit test, as we need to simulate proper order, proper payloads generated by other units, etc. Every such message produces a different state of the unit, so it is easy to wrongly write a test case and think that it tests the unit properly.  

What is more, coverage increases, so it looks like the code is covered, but it is not true. 

The above is one of the most common problems related to unit tests. 

Thoughts 

In this story, we wanted to highlight the real nature of unit tests – what can be a risk when writing them. However, it does not mean that they are always bad, because it all depends on project needs. Sometimes it is worth writing such kinds of tests, but we need to be careful if the whole project relies only on them. There are also options, for how to make them less risky. Additionally, there are plenty of other testing methods – module tests, system component tests, integration tests, or even manual tests. 

Solutions for unit testing and the pros and cons of every other testing method we will try to cover in the next stories. 

As RSC we help companies decide what is the best testing process for their software. If you feel that it is something with which we can help – just contact us.