Testing Best Practices

Unofficial Content

Unit testing is not about finding regression bugs

Well, it’s important to understand the motive behind unit testing. Unit tests are not an effective way to find application-wide bugs or detect regressions defects. Unit tests, by definition, examine each unit of your code separately. But when your application runs for real, all those units have to work together, and the whole is more complex and subtle than the sum of its independently-tested parts. Proving that components X and Y both work independently doesn’t prove that they’re compatible with one another or configured correctly.

So, if you’re trying to find regression bugs, it’s far more effective to actually run the whole application together as it will run in production, just like you naturally do when testing manually. If you automate this sort of testing in order to detect breakages when they happen in the future, it’s called integration testing and typically uses different techniques and technologies than unit testing.

"Essentially, Unit testing should be seen as part of the design process, as it is in TDD (Test Driven Development)”. This should be used to support the design process such that designer can identify each smallest module in the system and test it separately.

Tips for writing great unit tests


1. Test only one code unit at a time

First of all and perhaps most important. When we try to test a unit of code, this unit can have multiple use cases. We should always test each use case in a separate test case. For example, if we are writing the test case for a function which is supposed to take two parameters and should return a value after doing some processing, then different use cases might be:

  1. First parameter can be null. It should throw Invalid parameter exception.

  2. Second parameter can be null. It should throw Invalid parameter exception.

  3. Both can be null. It should throw Invalid parameter exception.

  4. Finally, test the valid output of function. It should return valid pre-determined output.

This helps when you do some code changes or do refactoring then to test that functionality has not broken, running the test cases should be enough. Also, if you change any behavior then you need to change single or least number of test cases.


2. Don’t make unnecessary assertions

Remember, unit tests are a design specification of how a certain behavior should work, not a list of observations of everything the code happens to do.

Do not try to Assert everything just focus on what you are testing otherwise you will end up having multiple test cases failures for a single reason, which does not help in achieving anything.


3. Make each test independent of all the others

Do not make a chain of unit test cases. It will prevent you to identify the root cause of test case failures and you will have to debug the code. Also, it creates dependency, which means if you have to change one test case then you need to make changes in multiple test cases unnecessarily.


4. Mock out all external services and state

Otherwise, behavior in those external services overlaps multiple tests, and state data means that different unit tests can influence each other’s outcome. You’ve definitely taken a wrong turn if you have to run your tests in a specific order, or if they only work when your database or network connection is active.

Also, this is important because you would not love to debug the test cases which are actually failing due to bugs in some external system.

By the way, sometimes your architecture might mean your code touches static variables during unit tests. Avoid this if you can, but if you can’t, at least make sure each test resets the relevant statics to a known state before it runs.


5. Name your unit tests clearly and consistently

Well, this is perhaps the most important point to keep remember and keep following. You must name your test cases on what they actually do and test. Testcase naming convention which uses class names and method names for test cases name is never a good idea. Every time you change the method name or class name, you will end up updating a lot of test cases as well.

But, if your test cases names are logical i.e. based on operations then you will need almost no modification because most possibly application logic will remain the same.

E.g. Test case names should be like:

1) TestCreateEmployee_NullId_ShouldThrowException

2) TestCreateEmployee_NegativeId_ShouldThrowException

3) TestCreateEmployee_DuplicateId_ShouldThrowException

4) TestCreateEmployee_ValidId_ShouldPass

This principle says that if you are testing Employee module than you should first test Create Employee module as it has the minimum dependency on external test cases. Once they are done, start writing Modify Employee test cases as they need some employee to be in the database.


6. Write tests for methods that have the fewest dependencies first, and work your way up

To have some employee in the database, your create employee test cases must pass before moving forward. In this way, if there is some error in employee creation logic, it will be detected much earlier.


7. All procedures, regardless of visibility, should have appropriate unit tests

Well, this is controversial indeed. You need to look for most critical portions of your code and you should test them without worrying if they are even private. These methods can have certain critical algorithm called from one or two classes, but they play an important part. You would like to be sure that they work as intended.


8. Use the most appropriate assertion methods

There will be many assert methods you can work with each test case. Use the most appropriate with proper reasoning and thought. They are there for a purpose. Honor them.


9. Ensure that test code is separated from production code

In your build script, ensure that test code is not deployed with actual source code. It's a wastage of resource.


10. Do not print anything out in unit tests

If you are correctly following all the guidelines, then you will never need to add any print statement in your test cases. If you feel like having one, revisit your test case(s), you have done something wrong.


11. Do not rely on indirect testing

Do not assume that a particular test case tests another scenario also. This adds ambiguity. Instead, write another test case for each scenario.


12. Integrate Testcases with build script

It’s better if you can integrate your test cases with build script so that they will get executed in your production environment automatically. This increases the reliability of the application as well as test setup.


Summary

Without a doubt, unit testing can significantly increase the quality of your project. Many scholars in our industry claim that any unit tests are better than none, but I disagree: a test suite can be a great asset, but bad ones can become the equally great burden that contributes little. It depends on the quality of those tests, which seems to be determined by how well its developers have understood the goals and principles of unit testing.

If you understood above guidelines and will try to implement most of them in your next set of test cases, you will certainly feel the difference.

 

8 signs of bad unit test cases


1. Test passes but not testing the actual feature

Believe me, I have seen such test cases in my previous projects which seems to doing lots of stuff in code, but in actual they were doing nothing. They were sending requests to server and no matter what server respond, they were passing. Horror !!

Beware of such test cases making place in your code repository, by strict code reviews. Once they make their way in your code base, they will become liability on you to carry them, build them and running them every time but without adding any value.


2. Testing irrelevant things

This one is another sign of bad test case. I have seen developers checking multiple irrelevant things so that code passes with doing SOMETHING, well not necessarily the correct thing. Best approach is to follow single responsibility principle, which says, one unit test case should test only one thing and that’s all.


3. Testing multiple things in assertions

This one seems to be similar to above sign, but it is not. Where in previous sign, test was testing irrelevant things, here in this sign, unit-test test the correct things but many of such things in one test case. This is again violation of single responsibility principle.

Well, please note that I am not encouraging the use of single assertion per test case. To test a single entity, you might need to use multiple assertions, do it as needed.

For example, one API which takes some parameters in post body and create Employee object and return it in response. This Employee object can have multiple fields like first name, last name, address etc. Writing a test case to validate only first-name, then another for last-name and another for address is duplicacy of code, without any value. Don’t do it.

Instead, write a single positive test case for create employee and validate all fields in that unit test. Negative test cases should be written separately doing only one thing and one assertion in this case. e.g. One test case for blank first name, one test case for invalid first name and so on. All such negative test cases can be validate using single assertion which expect exception in response.


4. Test accessing the testee using reflexion

This one is real bad. Trying to change the testee to suite ones need. What happen when code refactoring on testee happen. test case will blow up. Do not use this or allow to be used either.


5. Tests swallowing exceptions

I got my fair share of such test cases. They silently swallow the exception in little catch block at the end of test case. And worse is that they do not raise a failed alarm also. Exceptions are signals that your applications throws to communicate that something bad has happened and you must investigate it. You should not allow the test cases to ignore these signals.

Whenever you see an unexpected exception, fail the test case without failure.


6. Test which depends on excessive setup

I do not like test cases which requires a number of things to happen before they start testing the actual thing. Such a scenario can be like: an application which facilitates online meetings. Now to test whether a user of particular type, can join the meeting, below are the steps test is performing:

  • Create the User

  • Set user permissions

  • Create the meeting

  • Set meeting properties

  • Publish meeting joining information

  • (Test) User join the meeting

  • Pass/Fail

Now in above scenario, there are 5 complex steps which must be setup before actual logic is verified. For me, this is not a good test case.

A correct test system will have an existing user present in system for this activity, at least. It will reduce at least two steps in test case. If one can find appropriate, he can have some existing already created meetings also to make this test case really focused.

Another way to make it correct is using mock objects. They are there for this very purpose. Isn’t it??


7. Test compatible to only developer machine

This is not widely seen but sometimes visible when written by freshers. They use system dependent file paths, environment variables or properties in place of using common properties/paths, or things like this. Watch out for them.


8. Test filling log files with load of texts

They do not seems creating problem in happy days. But when rainy days come, they make the life hell by putting unnecessary text without any information in log files and make the life hell for debugger who is trying to find something hidden in those log files.

Tests are not for debugging the application, so do not put debug logs kind statements. A single log statement for Pass/ Fail is enough. Believe me.

These are my thoughts based on my learning in last few years. I do not expect you to agree with me on all above points. But might have some other opinion which is perfectly cool. But I will surely like to discuss, what you feel on topic.