Getting Strict With Mocks
Most mocking frameworks allow two mocking modes; standard (or loose) and strict. Today I’m going to look at both approaches with my mocking library of choice for C#, Moq.
A good definition of loose vs strict mocking can be found here:
The key thing to note here is that if we call something on the strict mock that has not been setup, an exception will occur. This is surely a good thing? Don’t we want to know when a method is called that shouldn’t be called?
There are a number of issues with this, some of which I’ll cover here.
The Sample Code
To highlight these issues, I’ve created a very simple little solution, consisting of a library project and corresponding xunit test project. The code under test comprises of two classes, one being the dependency of the other.
The horrificly named classes. Naming is hard.
The message processor has a public method that accepts a list of strings. The IMessageHandler has a method that accepts a single string. It performs some manipulation on the string that was passed in, and stores it in memory. I also have the equivalent ‘get’ methods to get the data back out. The MessageProcessor code ends up looking like this:
Next, in my xunit project, I have two namespaces, one for “Loose” tests and the other for “Strict” tests. We’ll use this to highlight the issues I’ve mentioned above.
Now, I’ve put together some tests for the MessageProcessor. Fundamentally, the only functionality here is to return true if all messages are successful from the handler, and false if even one messages comes back as unsuccesful. I end up with two ‘loose’ test cases, and the equivalent ‘strict’ test cases.
The ‘Loose’ mocks
In the strict version, we pass in the ‘Strict’ behaviour when instantiating the mock.
The ‘Strict’ version
Right now, these tests are proving the exact same thing. Let’s run them and see what happens.
All good at this point. You can see the code at this point if you look at the branch “1_InitialTests” in the Git repository.
The problems occur as your code base evolves. Let’s imagine that some new feature is required. We now need to send the messages to a Parser service. And now let’s imagine that we’ve implemented it in the following approach (there’s obviously better ways to do this, but I’m just using this convoluted example to prove the point). This can be seen in the branch “2_NewFeaturePain”.
I’ve added a new method called “SendToParser” to the MessageHandler, and I call it from within the processor loop (line 25):
So the interesting thing here is that I have not actually changed the main observable behaviour. I still return true if all messages process successfully, and false if even one fails. So therefore, my tests should all still pass. Now, I’ll obviously have to add a new test to verify I actually call out to the parser, but nevertheless, my existing tests should still be green.
Oops
Hmm. Some may argue this is a good thing. I strongly disagree. Strict mocking forces constant rewriting of test code. This is not good. Tests should verify behaviour not implementation. Tying it directly to the implementation results in a constant churn of rewriting tests. If I have to rewrite tests, then how can I be sure I haven’t broken some previously working feature?
Generally speaking, an additive change to the code should not break any existing tests. You can think of it in the same terms of a public API. Adding a new endpoint or method should not break integration with existing services, but editing or deleting a method will break existing integrations.
Unit tests should almost be black-box, despite the common misconception that they are white box. Just because you can look at the inner workings, doesn’t mean you should. In fact, doing so may influence the tests you write. It’s much better if the tests can be written independently of the implementation. (The preference might even be to not using mocking at all – but that’s a post for another day. There’s also TDD implications….but again, for another day).
This issue can become huge when you have lots of dependencies. I’ve seen unit tests of 120 lines (!), that might have 10-15 Moq setups. Imagine some underlying dependency changes. This becomes a maintenance nightmare and ultimately chips away at the confidence in your test suite.
Also, the difficulty in writing these types of tests results in avoidance of writing any tests! In the previous example, there was a single, 120 line-long, happy path test. The previous developer obviously felt the effort to implement the edge cases was just too great (which is understandable). If there is a barrier to doing, then it won’t be done. These things need to be easier to do, not more difficult!
There are other issues with Strict Mocking, which I will discuss in a future post, but for now the source code can be found here.
Published June 15, 2020June 16, 2020
