Introduction to testing and automation
As any experienced software developer will know, testing is one of the most critical areas of software engineering and continues long after the code itself is written. If you want your project to be a success, you need to ensure your plan includes a realistic amount of time for both internal testing and to support the clients’ own user acceptance testing. At Sinara, when estimating how long it will take to develop a piece of software, we build these factors in from the start.
Also, in the fast-moving world of modern software development, we don’t just do releases every year; for many of our systems at Sinara, we may need to do a release every few weeks to support our clients’ schedules. This means more repetitive testing, often even the very same set of tests that were run only a little while ago.
So:
- It takes time
- It’s often the same function over and over
- But it’s critical that it’s done correctly
If you put that all together, it’s asking for automation!
What tests do you automate?
When it comes to automation, there’s one big no-brainer for any developer: automated unit testing. Whatever the framework you use, you should be able to run all unit tests automatically without too much fuss. Ideally, you’d have some form of continuous integration etc (I’d even recommended it within your IDE – we use NCrunch by the way).
The problem is that with large complex systems, unit testing probably only catches around half of issues, or maybe even less. You’ll always need some element of system testing, and system testing is hard to automate. The thing is, manual system testing is often slow and painful, so I’m going to assert something that I think should become one of the key tenets of test automation:
You shouldn’t automate a test if it’s not going to save time in the long run, and if it will save time in the long run, you should definitely automate it
Yes, it’s a tautology and completely obvious, but it can be all too easy to overlook. Simply put, if a test will take 3 days to write, but over the next few months, you’ll run it so many times that it will take more than 3 days to do, then automate it. You could also factor in the cost of not testing it (as more-likely-than-not, the test wouldn’t be run with every iteration), as well as the fact that there may be a lead time before a bug is found (and the longer it takes for a bug is caught after development, the more painful it is to fix). If you do this analysis, then there are bound to be tests that will need to be automated.
How do you automate it?
This one will depend a lot on what you’re testing. We’ve used Selenium/WebDriver to good effect on websites, but if it’s a TCP interface, then you’ll need to write some way of driving it etc.
It’s a good idea to have a dry run at this point – just build a small app that does the functionality of the test (without it being a formal test), just to prove you’re heading along the right lines.
How do you make it into a test?
OK now we’re onto the heart of this blog post. Enter the Sinara TestDriver…
Sinara.TestDriver
Our TestDriver is a .NET library that can be embedded in an executable that allows you to automate system tests in the style of a unit-test. You have Test Setup, Test Steps and Verifications/Assertions.
Most of what makes the TestDriver different is how we do the “Test Setup”. In most projects I’ve worked with, we automate a web application, but the same principles apply and have been used to automate a desktop application, a web interface, a FIX feed, etc.
So what are the key components/concepts?
- The ‘driver’ – that drives the system
- Your ‘assumptions’ that puts the system in the correct state
Drivers
First, you always need a ‘driver’ – this is a small, standalone class that can drive your system tests. In fact, you often need two or three as your system may have several interfaces
Since we’ve been talking about websites, a website driver would have different methods – so on a chat app, your driver might have methods: SendMessage, ReadLastReceivedMessage, ReactToRecentMessage, ChangeRoom etc. You might even have Login/Logout depending on how you design it.
And don’t think of it as a quick test utility – this must be engineered to be as reliable as possible. A system test framework that fails for no reason half the time is not a desirable outcome! It’s vital that effort is put in to automate the app before you start thinking about testing.
A final point – you need to make sure your driver works in a ‘headless’ mode – that is, run by a backend service where there isn’t a user logged in at the time. It is all too easy to build an app that works well when logged in, but then fails when there is no GUI present, so make sure this is designed in from the start.
Assumptions
Assumptions are small attributes that are used to setup the environment for the system test – making sure everything is in the correct state etc. The surest way to start each test with a system in a good state is to tear it down and rebuild it every time, but we’ve found that this is often too slow to be useful. It’s much faster to just reuse the system from a previous test; but the previous test may have left the system in a half-working state. There’s no point testing how the system reacts to price feed movements if the last test disabled the price feed within its test steps.
Assumptions seem like a simple concept, but they can quickly get quite complicated for a complex system, so it’s worth planning the key ones out before you start coding. They generally have two elements – 1) check some part of the system state, then 2) if the state is wrong, update it so it is right. You’ll need to consider every feature that needs to have a setting applied, so this can build up quickly.
The Test
Once you’ve got your drivers and assumptions in place, implementing the test is usually quite easy. What’s great is that you can quickly create many tests from a small set of drivers and assumptions, so whilst it’s a steep investment, the payoff comes quickly.
The TestDriver then uses its own Dependency Injection logic to provide the necessary drivers to each test (a single test may only need a small number of drivers). Also, since we’ve traditionally used XUnit for unit tests, so the TestDriver follows its patterns with the Fact and Theory attributes.
You end up with something like the following:
[Assumption1(true)][Assumption2("another state")][Theory][InlineDataRelaxed(5, "test str")][InlineDataRelaxed(25, "test str")]public async Task RunTest(Driver1 driver1, Driver2 driver2, int data1, string data2){await driver1.TriggerEvent(data1);await driver2.TriggerEvent(data2);var result = await driver1.GetOtherData();Assert.Equal(42, result);} |
If you’ve designed your tests well, they should be readable and maintainable – and ideally should be able to even be linked back to your functional specification.
Continuous Integration
Finally, you need to link what you’ve done into your continuous integration tool. At Sinara, we use Jenkins, and anytime it spots a check-in to our source control system, Jenkins runs the system test suite seamlessly, giving us confidence the new code works correctly. Developers are immediately alerted in the event of any failed build, giving rapid feedback on their work.
Summary
One thing that is often hard to convey is how much of an investment true automated system tests can be. Being able to automate the testing of key workflows in an application provides huge value to a project, giving extra confidence to introduce new features and releases to meet client’s business needs.
Another issue to point out is the amount of effort needed for the maintenance of system tests. Obviously, if you change the behaviour of a feature, you need to expect the related system tests to be changed similarly. But if a simple change suddenly results in many tests not passing, then the resulting investigation can often help reveal design issues and drive improvement in the software.
There’s also the issue of jitter – where tests work 9 times in 10. This may be due to a GUI or network timing issue, or a missing assumption, but again, unless you’re diligent, systems tests may begin to fail more frequently.
Which brings me onto the final point. If the system tests frequently fail without any obvious reason, then they will lose their value. Make sure you put as much investment into your system tests as you do into the actual software itself, and they will become a cornerstone of your software development process.
