Starting A New Mobile Product: must-have components of proper continuous testing
Joining an ongoing project with an established testing process is quite different from diving into a new project without any existing testing. In fact, the latter can be significantly more challenging. When it comes to mobile applications, there are additional factors that contribute to the complexity of the quality assurance process. Continuous testing, in turn, plays a critical role in this process.
In today’s issue, let’s have a look at what are the must-have components of the continuous testing process we should not forget when starting a new mobile application project.
Disclaimer: in this newsletter edition, we try to make the information mobile platform-agnostic. In practice, implementation of the below-mentioned tips might have slight deviations between iOS and Android pipelines.
1. Identify testing stages and jobs in your app’s CI pipeline
This first but important step is very much dependent on the chosen testing strategy. If you base your testing strategy on the Agile testing pyramid, then a CI pipeline would typically have the following:
Unit and integration tests that are run inside the build stage. Usually, it is one of the very first stages in the mobile pipeline;
If there’re any platform-dependent integration tests that require real or emulated hardware, they are usually run during the next testing stage (after the build one but before building artefacts for manual testing & distribution).
UI acceptance and/or end-2-end tests that can be run during the same testing stage or a separate one, but still before building artefacts for manual testing & distribution. Due to the time-consuming nature of such tests, it could be a good idea to set up parallel execution of platform-dependent and UI tests to save some pipeline time.
Such pipeline structure ensures layer-by-layer testing and is aimed at reducing the cost of fixing defects by detecting them as early as possible.
Stay connected and get the latest updates by following us on Twitter!
2. Define what flavours of your mobile app are needed for testing
A mobile app can have a variety of flavours such as debug or release builds, with or without analytics, with/without security hardening, with or without memory leak detection, with different features toggles or testing backdoors, pointing to different development environments, etc.
It is important to define what precisely is needed for each testing activity in your continuous testing process. Different types and levels of automated testing may require different flavours of the application under test. Hence, your CI pipeline should be flexible enough to prepare the required binaries.
At the same time, it should allow the team to trigger building the required app flavours manually from the pipeline. For instance, a protected release-candidate build from the master branch for performing final exploratory testing before production.
3. Farm, farm, farm…
We’ve just mentioned platform-dependent and UI tests. At some point, a mobile product becomes quite expensive to test, time-wise but also resource-wise. The more users and features it gets, the more crucial it becomes to check the app’s functionality on different devices and OS versions. It can become an obstacle if your existing crew and available devices do not allow such scaling.
There are a couple of ways here: you either purchase lots of hardware, create and maintain your own devices fleet or rely on a cloud device farm and concentrate on writing good tests instead. Cloud devices farm solutions such as AWS Device Farm, BrowserStack, MS AppCenter, SauceLabs and others are a game-saver here. Usually, they take over the “dirty” part of the work: keeping devices in a secure and clean state, charging, adding new device models, recording video of test execution, maintaining infrastructure, etc.
Of course, not every mobile project requires a device farm. Whether it’s needed or not, should be properly assessed, and other factors (for example, cost efficiency) should be taken into account.
4. Empower your CI pipeline with extra tweaks
All above-mentioned testing jobs/stages should work towards the common goal - making sure that mobile binaries are well-tested before the distribution. There are a few more practices that can be utilized to make the process even more robust:
Think about disallowing failures when configuring testing jobs. It might look a bit harsh at first sight. But disabling further pipeline execution when tests are red - brings benefits in the middle term, such as more responsibility in fixing bugs/flaky tests and more confidence in releases.
Utilize the filtering abilities of your test framework. Usually, executing all available tests for every branch or commit is not a good idea for a mobile product (unless you have limitless scaling capabilities). But splitting tests into different suites (regression, smoke, nightly, etc.) by using tags or attributes can be an efficient solution. It allows running a different set of tests per development branch, scheduling execution of “heavy” tests at night, rerunning tests for a particular feature, etc.
Shift-left security checks. Yes, yes, you didn’t hear it wrong. In many cases, checking how robust a mobile app is, relies on penetration testing somewhere before the release. While nobody disputes the value of penetration testing, the cost of fixing issues at that point is quite high. Moving some security checks to the left of a mobile CI pipeline can reduce that cost and raise the alarm about vulnerable dependencies, leaked secrets, and weak encryption algorithms much earlier.
Don’t ignore code coverage and code style checks. While this part might seem to be more on the developer’s plate, it does contribute to the final quality of a product. Depending on the mobile platform and chosen pipeline configuration, such checks can be either a part of the build job or run in parallel with it. And, of course, if they fail, it is a good reason to block your pipeline from further execution.
5. Don’t forget notifications and reporting
One of the crucial tasks of automated testing is to provide fast feedback when something goes wrong, isn’t it? Timely notifying a responsible group of people about failed test jobs may decrease overall resolution time. Hence, it could be a good idea to check if your CI has integration with your internal communication tool (whether it’s Skype for Business, Slack, MS Teams or others), and give it a try by configuring notifications. If it doesn’t, there is a high chance a CI can still send email notifications.
Another side of the coin is reporting. Letting people know that their test build failed does not really guarantee they can figure out what’s wrong quickly, especially for mobile apps. If the architecture of a testing framework does not provide logs, screenshots or video for analysis, it could be very time-consuming for anyone to replicate a failure.
Thus, having proper reporting with info that helps engineers to debug failures (stack traces, device logs, graphics) is one of the essential components of continuous testing.
6. Automate end-2-end distribution of your mobile app
Testing of a mobile app does not end in a test environment. Both Play Store and AppStore offer the possibility to distribute mobile apps to enclosed groups of beta and alpha testers. This is where the final touches of testing happen before an app goes live to a broad audience.
An efficient CI pipeline should be able to upload production release candidates to the application stores. Of course, after all preceding checks are passed, whether it’s autotests or manual exploratory testing, or design verification session. Both application stores provide API for uploading, and it is well integrated with widely used platform build tools such as Gradle (Android) or fastlane (iOS).
Conclusions
Of course, tweaking and polishing your continuous testing process isn’t an easy task. But we do hope that our pieces of advice will come in handy and guide you through this journey. In future editions, we are going to extend this topic and share even more useful insights. Stay in tune!