Pixel perfect UI: How to automatize automation testing on Android
Have you ever modify a dimensions file and realized too late, that this change has affected unexpected parts of your app? This is a common problem, especially in large teams where is hard to know who is using (or planning to use) shared resources like dimensions, colors, strings and styles.
Imagine that every time you build a new UI component, an automation test is automatically created to protect your view from accidental changes. Imagine that those tests could detect any type of changes, color, sizes or fonts.
This may look like fantasy but it is really easy to achieve and it will only require you writing some extra lines of code.
One test to rule them all
So how can a test be created automatically? The answer to that question is Parameterized Test. JUnit Parameterized Tests allow us to run the same test multiple times with different parameters. Let’s see how it works:
This test will be executed 4 times, once for each element (parameter) returned by the data() function. The test will pass for arguments 2, 4 and 6 but it will fail for 7.
We plan to use this to run the same test for all our UI components. We only need to return a list of UI elements instead of a list of numbers. Something like this:
For this example, we are using layouts but this technique works with custom views (plus some reflection) too.
Visual Regression Testing
At this point, we know how we are going to execute an automation test for each of our UI components without actually writing a new test every time. The next step is to figure out how can we create a test that fits all our views.
Our requirements are simple, the test should detect any type of changes in our UI (size, color, typography, shape, etc…) but at the same time, it should be generic enough so it can be re-used by all our UI.
Ok, this may look like an impossible task but what our requirements are describing is called Visual Regression Testing. Visual regression testing is a type of test in which we check that our new implementation looks exactly like our previous implementation. This type of test is usually implemented using Snapshot Testing.
The idea behind Snapshot Testing is pretty straight forward. We take a picture of our UI and the next time we do a change in our code, we compare our current UI with the picture. This is great because humans are not that good at detecting these subtle differences in the UI.
Building the one and only automation test
Taking a Snapshot of a view in Android is not hard, these three lines of code take a snapshot of our view and save them as png.
We could take this path but then we will need to implement the logic to create, store and compare the images. Luckily for us, Facebook’s UI Android team has developed an open-source Snapshot Testing library that we can use. This framework has some handy Gradle tasks to generate and verify our snapshots.
Let’s see how our test will look like:
As you can see the test is not that complex. We inflate the layout passed as a parameter, we set up the view with the proper dimensions and we take the picture (snapshot) of our view. Everything else will be handled by the Gradle tasks provided by the framework. Let’s see an example.
Running our test
To generate the initial Snapshots we only need to execute the command:
./gradlew record<App Variant>ScreenshotTestLet’s say that we have a layout like this:
Generated using this xml:
When we run the task, it will return the path to a summary page with all the recorded snapshots:
If we open the file we will see something like this:
As you can see the content of the view is missing, there is no user picture and there is no information in the name, age, and description fields. This Snapshot will help us detect changes in TextViews with static content as well as changes in dimensions and margins but we can do better.
A smarter test
At this point, we know how to generate our Snapshots and how to run a test for each layout. What we need now, is to set content in our views and take snapshots of that content.
To do that, we need to introduce a new concept, the Variation. A Variation represents a part of our UI that can change. In our example above, the user’s image and the name, age, and description will have its Variation.
A Variation has two parts:
- A lambda function. The lambda function implements the logic to update the UI. For the
name, we will create a function like this:
(name, view) -> view.nameTag.setText(name)
- A list of values. Each value will represent a different configuration of the UI. For instance, if we decide to create a
Variationfor the fieldname, we could have two namesAhriandChargoggagoggmaggchaubunagungamaugg. This way we will know how ourTextViewwill look like with a small string and a large string
Our final variation will look like this:
Variation(
{ view, string -> view.name.text = string },
"Ahri",
"Chargoggagoggmaggchaubunagungamaugg"
)Using the concept of variation, we can create a configuration file with all the view’s variations:
We will use the variations defined in the configuration file, to create a test that takes a Snapshot of all the possible permutations:
In this example, we have 4 Variations, each one containing 2 possible values, this means that our test will generate 16 (2*2*2*2) snapshots. If we execute our new test by running ./gradlew record<App Variant>ScreenshotTes we will see all the possible permutations:
Now that we have our 16 snapshots we can save them by using:
./gradlew pull<App Variant>ScreenshotTestOnce they are saved, we can compare them with new changes. Let’s say that we accidentally change the text size from 16sp to 15sp. Before pushing our code, or even better in CI we can run the command:
./gradlew verify<App Variant>ScreenshotTestThe task will fail to tell us that the UI has changed and we need to check what those changes are. If we are fine with the changes, we can run the record and pull commands again and override the existing old snapshots.
Visual Regression Testing and Continuous Integration
To achieve the full power of Visual Regression Testing, we need to make it part of CI. Each team has different processes and tools but making these test run in CI is not hard, a simple approach that I have used in the past is:
- After creating the test, we call manually the
recordandpulltasks. This will generate the original Snapshots in a folder called/snapshots. We can commit these changes togit(or your favorite version control system). - We create a new job to run the
verifytask every time new code is pushed. - If the task fails, we can write a message in the PR (if you use Github) and send a failure status so its owner and the reviewers know that they need to check the screenshot report.
- If the changes were intentional, the reviewers can add a label “Accept UI changes” to the PR. A webhook will detect the label and it will trigger a new CI job to run the
recordandpulltasks.
With this simple setup, you can easily protect your UI from unintended changes and prevent costly production bugs.
