Pipeline for Automated Testing of Android App Using Self-Hosted Azure Linux Agent

Today, I would like to share my experience of developing a pipeline for automated testing of our Android app.

Motivation

Lately I have realised that during my work as Software Developer I sometimes encounter situations where a scenario that I am trying to execute, lacks compact information, has information scattered around or is completely missing.

This way I wanted to contribute to the community and hopefully patch some of these informational gaps so that the next person researching similar scenario can find the information faster.

I also like the idea of reflecting on my past work. I can come back and remind myself of the journey, reasons and thought processes. Also it helps me to tidy up in my head and conclude.

Before I move on with the article, I wanted to clarify few things.

First off, the term automated testing in this article refers to collection of E2E and functional test suites that we have for the app.

Second, this article focuses on enabling of the automated tests execution in the Azure DevOps pipelines and does not cover use cases around this enabler. For example, I will not be talking about where does the pipeline belong in the process or how to distribute test results.

Problem

At work we used to run the automated tests against Android emulator instances from our local machines. This approach is obviously not suitable after early stages of development.

We have often encountered problems where devs and QA experienced different outcomes of the tests and the speed depended on the capability of the local machines.

Despite that annoying inconsistency, the real reason behind the effort of moving the automated tests execution to the pipelines was the need for automated integration of the tests in the release cycle.

Starting Line

When going into this we knew we had only 1 sprint (3 weeks) to deliver enabler solution to run automated tests in the pipelines. This is significant time pressure and therefore the investigation into possibilities did not span too wide into the external providers like GitHub Actions etc.

The app was distributed only for Android devices so that simplified the scope of the solution quite a bit.

The tech stakc of the app development is .NET and the automated tests that we needed to run were written in .NET stack using Appium as a bridge to the emulator browser drivers. The app itself is an hybrid of Android native app (.NET MAUI Hybrid app) where majority of the app is written in Blazor in WebView layer. This test environment setup needed to be supported on the solution and the test configurations needed to be exposed to allow control from the solution environment.

The tests already worked against Android emulator instances on local machines, so we could get inspiration from some of that setup when it comes to Android emulator configurations.

We already had a pipeline solution to build the Android app but the agent was coupled to Windows image (because of some Windows dependencies in the app build process) which made building the app very slow compared to Linux images. Also the purpose of that pipeline was releasable app which lacked some features required for autmation testing.

We experienced various performance issues on local machines so the caution in this direction was high especially considering the purpose of the solution - to be run automatically within deployment processes so naturally performance is a important requirement.

The Plan

First impulse was to use something that would cover most of the heavy lifting for us. However, after looking around for a bit in Azure ecosystem, it was obvious that Azure does not have pipelines or tasks that would make the work much easier.

I have found some Gradle tasks and deprecated resources like App Center that used to allow such solutions of automated testing on mobile devices in pipelines. We could have gone for 3rd party mobile device farms/providers to eleviate the emulation environment part of the problem, but the complexity of managing such 3rd party integration within our narrow time frame was not reasonable. So the Emulator was the way to pursue.

When it comes to the pipeline agents, it is intuitive to think that an pipeline on an azure-hosted agent should be enough and everything could be defined in it and yes, it is but when in process of developing the solution the importance of having the environment, that is being developed and possibly after (for maintaining purposes), permanently accessible, explorable and maintainable is very practical. So spinning up self-hosted agent on a Linux VM was inmediate first step.

Execution

There were many trial and errors mostly around performance of the Android emulator and tweaking of its configuration but eventually the solution was pretty clear.

Infrastructure

The infrastructure required minimum of one VM for the self-hosted agent where the pipeline and the Android emulator would run.

This was Linux based VM using Ubuntu image 24.04. The performance and virtualization (emulation requirement) requirements pushed us to more constly VM option of D8sd_v4 of Dsd_v4 family that is priced at around $300/month if running 24/7 which would not be the case anyway, since these tests are not as frequent as lower ranks of tests like integration or unit tests.

I have noticed that the emulator performance improved drastically between D4 and D8 series which is 4 vCPU jump. Lower capabilities than D8 were causing the emulator to work in unpredictable way e.g. chrome tabs crashing, slow app startup, emulator crashes.

By the way I tried to look for an Azure VM that would also provide capability of hardware/GPU accelerated rendering but unfortunatelly where D VM-series shine is where N-series lack and vice versa. So it's either GPU and no CPU virtualization or other way around. Android emulators perform much better with software rendering and CPU virtualization than with hardware/GPU accelerated rendering but no CPU virtualization.

Access to the VM was setup through SSH. Additionally we required a virtual network setup to restrict the access and define communication between external dependencies (other VMs on the same network).

Install the Azure Agent

After having the VM up and running, the Azure Agent package was downloaded as per official Microsoft documentation. With the agent being setup on the VM, the only requirement for the agent to be discoverable from Azure DevOps pipelines was to have the VM running.

Environment Variables

Whenever the agent runs from the pipeline it uses it's own scope for environment variables and initialization scripts. This means that environment variables defined in the VM are not always applied.

To specify environment variables usable by the pipelines, it is required to either define them in the pipeline itself or in the .env file of the agent located at ~/user/agent-name/.env e.g. ~/vmadmin/myagent/.env. Since we wanted to define this agent in more definitive and purposeful way we opted for defining the main set of environment variables on the agent in the .env file.

Dependencies

As per dependencies we needed to build .NET MAUI project into Android APK which requires few SDKs to be installed:

Configuring Android Emulator

We used the pixel 7 image for the Android emulator on x86_64 architecture with Android 14 (API 34). This is what we used to run locally and that's what we required.

Such Android emulator image contains lot of unnecesarry performance-starving configurations that were not needed for the run in pipeline environment. So we tweaked the configuration a bit. The configuration file is usually found in the ~/.android/avd/name-of-the-amulator.avd/config.ini.

The following configurations were used:

Finally certificates were installed on the emulator to establish trusted connection with external APIs.

Pipeline

The pipeline steps were defined:

Future Work

For the future, now that we have stable emulator agent, the configurations and environment setup could be moved to the pipeline definition to decouple from physical VM as much as possible and leave the VM to only provide the capabilities. Although it is quite big performance gain to bake some depepndencies in it like maui-android workload or SDKs and emulators.

Another one of the next steps would be to distribute the test reports. When we ran the tests locally, we opted for little reporting tool Allure Reports distributed as NPM package. This reporting is sufficient but in ecosystem of Azure DevOps Test Plans we can leverage existing reporting that is generated from the DotNetCli task command "test" rendering the Allure Reports not as crucial anymore.

After this enabler is done it is also needed to define policies and use the pipeline where testing strategy requires them, whether it is before release, nightly runs or combinations of such.

Conclusion

I am not going to lie, it is quite awesome feeling to see this local complexity of running automated tests on laptops finally happen in the cloud. Even though there are improvements to be claimed, the deadline is met and the result is achieved and that makes me and the delivery plan happy.

Until next time..