Mobile Application Testing:
Test Automation Life Cycle is a subset of Software Testing Life Cycle. It is always presumed that through the process of automation, we can expedite the process of validating software. But of course, there are certain challenges which exist when automating application under test (AUT) is concerned. In this blog post, we will talk about some of those challenges faced in automation testing. (instead of the ones we usually talk about in manual testing).
Why Mobile Automation Testing?
Smartphone applications have experienced explosive growth since 2007. The combined number of applications in the Apple App Store and Google Play already exceeds a million. So many applications competing for users’ attention and therefore meeting up with high quality bar becomes quintessential for the success of an app. Poor-quality applications cause revenue loss and irreparable brand damage. Faulty and inoperative enterprise applications can lead to lost productivity.
Implementing quality test automation of mobile applications across all of these operating systems, device platforms and networks is a scary task. We would not hesitate in saying that effective mobile automation testing enables device makers and application developers to collect appropriate metrics which no doubt help improve product quality and help provide significant business returns.
According to ABI Research, “The mobile application testing market will be $800 million at the end of 2017.”
Key challenges in Mobile Test Automation space:
- Device Diversity:
- Multiple platforms and browsers.
- Varied applications having varied run times on mobile devices.
- Network Challenges:
- Multiple network types (e.g., GSM/GPRS/Wi-Fi/Wi-Max).
- Varied speeds of connectivity across geographies.
- Customized network features of multiple network operators.
- Hardware Challenges:
- Limitations in processing speed.
- Limitations of mobile memory size.
- Differences in device communication protocols (e.g., WAP/HTTP)
Let’s Nail Down The Problems:
Here are a few pain points or mobile testing challenges every Dev or QA manager should take into account while considering whether or not to implement test automation within their quality assurance phase.
Challenge #1: Testing on Multitude Mobile Devices and Platforms
With the proliferation of devices such as handsets, smartphones, tabs and wearable tech, the number of platforms and devices is so vast such that while it was impossible with manual testing, even with automation testing it is a big challenge.
Test Automation cannot guarantee if a tested application works well on a given device, it will work 100% on another device even if it’s from the same product family. This is because of difference in Memory, OS optimization, CPU as well besides the obvious screen resolution and hardware differences.
Creating separate tests for every device and platform will be a costly resort and it is a time consuming process. With Algoworks Application Testing Labs, the same test script can be executed across platforms, resulting in minimum scripting effort and that too without any shortcuts in test coverage.
Challenge #2: Dealing with a variety of connectivity modes
WiFi, 3G, 4G, 5G… the more the ways mobile devices can connect the more exponentially multiplies the number of tests you need to run. With such a wide variety of handsets, OS versions, and connectivity options Mobile QA and Automation Testing gets more difficult to the point where it can fail.
Challenge #3: Different Mobile App Types
Your mobile app may be a native app, a web app or a hybrid app. Each app-type’s behavior from installation to functionality is different from one another, hence their testing and test coverage will also be different.
Challenge #4: Automated Cloud Testing Challenges
Many cloud testing services allow you to run your automated mobile and unit tests in different environments and on multiple machines without having to build your own testing infrastructure. There are many cloud service providers like Sauce Labs, Mailosaur, Gridlastic,Tddium, CloudBees, CircleCI and TestingBot providing cross browser Selenium testng, unit testing and continuous integration testing.
On the contrary there are a number of issues and challenges in testing clouds and cloud-based software. Some challenges include on-demand test environment construction, lack of well-defined validation methods and quality assurance standards in order to address the connectivity protocols, service APIs provided by SaaS and clouds APIs.
Challenge#5: Unit Testing frameworks
We all know that unit testing is essential in order to ensure your code does what it’s supposed to do. Also there are many unit testing frameworks available (NUnit, xUnit.net, PyUnit / unittest, JUnit, TestNG, PHPUnit, Symfony Lime, TestUnit, RSpe). Though Automatic unit testing saves a lot of time and allows more robust testing, sometimes even automatic unit testing frameworks such as xUnit do not let you test much of your code. Most of you simply don’t have a code that may be adequately tested by even the most robust of open-source frameworks, hence leaving us and our business products/services vulnerable to bugs and technical debt.
Challenge#6: Variety of mobile automation testing tools
The more the options to choose from, the more the fear which one is the best of them all? Some of the most popular mobile test automation tools(Selenium, Watir, Windmill, Ranorex, SoapUI, Sahi, Tellurium) and automated GUI tools (Squish, Ranorex, TestComplete, Test Studio, eggPlant) that are available in the market end up making test automation difficult besides the dependency on various devices, multiple platforms and myriad browsers. As the user interface of your application might change regularly, therefore because of various differences and incompatibilities between browsers you would usually require added servers.
Challenge#7: Record and don’t playback
Test development environment provides the capability to record your interaction with the mobile application and generate the test implementation out of it. This feature may be quite tempting but you should watch out for the generated code quality. Usually it is not “developer-friendly”. You then will have no other option than to recreate the test from the scratch. This would eat up more and more of your time, hence keeping you from creating new tests for the emerging features in the mobile application.
Challenge #8: Mobile Test Automation tools fails to deliver high ROI
Most of the existing test automation tools including open source or commercial, have complex steps to be performed in order to setup the test environment. Many tools support working with real devices. But only few of these offer the same support for simulators/emulators. The support for real devices across different operating systems is achieved via jailbreaking which is considered as a bad practice. We must tell you that it takes immense pain to perform tests on non-jailbroken devices as one requires certificates and profiles when it comes to iOS devices.
References: Mobilelabsinc, Deloitte Digital, Smartbear Blog, Impetus Blog, Testingtools, Softwaretestinghelp
Ajeet Singh
