Combinatorial Approach to Testing Mobile Applications: An Experimental Investigation Brandi Amstutz Advisor: Dr. Sergiy Vilkomir This research is supported by the National Science Foundation REU Site Award 1262933 Introduction Importance of Research Mobile testing is a critical step in the development process where errors in an application can be discovered and corrected to increase reliability and functionality. Traditional software testing strategies can be modified and implemented to solve the challenges associated with mobile device selection. The amount of devices required to exhaustively represent today’s mobile market is extremely high due to a wide range of variability between available devices. This investigative research analyzes the efficiency of mobile testing based on a combinatorial approach to device selection. Introduction Research Questions Addressed Which device characteristics have the potential to influence application reliability? Both hardware and software components have the potential to influence an application’s functionality. ▫ Hardware components considered for this approach include: Mobile device type Screen resolution Hardware manufacturer ▫ Software attributes included: Android OS version RAM memory Introduction Research Questions Addressed How should device selection be executed to ensure an optimal set of devices? This approach suggests parameter value specification based on user profile data reflecting the current mobile market. Less popular parameter values can be grouped to reduce the total possible values without compromising Huawei, 1.2% proportionality Pantech, 1.8% Motorola, 2.4% Other (tvdpi and xxhigh), 5.9% Other, 9.5% Google, 2.5% Xiaomi, 2.5% Low 10% Amazon, 3.6% LG, 6.1% Extra High, 24.6% Medium 23% High 36% Figure 1. Screen Resolution Distribution HTC, 6.3% Samsung, 55.9% Sony, 8.1% Figure 2. Device Manufacturer Distribution Source: Unity 3D Mobile Statistics Introduction Research Questions Addressed How many devices are sufficient for mobile testing? Exhaustive testing strategies are limited by resource constraints and contain redundancy between device attributes. This research suggests a set of five devices selected based on device parameterization and combinatorial testing techniques for maximum coverage Materials and Methods: Step One Device Parameterization Based on user profile data we suggest to use the following device characteristics and their parameter values: Type Device Manufacturer Android OS Version Smartphone HTC Honeycomb (3.0-3.2) Tablet Motorola Ice Cream Sandwich (4.0) Samsung Jelly Bean (4.1-4.3) Other Other Screen Resolution Low less than 426 dp x 320 dp Medium at least 470 dp x 320 dp High at least 640 dp x480 dp Extra High at least 960 dp x 720 dp Table 1. Selected Parameter Values RAM Memory < 768 MB ≥ 768 MB Materials and Methods: Step Two Device Selection Through Each-Choice coverage techniques and consideration of current market statistics we suggest the following two sets of selected devices: Table 2. Selected Device Sets Based on user profile data the following devices were included due to their high level of popularity: Set A: Galaxy S2 i9100 and Galaxy Tab 2, 7.0 Set B: Galaxy S2 i9100 and Galaxy Tab 10.1 All other devices have been added to provide total coverage of device characteristics Materials and Methods: Step Three Application Selection Three applications with similar functionality were selected for testing. The following applications used provide access to search, book and review travel accommodations ranging from car rental to discount vacation packages: Hotwire Travelocity BookIt.com Materials and Methods: Step Four Device Access Perfecto Mobile cloud services were used for remote access to devices for testing purposes. Tests were recorded and screen shots were saved for later analysis. http://www.perfectomobile.com/ Remote testing procedure using SaaS cloud-based device access: User provides test input Input is sent to devices selected by user Audio & video of live test execution is recorded Test results & video are sent to user for analysis Materials and Methods: Step Four Device Access Figure 3. Perfecto Mobile Dashboard Materials and Methods: Step Five Additional Devices For comparison, two sets of random devices equivalent in size to the Each-Choice sets were selected. The same testing procedures were executed on the ten additional devices: Table 3. Random Device Sets Results Screenshot Analysis Screenshots were analyzed from all sets of devices for fault identification Figure 4 (left) illustrates Fault H5 for the Hotwire application. The expected result is that the number of rooms should update on the main page when a user selects a new number of desired hotel rooms from the drop down menu. The fault occurs when the room quantity is not updated appropriately Results Screenshot Analysis Figure 5 (right) illustrates fault T1 for the Travelocity.com application. The expectation is that the Travelocity menu bar with their gnome icon is displayed at the top of the home screen. The fault occurs when the menu bar is missing from view. Results Screenshot Analysis Expected Fault Figure 6 (left) illustrates fault B1 of the Bookit.com application. The expectation is that a menu bar with current query's information and reference number is displayed at the top of the screen. The fault occurs when this menu bar is missing from view. Results Fault Detection Fault detection totals are as follows: Hotwire: 7 2 device independent 5 device dependent Travelocity: 7 1 device independent 6 device dependent Bookit.com: 4 1 device independent 3 device dependent Results Fault Detection Table 4. Fault Detection In 7 out of 12 cases (or 58%) the effectiveness of our approach was greater than the results of random testing. In 4 out of 12 cases (or 33%) the effective ness of our approach was equivalent to the results of random testing. Conclusion Conclusions The quantity of application faults detected by the selected set of devices was equal to or greater than the quantity detected by the random set of devices. Additionally, the random set of devices did not yield a fault that the selected set was not able to identify. This suggests that the device attributes selected were suitable based on their expected influence on application reliability. Combinatorial testing strategies provided sufficient parameter value coverage with a test suite of five devices. This research suggests that this small set of devices is effective for mobile testing while also reasonable enough for resource constraints. Questions?