Rethinking Testing for the Mobile-first Era

Mobile phones are an integral part of our daily lives. Studies indicate that tech-savvy Millennials and Gen Z spend over 3 hours on their phones every day and have 80+ apps installed. With innovations in payment gateways making it easier for people to pay on the go, it is expected that nearly half of all e-commerce purchases will be made through mobile phones by 2024. The recent Covid-19 pandemic has further emphasized the importance of mobile phones for a range of activities, from shopping and payments to banking and entertainment. The convenience of using mobile phones from anywhere, at any time, has made them a crucial part of people’s daily routines.

In response to the skyrocketing use of mobile devices, Google switched to mobile-first indexing in 2018. Google now focuses only on the mobile version of any website for indexing and ranking instead of the desktop version. Similarly, Google’s app store now prioritizes app quality and user experience in order to rank apps on the Play Store. They are actively making high-quality titles more visible on the app store, steering users away from lower-quality ones. The mobile experience of websites and apps, therefore, has a direct impact on SEO ranking and discovery.

A mobile-first strategy focuses on delivering a seamless, optimized mobile experience to all users, driving engagement and customer loyalty. Businesses that prioritize mobile experiences often stay ahead of the curve in terms of innovation and technology, enabling them to respond quickly to changing consumer needs and preferences. A mobile-first strategy can, therefore, help businesses retain a competitive edge and stay relevant in today’s dynamic digital landscape.

Design, Development, and Testing are the three main aspects of a mobile-first strategy.

The mobile-first framework focuses on optimizing design, user interface, and information architecture for smaller screens and then scaling up for larger screens. This approach ensures that websites and apps are accessible and usable on any device and provides a better user experience for mobile users, who make up the majority of internet traffic.

Responsive design, which enables websites and apps to adapt to various devices, screen sizes, and window sizes for optimal usability and consistency, has been around for a while. However, it is not the same as mobile-first design. With responsive design, the content, navigation, and layout are adjusted to fit smaller mobile screens, but it is still primarily optimized for traditional desktop websites.

On the other hand, mobile-first design starts with the smallest screens, specifically mobile devices, and is then progressively enhanced to fit larger screens, such as tablets, laptops, and desktops. The approach enables businesses to deliver the most essential content and features to the majority of their user base while simultaneously providing the best possible experience to those who satisfy all the more advanced hardware and software requirements of the website or app.

When considered as a flywheel, a mobile-first strategy can bring continuous improvement in the mobile experience and lead to increased usage, which in turn generates more data and insights that can be used to improve the experience further. This creates a virtuous cycle that drives continuous improvement and value for all users.

Adopting a mobile-first approach requires businesses to rethink their product development processes, design frameworks, and technology stack to ensure that mobile is at the center of everything they do. Companies must also have a deep understanding of their mobile users, their habits and behaviors, and their needs and pain points.

Here’s a look at the various steps involved in the mobile-first strategy flywheel:

Gather Requirements: Identify and document the needs and goals of the project.

Organize Content: Determine the most important content for the mobile experience and prioritize it.

Build a Wireframe: Create a basic layout that outlines the placement of content and UI elements.

Design: Develop the visual design of the mobile layout, taking screen size, touch-based interaction, and load times into consideration.

Develop: Write the code for the mobile layout, including HTML, CSS, and JavaScript.

Test: Verify that the mobile layout works accurately and meets the requirements, testing it on a variety of devices and browsers.

Deploy: Launch the mobile-first website or application.

Optimize: Continuously monitor, review and improve the mobile layout to ensure it remains optimized for the latest devices and technologies, including accessibility and loading speeds.

Without a mobile-first approach, businesses risk missing out on the rapidly growing mobile market and the opportunities that come with it. Here are some of the key things that can be missed without a mobile-first approach:

Mobile-intuitive user flow: The design and functionality might not be optimized for smaller screens and limited inputs.

Touch interactions: Support for multitouch/multi-finger interactions might be missing or inadequate.

Mobile-specific best practices: The design and functionality might not take into consideration mobile-specific constraints like smaller fonts sizes, limited real estate, limited battery usage, etc.

Mobile browser support: Many advanced styles and HTML elements may not be supported by mobile browsers.

Cost-effectiveness: Not having a mobile-first approach from the get-go can result in increased development and maintenance costs, as the need for re-development and additional testing for mobiles may arise.

Performance: Failing to account for the limited processing power and memory of mobile devices, in contrast to desktop computers, could result in a negative impact on performance.

Accessibility: Mobile devices are used by people with different abilities. Without a mobile-first approach, accessibility features such as larger text sizes, high-contrast mode, and text-to-speech might be ignored or inadequate.

Location-based services: The design and functionality might fail to leverage GPS and other location-based services to provide personalized experiences.

Content architecture and layout designs meant for mobile devices differ from desktop ones. Using one for the other often leads to user experience inconsistencies.

Organizations and website owners must prioritize mobile-focused content and design to deliver outstanding user experiences and reach the expanding mobile-dominant user base.

Fragmentation in the mobile devices market – different screen sizes, resolutions, operating systems, hardware capabilities, etc. – and global audiences can impact the rendering, performance, and functionality of software, apps, and websites.

Comprehensive testing must be conducted on a wide range of devices to ensure that apps and websites function seamlessly and provide a consistent user experience across all platforms.

Fragmentation of devices and distributed audiences make testing a massive challenge, as it requires significant time and resources to thoroughly test apps and websites on each device, leading to increased development costs and longer time-to-market.

Without scalable and dependable mobile-first testing tools, testing can become a bottleneck, causing delays and leading to bugs that can negatively impact user experience. It also makes testing difficult, ineffective, and inefficient, undermining a mobile-first approach.

The Usage-Complexity Framework below can help businesses understand how testing needs may change over time as the app or website grows and user behavior evolves. By understanding these factors, businesses can adapt their testing strategies and tools to ensure that their app or website remains functional, user-friendly, and future-proof.

Parameters considered to define the Usage-Complexity Framework:

Number of users

Usage patterns and engagement change over time. The number of people who use the app or visit the website, the types of devices, browsers, and operating systems they use, as well as their geographic locations, impact testing needs.

Number of functionalities

Depending on the type, growth stage, and purpose of the app or website, it can have various features and functionalities. The general rule is that the more integrations and data inputs required, the more complex the digital asset and the more comprehensively it should be tested.

Testing Needs

Testing needs exist at all stages of app or website growth. But as the app or website progresses from the initial launch stage to the growth and maturity stages, new features are added, and the number of users grows. Testing requirements evolve accordingly, corresponding to the different stages of growth.

Variables that make testing complicated

• The complexity of the system being tested
• Number of user devices being tested
• Operating System versions
• Screen sizes and resolutions
• Processor variations
• User locations
• Network conditions
• Mobile-specific features
• Integration with other systems and components
• Third-party integrations
• Volume and variety of data input required
• Battery capacity variation
• Variations in use case and user behavior

The three most popular approaches for mobile testing are discussed below:

In-house device labs

In-house device labs are physical or virtual testing environments established and maintained on an organization’s premises for testing purposes. These device labs can include a range of hardware, such as smartphones, tablets, laptops, and desktop computers, and are typically managed and operated by the organization’s development team.

In-house device labs provide developers with a controlled testing environment where they can quickly test and debug their code. However, this approach is impractical in a remote work setup with distributed teams and has multiple challenges:

• Cost and effort required to set up and regularly maintain the lab is high
• Not enough device coverage due to fragmentation
• Automation doesn’t run out of the box without additional time and effort for framework support

“Even if we were to maintain an on-premise device farm of only the top 10 user devices, we would have to spend upwards of $50,000 a year, and we would still not have any automation capabilities out of the box.”

Todd Eaton, Head of Consumer Product DevOps & QA, The Weather Company

Emulators and simulators

An emulator is a software program that mimics the behavior of a real mobile device. A simulator, on the other hand, mimics the behavior of a real device’s software environment but does not fully emulate the hardware components of the device. Emulators and simulators can be used to test website or app performance, functionality, and compatibility across different mobile devices and operating systems.

While emulators and simulators are easily available and competitively priced, they have several limitations when it comes to mobile testing. They fail to simulate real-world use cases and device characteristics, leading to potential inaccuracies and disruption of user experience.

• Emulators and simulators do not fully replicate the hardware features, performance, and behavior of real devices, leading to false results.
• Network conditions, such as speed, latency, and bandwidth, can significantly impact user experience but are not accurately recreated on emulators and simulators.
• Differences between the emulated or simulated OS and the actual OS can impact the functionality and behavior of the app.
• Touch and gesture inputs on physical devices may differ from those on emulators and simulators.

“With emulators and simulators, there was always a percentage of error that had to be accounted for with respect to real devices. The error rate has reduced a lot from the time we have shifted to BrowserStack.”

Ravinder Singh, QA Team Lead, OLX Autos

Real device clouds

Real device clouds provide instant access to a large pool of real mobile devices for testing and debugging of mobile websites and apps. These devices can be accessed remotely over the internet, making it easier for developers and testers to test their applications on various device models, operating systems, and screen sizes without having to physically access the devices. Real device clouds are a cost-effective solution for organizations to test on a wide range of devices without the need for expensive hardware setup and maintenance.

Testing on a real device cloud is the only solution for high-quality releases and user experience, which is why over 6 million developers and 50,000 teams test on BrowserStack.

“Our quality is going up. We now test against many different devices, especially the ones we couldn’t verify against before due to our small device pool. BrowserStack’s regularly updated cloud with a wide range of browsers and devices is a flexible and efficient solution that helps us maintain quality levels our members expect.”

Tim Gould, Engineering Manager, Nationwide Building Society

BrowserStack gives you instant access to 3000+ real device and browser combinations across manufacturers, models, OSes (both major and minor), and browser versions to test on. BrowserStack doesn’t use emulators or simulators; all tests are run on real devices to provide real-world results and avoid any false results. This enables your team to understand exactly how the product works on all the devices their customers are using to deliver predictably good experiences every time, boosting customer acquisition, satisfaction, and retention.

BrowserStack enables your team to automate and significantly expand testing coverage to ensure the products they bring to market are bug-free. The scalability of the BrowserStack platform enables you to identify issues as soon as they are introduced (Shift Left testing), creating a virtuous cycle that allows teams to automate more, innovate more, and deliver more, which leads to more revenue.

BrowserStack eliminates the testing bottleneck by enabling hundreds of tests to run in parallel. This allows teams to test at the speed of today’s modern CI/CD pipelines without compromising on quality. BrowserStack’s elastic cloud helps teams scale up tests on-demand – there are no device reservations required. BrowserStack’s platform is powered by 19 data centers across 13 locations to give organizations unmatched performance, scale, and reliability to support faster builds and faster time to market.

BrowserStack’s data centers and platform are SOC2 Type II and GDPR compliant, giving organizations a testing infrastructure that meets their most stringent security and compliance requirements. With default HTTPS implementation, data controls, and access management policies, enterprises can be confident their data is safe and secure when using BrowserStack. Once a test session is complete, all test data is destroyed, and devices are put into a factory reset state. In addition, BrowserStack’s globally distributed data centers enable customers to meet in-region data privacy and sovereignty requirements.

BrowserStack keeps teams focused on what they do best, allowing them to spend their time and energy on testing, not test infrastructure. With BrowserStack, testing is fast, reliable, scalable, and hassle-free, requiring no overhead and no maintenance. In addition, the BrowserStack platform is future-proofed, giving enterprises immediate access to popular new devices and browsers the moment they are added to the platform at no additional cost.