Mobile App Testing: A Guide to Performance, Security, and AB Testing | Rootstrap

You could compare the speed and innovation of the mobile app industry to that of a world-class factory line. In the blink of an eye, a new app, technology, and a device have been created on a never-ending belt of new ideas. Off they go to be consumed but to also face the possibility that they will be replaced by the next round of even more up-to-the-minute designs. 

That factory analogy isn’t far from the mark based on the current state of things. It seems like a new app is launched every single day. How can you even attempt to keep up, let alone perform devops to maintain your products?

It all comes down to the trust users have in your app, and how you can gain that trust is through ensuring your app is top of the line concerning performance, security and AB testing. 

What Are the Different Kinds of App Testing

• Functional 

This type of testing, well, you guessed it - tests function. Functional testing works to ensure that the app is working as it should, based on the set requirements. It will largely be performed on the user interface, and occasional call flows. This stage of testing requires a large amount of human consideration. What we mean by that is that with an automated functional testing system, you’ll need to employ a good bit of time and human resources to ensure that testers are fully equipped.

• Performance

While functional testing focuses on the behavior of the app, performance testing looks at the actions of applications in regard to predictable challenges. This form of app testing is really more a game of predictability; as in, how many possible scenarios and challenges could a user face when on the app? Core considerations here are things like; how much of a battery does the user have? If they have a low battery, does the app perform up to its regular standard? What about if the user is nearly out of memory on their device? Or, what happens when the network coverage is poor in a certain area? For this stage of mobile app testing, you get into a user persona’s head as much as possible. 

• Memory Leakage 

The list continues! Another form of app testing is good ol’ memory leakage. Doesn’t sound particularly appealing now, does it? To sum it all up, mobile devices have very limited memory, and mobile operating system applications use up most of this space. So, you’ve absolutely got to check to make sure the app you’re building is optimized in terms of processing memory.

• Interrupt 

This form of testing has to do directly with how an app interacts with other standard functions of a mobile device. Say someone has your app downloaded on their phone, and they get an incoming call or message? Does this directly impact performance, and how? Or is it that those functions like calls or messages don’t register on a device because your open app has interrupted these? Your app should be able to stay in a suspended state when interrupted, and immediately resume when said interruptions are concluded. 

• Usability 

UX is the magic word here; it’s all about user experience. Our developers at Rootstrap will tell you exactly the same, which emphasizes why this type of testing is so imperative to the long-term success of your app. How easy to use is the app? At the usability testing stage, that’s the question you’re focusing on answering - and further maximizing ease of use where you can.

• Installation

All mobile devices come with two types of applications. The first is the one that installs directly when you launch your operating system. The second is one that you need to get installed at a store. As you’d expect, this form of testing checks how smoothly the installing process goes with your app. 

• Operational

Ever found yourself working on a huge work project, or paper for school, when suddenly your entire device crashes? In that same scenario, for those of us lucky enough to have a great recovery system, the device also gave us the option to re-open an automatically saved copy. When it comes to operational testing, this is used to test that recovery process on a device, and how it relates to your app. 

• Security

Last, but certainly not least, is security testing. This type of testing is all about network security; is the network security or data responding as it should? You’ll be comparing results to the app requirements at this point. 

Why Mobile App Performance Testing Matters

The biggest reason why mobile app performance testing matters go back to the sole most important part of your business - the customer. Testing sets out to verify that a user can trust your application to work in the way it’s intended and won’t compromise their device. 

The Importance of Security Testing

It’s a simple equation: if an app is prone to security threats, then people won’t use that app. This is one of the most important parts of a solid QA process. Case and point. Security testing is all about proving app credibility; if you can verify that your app has been comprehensively security tested, then you can give your potential app users peace of mind. That will go a long way in the highly competitive mobile app industry. Win the trust of your app users by having an app that has been properly security tested, and is in no way vulnerable.

How Often Should You Conduct Testing on Your App

If you’re completely new to mobile app testing, you may instinctively wonder the following: “If my app is tested fully the first time, is there a need to continue testing?” Anyone who is well-versed in the mobile development process, knows the resounding answer to that question is YES. Routine testing of your app is something that is vital to your testing strategy, from ideation of the testing process to an ongoing check. The focus on these check-ups should be to continually debug the app. 

Newsflash - there will always be ways to better the user experience of your app. You should continually be working to test these and improve any existing or developed issues. 

Finding the Best Way to Conduct Tests on Your App

Now that we know the importance of application performance testing, it’s probably a natural idea to wonder - well, what in the world are the best ways to manage these tests? Don’t fret - our developers at Rootstrap have got you. We brainstormed some vital tips from our greatest app-loving minds to yours, which should help you on your journey to ensuring customer loyalty by providing the best digital user experience. Hang on tight, because here we go:

1. Monitor Patterns

So, how do you find the best way to track user patterns in your app? There’s an app for that. Kidding (sort of). By “app” we mean there are loads of open-source tools that can help you monitor user patterns, and better improve the experience. These enable you to really grasp the usability of the app as a whole. Top tip to add to this step - consider the questions the developer doing the testing should answer. Which is the most commonly used screen, and the most used area in it? Where does the app crash? Are there any patterns in an area where it crashes the most?

2.Virtual Simulation

Say what? You read that correctly. Tip number two has all to do with testing all sorts of functional scenarios on your app. Or, as we like to call it, virtual networks. A fairly standard step is just simulating all sorts of network experiences; i.e. different types of connection types, locations and even the mobile phone operators a user could be employing while on your app. This is a fail-safe way to improve your confidence that the app can handle any sort of network situation.

3. Production Environments

It’s a short one, but test in a production environment, or in a simulated one. A necessary way to see what really happens when users go on your app. 

4. System Performance Monitor

Here’s a golden bit of advice - keep those performance scripts running constantly. By leveraging these, you will do your whole business a favor. It allows an in-depth evaluation into usage scenarios and how these impact revenue. System performance monitoring is a huge aid to improving overall business intelligence. 

5. Backend Monitoring

While you can’t argue the sheer benefits of front-end continual testing to ensure your users are having the easiest time possible on your app, you can’t forget an equally important aspect - that back-end database. How’s the ongoing performance of the database doing? Production tests and real user response times are a good shout while you roll out the app, and even after to ensure database stability.

6. Involve Everyone

You’d be surprised at how often businesses overlook other necessary input of a team behind a mobile app aside from the development stand-point. What a massive, massive mistake. Application performance is viewed differently by different parts of the business. As in, you simply cannot just have the developer perspective on what is “good” without taking into account what Operations or those behind User Experience have to say. 

A useful reminder here is who your app is for; the customer. Frankly, the developers behind the magic are too far away from the actual users who will be the ones to decide whether or not your business will make it. Bring in Operations and User Experience to fully collaborate with your Development team in scoping out an application performance baseline. In fact, it’s even better to build this strong foundation before you even initially launch the new app, so as to ensure everyone in your team agrees on what you want to achieve every time you application performance test.

7. Crowd Testing

Last, but certainly not least, is our handy secret ingredient to a successful app: crowd testing. It’s an amazing way to understand exactly how your app is performing, across various locations, demographics, and countless networks. Orchestrate the occasional session where you have as many users as possible to use your application in load slots. If you can get these crowd testers to report back to you the issues they experienced (we recommend nifty gift incentives or the like), the in-app insight you will achieve from this method of testing is priceless.

What Are the Benefits of User Beta Testing

Beta in mobile app lingo translates to the stage right before you're ready to launch your app. Here is when you focus on two key areas; 1) debugging 2) reviewing the pre-released app. The core function of user beta testing is checking for errors so that these can be taken care of ahead of market release. 

How do you hand-pick the users who will beta test and come face to face with your app’s errors? You’ll ideally want to identify loyal users who have been on the previous version of the app. If the app you’re testing is entirely new, you’ll want users with relevant app experience.  

What Are the Drawbacks to User Beta Testing

There are a few drawbacks to beta testing worth considering. The first is an investment; beta testing isn’t free. In fact, there’s a pretty price tag which comes with this type of mobile app testing. Why does it cost money to beta test? Think about all that’s involved.

You need to identify and select a number of testing participants, set a duration for the testing period, you need a wide range of demographics to properly test, and finally, you’ll have to cover any other associated costs.

The Extra Benefit from Hiring Testers

Manual testers bring a significant amount of value to a testing process. Why? Well put plainly - they save you a lot of money. You’re probably wondering how that is even possible. In a long-term perspective, any bugs found in development will cost much less than those found once your app is live. 

Another benefit from hiring testers? You free up your developers to do what they love most - code. Not only will your team love you for that, but so will your app. 

We would recommend that the following types of projects should absolutely use manual testers:

1. Start-up projects, because everything is aspects and function is continually altering 
2. Speedy projects, where things need to move rapidly forward
3. Any type of budgeted project where automated testers aren’t a fiscal reality. 

Good Interview Questions for Prospect Testers

You’ve decided to go manual. Go you! Next immediate step, naturally, is to hire some testers. You want to make sure you get the best of the best in to test your software. But how do you know who is right for the job, and who is not? These are some top interview questions you should ask prospect testers before hiring:

1. What needs to be done to reproduce a bug?
2. What does a tester do when coming across a bug in testing?
3. Tell me - how do you go about launching the test cases?
4. Define the key differences between preventative and reactive approaches to testing.
5. What’s exit criteria?
6. How do you assess the level of risk?
7. When does Decision table tasting get used?
8. How do you put together a bug report?
9. Tell me - what’s a bug lifecycle?’
10.  How do you know when a bug is properly fixed?

What Kind of Training Should You Offer Your Testers

When it comes to available training for testers, there are some critical things to do to ensure your new tester is ready to get started.

First and foremost, know their background. Any prior testing experience? First new role? Or perhaps they could have worked in a similar technical position? What are their personal preferences when it comes to actual workload? 

What Is Automated Testing and How Is It Used

Automated app testing is a whole other ball game. Rather than hire a person to manually test your app for you, you hire a system to do it for you. Struggling to know which way to go with testing your app? Fear not. In terms of advantages v. disadvantages of automated testing, we broke that down for you:

Advantages:

Speed of process

• The running process is much speedier. Nice.

Long-term cost

• It’s effective if you’re thinking future payouts; i.e. less errors in live. 

Reusable

• Automated cases are easy to use again and again. Winning.

Performance 

• For certain kinds of testing, automated mobile testing is a must. 

Sharable

• The results are easily displayed and sent to others. 

Disadvantages

Limitations:

Time-consuming 

• These tools can often take a lot of time to complete testing. If you’re in a time crunch, manual testing is better.

Lower efficiency

• When it comes to user experience or ease of use, automated testers just have less efficiency. 

More negative customer experience

• As well, they lack an overtly clear positive customer experience.

Some Easy to Use Low-Cost Automation Testing Tools

Low-cost and automation - have two words ever sounded so sweet? We got the scoop for you on some fantastic, affordable automation testing tools you should spring for. 

1. Selenium 

First up is everyone’s favorite, Selenium. An open-source testing tool, this bad boy is free as a bird. Not only that, but the tool accepts many languages. These are: C#, Java, Perl, PHP, Python, Ruby and Groovy. Selenium is also deployed on Windows, MAC OS, and Linux.

This low-cost automation tool can be learned in two parts – Selenium IDE and Selenium WebDriver or Selenium RC. Selenium IDE is for more practice of functionality. So, we recommend employing Selenium Web driver for testing any web apps. Selenium WebDriver is a more intriciate version of Selenium RC.

Most of all, Selenium is easy to use, folks. Get it while it’s hot.

2) Telerik Test Studio

We can’t talk about beautiful automation tools without mentioning Telerik. By far, it is the most simple tool available today. This level of ease does come with a price, however. The cost isn’t too sky-high though, averaging at $170 every month, and with this purchase you get functional and performance testing solutions, amongst a few others.

Caution - Telerik comes with a plug-in of Visual Studio. What does this mean? You’ve got to know the visual script. 

3) Sahi

Another open-source automation testing tool is our friend Sahi. Yes, it’s easy to get going and navigate and is all about low-cost and low-time spent during testing. It is specific though, as it’s a tool for testers using Agile. 

The awesome thing about Sahi is that it meshes well with manual testers. You don’t find this smooth collide of worlds often, and it only works like that for Sahi because of the ease of tool navigation.

The Pros and Cons of Hiring Testing Services

Are you asking yourself, “do I actually need to outsource testing?” If so, you came to the right place. Not because we have the right answer for you - as it’s imperative to remember that every project, business, and app is different - but because we can spell out the pros and cons to help you on your path of decisions. 

Let’s start with the pros of outsourcing your testing services:

Cheaper

• It’s just a lot cheaper to outsource, in most cases. That’s all there is to it!

Efficiency

• Software testing service providers offer testing professionals who are experts in specific types of testing. Hence, software testing projects are likely to be carried out efficiently, taking up less time and resources.

Time

• No need to test internally or supervise testers? Time is suddenly on your side. Use that extra free time to prepare for that new app launch!

Now, for the cons: 

Security

• The biggest of all concerns around outsourcing is security. There are so many factors to consider here: do you trust this company with your confidential information, a lot of which they’ll come across throughout the app testing process? Are they doing the right things to ensure safety and confidentiality on your behalf? Worthy questions to think about.

Trust

• How do you pick a partner you can trust? This is another factor of the case against outsourcing. Can you really give that responsibility to someone outside of your nest?

Quality

• This is simply a potential if the business that is outsourcing doesn’t do a lot of micro-managing throughout the process. Just watch closely, to ensure the standard of work is up to par.

Loss of Control 

• It almost goes without saying, but when you outsource software testing, you also outsource a lot of control. Going back to the trust point here - this is why you need a partner you know can deliver. 

Overseas

• Where’s that outsourced business you’re considering located? Most are across the world, in countries as far away as India or China. Of course, this level of distance can impact a lot - the biggest concern being time zone differentiations. 

Some Examples of Testing Services

Those pro’s above convince you that outsourcing a testing service is the way to go? We’ve got some top recommended services online by other businesses. But first, our criteria in selection:

Flexibility 

• Can they address all parts of the big project? If yes, then this box is ticked.

Experience

• What’s their CV look like? How many years, and what companies have they worked with?

Location

• Are there options on where testers are coming from? If you have the choice between on-site work, off-site nearby, or in a foreign country, you’ve got a strong service there.

Our top recommended services are:

UberTesters

Based in New Jersey, UberTesters has proven it’s trustworthiness within its user base for quite some time now. The company handles beta-testing management and crowd-sourced testing, which it’s obvious they do really well, considering their incredibly high user rating on Clutch.co. 

Their also the most affordable of the bunch, ranging in around $1,000 and upwards. 

OnPath Testing

This service is based in Boulder, Colorado, and offers full application testing. Also praised by its users online, OnPath demonstrates a very strong work ethic and gets clear results. The service is a bit more expensive, ranging at $5,000 and up. Still, you get what you pay for!

A Quick Mobile App Testing Checklist

When you Google, “best ways to test your mobile app,” you’ll get around 680,000,000 results. Overwhelming, huh? Although you could easily find yourself sucked into a vortex of hundreds of different ways to test that mobile app, we reckon what you’d rather do is find a list of all the steps that nearly every developer universally takes in this process. That also will allow you the opportunity to take those hours of Internet perusal, and instead get started on implementing your very own mobile app testing. With no further ado, here is Rootstrap’s go-to mobile app testing checklist:

1. Mobile-Platforms

You know that a device can operate on many different mobile platforms, right? This is what the end goal is with mobile development, as any business wants their app to work in a cross-functinal, cross platform type of way. Make sure to include the platform and the device the app is targeting here. What this will do is help any testers better evaluate performance of this cross-platform functionality.

1. Devices

Oh, the world of the digital phone these days is just downright revolutionary. One can’t help but compare the possibilities and variations of mobile devices to the vastness of space itself. But, rather than getting lost in that analogy, we recommend you check feature compatibility of your app in this testing cycle. The performance of an app absolutely varies depending on types of devices used, so make sure you know which ones it meshes with, and which ones it doesn’t.  

1. Features

Continuing on with the complexity of all those mobile devices out there, another thing to check is hardware. As in, what hardware feature does your app need in order to work with a device with a specific requirement? It’s all about the specifics!

1. Customization

May it be big screens or small screens, your app needs to be prepared to function and perform exactly the same. We call this step: custom optimization. Get your app ready so that your company can be proud of the description: mobile-optimized. 

1. Security

Everyone thinks about security, in all aspects of a business. Especially with Europe’s new GDPR regulations, the awareness of the world in general is heightened regarding data privacy. However, you choose to test your app, pay extra attention to data encryption and storage. Make sure your team of testers are performing all of the right security checks in order to assess these. 

1. Interruptions

Much like when a person interrupts you mind-speech, an app getting interrupted could be considered as equally irritating. Also like real-life interruptions, this is inevitable and happens often for any app. There could be many reasons for an app getting interrupted, but what you want to focus on in this portion of testing is how your app behaves in response to these interruptions. 

1. Network 

We’ve already mentioned it once or twice in this article but check for network connectivity and how different levels and types work with your app. You won’t regret this necessary testing step once you see the positive results of a mobile app optimized for all types of networks. 

1. Customer Satisfaction

User experience is really something, isn’t it? We can’t preach the importance of it as the fundamental focus of any app development enough. Going back to that space analogy, the app marketplace is incrementally large, and one that you and your app will have zero chance at keeping a user if you don’t work really hard at presenting the best possible UX. 

Looking for More Help 

We do really hope this guide helps you and yours on your next mobile app testing venture. While we tackled relevant topics such ways to test application performance, security and even AB testing it all, we also know how demanding mobile app testing can be. Feel free to give us a shout if you’re interested in getting the help of specialists with your mobile app development - our agile ninja developers are all over the U.S. ( L.A., NY, Austin, Chicago, Atlanta - you name it!) and even in Montevideo, Uruguay, as our reach is pretty much the Western hemisphere. Interested? Learn more about what we do, how we do it, and the faces behind the action here.