Skip to content

Real Devices or Emulators/Simulators - Which is more appropriate for Automated Testing?

The predominant question which arises on every tester's mind before kick-starting the automation of the testing for web apps when accessed from mobile devices or for native/hybrid mobile apps is whether they should execute these tests on real devices or using mobile emulators (or simulators). Emulators, as you may be aware, are software programs designed to provide simulation for important features of a smartphones (or tablets).

Given an unlimited budget and time-frame, testing across the multitude of 'real' devices available would be ideal. However, when facing the reality of the wide range of Android and iOS devices currently available, it’s almost impossible to procure and test on each and every type of device. 

Best practices for mobile app development would include both emulators and real devices. “A recommended approach is to find a healthy mix of emulators, simulators and real devices to get the best out of your test automation.” (Gartner, Market Guide for Mobile App Test Automation Tools, Maritess Sobejana).

One usage pattern is that, ideally, one would use emulators in the early coding and debugging phases and 'functional/regression' testing, and real devices for the later 'usability' and 'performance' testing phases.

Another pattern that is the most plausible and widely used involves:
  • Using real devices for functional/usability testing of the top 'N' devices that are used by your customers/users, and
  • Using emulators to test the rest of the devices

Challenges & Limitations of Testing with Emulators:


Even though device emulators give you the ability to perform functional/regression testing across a wide variety of devices at a significantly lower cost, there are some drawbacks to using emulators. 

The list provided below highlights a few reasons why testing on real devices usually result in better testing outcomes than using emulators.

Situation Based Testing: You cannot emulate real life usage with software. 
  • How does the app look when you are outside in the sun, or when it’s raining? 
  • Can you use the app while walking? Does the interface translate well with swipes and finger usage? 
All these situations are impossible to replicate with emulators.

Battery Testing: It is impossible to test the effects of your app on the battery life of a device with an emulator.

Interrupts: Things like receiving a text message or phone call while using the app cannot be tested while using an emulator. Since the mobile device (a phone or a tablet) can be a phone (has a phone number that can ring), network-related events (e.g., incoming call, text message, etc.) must be tested to determine their impact on your application. This is very hard to accomplish using an emulator, because quality of the network varies between carriers, states, countries, and regions. Since emulators are not connected to the mobile network, they are not capable of providing an answer for how these will affect your application.

CPU & Memory Related Issues: Emulators tend to have a great deal more memory and/or CPU available than real devices. This is because they are not multitasking the way a real device is. The performance of your app on an emulator may be much better than on a real device. This could cause a misconception of how quick your app responds.

Display/Resolution Variance: The look and feel of the application may be different on an emulator as opposed to a real device. How bright or dim the app appears, and the changes in resolution across devices may differ with respect to the emulator. It’s also very hard to simulate light and dark situations on an emulator.

Different Network Environment: In terms of network configuration, mobile emulators run on the PC, connect to the LAN and access the Internet via your personal/corporate firewall. Using real devices, the network is connected to the radio interface and from there to the Internet. In many cases, the low level network stack itself is completely different. Regardless of whether this is good or bad, the network environment is different and will result in your application behaving in another way.

False Positive Outcomes: Even if your automated tests run perfectly well on emualtors, you cannot be 100% sure that the functionality covered within your tests will work well when your customers/users access the same functionality on a real device. This raises the question of which tests need to be double-checked on a real device even after the tests pass on emualtors and which tetss can be assumed reliable and hence don't need testing to be conducted on real devices. Also, in the event of a test that fails on the emulator, testers need to decide whether to perform the same test on a real device or simply assume that the function will not work on real devices as well and hence initiate correction of code to rectigy the issue(s).




Feedback and Knowledge Base