From the Blogosphere
Considerations for Choosing a Cloud Load Testing Solution
How to pick a tool to load test in the cloud
By: Steve Weisfeldt
Jul. 10, 2012 04:00 AM
In a previous post, I've discussed how load testing with the cloud can enable your team to conduct more efficient and more realistic large scale tests while saving you (time and) money. With this all in mind, how do we go about choosing the correct cloud load testing solution?
There are many cloud load testing solutions that will enable you to make use of the cloud. However, very few enable you to capitalize on the opportunities of load testing in the cloud. As I've discussed, load testing with the cloud offers clear advantages over traditional load testing in certain circumstances, but the tools you use are even more important to the quality of your tests.
When considering a cloud testing solution, ask the following questions:
Much of this is outlined in the Neotys Cloud Whitepaper, but for the purposes of this blog, let's briefly talk about each of these:
Integration with the Cloud Platform
Load testing solutions that offer integration with the cloud simplify and accelerate the steps needed to use the cloud infrastructure. With the proper solution, connecting to, and utilizing the cloud should be possible with just a few mouse clicks.
Realistic Load Tests
When choosing a load testing solution, look for one that provides a bandwidth simulation feature that limits bandwidth to ensure that the virtual users download the content of the web application at a realistic rate. This capability is particularly important when testing mobile applications, because mobile devices typically operate with less bandwidth than laptops and desktops.
Unified Lab Testing and Cloud Testing
When choosing a load testing solution, you should make sure that it supports lab testing so you can confirm the performance on your application(s) internally, before you introduce additional variables by testing over the internet. This allows you to shake out any early performance issues with smaller loads before you execute tests from the cloud. Finding these issues earlier in the application's lifecycle will lower costs associated with not only the cloud but also those attributed to "late stage bugs".
Making sure that you use a single solution for both "internal" (within the lab and/or firewall) and "external" (via The Cloud) testing allows your engineers to reuse scripts across both types of tests. If you choose different solutions, you may have to create a new set of scripts for your external testing, costing you both time and money.
Ease of Use
Analysis, Monitoring, and Reporting
Make sure you choose a solution that allows you to easily analyze the collected data, creating actionable reports that describe the performance of the tested application. I've seen too many a test engineer spend hours, if not days taking raw data from a load testing tool and pushing it through Excel or a statistical tool to create graphs, charts and reports. A tool that allows for easy analysis not only makes this process pain-free but can increase the collaboration among the different stakeholders of the application's infrastructure.
Making sure your load testing solution contains a comprehensive monitoring system is also essential when you need to find the root causes of a problem. If you are only looking at performance measure collected from the end-users point of view, you're missing half the picture. Being able to use a monitoring system allows you to understand what was happening within the infrastructure of your web application is critical in identifying any root issues. Moreover, the selected load testing solution should allow for easy correlation between data collected from the end-users point of view and data collected from any back-end servers. Without this ability, results analysis becomes extremely cumbersome as well as potentially inaccurate. Too many times in my load testing career I've tried to correlate performance data collected from a load testing tool with that of data collected from separate application server and database server monitoring tools. An almost impossible task.
Support for Web Technologies
While considering the above topics to search for a cloud provider, you may also want to consider using MULTIPLE cloud computing providers. There are several advantages to this approach.
First, multiple providers may allow you to test from more geographical regions. Earlier, we've talked about how testing from multiple regions can provide a more realistic test scenario. With this in mind, combine the regions available from multiple providers can lead to an even MORE realistic scenario. In other words, if a single provider does not give you the geographical coverage you need to emulate (based on where the real world users of your application "live"), aggregating multiple regions can be very useful.
Second, if you are executing exceptionally large scale tests, engaging multiple providers simultaneously allows you to bypass any limitations that a single provider may place on bandwidth or the number of machines in use.
Finally, using multiple cloud providers enables you to detect potential network issues at the cloud provider level. Let's say that during results analysis of particular test run you noting that there is significantly worse performance measure from load generators from a particular cloud provider while all other load generator show acceptable performance. With this data, you can safely conclude that there is a problem (temporary or not) with that provider and not your application. If you had locked yourself into a single cloud provider, you might be limited in your ability to conduct accurate large-scale tests.
Latest Cloud Developer Stories
Subscribe to the World's Most Powerful Newsletters
Subscribe to Our Rss Feeds & Get Your SYS-CON News Live!
SYS-CON Featured Whitepapers
Most Read This Week