Practical Web Load Testing

Testing from cloud

2009-10-22T14:55:00.003-04:00

I am pleased to announce the primary feature of the Load Tester 4.0 release - generating load from the cloud.

On the services side of the business here at Web Performance, we do a lot of load testing for our customers. One of the advantages we have is our network of load generating engines deployed across the continent. This makes it easy for us to generate load from outside the customers network. Maintaining this network would not be cost-effective if we weren't using it on a daily basis. So for our software customers who want Load Tester to generate load against their system from outside their network, it can be both cost- and time-prohibitive to setup the needed infrastructure.

Load Tester 4.0 changes that! At the press of a button, any tester can have tens or even hundreds of load engines ready to throw load on their website. Now, we are not the first company to offer this ability. But we are the first to combine that with the maturity of Load Tester's complete feature set - such as the rich analysis reports and the ability to handle complex applications. Also, since Load Tester is not a hosted application, you can use the same tool for performance testing both inside and outside your network.

If you would like to learn more, try our video and tutorial!

I hope you are as excited about these new capabilities as we are - and I look forward to hearing your feedback!

Chris

Testing the Zend Optimizer

2007-11-06T13:51:00.000-05:00

I have recently completed a series of tests of various PHP optimizers. We measured the performance impact by load testing a reference PHP application (SugarCRM) and calculating the capacity of the optimized and un-optimized systems based on objective performance goals. The first test subjected the Zend Optimizer to a round of tests and concludes that the optimizer had no measurable impact on system capacity.

For more details, read the full article.

Coming soon: testing the performance impact of the Zend Platform. Will it fare any better?

Response to VMware blog response

2007-09-24T16:29:00.000-04:00

A few months ago, we posted our test results of some testing we had done in our test lab comparing native-to-virtual performance using VMware Server. Our report could have been more clear and the title could have been better, resulting in quite a stir on Slashdot that morning.

A VMware blogger responded to our article here: http://blogs.vmware.com/vmtn/2007/04/response_to_loa.html.

The author called the test an apples-to-oranges comparison, but is mistaken in a number of ways.

The author assumed that our report tested a VM configured with a single CPU. This was not the case - the VM was configured for dual CPUs in order to match the native configuration. We thought that was pretty obvious...but we should have been more clear.
The author implied that the host OS would be swapping because we had not configured the memory parameters correctly. This was not the case and we said so in the article (specifically we said the VM had 2G of RAM and the host had more). We didn't state exactly how much, but for the record it had 4G of RAM, so the host OS (Linux) was most definitely NOT swapping.
The author claimed that the performance drop was only 16% instead of 43% because we should have been using the hyper-threading-disabled results for the native machine. This is not in the spirit of the test, which was to determine the virtualization overhead on the same hardware. To purposely handicap the hardware during the native test would have drawn invalid conclusion because (1) this is the default configuration for this hardware and (2) anybody who compared with and without hyper-threading would be running the server with this native feature enabled.
Item number 7, "make sure the memory, disk and networking configurations for both physical and virtual environments match" was done in the original tests as well.
The suggestion to run multiple VMs instead of one (#8) seems a bit odd in this context. The point of the test was to compare the performance of a single web application in native mode and virtualized. Adding the overhead of a second VM (and a second guest OS) does not seem like something that would improve performance. Maybe it's worth testing, though.

We had initially planned to re-run the tests with the suggestions from VMware. But after more internal discussion, they do not seem to align well with the purpose of the report, which was to determine the performance cost of virtualization using VMware Server measured by the capacity of the system's ability to serve multiple simultaneous users while meeting specific performance requirements.

Load Testing a Web Service

2007-06-15T11:42:00.001-04:00

I ran across an article today describing the process of load testing a web service:

http://webservices.sys-con.com/read/284564.htm

Overall, it is not a bad introduction to the topic for readers new to load testing. However, I think it missed out on a few key factors:

1. Ramping load tests - The tests ran by the authors stepped immediately to the specified load level. This is mildly interesting if that load level is the only thing you care about. But in most cases, we want to know more - specifically we want to know how the performance changes as that load level is attained. We usually recommend ramping load tests to our customers - as this conveys a lot more information about the performance of the system. Instead of giving you a yes/no answer to a simple question (are the performance goals satisfied at 100 transactions/sec), you get much more information about the performance profile of the system. If the system performance has not degraded at all up to the maximum tested, it is a good indicator that they system has considerably more capacity available. If the system has degraded, then you know the load level at which the performance started to degrade...and may even have a feel for the maximum capacity.

2. Multiple scnearios - Few systems are likely to offer only a single web service - they will typically offer many. The selection of which scenarios (services) to test and the proper mix of the scenarios is critical to performing an effective load test. I was disappointed that the authors did not mention this point at all.

Measuring the performance cost of virtualization

2007-03-28T09:14:00.000-04:00

We've recently virtualized many of our internal servers using VMWare Server. We did this primarily to ease maintenance and backups. We were able to do this cost-effectively because all the servers, in physical form, were grossly underutilized -- IOW, they ran at 0.1% CPU 99.9% of the time. So this had the added benefit of letting us turn off a bunch of servers...hopefully reducing our energy consumption a little, too!

As a performance testing company, it seemed natural for us to be concerned about the performance impact, so we decided to measure it. We did the tests and wrote the report back in January. A number of incidents conspired against getting the article up on our site in a timely fashion...but, better late than never:

Load Testing a Virtual Web Application

Load testing outline

2007-03-26T13:42:00.000-04:00

Load testing can be a daunting task for those approaching it for the first time. Since load testing is most effective when applied to an entire system, it brings all the subsystems into play - requiring analysis of application use cases, applications and frameworks, servers, operating systems, databases, networks and more. There are lots of things to measure and analyze...and lots of opportunities to get off track.

The user experience
Because it is so easy to get lost in the details of a complex system, it is important to keep focused on the real goals of the project. With few exceptions, the goal of load-testing a web application is to determine if the user experience will be satisfactory. The user experience is what matters - and your testing activities should be focused on gaining a better understanding of how the application performance will effect the user experience. Keeping this in mind, let us examine some of the basic steps involved in load-testing a web application.

In my experience, the load-testing process should generally follow these steps:

Determine the purpose of testing
Choose what to test
Prepare the test environment
Measure the baseline performance
Model and test individual scenarios
Analyze results
Model and test the user population
Analyze results

One might think these are pretty obvious, but I've found that when you're down in the trenches and somebody is screaming for results, many of these steps are skipped. Most commonly, far too little thought is put into #2 and everything else before step 7 is skipped entirely. Note that practical load testing is focused on results - and good results require planning. On the other hand, don't mistake planning for needless paperwork and pointless meetings. Practical testing is the best of both without the worst either - agile, quick and to-the-point. Skipped steps can only lead to mistakes, wasted effort or misleading results.

Here's a quick summary of each step:

1. Determine the purpose of testing

Question: Why are you going to run a load test?
Answer: Somebody, somewhere, has questions about the performance of the application.

These questions need to be answered before certain higher-level decisions can be made. If you are not sure what these higher-level decisions are, then you might end up answering the wrong questions...or even worse, providing the misleading answers to the questions. I encourage any tester that is not sure of those higher-level decisions to ask the higher-ups for a better understanding of the context of the task. If necessary, provide gentle reminders that knowing the greater context of testing effort will help you to design tests to answer the questions quickly and accurately.

2. Choose what to test

This is one of the most important prerequisites for practical load-testing:

Testing the wrong things can result in useless or misleading answers.
Superficial testing may fail to reveal important problems.
Testing too thoroughly can waste time and money or needlessly push back project deadlines.

Choosing which scenarios to test can be one of the hardest parts of the process. To do this effectively, the tester will need a good high-level understanding of the application and it's users...or access to somebody else with a solid understanding. I'll address this topic in more detail and hopefully I can help alleviate some wasted effort.

3. Prepare the test environment

The effectiveness of the test effort is limited by the test environment and tools available to do the job. Picking the wrong tool or having an insufficient test environment can jeopardize any testing effort. In a less-than-ideal test environment, knowing the shortcomings and how to deal with them will maximize the effectiveness of the tests.

4. Measure the baseline performance

In many cases, the test results cannot be interpreted correctly without understanding of the baseline performance of the system, i.e. with a single user accessing the application. There are some easy ways to measure and analyze the baseline performance - I'll have some tips in a later post.

5. Model and test individual scenarios

Each scenario to be tested must to be checked closely for accurate simulation. There is no point running a load test before you are confident that the testcases are correct. Additionally, load-testing each scenario individually will help to put the results from multiple-scenario tests into perspective.

6. Analyze results

While the purpose of this step is pretty obvious, the path to understanding frequently is not. Even small load tests can generate massive amounts of data - determining what is important and what is not is key to practical load testing. At this point in the process, you will be primarily focused on the big problems -- which are not difficult to spot when you know where to look.

7. Model and test the user population

Once you have individual user scenarios modeled correctly and understand their base performance, you are ready to test the entire application. If any questions about the composition of the user population were not answered during #2, they will need to be answered before completing this step.

8. Analyze results

Here is where you will start getting the answers that management needs to make strategic decisions. In later posts, I'll explain how we summarize the massive amount of data collected in a series of load tests into something that management can use to make critical decisions.

Coming soon...
In the next post, I will address #1 - Determine the purpose of load testing. Determining what the goals of the effort should be...and maybe more importantly, what they should not be, is one of the keys to practical and effective load testing.

What it's about

2007-03-16T08:51:00.000-04:00

Before I start boring anyone with the minutiae of load testing, it would be prudent for me to define what Practical Web Load Testing means to me.

Practical - maximizing the benefit received for the effort expended.
Web - Browser-based applications using the http/https protocols.
Load Testing - simulating a realistic usage scenario to determine performance and/or capacity.

Rolled together, Practical Web Load Testing means efficiently and effectively testing a web application to determine if it is likely to meet the performance requirements.

Load Testing can encompass a wide arena of testing with varying terminology. Some of the more common variations are:

Performance testing - determine if the system meets the required performance goals at the expected load levels.
Capacity testing - determine the peak capacity of a system
Stress testing - testing at higher-than required performance (e.g. more hits/sec, more users, more bandwidth, larger transactions, etc).
Failure-mode testing - testing in various ways that cause system failure - usually for the purpose of determining if the system fails gracefully.

For purposes of this blog, Load Testing will be used to refer to Performance Testing and/or Capacity testing. Why this focus? For most development organizations, the driving factor behind load testing usually comes from one of these questions:

How will the website perform with the expected number of users?
How many users can the website handle and still meet performance goals?

The first question is frequently asked at the end of a development cycle - just prior to deploying an important new IT system. The second is frequently asked during planning stages. For instance, just before making an existing application available to a much larger audience. I hope that articles here will help you determine how to answer those questions quickly and effectively.

BTW, Who am I?

Almost forgot -- my name is Chris and I work for Web Performance. I've been helping organizations load test their web applications for about 6 years. Naturally, I am more than a little biased towards our load testing tool, Web Performance Load Tester. We work hard to deliver the most practical load testing product available -- that is, one that delivers the features you need at a price you can afford. I hope you'll give it a try!