How many users should I load test my system with?
August 20th, 2008When customers are preparing to buy our load testing software, they frequently ask “How many users should I load test my system with?”. Naturally, we can’t answer that question without a lot more information about the performance requirements and the expected usage of the web application.
If the application will service a known number of total users (common with internally-deployed systems), then it is a simple matter of estimating how many users are likely to be using the system at the same time. For example, if no more than 10% of the employees are expected to access an HR web application at the same time, you would test up to 1000 virtual users in a company with 10,000 employees.
However, the customer frequently has only rough estimates of expected traffic. For instance, they may know how many page views/hour that they want to handle or there may be web server logs from an existing system that can be analyzed. While that information is not ideal, it is usually enough to get a ballpark figure for initial testing.
In either case, the first step is to determine which usage scenarios will be tested. Each usage scenario defines a series of pages on the site that is visited in order to accomplish a task. For instance, a typical scenario in an e-commerce application might consist of these steps:
- Navigate to the homepage
- Search for a product
- Choose an item from the search results
- Add the item to the shopping cart
- Complete the checkout process
If server logs are available, these may help to identify the most commonly visited pages - and from that it may be possible to identify the most common scenarios. In most applications, there will be a number of scenarios to be tested, but for the purposes of this discussion we will assume there is only one.
The next step is to determine how much time each user spends at the site. If server logs from an existing system are available for analysis, a log analysis program may reveal how long, on average, a user spends on the site. That would be a good place to start. Another common technique is to recruit 5 people who are not familiar with the system and have them perform the usage scenario. An observer would measured the amount of time spend on each page during the process. If that is not possible, then some guesswork will be required. For our example, we will use a visit duration of 2 minutes.
Armed with a number of pages per testcase and a testcase duration, we can now use the desired number of page views/hour to compute the number of virtual users. In our case, we will use 100,000 page views/hour as the target load level.
For our calculation we will use these variables:
PR = page rate (pages/hour)
TD = testcase duration (minutes)
NP = number of pages in the testcase
VUs = the number of virtual users required to generate the desired load
VUs = (PR * TD) / (60 * NP)
In our example:
PR = 100,000 pages/hour
TD = 2 minutes
NP = 5 pages in the testcase
VUs = (100,000 * 2) / (60 * 5) = 667
667 Virtual Users would be required to generate 100,000 page views/hour in the above scenario.
Note that this assumes the system performance does not degrade from the baseline performance. As the performance degrades, each user will naturally generate a lower page rate, since each page will take longer to complete - thus raising the testcase duration. Depending on what the expected or acceptable level of degradation is, the number of VUs may need to be raised to achieve the desired page rate.
If each of the 5 pages was expected to degrade by 3 seconds, that would raise the testcase duration to 2:15 (or 2.25 minutes). Plugging the new testcase duration into the above formula would raise the VU requirement to 750 simultaneous users for the load test.
Good luck!
Chris
