Modern Application Performance Testing
Everyone agrees that application performance matters. The more difficult issue is defining application performance. Is it how fast the servers respond? The time from user click to server response? How well the system handles five hundred or ten thousand users? Depending on the application, the answer can be any or all of those. Therefore, what matters most is the ability to test applications to ensure they provide performance appropriate for each definition’s use.
In order to understand if performance matches needs, testing is a necessity. While there are many areas that help define testing parameters, three overarching testing concepts must be addressed in order to provide appropriate performance for modern applications: your users, your data, and your environment.
What Are Your Users Doing?
While almost everyone recognizes that it’s important to understand your users, that knowledge is not often formalized. When it is, the formalization often happens within the testing groups, almost completely separately from development and operations. Performance testers have a unique opportunity to work with development and operations teams to fully understand how the application is used in production.
Therefore, it’s important to focus on conducting a production traffic analysis early in order to build user cases and performance simulations that model how the business workers do the work the application is going to assist or replace. The analysis can be completed using logs, custom metrics, or monitoring tools and should involve the operations team, as they have knowledge of the real-world issues that need to be included in testing. Performance test models should be assessed annually, at a minimum, and compared to production to ensure that performance tests grow and mature along with the application. This activity opens the lines of communication among all teams involved in the software development lifecycle and helps create a feedback loop, which is the cornerstone of the DevOps movement.
The goal is to ensure a clear definition of what performance metrics matter to the business and to guarantee that your performance metrics are aligned with ever-changing real-world needs.
Leveraging Test Data Management
Data has always been important, but the growing complexity and volume of data in modern applications means that test data management is gaining focus. Formalizing what data is available as well as how it is accessed from sources and delivered to customers is critical to providing applications that meet performance objectives.
In an attempt to keep on schedule and minimize the impact on production, test data is often generated behind the scenes. That can result in clean, generic data, sometimes with “unimportant” fields left empty, or little variation and not enough outliers. Other times, data is generated to check certain validation routines in unit testing, then left in the performance testing data sets, providing nonrepresentative data and introducing a large risk to the validity of any testing completed. It is critical to get full data sets from operational systems to provide system testing that replicates the real data that will be managed in the system as closely as possible.
The size of the data sets is also critical. The volume of data in use by applications is increasing rapidly, and performance doesn’t change in a linear fashion. It’s important to make as few assumptions as possible about scaling to larger data loads. Test with a data set that is as large and realistic as possible. Even with larger test sets, the difference between test and product data volumes and variability are such that you must always be prepared to continue testing in production to understand true application performance.