A Maturity Model for Application Performance Management Process Evolution
A model for evolving organization’s application performance management process
Jun. 23, 2011 10:30 AM
As IT systems form the backbone of business operations, their performance plays a key role in business growth. Understanding this fact, organizations work toward obtaining best performance from the software systems to maximize ROI on IT. Now an application's performance can be improved by tuning numerous factors like the underlying infrastructure, deployment configuration, application architecture, design, workload, etc. Yet there is another important factor driving the performance of all applications of an organization - the performance management processes adopted by an organization. The performance management process consists of activities performed to get a better understanding and control of application performance.
Here we present a maturity model that will help organizations evaluate and evolve their processes on certain key dimensions. The scope of the model presented is limited to the activities that have to be carried out as part of the performance management process. In addition, as the people and technology used for process implementation are equally important to achieve the required success, equal emphasis has been given to them. The model describes a six-level evolutionary path to progressively mature performance management processes in an organized and systematic manner. With the increasing maturity of the performance management process implementation, organizations can see a positive impact of the performance engineering adoption on the business.
Levels of Performance Management Maturity
Figure 1: Performance Management Maturity Model
Next we detail the characteristics of a project at various maturity levels. The necessary activities, team, and technology required to achieve higher maturity are also highlighted.
Level 0: Ad-hoc
Transition to Level 1: Systematic Problem Resolution Mechanisms
Systematic performance resolution processes with phases like discover, detection, isolation, and resolution are used to fix the problems. Though a process is followed, it focuses only on performance issue resolution. The same tools used at maturity level 0 are used for problem identification and resolution, but in a more structured manner. Performance management strategy is limited to bottleneck analysis and tuning, so technology domain experts are called in to fix the performance problems in production. They follow defined methodology to do application (configuration) tuning until the performance targets are achieved.
Though using a systematic process reduces the application downtime, performance problems and occurrence risks still persist. The steps taken are still for problem resolution, not for identification and the removal of imminent performance problems. So under unusual circumstances, such as holiday transaction volumes, performance SLA violations can occur.
Transition to Level 2: Robust go-no-go gating mechanism
To ensure this, a comprehensive performance testing methodology is put in place so that all applications are deployed in production without any performance bottlenecks. Such robust gating mechanism ensures that applications are tested thoroughly for all the business-critical steps under production-like situations.
At this level, performance testing is recognized as a separate function of the software development process. The focus shifts toward a more planned process of problem identification before the deployment of an application. During performance testing, various load testing and monitoring tools are used for performance problem identification. Exhaustive performance test plans are prepared by experienced performance analysts who understand the application and its performance goals. Performance test engineers, skilled in using performance testing and monitoring tools, record and run the scripts. In addition, the performance experts analyze the performance test results and recommend the appropriate steps to be followed to resolve the performance bottlenecks. Thus, there is a cost for the specialized people involved in the project.
Though a problem is detected and resolved before it occurs, it's a onetime activity performed at a much later part of the software development life cycle. For early detection of performance issues, there is a need for a more strategic method that puts a planned performance analysis process in place throughout the application development.
Transition to Level 3: Early detection of performance issue through validation
At maturity level 3, organizations use processes that can help them identify problems as the application is built by validating each stage of an application development life cycle. The defined architecture, along with each of the individual components and modules, are validated for performance. The deployment configurations and hardware capacity are also validated before procuring them.
The "V" model performance validation process is followed to validate the performance at each development life cycle phase. The process can include activities like NFR analysis, workload analysis, architecture validation using performance modeling, and code profiling. Performance profiling, testing and monitoring tools are used to continuously track the performance, which is an important task for performance validation. Performance modeling and simulation tools are used to carry out the architecture and design validation. Here the performance analysts are involved in all phases of the development life cycle, unlike level 3 where they are required only at the system integration phase. The technology experts are required to review the design and code for performance. Also the people knowledgeable about performance modeling, profiling and testing are required to verify the performance at all stages.
Even if the strategy followed ensures that the performance issues are identified and eliminated quite early in the development life cycle of an application, it's still carried out as a reactive process. It, thus, remains a onetime reactive activity performed a little earlier as compared to the level 2 maturity processes.
Transition to Level 4: Proactive Holistic performance engineering solution
At this level, a proactive and holistic performance engineering process is followed to take care of application performance during all phases along with the validation phases to further ensure their absence. The exercise starts right at the requirement gathering stage wherein performance requirements are also captured. Then throughout the development life cycle and across the technology stack of an application, it's tracked for how performance is getting engineered into an application by following available best practices.
A performance engineering process is integrated along with the development process to build the performance into an application. During the requirements-gathering phase, performance requirement gathering is included. The architecture and design phase involve architecture creation using patterns, anti-patterns, modeling and validation. The build phase follows technology best practices and includes code performance review and profiling. The system testing phase is followed by performance testing and analysis phases to deliver a scalable application.
The technology adopted is more for the engineering exercise starting right at the beginning of the application development life cycle. Non Functional Requirements gathering is done using workload modeling tools and techniques. Then performance modeling and simulation tools for architecture validation are used. Later stages use profiling tools for code validation, and performance testing/monitoring tools to benchmark the application performance.
Performance experts are involved in the project from the requirements gathering phase to focus on the application performance. Architects, experienced in the underlying technology, create the application architecture and performance architects review it. People with performance modeling knowledge evaluate the architecture for bottlenecks and scalability. Next the performance analysts review and test the design and code. Last, but not the least, specialized testers, engineers and architects do the performance testing, analysis and tuning.
Though here the organizations have a proactive approach for preventing performance-related problems, it ends once the application is developed and deployed at a production environment. It lacks a feedback mechanism for converting a proactive approach toward performance to a continuous one.
Transition to Level 5: Proactive continuous performance optimization solution
Organizations, at level 5, make sure that the application's performance goals are always met in production even as the time passes. They establish continuous monitoring, enhancement and optimization mechanisms. An application's performance is continuously monitored for tracking the performance behavior. Performance modeling and simulation exercises are carried out to foresee any imminent performance issues. A feedback mechanism is put in place to take corrective measures before the performance starts degrading.
At this level, the Performance Management process is considerably evolved, with a proactive and continuous vision toward application performance maintenance.
A proactive and continuous performance engineering process and capacity planning process is put in place to take care of the application development, re-engineering and problem resolution.
A thorough capacity management solution, including tools for monitoring, modeling, forecasting and simulation tools, is used. These tools help convert a simple performance engineering exercise as follows at level 4 to a continuous and proactive engineering solution.
The profiles of people involved in project development is the same as at level 4 as all the activities are done here as well. In addition, at this level, system administrators can use the available set of tools for continuous monitoring and will understand the alerts generated at various stages.
The performance management process at maturity level 5 has the following characteristics:
Table 1 summarizes the performance management characteristics across the dimensions of each level of maturity.
Alignment Among Dimensions for Transitioning to Higher Maturity Levels
Figure 2: Alignment of 3 Dimensions in different organizations and their maturity level
Impact of Maturity Levels on Businesses
Figure 3 depicts the application's cost distribution across various buckets at each maturity level of the performance management process. The cost is divided into percentages spent at each stage of an application's life cycle. Organizations at lower levels do not consider the business impact of the abrupt degradation of an application's performance and subsequent ad-hoc responses. So at level 0, the majority of performance management exercises are carried out at the production stage, mostly as a response to a performance issue in an application. As the cost spent on performance is on an ad-hoc basis and highly unplanned, it becomes difficult to keep track of the ROI on system development and maintenance. As an organization moves higher up the maturity level, the importance of performance management becomes apparent, so the cost spent on performance management increases at earlier phases of application development life cycle and reduces in production.
As seen in Figure 3, the cost spent in production reduces across the maturity levels while for other phases involving design, development and testing, it slightly increases with increasing maturity.
Though there are no immediate returns from the investments on performance exercises during the earlier phases, there is a long-term vision in place. This guarantees a lower overall expenditure on performance issues resolution, as shown in Figure 3.
Figure 3: Change in cost distribution across SDLC for different maturity levels
As an organization's processes moves to level 2 and 3 using performance validation to filter out critical performance issues, the risks of getting the performance issues in deployment decrease. Yet performance is not very predictable, any changes in the application's usage, enhancements and/or environment can increase the risks of failures.
To tackle this problem, the next level process involves system architecture, design and code reviews and validations to ensure performance for changing environments, thus lowering the risks.
At the next level, system behaviors, metrics affecting system performance and other business measures are captured, reported, analyzed and predicted to help find ways to deliver good performance continuously, thus minimizing the risks of failure proactively. Consequently there is a lower risk of failure and customer dissatisfaction.
Summary and Conclusion
Reader Feedback: Page 1 of 1
Latest Cloud Developer Stories
Subscribe to the World's Most Powerful Newsletters
Subscribe to Our Rss Feeds & Get Your SYS-CON News Live!
SYS-CON Featured Whitepapers