Performance management is a challenging task. We often encounter development teams that acknowledge the need for improving performance but always have something on their plate that is more urgent — whether it’s completing the current sprint or finishing necessary environments or tests. In this post, we’ll look at different tools and process options for how to effectively address performance management.
Do I need to invest in performance?
Before jumping in and building custom testing frameworks and solutions, the most important thing to do is understand the real impact of performance on your end users. Bringing a new product to market quickly and sacrificing performance for the ability to validate ideas rapidly is fine. You accept the short-term risk and communicate it to all stakeholders.
However, if you are operating a large e-commerce business, for example, application speed and availability is critical. In any customer-facing application where downtime is directly translatable to lost revenue, performance is a high priority. Good questions to ask yourself to understand the investment you are willing to make are:
- • How much does one hour (or one day) of application downtime cost me?
- • How many performance incidents have we had in the past month or the past quarter?
Performance is a priority. Where do I start?
Adding an APM (application performance management) tool to your application is the first step. This allows you to monitor how the application is behaving in its production environment and presents possible performance bottlenecks.
APMs take snapshots of the system based on time intervals. With enough hits, APMs can provide a statistical overview of the application. The most common items measured are:
- • Request time
- • Number of requests
- • Errors
- • Uptime
- • Meta-data about the user (OS version, country, browser version, etc.)
APMs will alert you if something is misbehaving. This is where the engineering team needs to react. Depending on the organization, the production environment may or may not be under the control of the development team, so once an issue is discovered, it’s investigated by a team who needs to determine if it’s the environment configuration, the application code itself, or something else causing the problem. If it turns out to be an application issue, it’s fed back to the engineering team who will provide the fix. The engineering team might use additional tools like profilers to understand the root cause and to implement a fix.
The main benefits provided by using an APM include:
- • Automatically discovering issues rather than waiting for users to complain
- • Determining the impact of the issue directly
- • Providing an overview of the health of the production environment
- • Guiding the investigation into an issue
The big players in this space are Dynatrace, New Relic, CA, and AppDynamics.
I already monitor production. What’s next?
If you already have an APM, there is already an established process around monitoring and handling performance issues. This is good news. However, you are still vulnerable to playing catchup. If the development team delivers a feature and then hears about performance issues, chances are they have already impacted your users. Cycles are then spent to investigate and figure out the root cause. As APMs only take snapshots of the system, the root cause can be complex to diagnose. You are being reactive about your performance issues. Your next step is to become preventive.
From reacting to preventing with QRebel
Taking the next step forward is to start thinking about performance during development and testing. Depending on the available budget, you may have a specific team responsible for performance. They will build and maintain environments and provide thorough testing before any change is published to production. This removes the burden from the engineering team to find any issues themselves.
According to RebelLabs Developer Productivity Report, if a company has a dedicated performance team, only 32% of those teams will fix the performance issues. Engineering teams without a dedicated performance team will be responsible for delivering performance fixes in 96% of the cases, so, in most of the cases, assume whoever creates the bug fixes it.
There are also third-party solutions, such as QRebel, that focus on finding the performance issues early. QRebel finds performance regressions between different builds using automated tests. Actionable reports are then provided, down to the method-level. This enables engineering teams to understand the issue and fix it without impacting end users.
The RebelLabs Developer Productivity Report identifies that the top three performance issues come from:
- • Slow database queries
- • Too many database queries
- • Inefficient application code
With QRebel, you can find up to 86% of performance issues. You can also use static code analysis tools, like Klocwork, to directly analyze the source code. Whether run in test environments or in the developer’s machine, these tools can point out inefficiencies as well as security problems.
The benefits of starting performance management earlier in the development cycle include:
- • Preventing performance issues from impacting end users
- • Speeding up the process to fix any issues identified
- • RebelLabs found that issues coming from production take an average of five days to fix, whereas issues found before production take only three days to fix
- • Checking code changes for performance issues before pushing to production
Adopting solutions to improve the development process
There are many different solutions that can improve performance in production and in development. Having a process built around the tools is just as important as having the tools themselves. Tools and solutions that aren’t adopted into the everyday development process provide little help and can be a waste of time and money.
The key here is automation, starting from a continuous integration pipeline where each developer can get feedback fast. The next step is transitioning from continuous integration to continuous delivery. This enables development teams to push changes to production automatically once they’ve passed quality checks.
Having built a continuous delivery pipeline, the engineering team can start moving toward a DevOps model where:
- • The engineering team takes full control from development to production
- • Manual work is minimized through heavy automation
- • There is strong ownership
According to DORA’s “State of DevOps”, elite teams are able to deploy on-demand multiple times a day, with up to a 15% change failure rate, and it takes less than an hour to restore the service. It also takes less than an hour for a code commit to make its way to production.
Source: DevOps Research and Assessment, Accelerate: State of DevOps, 2018
In a DevOps model, with heavy emphasis on automation and integration of different tools into their continuous delivery, teams have the capability of delivering quality code fast. APMs provide information about the health of their production environment, and tools like QRebel prevent inefficient code from impacting end users. Use these tools as automated quality checkpoints in your continuous delivery pipeline.