Important things you should know about application performance

While working on multiple performance improvements, I worked on a range of product that varied from legacy enterprise apps to startups. I spent enough time identifying bottlenecks in the applications, as well as fixing them, which is a big learning for me.

When I heard a team that just does performance improvements, I always thought how is it possible to give you 100% improvement in a use-case? I come from a background where every millisecond of the application was measured and everybody always thought about load while designing applications. So I had never thought that there will be such inefficiencies in the applications.

Of course, after working with a range of products, I realized that even if you have best developers of the world working for you, you may still end up in inefficient code. Here is why.

I see two categories of organizations that have such junctions of inefficient code.

1. Legacy code →

case 1 → I see a lot of organizations that hail legacy code from Java 1.2. Some of them even use HashTable and Vectors as their primary data structures. So what happens is even if the code is highly efficient at the time it was written, it becomes outdated after some time. If you do not upgrade your code to be compatible with latest development technologies, slowly you start missing the special code structures that language provides. So 10 years down the line, your code is highly inefficient compared to its peers.

Case 2 → This scenario deals with the databases. Sometimes the products are reluctant towards upgrading to latest database related technologies. It may be just upgrading versions, or may be just migrating to a new system that suits your needs. There is another case, too. Sometimes people upgrade the database when it releases a new version, may be because the updates are free, or because project has more budget. But they do not really evaluate if there are any new features they can use. e.g. I recently heard a usecase from my colleague where he got 5 times improvement just with changing a database query to take advantage of latest features released by SQL server.

Case 3 → Sometimes the vendor providing you the technology stops providing support to the technology. Or in case of open source technologies, the community support drops down. I remember an example here, there was a project that was using gwt as a UI framework.

It was good in old days where functional programming wasn’t a buzzword, and Angular hadn’t picked up. But after evolution of Angular like frameworks, the developer support of gwt dropped down significantly, which put us in a big challenge when we got a gwt usecase to improve.

We had almost no material available on the internet to tackle the issue we were facing. GWT indeed had a workaround to the problems we were facing, but it took just a complete week to rectify the same.

So in a nutshell, in case of legacy codes, the reluctance of team to adapt the code to technology advancements leaves the code behind the world.

2. Fast moving startups

Startup theoretically has everything. They usually have world class talent to work for them, they work on cutting edge technologies, they are agille, tool. But still they face performance bottlenecks.

Here is why...

In case of startups, the pace of development is the key pain. Because business is not setup, the domain always keeps changing, and so the requirements. This leaves them to find out the tools that can help them do the rapid development. Often these tools have an overhead, and put some load onto the server.

Recently I worked on a project, where it had several issues in spite of being a startup.

It was a typical Spring + angular stack, and spring services used to return the JSON data which angular application used to consume. Now, definitely at some point of time, they must be moving at a huge pace, and data being returned from server was not defined, and at a situation, Angular client might need any data at any point of time. The solution that they found was interesting.

They started returning the complete related data model as part of the response. This worked well, but that came along with another issue of data abstraction. People were able to view the data they were not authorized to. The solution that was introduced to handle the case introduced a huge pullback onto the performance.

What they did was, they introduced ACL through annotations onto every field in the datamodel, and a generic response processor in the response pipeline that will hide the fields the user is not authorized to view.

While this looks like a simple solution, it actually had a serious drawback. The response processor used runtime annotations based processing, which would actually use java reflections to find out the access level of the object. This was Ok until they had less number of users on the platform, but once the platform had sufficient number of users, the response had millions of fields and processing of such million fields actually took seconds to execute, where actual execution of business logic was taking milli’s. Its needless to say the fix was simple. We just created few response objects based on usecase, and turned off the response processor to get ~8 times improvement.

Based on my experience, here are the checklist to look at

1. Avoid lazy loading of fields → Its true that lazy loading helps you load related objects on demand, and at the same time saves you from writing complex queries, but when you are planning to process thousands of such objects, every lazy load is essentially a db trip, which results in delayed response. So if you are processing 1000s of objects, its better to load them early than on demand.

2. Avoid heavy joins → Of course there can be an argument from other side, heavy joins on the tables that have millions of records causes bottlenecks on the database. It may be because of database configurations, but on a shared database where resources are limited, its better to avoid such joins. In one of my use-case, I had got 3 times improvement just with decomposing a four level join into corresponding SELECT statements with IN clause. Moreover modern databases are now capable of handling even 25K parameters for IN clause, so practically there no limitation on it.

3. Make sure you have correct indices → Index scans are much faster than unindexed scans, rather indices are designed to facilitate faster access to the table. If you have any combinations that are accessed frequently, you can consider creating index for them, it can effectively give you 100 times better performance than its counterpart.

4. Avoid expensive Aggregations on application server→ There was a time when my manager used to tell us that no matter what you do, do not put too much load on the database, because you know you can scale application server horizontally, but scaling database server is a problem. While this statement still holds true, lot of things have changed by then.

Aurora servers on aws give you virtually infinite capability of storage, and you can extend it based on your need. In such cases, sometimes it makes sense to execute aggregations onto database rather than Application. Consider a usecase, where you are trying to track the events done by users. If user clicked on your page 5 times, you choose 1, and if he clicks 5-10 times you choose 2, and so on.

And all you need is the sum of all these 1, 2’s. Now, for every user, you have 1000 records, and you are trying to fetch data for 100 users. In such case, if you still stay firm on avoiding complex queries, and you load everything onto application through simple queries and left aggregation to application, you will end up in a problem.

In spite of correct indices, loading of such data still take few seconds, provided a shared database server on a cloud. Along with the time spent in network trips, it can also create a memory spike onto database and application side, which can affect both time and space complexity of your application.

5. Effective use of caches → This problem is often seen on legacy products. Most of the times, there are few tables that have less data, and accessed very frequently. Not sure why, but I have seen few legacy products not implementing caches for such obvious scenarios. All you need to do is to verify if you have caching available anywhere, and use it. In worst case, just a plan WeakKeyHashMap based in-memory cache can also give a significant improvement.

6. Effective use of partitions → Partitions are a good way to improve the load on database, but not always. Especially with servers like MySQL, where it does not support partitioned joins, partitions can practically become a headache. Partitions are extremely helpful when

1. You have really really huge data, rows more then 250m

2. You do not have complex joins, or the server supports partitioned joins.

3. You have query that does not go across partitions. May be loading data from 2 partitions out of 10 is acceptable, but definitely if your query is scanning 5-6 partitions out of 10, then its a problem. So in a nutshell, partitions are extremely useful in targetted queries, but not really good in case of bulk loads, At least with MySQL. Oracle and MSSQL have more efficient partitions as I read.

Partitions, if used wisely, can give you sizable boost, and if not used so, can give you a setback too.

7. Evaluate client side waterfall → Client side waterfall is basically the sequence of web calls made by browser. Sometimes these calls have a dependency within each other, Sometimes not. Executing these calls in parallel helps inprove overall wait time for user. At the same time Browsers have a mimitations of executing only 6 calls in parallel per server. So if you put 20 calls for a server in parallel, they are efeectively 6 calls at a time. Tuning this scheduling can give you a boost, too.

8. Try compressing HTML response → Sometimes you cannot avoid the amount of data you want to send from server to client. But you can send compressed data over the wire so that it doesnt eat up network bandwidth. Typical ratio of text compression is < 5%, which means if you compress the response, you can achive the same data within 5% of network time. Although enabling compression is a judicious call. For small responses, compression is an overhead, so its always a trade off.

9. Avoid use of HTML tables → HTML tables are rendered as All at once, and thus rendering table involves lot of computations. If you have a table that holds 1000 records, rendering it through Tables is 10 times slower than rendering it through divs. Along with it, adding a row in the table on the fly is equally expensive. Rendering HTML divs is much cheaper than it, and should be always preferred over HTML tables.

10. Parallelize the IO calls → If you are making thirdparty calls that take lot of time, you should consider parallelizing them. Webservice cals just keep the thread blocked, but do not put additional load onto server, so you can very well try to execute them in parallel. Where as the other calls like DiskIO may not leave tham much room for parallelizing.

11. Avoid heavy style changes in browser → Its Ok to use render data or make small style changes, however, when you want to replace complete layout of the page, its always good to use HTML template rather than changing the DOM/styles in JS. Browsers are not created for such computations and it severely affects user experience.

On the other side, here are few tips if you want to save your application from being “slow”

1. Think of performance as “infrastructure cost” → Bad performance doesn’t just mean bad user experience, But it also results in more infrastructure to serve the same load, which can actually sometimes can pay few extra developers. I have seen organizations that saved millions of dollars just by optimizing aws usage.

2. Keep yourself updated with technology advancements → You may not want to take every update the technology has, but at least you should know what it has. e.g. as I mentioned above, in one of the examples, the application was compatible with latest SQL server, but it was not using all of its capabilities. If you evaluate the new features in technologies time to time, it can help you take this advantage ASAP.

3. Avoid same query being executed on database multiple times → If you have the same query going to database again and again, then you have a design problem. 1 db query is 1 round trip, and if table is not available in cache, loading the data again. If you make such 1000 trips to the database, you will spent 90% time in just IO and 10% time in actual processing. Where as 1 consolidated query can result in saving these 90% IO time, which can save few pennies for you.

4. Make sure you have load tests for critical paths of the applications → Critical paths of the applications are the ones that frequently accessed, or which have a vital role in overall user experience. Any adverse impact on these paths can cost in terms of infrastructure or user experience. So you should have test cases to identify them early.

5. Prefer direct method calls over reflections → Even though modern Java reflections API’s are much efficient than earlier ones, they are still not efficient than plain mnethod call. Hence if a method id called thousand times via reflection, it has a significant impact on performance than its counter part. There are a bunch of libraries that give you power annotations via codegen, rather than reflections. Lombok, mapstruct are well known examples of the same. You still use annotations based language style, but instead of using reflections to resolve the annotated functionality at runtime, these annotation processors generate few more lines of code during compilations that can effectively save some time for you.

6. Use in-application profiling tools like newRelic, which are lightweight, and still can proivide deeper insights in the applications. These tools can be deployed even in production, and provide you realtime status of the application, and also can discover few bottlenecks which can never be found in development.

Caution: NewRelic instrumentation on ResultSet.read() methods can cause serious performance inefficiencies if you have sufficiently larger data set, Of course newRelic provides configuration to turn it off, but one should be aware of it.

7. Give back → If you are using an opensource project, and you found an issue, you should actually contribute to the project solving it. Such people contributing open source keep the project alive, and eventually help you getting new improvements.

A quick step by step guide to drill down performance problems