Optimising Custom Applications on Force.com

One of the greatest challenges of developing on someone else’s PaaS offering is performance optimisation. When I built large-scale enterprise web systems from the ground up there were so many levers we could pull to improve performance without changing a single line of code: more/better CPUs, more memory, more/better use of indexing/caching/threading, and so on. And if we wanted to optimise code we could choose where to apply our efforts, from the lowest of the low-level infrastructure code to the client code running in the browser.

But when you build code that runs on someone else’s platform you have only one thing you can optimise: your own code.

One of the things that amazes me about building on force.com is how infrequently we need to do performance optimisation. Create a simple custom page with a record or two’s worth of data and a smattering of dynamic client-side functionality and an end user will be hard pushed to tell that its not a native force.com tab. Even more complex pages, with more than a couple of records and more than just a smattering of client-side functionality render pretty damn quickly and are perfectly usable. But the Singletrack system also has a few pages that are really very complex, cover a few hundred records and provide a lot of client-side tools. This post covers the specific topic of how we optimised a custom force.com page that took ~40s to render in its first version and got this down to < 2s.

The problem

Deliver information about a list of contacts, usually around 150-300 in length. Which contacts are returned may be manually set or may be dynamically chosen using a set of non-trivial criteria. What information is delivered is configurable on a per-user basis but typically consists of ~20 fields covering the Contact, its Account, and recent Activity History. Add in a number of tools for manipulating and interacting with the information on the list. If it helps, think of it as a Salesforce list view on steroids.

The first solution

Work out what information the user wants to see about each contact (from Custom Settings). Work out the criteria for selecting the contacts (stored on the record underpinning the view). Dynamically construct the query string and execute. Render as a table using Visualforce (think JSP/ASP/ERB for none-force.com’ers) along with embedded Javascript for all the client-side functionality. The result: ~40s from request to availability in Chrome.

The first optimisation … and some lessons learned

Rule #1 of optimisation; profile the shit out of the system and work from facts not opinions. But profiling isn’t well supported in force.com (basically, debug statements with timestamps are required) so we made some guesses as to where we thought the problem was likely to be in order to focus our instrumentation efforts. Given we were still quite new to force.com at the time we were probably a bit too influenced by our fears and immediately set about instrumenting all the querying. Waste of time, even increasing the number of contacts tenfold the querying accounted for less than 300ms of the request. And in general the server processing was really very fast.

Lesson #1 of optimising in the cloud: profile the shit out of the system and work from facts not opinions.

Instead we turned our attention to the page rendering and this turned up a surprising result. We needed two <apex:repeat /> loops to construct the table; one for the rows and one for the columns. Rendering a table of 3000 rows (requiring 2000 iterations in one loop) was pretty fast, rendering a table of 400 rows with 5 columns (also requiring 2000 iterations but two loops) was not. In fact it was 10 times slower and rendering 200 rows with 10 columns – our most typical use case – was much slower still.

This is when lesson #2 of optimising in the cloud really hit home: you can either do less or you can do differently, you don’t have the option of adding more of a vital resource. We could remove the ability of users to choose which columns they saw (making the column set fixed and removing the need for the nested loop) or we could change the way we rendered the table. In the end we decided to do differently and return all the results as JSON data and construct the table in Javascript on the client. Our first version of this approach gave us a 100% improvement in performance: 20s from request to availability.

However we also quickly worked out that our JSON based solution (this was before force.com released native support for JSON) was still pretty slow due to using ‘+’ for String concatenation in creating the JSON string. Replacing this with extensive use of String.format() gave us another 100+% performance improvement: 8.5s.

The second optimisation … and more lessons

We lived with this for a while. 10s wasn’t great but it was no slower than opening up an Excel spreadsheet (what users had been doing before they used our system) and general consensus was 10s was okay. Of course, what seems acceptable on day one rapidly becomes irritatingly slow and within a couple of months there was a lot of grumbling about performance especially as some people were reporting that the page ‘regularly’ took 20+s to load. This turned out to be rooted in some environmental issues: i) browser caches were being cleared out every time the browser was closed – a not uncommon admin setting in our customers’ domain and ii) the ISP routing for salesforce.com subdomains (used for accessing custom code in managed packages) turned out to be less than optimal – sometimes adding 6-8s to a request. The latter was a real eye-opener and we still haven’t got to the bottom of why that was the case but switching to a back-up ISP and resolving the browser cache issue ensured the customers’ were getting consistent 10s response times.

Once we’d resolved these problems we noticed that the latency in requesting Static Resources from force.com could be pretty high: ~1.5-2s in some cases (as is commonly the case all our Javascript and CSS files were packaged up as a zip file and deployed to force.com as a Static Resource). By moving our Javascript and CSS outside the package and delivering it via Amazon Cloudfront we won’t improve performance in the common circumstance of someone accessing the system via a browser with a fully populated cache but, we can shave a second or two off the overall response time for a browser with an empty cache as well as isolating ourselves from whatever the circumstances that cause force.com to update the timestamps for Static Resource caching (it seemed that every new force.com release required all browsers to completely repopulate their caches causing a rash of performance complaints directly after a major Salesforce upgrade).

Lesson #3 of optimising in the cloud: there’s not much you can optimise outside your own code, but there is more than nothing.

This round of work got us looking at our implementation again. One thing that really stood out was the amount of time we had a ‘blank page’ before any of our data was being downloaded, even ignoring client-side processing of it: 4s. Create a custom Visualforce page with just some text in it and it will have sub-second response time. Given we knew that querying and processing the data took only ~300ms it seemed surprising that it was taking ~3s to download the data. Some investigation here turned up a very surprising result – that the viewstate for the page was huge even though there wasn’t much real state within the page. By ensuring that the scope of the <apex:form/> tags was as narrow as possible (just encompassing the fields and actions within the form) and judicious use of transient variables we were able to significantly reduce the size of the viewstate and this brought the ‘blank page’ time down to below 2s and overall response times of ~4.5s. Additionally, the psychological effect of not staring at a blank page for a few seconds meant that people felt the page was a lot faster than you might expect from a 60% improvement.

Something else we looked at here, after realising that Static Resources can be a bit slow, was how long it took for force.com to download its own Javascript and Stylesheets. By removing the Salesforce header and sidebar, and opting not to load standard style sheets we took another 0.5s off the page load.

Lesson #4 of optimising in the cloud: a bit of understanding about how the platform works goes a long way to spotting areas for optimisation even when you think your actual code doesn’t have a lot of room for improvement.

With the use of the new native support for JSON (worth ~1.5s over our homegrown implementation, and resulting in vastly simpler code) and a few other minor tweaks we’re now down to a steady 2s for page load; a whopping 2000% improvement over that first, not-too-naive implementation.

Summary

Far from just having whatever performance you get out of force.com on your first effort, there is plenty of opportunity for optimisation if you find you need to do it. However, don’t look too much to the platform for bottlenecks (apart from a few specific cases mentioned) the answer lies in your use of the platform and in your custom code. String concatenation and viewstate in particular seem to be areas where you can gain some quite significant improvements with relatively little effort and, with the new support for JSON, shipping compact JSON strings that you then render on the client rather than having Visualforce do all your rendering on the server side for you is definitely a good option even if you don’t need nested <apex:repeats />. And if you see inconsistent performance in your customers’ environments there are certain things to go looking for there that can make a dramatic improvement.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s