Today companies are continuously facing challenges in managing increasing data volumes and growing customer expectations. Customers and prospects are pushing the limits of existing applications by adding massive datasets and concurrent users to meet and exceed expectations. An inefficiency in solving these challenges could lead to missed deals, dissatisfied customers, reduced profitability, and market leadership. Digital transformation, marketplace disruptors, changing business landscape, and the rise of the customer are just a few of the many factors that are pushing companies to innovate and scale at a pace like never before. Enterprises must, therefore, be future-ready while scaling rapidly. To ensure enterprise readiness and scale with our customers along with meeting their personalized definition of product success, Propel has joined forces with the Salesforce ISV enablement team.
This blog post summarizes our enterprise scaling journey and aims to help other Salesforce ISVs embark on their very own Enterprise Readiness initiatives.
Theodore Matthews, Quality Engineer II, Propel
Ron Hess, CTO, Propel
Propel experienced a 92% improvement in search and 82% transaction performance as a result of our efforts to ensure the enterprise scale readiness of our application. Beyond the confidence our application can scale to large enterprise user and data volumes, this process uncovered various tips and tricks to drive performance improvements that future-proof our existing application. We also discovered that Salesforce offers valuable benefits to ISV partners, including performance testing apps and recommended best practices. This whitepaper summarizes our recommendations for other app providers considering enterprise scale readiness, as well as our experience working closely with the Salesforce ISV team over the past several months.
Why Test for Scale?
- Confidence that your app will scale — uncover and fix performance issues in your app before your customer discovers them.
- Competitive advantage on the AppExchange — successfully testing your application for enterprise-scale can be advertised on your AppExchange listing.
- Increased customer satisfaction from scalable and performant apps.
Where to Begin
We launched our process by scoping and defining the limits of our tests. We scoped our tests using critical user journeys (CUJs). This allowed us to scope out inconsequential code paths and target paths needing optimization. We limited tests by mirroring our largest customers and prospects. Through mirroring, we determined eleven hundred users would be the appropriate load and identified that a one to ten ratio, full CRUD to read-only CRUD, was representative of user personas. Moreover, data consisting of more than ten million records would be consistent with a large customer or prospect. We aimed to load an equivalent amount. Appropriately defining the scope and the limit of your testing ensures that the results of your scale testing are impactful.
It is important to note that testing should be determined by the constraints of your application and the needs of your customers and prospects, not the Salesforce Platform. Additionally, you are not stress testing the Salesforce Platform, doing capacity or failover testing, testing network latency, browser performance, or third-party integrations.
After constructing tests and loading data, you should understand how to scale tests and measure transaction performance. To determine our scaling methodology we used the Enterprise Scale Testing article¹ from the Salesforce Partners community.
It outlines the steps to kickstart your performance testing and highlights resources to use for additional information. To measure transaction performance, use transactions per second, or TPS. We calculated TPS by dividing aggregate transactions by total test time. To analyze TPS, we used Event Monitoring2. Event Monitoring is a Salesforce Partner resource that collects application and platform actions as files and categorizes those collections as event types. We utilized the URI event type to measure our TPS. With a means to scale tests and measure TPS, we can effectively test our app and critically analyze results.
Testing and Analysis
We started our testing by optimizing our application for a single user in each persona. In our preliminary tests, we discovered many of our SOQL queries built to handle non-indexed fields were taking up to 12 seconds and in some cases causing Apex Timeouts. In order to combat this issue, we enabled SOSL search capabilities for all our customers. We targeted SOQL searches by adding flags in our code to control when customers use SOSL. When the flag is on SOSL queries are executed, when the flag is off only SOQL queries are executed. This allows the opportunity for smaller customers to continue using SOQL and empowering larger customers to have quick searches. The result of switching from SOQL to SOSL was a reduction of search times by 92%.
Next, we tested and optimized for approximately ten users, in our case we used the one to ten persona ratio. Using the URI event type, we estimated TPS for a thousand users at 200. Speaking with the Salesforce Partner team, 200 was clearly too high; most applications estimate around 40 TPS. From this, we targeted 40 TPS to be appropriate, but this left us with the issue of decreasing TPS. To solve this problem, we utilized the Network tab provided by the Google Chrome Developer Console. We discovered that many remote actions are called separately, but are in the same execution. We decided to boxcar transactions enabling us to mimic our old behavior with less over the wire transactions. Using boxcarring we were able to reach an estimate TPS around 40.
Having determined an appropriate estimated TPS, we opened a case with Salesforce two weeks ahead of when we would begin testing at scale. During this two week period, we started building our large concurrent tests using JMeter. In order to call apex remote actions through JMeter, you need to fetch the session ID and the cross-site request forgery (CSRF). We utilized correlation to pass the CSRF to each HTTP Sampler, which provided a means for stable tests. We also implemented to ramp up and add think times to match the TPS reported to Salesforce. Once our ticket was resolved we began our concurrent tests with one thousand users. The results of our tests yielded a measured TPS of 36. Through boxcarring, we improved our TPS by 82%.
In addition to boxcarring, if you are having trouble improving performance or issues decreasing TPS, consider leveraging the Salesforce Partner resource Platform Cache. Platform Cache is an in-platform cache technology that can be used to target cacheable data. The setup is straightforward, request a trial, create partitions, and begin writing code. The implementation of cache means fewer calls to the database via decreased remote actions. We explored this technology by searching our code base to find specific queries and business logic that were costly to retrieve or generate and that did not need to frequently change.
Through this process, we not only saw performance improvements in-app but also ensured enterprise customers and prospects that are capable of handling their users and data volumes. More importantly, without the invaluable resource and aid of the Salesforce ISV team, we could not have had such a success.
1. Enterprise Scale Testing Article
2. Event Monitoring Application