Food Service Chain Achieves 30-Fold Performance Gains for Cloud-Based Mobile Order Processing

Summary 

Our Fortune 500® food service client needed to re-architect their mobile order system from an in-store, point-of-sale-hosted solution to a cloud-based implementation to support their increased order transaction rate.

WWT established a cloud-based, production-like environment so testing could be performed to validate performance and quickly identify fail points. This aggressive approach to performance monitoring saves enormous and unnecessary costs in production downtime and guides production scaling decisions that could lead to further cost reductions.

The new cloud deployment, an APM in place and the ability to monitor overall system performance tripled the number of orders processed. It also provided stability and significantly reduced the failover needs. These results provided a 220% gain over the target goal and far exceeded the client's original expectations.

Challenge 

The client's system was architected and built, microservices were deployed and approximately 1,800 low-traffic stores had been migrated to the cloud. However, the system's inability to sustain transaction rates of ~22 orders per second, far below the 200 orders per second target, resulted in: 

• Frequent outages
• Lost and rejected orders
• Overwhelmed support team
• Missed feature development dates

The client needed the ability to load test the system and monitor cloud resource utilization and application software performance while the system was under various load levels. This required deployment of full cloud technology, an Application Performance Management (APM) tool, visibility to memory usage and log monitoring capabilities to: 

• Move 10,000 stores into the cloud environment
• Enable 20,000 devices to process a minimum of 200 orders per second
• Monitor system and app performance to identify bottlenecks and deficiencies

Solution 

WWT established a cloud-based, production-like environment against which testing could be performed to validate performance and quickly identify fail points adequately. This aggressive approach to performance monitoring saves enormous and unnecessary costs in production downtime and guides production scaling decisions that could lead to further cost reductions. This methodology included: 

• Developing an environment in which to perform rapid feedback and sustainability testing:
  • Appropriately distributing the test environment to prevent performance limitations
  • Capacity: test the system's stability by ramping from starting load until the target load is reached
  • Load: assess the system's ability to sustain the load for a short time
  • Soak: gauge the system's ability to support the load for an extended period, up to 30 hours
• Creating executive-level dashboards for visibility to system and application performance to:
  • Monitor load test times
  • Identify instances of high memory usage
  • Obtain performance issue indicators and alerts
  • Discern potential bottlenecks within dependent systems

Results 

With the new cloud deployment, an APM in place, and the ability to monitor overall system performance, tripled the number of orders processed, hitting a maximum of 660 orders per second. Maintaining stability with the client's regional failover needs to be reduced to ~440 transactions per second – a 30% gain from where the client began and a 220% gain over the target goal. The outcome far exceeded the client's original target and resulted in: 

• 38% increase in profit margin
• Ability to process over 440 orders/transactions per second
• Executive-level reports and dashboards
• Total visibility of system and application performance
• Identical prod / non-prod environments managed through IAC (infrastructure as code)
• New environments are being spun up in a few hours rather than a few months