Streaming Platform Development for a Global Financial Services IT Operations
In this case study
Overview
WWT was commissioned to develop a streaming platform for a Global Financial Services IT operations team. The platform required the ability to collect and stream data from multiple source systems and make the data available to multiple consumers through a publish/subscribe messaging model.
This case study highlights how WWT was able to provide our client with a solution that would optimize their operations team and provide near real-time health check updates.
Challenge
The Global Financial Services client was experiencing an increase in technical environmental complexity, creating a need to evolve the way in which internal engineering and operations teams minimize critical infrastructure downtime and optimize performance. To scale their internal engineering and operations capability, they were seeking to leverage their IT organization's array of operational and system data to uncover areas for potential optimization and value creation (e.g., time, resource, cost) through a holistic analytics capability.
Objectives of the engagement
The client's objective was to build an initial streaming platform ready for production release. The streaming platform needed the ability to collect and stream data from multiple source systems and make the data available to multiple consumers through a publish/subscribe model. In addition, it needed to be vendor neutral, reliable, secure and scalable for future global rollout.
The engagement was structured so that the WWT team would also train and transfer knowledge to the client's internal team members so that they could continue to develop the platform after the initial release. This would enable the client to evolve the use cases to focus on minimizing downtime and optimize performance and cost of the environment.
Delivery
WWT was engaged in a 16-week activity to support the client. The engagement was broken down into three phases:
- Diagnostic and design
- Build
- Document
Diagnostic and design – WWT undertook discovery activities to understand the current-state system and desired end-state needs and create design documentation to prepare for the build of the streaming platform's initial production-ready release. Four use cases were initially identified, and through a value-prioritization exercise two were selected:
- Infrastructure Health: provides real-time telemetry data for CPU, RAM and Disk for the hosts that were installed in the environment
- Agent Hygiene: provides real-time health information for Tanium and Puppet agents for the hosts that were installed in the environment
Build – WWT built an initial streaming platform in the client's internal cloud. The streaming platform was created in their development environment and had the ability to collect and stream data from multiple source systems, perform transformations and processing, and make the data available to multiple consumers. It was vendor neutral, reliable, secure and scalable. The data sources necessary to monitor the two selected use cases – Infrastructure Health and Agent Hygiene – were added to the streaming platform.
Document – As the work progressed, WWT ensured that all activities were documented (including all final as-written source code), and regular hands-on knowledge transfer sessions were held with the client to ensure that they had the ability to continue to develop and scale the streaming platform globally following the initial two use cases.
The key deliverables are summarized below:
Benefits
The WWT Delivery Team aligned with the client stakeholders to design and deliver a development streaming environment with two working use cases, Infrastructure Health and Agent Hygiene.
The Streaming Environment provides the ability to collect and stream data from multiple source systems and make the data available to multiple downstream consumers. This offers excellent opportunities for further use cases by having the capability to perform complex aggregations and transforms in real-time from multiple data sources. In addition, by being able to connect, aggregate and merge multiple data sources, the streaming platform was able to provide the client, in a single dashboard, a specific level of information and data per server not available prior to the project.
The initial Streaming Environment provides a tangible foundation, with hands-on experience from building two use cases, that will enhance the client team's ability to continue to expand the environment and use cases leveraging near real-time access to data.
Conclusion
The client has a long-term vision of having a full data-centric organization, and the next step is to create a global architectural design to productionize the streaming environment to enable the realization of further analytics use cases.
Next steps to success
If you are looking for help to create a Streaming Platform, reach out to us today. We would welcome a discussion to understand how we can support you to deliver transformation to your IT Service or end-user environment.