TickSmith’s GOLD Platform– Resilient During the Crisis
The Data Behind The Data
The use cases for TickSmith GOLD focus on the collection of historical market data processing for distribution and analytics. Simply put, we get the data ready for our clients so their end users can use it. The GOLD Platform also ingests data in near real-time to support and create analytics that tell traders how well their trades are performing to improve their algoes.
A NEW NORMAL?
We power systems for the largest exchanges in the world, with massive historical data dating back to the 1970s. So from a bird eye’s view, we can state that we have never seen data volume of this kind across the exchanges. Our clients have not seen anything like this in all of their history.
In the past, we’ve seen spikes of data from a certain exchange on a certain day for certain reasons. However, we’ve never seen a spike like this from all the exchanges globally, and all at once. This would be a new normal that hasn’t occurred looking back at fifty years of data. How did we adapt our GOLD Platform to rise to the challenge?
PAY AS YOU GO PROCESSING ON THE CLOUD
TickSmith’s GOLD Platform is elastic because it is built for the cloud. One benefit of the cloud is the capability to provision more or less resources as needed. For example, during the crisis, we are able to double the maximum number of machines available for processing with one click.
As a result, the platform is capable of dealing with the volume of data coming in that day. In addition, using the technology of Spark allows for the distribution of workload across machines in an intelligent manner, rather than having it sit on one machine. This makes for quicker processing.
Our clients have the flexibility to make the final decision on how they want their data processed, to either prioritize cost, speed, and/or efficiency. Clients decide on the necessary levers, but the reliability of GOLD is still maintained no matter what.
WHY SCALABILITY IS IMPORTANT
The GOLD platform has the ability to scale up and down as necessary. Our clients have benefited from seeing minimal disruption to the processing and delivery of data even though most exchanges have seen a record in trade volume. Because the cloud allows for processing based on the volume of data needing to be ingested, the constant adaptation to a higher trading volume has been seamless.
One of our development teams is currently working on a client use case that is seeing up to 8 times increase in data ingestion with real-time capabilities needed. However, when dealing with real-time, if the necessary infrastructure is not in place to collect the data in the first place, the data can be lost forever. That’s why the ability to scale instantly is an important feature in any data management system.
The results would be numerous backups, delayed delivery to clients, and contract changes, if exchanges continue to use legacy systems that do not have the ability to scale seamlessly. This is especially a problem because clients need market data now more than ever. Minutes, hours, and days of delay could cost data consumers millions of dollars.
IMPROVEMENTS SINCE THE CRISIS
We’ve seen how resilient the GOLD Platform is when it comes to scalability. However, restrictions to scalability can be limited by our client’s budget. As a result of the crisis, we plan to implement customized levers for our clients’ spending models. We’ve learned that it’s better to prepare our clients beforehand to adjust to changes coming from a potential market crisis– even if it’s considered a black swan.
We’re also looking into improving our processing efficiency by introducing a two cluster system based on cost and performance. So the minor improvements needed for GOLD revolve around a more streamlined system for customer levers.
A TIMELY FEATURE
In early January, TickSmith introduced a new feature called the User Configured Data Output File, for end users to be able to manipulate and load subsets of market data. This feature proved useful during the crisis because the dataset files are too large, and most end users were unable to load the files. We didn’t create this feature specifically for the crisis, however, we discovered that it works perfectly for one.
THE BLACK SWAN OF BLACK SWANS
Just before the crisis, we did an analysis of one of our client’s data dating back fifty years. We found only six instances in history where the data files were greater than 40 gigabytes. So we customized and sized the clusters on their platform accordingly, for cost effectiveness.
The pandemic hit the next week and during that week, we saw over thirty instances of files being over 40 gigs. We’ve learned that when it comes to a black swan, history doesn’t always provide the necessary data.
No doubt, the pandemic is history in the making and will leave a unique mark on the world. We have seen how technology has come to the rescue though. For example, online applications allow workers to work from home and connect with each other, while working remotely.
Being on the cloud can be seen in a similar light. Besides the obvious advantage of scalability, a side benefit is how it enables data to be distributed quickly to entitled users– no matter where they are.
In an ever changing world, the benefits of being data-driven will not change. Data is the new gold, and those extracting insights from it will gain the competitive advantage in the future.