Once clients realize the rich dataset our network packet capture and analysis provides they ask us to answer questions about regulations, risk, and cybersecurity, strategy execution and a myriad other business-level topics.
I had a chance to hear our CTO Raymond Russell in a panel discussion on the changes in packet capture and network data analysis at NYC STAC Summit. One of the key points made was that while most trading firms and exchanges are doing packet capture to some extent, the use of the captured data is changing dramatically.
Corvil’s history shows ample evidence of this. Our traditional use cases are all about latency. However, in order to actually measure latency reliably you have to accurately timestamp and decode all the data flowing across the network. By all the data we mean that we can decode order details, number of shares changing hands, the activity dollar value and more. Once our clients realize that they have ready access to a rich dataset through capture and analysis they inevitably start asking us to answer questions about regulations, risk, and cybersecurity, strategy execution and a myriad other business-level questions.
Raymond noted that data architecture is often the primary challenge trading firms face when answering those questions. When you are asking a question like “What’s the hit rate for Client X on Market Y for Symbol Z?” the data you need to answer it is scattered all over your infrastructure and the data is far too voluminous to move around efficiently. One of the lessons learned from big data projects is to leave the packet data where it is… which opens up the question of where to put the analytics.
The panel generally agreed that having a network capture layer deployed which supported multiple types of analytics would be a more cost effective than having multiple analytics solutions each with their own packet capture capabilities.
Raymond went on to outline some practical deployment considerations. For example, some performance analytics are tightly tied to the location of the capture point on the network. If you are trying to determine what is compromising this trading strategy – i.e. why is it not behaving as expected – you need to understand the performance profile of the trading transactions across every network hop it traverses. Replacing the performance analytics at those points with some other analysis engine could create a blind spot in your performance measurements.
Other analytics may not be tightly tied to location, Raymond noted that it depends on the questions you are answering. In some cases your questions may not need the entire data stream from the packets but only a small subset. In those cases, having the full analytics for that question at every capture point may not be the most effective option – particularly for companies with global operations and datacenters. Instead streaming an indexed subset of data to a centralized analytics engine may work best. The key point is that architectural flexibility and scalability are a must for getting a variety of insights out of your network data.