TechnologyData Quality Management

Addressing Network Data Quality problems

The network is an extremely rich and untapped data source on the real-time performance and state of your business and services. Tuning in to that robust data set is highly valuable, but not always easy to achieve. In fact, because of its scale and speed, network data is notoriously difficult to manage and extract value from, until now.

At Corvil, we believe the benefit of understanding what your network data is telling you is worth the effort of making it easy to tap and quickly interperet that data, and we have engineered our system to ensure that the network data we extract and analyze is always complete, timely, and actionable.

Today, there are many options for extracting network data — you can duplicate and forward a copy of traffic via a switch mirror port (SPAN), or you can insert an inline device (TAP). There is also a growing trend towards more powerful and flexible architectures, involving mediation of SPAN and TAP via aggregation devices that can add timestamps and ingress port identifiers, and provide filtering, deduplication, and forwarding to multiple monitoring devices.

Each of these approaches has its advantages, but also carries the potential to corrupt the data. If you want to avoid the “garbage-in, garbage-out” problem in your network data analytics, you need a solution that can identify, flag and correct, where possible, the incoming data stream.

Corvil offers a powerful suite of features to ensure that the network data being received is complete, accurate, and glitch free. Corvil Data Quality Management is the bedrock on which the trusted analytics are built.

Identification of SPAN congestion

SPAN ports and aggregation devices can become temporarily overloaded due to spikes in the traffic known as microbursts. These microbursts cause congestion and distort the latency profile of the underlying traffic before it reaches the analytics appliance. This can lead to false and misleading performance metrics. It can even result in website responses being seen “before” the corresponding request. Corvil’s unique SPAN Congestion Detection continuously monitors and automatically alarms on any congestion occurring on the measurement port prior to the data being received. Corvil will flag any impacted measurements whose integrity cannot be guaranteed.

Detection of loss in the data acquisition fabric

One of the aims of SPAN and aggregation layers is to efficiently ‘overbook’ the physical monitoring ports receiving data — for example using a single 10G port to monitor data from 10 x 10G of physical ports. This can reduce cost, but may introduce drops in the acquisition layer (in addition to the accuracy-impacting congestion mentioned above). Corvil can detect data dropped within the data acquisition fabric. This allows congestion points to be identified and remedied.

De-Duplication of traffic

It is common to monitor the network traffic at multiple points so that transport latency and loss can be reported (such as across a firewall and load-balancer). It is also quite easy to mis-configure a SPAN session so that it forwards multiple copies of every packet. Corvil has extensive support for handling duplicate copies of traffic, whether deliberate or accidental, so you can remove unwanted copies yet still cleanly monitor a flow at multiple observation points.

Aggregation layer integration

Aggregation devices can reduce costs and provide greater flexibility for network data collection. They allow network data streams from multiple observation points to be multiplexed and forwarded to analytics and monitoring appliances on a single port. The monitoring appliance needs to be able to understand the metadata provided by the aggregation layer to demultiplex and handle the duplicate copies of traffic appropriately. Corvil have worked with all the major aggregation switch vendors over several years to ensure that our customers can get the benefits of aggregation switches without the headaches.

Misconfigured SPAN and aggregation sessions

Mistakes happen. It’s all too easy for a stray configuration to result in only one direction of a TCP flow being monitored, or for a VLAN tag to be applied to monitored traffic in just one direction of traffic through a router. Corvil detects and alarms on these problems, and equips you with a rich set of tools to understand and address misconfigurations.