STREAMS

Event data is represented in TuningFork as streams. Streams can either be base or derived. A base stream is computed directly from the actual events in the feed, for instance "Start GC at time 8.978s" or "Allocate Memory 37 KB at time 13.854s". Derived streams are defined by applying Stream operators to one or more input streams.

Streams primarily come in two flavors: time interval streams () and sample streams (). Time interval streams contain events that are begin/end pairs and delimit a certain kind of activity (for example "GC Phase"). Sample streams consist of a timestamp and a quantity (like the Allocate Memory example above).

 Filters and Derived Streams

A derived stream is created from other streams using a filter or combining operator. Derived streams often provide data that is far more useful for analysis purposes, and avoid the overhead of computing the information inside of the running real-time application.

An example of a derived stream is Allocation Rate, which is computed from the Allocate Bytes events which report the amount of memory in use at a particular time. The easiest way to explore this structure is to use the Feed Explorer, starting at the Allocation Rate node under Streams. By opening up the tree, you can see how streams are combined to derive allocation rate from the low-level stream of individual allocations. In fact, Allocation Rate is a multi-stage derived stream:

  1. A summing stream adds Allocate Memory events and produces an Allocated Memory Wallclock stream, which shows the allocated memory changing over physical time.
  2. An exclusion filter takes the Allocated Memory Wallclock events and the GC Thread Running events and removes the latter time intervals from the former, which produces an Allocated Memory Virtual stream which has a virtual time axis in which garbage collector execution is not considered, since we are interested in calculating the inherent allocation rate of the application when it is not being interfered with by other components.
  3. A differentiator stream differentiates the Allocated Memory Virtual stream to produce an Allocation Rate stream.
  4. Finally, a convolution filter can be applied to smooth the Allocation Rate stream, for instance to allocation rate over 1/10 second intervals, to produce the Allocation Rate Smoothed 0.1 Second stream.

While the above sequence may seem rather complicated when described in words, the structure of any derived Stream can be interactively explored in the Feed Explorer. The Feed Explorer can also be used to interactively define new Streams by right-clicking on an existing Stream or Event. The ability to easily create and visualize Streams from the FeedExplorer is one of the most powerful data exploration and analysis features of TuningFork.