# Data Browser

- Overview - Getting Started - Common Tasks - Perspective - Toolbar - Properties Panel - Annotations - Search - Export - Import - Inspect Samples - Waveforms - Preferences - Older Data Browser - History -

## Overview

The Data Browser is a trending tool. It can display the values of 'live' Process Variables (PVs) as well as historic data in a Stripchart-type plot.

Fig. 1: Data Browser Screenshot

### Live Data

For each PV, the Data Browser collects 'live' samples from the control system and plots them over time. The size of the live sample buffer is configurable, as is the sample period.

### Historic Data

The Data Browser queries archive data sources for historic samples to obtain data for time ranges 'before' the live sample buffer. You can configure one or more archive data source for each PV.

## Getting Started

To create a new plot:

1. Click on the "Open new Data Browser plot" icon in the CSS tool bar (top row of the CSS window).
2. Open the plot's context menu by right-clicking into the plot. Select "Add PV".
3. Enter the desired PV name, press "OK"
4. Use the Toolbar or Properties Panel to change the displayed time range, zoom in/out etc.

## Common Tasks

- Add PV or Formula: Right-click on plot or trace table in Properties Panel, select "Add PV" or "Add Formula".
- Delete PVs or Formulas: In Properties Panel, right-click on selected item in trace table and select "Delete Item".
- Find PV Name: Use the Search Panel
- Add archive data source: Select PV in Properties Panel. Right-click to "Add Archive". Or drag archive from Search Panel onto PV to cause that PV to be read from archive, or drag archive from Search Panel onto plot to cause all PVs in plot to be read from that archive.
- Remove archive data source: Select PV in Properties Panel. In lower "Archive Data Sources" list, right-click and select "Remove Archive Data Source".

## Data Browser Perspective

In addition to the Browser plot window itself, you typically use the Properties Panel to change colors etc., or the Search Panel to locate the names of PVs with historic data.

The Data Browser "Perspective" provides a suggested arrangement of these additional panels around the plot window, similar to the screenshot in Fig. 1 where the Properties and Export panels are below the plot, and a Search panel is to its left.

To switch to the Data Browser Perspective in case it is not currently selected, right-click on the Data Browser plot and select "Data Browser Perspective".

While in the perspective, you can then close the search panel, move the properties panel around as desired. If you later want to restore the default arrangement, again right-click on the Data Browser plot and select "Data Browser Perspective". Since it's already active, it will ask if you want to reset the perspective to its original state.

## Plot Toolbar

The toolbar is displayed on top of a Data Browser plot. If it is not visible, right-click on the plot to open the context menu, then select "Show Toolbar".

Fig. 2: Data Browser Plot Toolbar

Tool tips with brief explanations will appear if you hover the mouse pointer over a toolbar button for a few seconds. Explanation of toolbar buttons:

- Configure Settings: Opens dialog for configuring the plot.
  Note that these are runtime settings which include things like font settings that will not be saved as part of the Data Browser configuration.
- Add Annotation: After configuring the annotation to for example 'snap' to a selected trace, the next mouse click into the plot will add an annotation.
- Remove Annotation: Prompts for an annotation to remove from the plot.

- Stagger Axes: Arranges the value axes such that the traces on different axes don't overlap.
- Rubberband Zoom: Click into plot, hold mouse pointer to drag a 'rubberband' rectangle, release to zoom into that area.
  Can also be used on a value axis or the time axis to select a start/end range on the axis.
- Horizontal Zoom: Click into plot at desired start time, hold mouse pointer to drag a 'rubberband' rectangle to the desired end time, release to zoom into that area.
  Can also be used on the time axis to select a start/end time.
- Vertical Zoom: Like Vertical Zoom for vertical (value) axes.
  Can also be used on a value axis to select a start/end range for that axis.
- Zoom In Horizontally: Click and hold mouse pointer to zoom into time axis.
- Zoom Out ...: Click and hold mouse pointer to zoom out of time axis.
- Zoom In Vertically: Click and hold mouse pointer to zoom into value axes.
  Can also be used on a particular value axis zoom into that value axis.
- Zoom Out ...: Click and hold mouse pointer to zoom out of value axes.
- Panning: Click into plot or axes, hold and drag to desired position, release mouse button.
- None: For most of the other toolbar buttons, clicking or holding the (left) mouse button will for example activate a zoom operation. The "None" option returns the mouse pointer to an inactive state so that clicking into the plot has no effect.

- Undo: Undo the last operation. Click again to undo the previous operation, the one before that and so on.
- Redo: Un-do the last undo, i.e. re-do.

- Save Snapshot: Prompts for file name under which an image of the current plot is saved.
- Enable/Disable Scrolling: Turns the strip-chart behavior of the Data Browser on or off: Scroll such that the right end of the time axis is always "now", or freeze the time axis.

## Properties Panel

The Properties panel allows configuration of the plot, its axes and traces.

The same Properties panel is shared with other Data Browser plots or even other CSS tools. When you have multiple Data Browser plots open, the Properties panel will show the configuration of the plot that was last selected via a mouse click. When no Data Browser plot is the active window, the Properties panel will be empty or even show the configuration of a different CSS tool.

You can force the Properties panel to stay attached to a specific Data Browser plot by selecting the " pin" icon, or open more than one Properties panel (which you can then 'pin' to specific Data Browser plots) via the " drop-down" menu of the panel.

### Subsection 'Traces'

Fig. 3: Properties panel, subseciton 'Traces'

Configure the individual traces in the plot via the table of traces. Right-click in the table to open a context menu for adding PVs or Formulas, deleting selected items, or adding archive data sources to PVs.

- Trace: Allows to temporarily hide a trace from the plot. If the trace is really not needed any more, consider deleting it.
- Item: Edit the item's name. For a PV, this would be the control system process variable. For a formula, this is the name under which the formula can appear as an input to other formulas.
- Display Name: The name shown in the plot legend.
- Color: Click to select the trace color
- Scan Period: Control system PVs can be scanned at some periodic rate, specified in second, to collect 'live' samples in addition to the data received from an archive. When entering 0 seconds, the Data Browser will show every update received from the PV, and the PV's configuration decides when to send such updates.
- Buffer Size: The Data Browser maintains a fixed-size ring buffer for 'live' PV samples. When reaching the end of the ring buffer, older samples are deleted to prevent running out of memory. You can configure the ring buffer size for each trace.
- Width: The line widths of the trace in the plot. Also used for the size of markers.
- Axis: Click to select the axis on which to plot the trace.
- Trace Type: Click to select how the trace should be displayed: Line, markers like "Square" etc. The size of the lines respectively markers is controlled by the "Width".
- Request: Click to select 'Optimized' vs. 'Raw Data'.
- Index: Index of an element in an array (waveform record). Change this index to see trend of a specified element in an array. The index is zero-based numbering. The index for the first element is 0, the index for the second element is 1, ... If the index is out of range of the array, the corresponding value will be plotted as NaN which shows a marker on the bottom of the plot.

Fig. 4a: Traces with 'Optimized' Requests.

With 'Optimized' requests, the Data Browser requests historic data as a reduced (optimized) set of min/max/average samples. When using 'Area' traces as shown in Fig. 4a, the average values are plotted as a line, and the min/max outline is displayed as a light-colored area. This not only reduces the amount of data compared to plotting every single raw sample from the archive. In addition it makes it obvious if a value was fairly stable over time, or if there was a lot of movement in the data as around 06:00:00 in Fig. 4a.

The number of requested min/max/average samples is configurable in the Preferences. When there are less 'raw' data points than requested min/max/average samples, the 'Optimized' algorithm falls back to returning the raw data, meaning: When you zoom into the data far enough, you will eventually get raw data.

Fig. 4b: Blue traces switched to 'Raw' Request.

When switching to 'Raw Data' requests as in Fig. 4b, each original sample is requested from the history. Not only is this usually more data, taking longer to receive and then plot to. It also uses more memory, and when you try to look at raw data for a long time range your computer will eventually ran out of memory. The 'Raw Data' requests should therefore only be used when necessary because of shortcomings in the 'Optimized' algorithm.

### Subsection 'Time Axis'

Configure the start and end time of the time (horizontal) axis.

### Subsection 'Value Axes'

Add/remove value ('Y', vertical) axes, change their label or color.

### Subsection 'Misc.'

General plot settings: Background color, refresh rate.

## Annotations

The Toolbar contains buttons to add and remove Annotations.

When you add an Annotation, you select if and to which trace it should "snap". You can enter an arbitrary annotation text ("Name") and show that with the Annotation, and/or have it display the plotposition or Sample Information.

Move Annotation around the plot in the Panning or inactive zoom modes.

Change the Name, color, "snap" behavior of existing Annotations via the Configure Settings button in the toolbar.

## Archive Search Panel

The Archive Search Panel is part of the default Perspective. If you cannot find it, try to reset the perspective, or open a new one by right-clicking into the plot and selecting "Open Archive Search Panel".

To locate Process Variable names (PVs) in an archive:

1. Optional: Select an archive URL other than the default.
2. Optional: If the archive has more than one sub-archive, select one or more sub-archive to search.
3. Enter PV name pattern, see details below.
4. Press "Search".

To show data for the resulting PVs, either right-click on PVs of interest and select "CSS/Data Browser" to open a new Data Browser Plot for them, or drag/drop the PVs from the search result table into an existing Data Browser Plot.

### Simple Pattern

By default, PV name patterns can contain a question mark '?' as a place-holder for any single character or an asterisk '*' in place of any sequence of characters.

### Regular Expression Pattern

When enabling "Reg.Exp." pattern search via the corresponding check box, a more powerful but also slower regular expression can be used:

Reg.Exp.. | Meaning
--- | ---
. | Any Single Character
.* | Zero or More Characters
a? | Single 'a' or no 'a'
a* | Zero or more 'a'
a+ | One or more 'a'
.+ | One or more arb. characters
a|b | Either single 'a' or 'b'
[0-9] | '0' or '1' or ... or '9'
() | grouping of subexpressions
Temp | Any Channel containing 'Temp'
Temp$ | Any Channel ending in 'Temp'
^RFQHR | Any Channel starting with 'RFQHR'
Tin_[0-9]+_sig$ | Channels ending in 'Tin_<NUMBER>_sig'
^XYZSys_Temp[2-3]$ | 'XYZSys_Temp2' or 'XYZSys_Temp3'
Pwr_(in)|(out) | Channels containing 'Pwr_in' or 'Pwr_out'

## Data Export

Use the "Export Samples" panel to write data into files suitable for spreadsheet programs or Matlab.

### Time Range

By default, the export will use the time range from the plot, but you can export data for a different time range.

### Source Options

By default, the export will use the time range from the plot but fetch raw data from the archive, not the optimized, reduced min/max/average data that is displayed in the plot. You can modify the time range of the export to differ from the range of the plot, and also select to export either the exact data that is displayed in the plot, or request optimized data with a different number of "bins" from the archive.

- "Plot": The exact samples currently displayed in the plot will be exported.
- "Raw Archived Data": The exported data will contain data that is requested from the archive without modifications.
- "Optimized Archived Data": To reduce the number of exported samples, optimized data is requested from the archive. You specify the number of desired samples, and the export will then request roughly this number of averaged samples from the archive data source.
  For example, when requesting 800 optimized samples, the start..end time range is split into 800 "bins". The the minimum, maximum and average value of the raw samples within each bin is then exported.
- "Linear Interpolation": As an alternative way to reduce the number of exported samples, the exported samples are computed via linear interpolation from the raw data. You specify the interpolation interval in hours, minutes and seconds, for example 00:10:00 to obtain one sample every 10 minutes.

### Format Options

The format of the exported file defaults to a spreadsheet suitable for import into most spreadsheet programs. You can modify these settings:

- "Tabular": Instead of a table that lists the samples for all channels, you can export the samples for each channel separately.
- ".. with error columns": The min...max range of optimzed data is by default export as "error" columns, suitable for an error-bar display.
- ".. with Severity/Status": The Severity and Status of each sample is by default exported, but you can omit it if not needed.
- "Default format": The number of decimal digits will be obtained from the data source.
- "Decimal format": You specify the number of decimal digits.
- "Exponential notation": Use exponential notation, where you again specify the number of decimal digits.

### Excel Spreadsheets

The exported data files are in a text format with TAB-delimited columns suitable for import into spreadsheet programs like Microsoft Excel.

Assume you chose a filename of "test.dat" on your Desktop, follow these steps for import into Excel:

1. In Excel, use File/Open to open the file "test.dat".
   You might have to select "Files of type: All Files (*.*)" in the file "Open" dialog to do this.
2. A "Text Import Wizard" should appear, and the default settings should already be set to
   - "Delimited - Characters such as ... tabs ..."
   - "Delimiters: Tab"
3. Excel will default to a less useful format for the first column, the "Time" column, and only show time down to minutes, omitting the seconds or microseconds.
   Fix this by clicking on the "A" table header, i.e. selecting the whole first column; right-click to get the "Format Cells..." dialog, and enter a "Custom" format:

         m/d/yy h:mm:ss.000

When performing computations on the data, values marked "#N/A" which have non numerical value because they represent a status or error should be ignored by Excel.

For plots, the "X/Y scatter" plot type using time as the X axis and the value column for the Y axis tends to work best.

Note that Excel is limited to about 65000 lines. If your data file includes more lines, those will be lost in the import. You can work around this by exporting a smaller time range into separate files, or by exporting averaged data.

The value columns for averaged data will contain text like "5 [0 ... 10]" to indicate that the average value for that time was 5, with a minimum/maximum range of 0 to 10. To perform computations in excel, it might be useful to select the column and perform a text replacement of "[*]" with "" (nothing) to delete the min/max info.

### OpenOffice.org/LibreOffice Calc (Spreadsheets)

As long as the exported data file has a ".csv" suffix, OpenOffice.org/LibreOffice should open the file with an "import" dialog similar to MS Excel, where you select "tab" delimited columns as explained above.

File names ending in ".dat", ".txt" etc. might only open in the OpenOffice.org/LibreOffice Writer (word processor), so you have to use the correct file suffix.

### Matlab

The Matlab export can use two different file formats:

- To create binary Matlab data files, use a file name ending in ".mat".
- Matlab text files are created when the file name ends in ".m".

#### Binary Matlab .mat Files

The binary data file like "example.mat" is suitable for loading into Matlab 5 and newer like this:

    load /path/to/the/example.mat

In your Matlab workspace you should then find structures named "channel0", "channel1", ..., one for each channel that was loaded from the file. Each structure has the following elements:

- name: Channel name
- time: Time stamps
- value: Numeric value
- severity: Severity

The time stamps are saved as text. In reasonable modern versions of Matlab you best convert them to Matlab serial date numbers, which you can then for example plot over time like this:

    channel0.date = datenum(channel0.time, 'yyyy/mm/dd HH:MM:SS.FFF');
    plot(channel0.date, channel0.value);
    datetick('x',0);
    title(channel0.name);

#### Matlab .m Text Files

The text file like "example.m" is suitable for loading (executing) in Matlab R2006b or newer, creating one 'timeseries' object per trace. To load it into Matlab, you execute the file like this:

    cd /path/to/the/file
    example

Matlab will then execute the commands in the file which define the data in the Matlab workspace and also plot it. With Matlab releases older than R2006b you will have to edit the generated file to suit your needs, for example use simply "plot(v);" to show the values.

## Data Import

There is basic support for importing data from data files. The data file is fundamentally treated like as archive data source. The data file must reside in the workspace, i.e. it must be visible in the file Navigator.

A channel based on a data file can be added in two ways:

- 'Drag' the file into a plot, or
- Select 'Import' from the plot context menu.

The data file must only contain data for a single channel. The lines of this data file must match the following format:

    # YYYY-MM-DD HH:MM:SS.SSS   value
    2011-08-09 15:51:16.734     299.634000

    # or
    # YYYY/MM/DD HH:MM:SS.SSSSSSSSS   value
    2012/12/06 07:38:05.764764346     26.6504 OK      OK

The time stamp and value may be separated by spaces or tab characters. Lines that do not match this format are ignored.

One example of a suitable data file is the format that results from 'exporting' data for a single channel.

When saving a *.plt file that includes channels with imported data, the name of the data file is stored, not the actual sample data. The next time such a plot is loaded, data will again be imported from the file.

## Inspect Samples

The "Inspect Samples" panel, accessible from the context menu of the plot, provides a tabular view of all samples in the Data Browser memory. It is mostly used for debugging.

For performance reasons, the table does not update with each received sample. Instead, a manual update via the "Refresh" button is required to show newly received "live" samples or to force an update of a table that appears inconsitent.

## Waveforms (Arrays)

Support for waveform (array) type channels is limited. The Data Browser will only display one array element of a waveform channel over time. By default this is the first array element of each waveform sample, array element zero. The array index can be changed via the "Index" property of the channel in the Properties Panel.

The context menu of the plot has an entry to open the "Inspect Waveform" view. It can display each waveform sample individually.

The fundamental problem with waveform channels is that the channel data itself does not provide sufficient information: Is the array sample

- a readout of a fast device, where the time stamp identifies the time of the first (or last) array element, and the other array elements were obtained at small, fixed time intervals after (before) the sample's time stamp? This type of waveform could be displayed in the main Data Browser plot over time, but the time detail is not known from the channel data.
- a readout of for example a beam position monitor or histogram or multi channel analyzer, where each array element represents the value at a certain position, energy, ...? To display this, one needs to know the "X Axis" information.
- other structured data that was encoded in a waveform, because the underlying control system (EPICS Version 3) does not support data types for images or structures? To display this, the data encoding needs to be known.

## Preference Settings

Use the menu Edit/Preferences/CSS Applications/Trends/Data Browser to change various settings including the default time range of a new plot, the number of 'Plot Bins' used by the Optimized request type.

## Transition from older Data Browser

Even though this Data Browser code is quite different from the original code, you will notice a very similar look and overall behavior. The data browser also loads old "*.css-plt" configuration files.

When you first open the new Data Browser, you should reset the perspective.

The main differences:

- The plot's Toolbar has different options
- The "Data Browser Config" panel has been replaced by the standard Eclipse "Properties" Panel

## Product History

At LANL, Craig McChesney and Sergei Chevtsov with contributions by Peregrine McGehee implemented the Java Archive Viewer for displaying Channel Archiver data (2004).

The first CSS Data Browser was based on Archive Viewer ideas but added live data handling as well as CSS integration and pluggable support for more than one type of historic data source (2007)

After many changes to the code, a rework was required (2010):

- Code cleanup using UML diagrams, 'Model' with standalone Unit tests...
- New XY Chart:
  - Pan axes or whole plot via mouse
  - More zoom in/out options
  - Annotations can 'snap' to trace or be freely positioned
- Undo/redo support for almost everything
- PVs can scan at individual rates, or log every received change.
- Formula handles min/max samples
- Export supports spreadsheet as well as MatLab file format
- Use of "Properties" View allows shared as well as multiple configuration panels
- New archive reader API that is 'smaller', with better support for cancellation, and support for faster Oracle data readout via 'stored procedure'.