Section 8: Dynamic Data Access Explained

Dynamic Data Access refers to the ability to request and present charts, tables, and web pages that contain up-to-date data without the need for manual intervention to create those objects. In a true dynamic data access environment, the most recent data is automatically combined, at runtime, with defined presentation templates. The result is what’s really at the core of NetCharts Server– an application that provides for accurate, timely, and customized information, available to distributed users on an as-needed basis.

NetCharts Server provides developers a great deal of flexibility to decide how and when data is to be retrieved, processed and displayed. Developers can manage all the data retrieval functions independently using third party software, or have the NetCharts Server data clients make the requests directly. Whatever the data architecture, NetCharts Server can address it.

Although a great deal of this section deals with the Named Data Set (NDS) resource and associated NDX template files, developers are not required to use NDS to retrieve data from the data source. Data can be retrieved using third-party data requests and passed in dynamically using CDX variables or the data can be put directly into the NDS cache. Developers should choose the appropriate method based on their architecture and application goals. A general guide is provided here:

Your Application / Environment Our Recommendation
Data never or rarely changes. Inline data directly into chart file
Environment contains an Application Server ( IIS/ASP) that manages database access. Format and pass data to charts using CDX variables, or use the putDataCacheEntry API to push data into the server and the NDS Cache plugin in the NDX files referenced by charts and tables.
Environment contains custom software to manage data/database access. Format and pass data to charts using CDX variables, or use the putDataCacheEntry API method to push data into the server and the NDS Cache plugin in the NDX files referenced by charts and tables.
An HTML form is used to collect data and/or parameters from the user for rendering Format and pass data to charts using CDX variables, or use the putDataCacheEntry API method to push data into the server and the NDS Cache plugin in the NDX files referenced by charts and tables.
Custom query attributes are collected from the user for database retrieval before processing Use the NDS Resource with defined variables. Pass query attributes to the NDS resource at runtime.
Data resides in Oracle, Sybase, or an ODBC compliant database. Use NetCharts Server NDX functionality. Customize the .ndx property file to use the other JDBC driver and supply database specific parameters (logins, passwords, etc) and dynamic or static query information.
Data resides in a database for which there is no JDBC driver available and does not support ODBC, but does support a TCP/IP connection. Use NetCharts Server NDX functionality. Extend NDSPlugInAdapter class to pass instructions to the database. Write a .ndxtemplate file to use the plugin and pass configuration information.Contract with Visual Mining to develop custom plug-in or JDBC driver.
Data resides in a “flat” text file where there are character file delimiters or fixed field sizes Use NetCharts Server NDX functionality. Customize .ndx property file with the appropriate filename and delimiter information.
Data resides in a binary file or unstructured text file. Use NetCharts Server NDX functionality. Extend NDSPlugInAdapter class to incorporate appropriate file parse engine.
Data is formatted as XML Use NetCharts Server NDX functionality. Apply a transform to the data to product NCXML format, or supply a custom XML parser for the existing format.
Raw data requires business logic (to apply algorithms or rearrange the data) processing before display. Use NetCharts Server NDX functionality or third-party data retrieval tools to get the data. Use Process Modules to post-process and apply the required business logic th the result set.
NOTE: ODBC-compliant databases include MS-Access, SQL Server, MS-Excel, Lotus Domino, and many more.

Data Clients
A data client is a page object that requests and/or displays dynamic data. Within NetCharts Server there are three types of data clients: charts, tables, and the API toolkit. All data clients can request data from the Named Data Set (NDS) Resource by using NDS property files. Chart data clients can also accept CDX data variables.

Chart Data Client

When a chart is requested, the chart data client loads the parameter file (usually with a .cdx extension) named in the request. If CDX data variables are passed as part of the request, the variables are substituted. Then, if the parameter file contains one or more references to Named Data Set (NDS) property files, those data sources will be resolved prior to rendering the chart.

The chart parameter file may contain multiple references to the same NDS property file. This allows charts to share the results of a single data request in order to populate a variety of chart content including data points, axis labels, and legend items.

Table Data Client

When a table is requested, the table data client loads the table property file (with a .tbl extension). The table property file will usually contain the name of an NDS property file and perhaps filter information. Alternatively, developers may opt to pass along the entire table dataset as part of the table request.

ToolKit API Data Client

The Developer may request data directly from NetCharts Server by using one of the ToolKit API’sgetNDSData() methods. The Developer may then modify the data, plave the data back in the server’s cache, use it to populate other page elements, or use it to construct data variables to pass to charts.

Named Data Set (NDS) Resource

The Named Data Set (NDS) Resource manages data requests, the data cache, and a variety of other datarelated functions. NDX files are templates that contain NDS parameters along with variable definitions. NDS provides an important abstraction layer between the underlying data and the post processing and presentation.

NDS Overview

The basic concept of NDS is: a data client (e.g. chart, table, or toolkit API) requests data. The data is read from a backend data source using configurable data access programs called “NDS plugins”. The configuration parameters are name=value properties that are stored in property files and/or passed as part of the data request. The NDS plugin returns the data in a two-dimensional object array. The data is processed and returned to the requesting data client. Row/Column filters identify subsets of the data. Variable-izaton of these filters allows a single NDS to produce different data for different data clients.

NDS properties can be defined in a Data Property Template object. Data Connection Templates are text files that end with a .ndxinclude extension. They contain NDS parameters that can be made common to multiple NDX files. The purpose of Data Connection Templates is to manage common properties of individual NDX files from central definitions. For example, a Data Connection Template can define the host and port for a database. When this Data Connection Template is referenced in multiple NDX files, then changes to the host or port information in the template will automatically be available to all referencing NDX files.

Data Clients specify row/column filters at access time. Filters allow a data client to choose a subset of the data,and thereby assign subsets of the data to specific elements of the chart. For a given result set, a row filter of 0 may identify the legend items for a chart , and a column filter of 0 may identify the axis labels. By using the cache and row/column filters, multiple data clients can share the same datasets.

The diagram below shows various components in the NDS Resource architecture as well as the data flow from the raw data sources to the data clients.
 
dev
 
There are many advantages to using NDS for data retrieval; the biggest among them is separating the details of data access from business logic and presentation layers. In fact, NetCharts Server, via the toolkit, provides the ability for the data access to occur via a third party, and then have the data be put into the cache for subsequent processing and display.

Data clients get data by requesting an NDS property file from their project. Each data request involves three components, the NDS property file, runtime NDS properties, and the NDS Plugin.

NDS Plugin

An NDS plugin is a small Java program that can connect to a data source, read input, and parse the input into a common format. NetCharts Server ships with out-of-the-box plugins for JDBC (which providesODBC support as well), Oracle, CSV/Flat File, XML, and Cache. Each of the supplied plugins extends the NDSPlugInAdapter class. NDSPlugInAdapter is a superclass used to support NDS properties for caching, filtering, sorting, aggregation, and transposition. The following diagram shows the object hierarchy for the NDS plugin classes.

NOTE: The common format used is a two-dimensional object array.
NOTE: The Cache NDSPlugIn is unique in that it does not actually retrieve raw data. It provides a mechanism to treat cache content like any another data source.

 
dev
 

NDS Property Files (.nds or .ndx)

An NDS property file is a plain-text file that contains a number of name=value directives which describe how to get data from a data source. The most important directive is the class parameter. The class parameter identifies the NDS plugin to be used. When a property file is requested, the NDS Resource instantiates a copy of the NDS plugin named by the class directive. The remaining file directives are used to configure the plugin.

NDS property files are identified by their extension, either .nds, or .ndx. NetCharts Server introduced the .ndx type in order to simplify variable passing. Using .ndx, developers can dynamically modify fragments of an NDS property. This simplifies many common tasks like modifying the “where” clause of an SQL statement. With the older .nds extensions, the entire SQL statement had to be resubmitted.

Data Connection Templates (Excluded from the diagram above) are collections of NDS properties which are dynamically included by an NDS prior to execution. NetCharts Server introduced the Data Connection Templates, or NDXINCLUDE files. Use of Data Connection Templates influences he behavior of the architecture diagramed above by supplying properties not defined in the NDX just before execution. NDXINCLUDE files provide both a dynamic element to the execution of the data object, and a central place from which to manage general properties such as database names.

NDS property files and Data Connection Templates can be created easily using the forms-based file creation pages in the Developer console. Just select “Data/NDS” as the file type. If creating anNDXINCLUDE, change the suffix to .ndxinclude

NDS Runtime Properties

When an NDS property file is requested, runtime parameters may also be provided to the plugin. Runtime parameters supplement or replace the properties in the requested property file. This has many advantages, the clearest being the ability to make one data request and then filter the output as needed for different data clients. The chart data client passes runtime properties via the NdsParams CDL parameter. The table and toolkit data clients pass runtime properties via a hashtable in the data request.

Direct NDS Access from the ToolKit

The toolkits for Java, ASP/.NET, and Cold Fusion have methods that allow a developer direct access to the NDS resource. These methods allow the developer to request data from the NDS resource. Examples of these methods can be found on the Developer Console toolkit pages.

NDSPlugInAdapter Feature Set

All plugins supplied by Visual Mining (JDBC, CSV, XML, Cache) extend the NDSPlugInAdapter and thus inherit its features. These features, described below, can be activated by specifying the appropriate NDS properties in the property file or the runtime properties.

NDS Filters

When data is extracted from its source, it is represented as a two dimensional table. By using the NDS filters, the developer can explicitly extract specific columns and rows. There are two stages of filtering: the “raw” data filters and the post-processing filter.

The raw data filters, if specified, are applied immediately after data has been extracted from the source and before storing in the cache.

The post-processing filters, if specified, are applied after sorting and aggregation, but before any CDL formatting.

The notation for the columns works like this:

  • Row/Column numbering starts at ‘0’.
  • Row/Column numbers and ranges are delimited using commas.
  • Row/Column ranges may be shown using a dash, as in ‘2-5’.
  • Row/Column numbers ending with ‘*’ indicate all remaining columns.

The properties used to express the filters are rawRowFilter, rawColumnFilter, rowFilter, and columnFilter. Typically, rawRowFilter and rawColumnFilter will appear in the .ndx file, and rowFilter and columnFilter will appear as a runtime property in the data client request.

Some examples:

Sample Meaning
rawColumnFilter=0* All columns from the original dataset
columnFilter=0,1,2 The first three columns in the dataset, after sorting and aggregation have occurred.
rawRowFilter=0-4 The first five rows in the original data set
rowFilter=2-8,20* Rows 2 thru 8, then rows 20 and all after in the dataset, after sorting and aggregation has occurred.

 
NDS CDL Formatter

Chart NDS requests have data results automatically formatted into Visual Mining’s Chart Definition Language (CDL) parameters before returning. Some particularly noteworthy behaviors of the automatic formatting capability are:

  • Automatic DataSet Creation – DataSet parameters are created automatically based on the input data. For example, bar chart data is one-dimensional. If multiple columns of data are passed, then each column represents a separate DataSet. In this case, the CDL formatter will also generate a new DataSets parameter and set the color element to “null”, thus using the ColorTable to color each DataSet. Note that developers may be required to configure display attributes (colors, symbols, line styles, etc) for each of the expected sets.
  • Row or Column-Major Processing – The orderBy property, also known as the Row-Major property, determines the method by which data is extracted from the data set. Consider the following data table:
     

    1

    2

    3

    4

    5

    6

    7

    8

    9

    10

    11

    12

     
    In most cases, NetCharts Server extracts the data using a column major order, that is, reading top to bottom and then left to right. In the above case, using a Column-Major order, the parameters would be:

    Set1 = 1,5,9;
    Set2 = 2,6,10;
    Set3 = 3,7,11;
    Set4 = 4,8,12;

    In a Row-Major order, however, the data would be formatted as:

    Set1 = 1,2,3,4;
    Set2 = 5,6,7,8;
    Set3 = 9,10,11,12;

    Note: For multidimensional data (e.g. X/Y, Bubble, Stock), when in Row-Major or Column-Major
    mode, the nth dimensions are always extracted from adjacent columns. For example, consider an
    X/Y chart using the above data set, first in Column-Major (default) mode:

    Set1 = (1,2),(5,6),(9,10);
    Set2 = (3,4),(7,8), (11,12);

    and then in Row-Major mode:

    Set1 = (1,2),(3,4);
    Set2 = (5,6),(7,8);
    Set3 = (9,10),(11,12);

The CDL formatting is automatic with the orderBy directly influences the formatting. By default,orderBy is set to COLUMN (for Column-Major), but may also be set to ROW (for Row-Major).

NDS Cache

The data cache parameters are used to control how long the data remains in memory. By using the data cache and data filter, developers can prevent data clients from making unnecessary repeat calls to the underlying data source.

The cache works by specifying a name, or cacheKey, to associate with the data, as well as the cacheLifeSpan, which determines the number of seconds the data will remain in memory.

NOTE: Particular care needs to be taken if the cache is used in conjunction with NDS variables. Variables are applied to the .ndx file before the data is put into the cache, so if two requests are made, with different variables but the same cache key, the second request’s variables will not be applied. A convenient way to avoid this is to attach the variable name to the cache key. So if you have a variable named WHICHFIELD and a cache key named myproject-data, changing the cache key to myproject-data-WHICHFIELD will solve the problem. NetCharts Designer does this for developers automatically.

The properties used to control the cache behavior are cacheKey, cacheLifeSpan, and cacheRefresh.

Some examples:

Sample Meaning
cacheKey=Q1Data Name the cache entry for this dataset “Q1Data”
cacheLifeSpan=3 This cache item is valid three seconds from the time it was put into the cache.
cacheRefresh=true Force the cache to refresh with current data

 
NDS Sort

The NDS Sort function allows the developer to identify a column in the output to sort on. If the column number ends with “R” (e.g. 4R) the sort order will be reversed.

The properties used to control the sort behavior are columnSort, skipFirstRowSort, skipLastRowSortand simpleDateFormat.

Some examples:

Sample Meaning
columnSort=3 Sort the data set by the fourth column. (First column is ‘0’)
skipFirstRowSort=true Do not include the first row’s value in the sort. This is typically used to avoid having column labels included in the sort.
skipLastRowSort=true FoDo not include the last row’s value in the sort. This is typically used to avoid having column totals included in the sort.
SimpleDateFormat=dd/MM/yyyy The sort column is in 01/Jan/2002 format and should be sorted
using date values (vs. alphanumeric sorting).

 

NOTE: SimpleDateFormat format uses the Java language SimpleDateFormat.

 
NDS Data Aggregation

The NDS data aggregation capability provides some basic utilities for applying data aggregation functions to columns or rows of data.

The properties used to control the aggregation behavior are aggregateFunction, aggregateInclude, aggregateLabel and aggregateBy.

Some examples:

Sample Meaning
aggregateFunction=SUM SUM the data. Other functions are AVG, MIN, and MAX.
aggregateInclude=RAWDATA Include the raw data with the aggregated result.
aggregateLabel=Total Include the string “Total” in the corresponding label element of the output. This is useful when the data is to be displayed in a table.
aggregateBy=ROW Aggregate the data left to right into a new column.

 

NOTE: In cases where the aggregated data is used in a chart, be aware that some confusion can occur because the aggregation will automatically use the orderBy NDS flag if set in the chart’s NdsParams (or via the wizard’s Row-Major flag). To avoid this issue, ensure that the aggregateBy NDS flag is set appropriately in the chart’s NdsParams.

 
NDS Data Transpose

The data pivot function is provided to simplify some data display tasks. When transposeData is set to true, the contents of the data table are transposed so that columns become rows and vice versa. The transpose data function is applied before data is put into the cache or other processing occurs.

NDS Pivot Function

The NDS pivot functionality differs slightly than the data pivot function discussed in the previous section. When pivotFunction is set to true, the NDS processing engine will use the related NDS parameters to modify the data structure to create a “Pivot Table” view of the data. The NDS pivot functionality allows the control to specify various data columns that will be included in the resultant table.

The properties used to control the aggregation behavior are pivotFunction, pivotColumn, pivotRow, pivotDataColumns and pivotRowHeaderName .

Some examples:

Sample Meaning
pivotFunction=true Instructs the NDS processing engine to perform the pivot function.
pivotColumn=1 Specifies the data column that represents the “pivot column”.
pivotRow=0 Specifies the data column to use as the first column in the resultant table. The unique values of this column will become the row headers in The resultant table.
pivotDataColumns=2,3 Specifies the data columns that represent the data to include in the
new “view”.
pivotRowHeaderName=Date The text to include in the first column of the first row.

 
NDS Variables and Default Values

The .ndx file extension, introduced in NetCharts Server 4.0, provides for dynamic replacement of NDS property file content. If the variables property is set, the variables listed will be expected as part of the data request. The contents of the properties file are searched for instances of the variables, and replaced by the values sent in as part of the request.

An example:

Consider a .ndx file for ODBC that has the following content:
 

Class=chartworks.es.plugin.JdbcPlugIn
driverName=sun.jdbc.odbc.JdbcOdbcDriver
serverURLBase=jdbc\:odbc\:
ndsType=ODBC
includeColumnNames=true
columnSort=
dbName=Sunshine
variables=DataField,TheLocation,CoffeeType
defaultvalues=AllBudget,All,All
sql=select CostType, sum(DataField) as Cost from ProductionCosts\r\nwhere
location\=’TheLocation’ and\r\nCaffeinated like ‘CoffeeType’ and\r\nCostType <>
‘All’ and Caffeinated <> ‘ALL’ group by CostType

Note that this .ndx file identifies three variables, DataField, TheLocation, and CoffeeType. By changing the call to this NDS we can change the result. Assume we pass the following variables and values:
 

DataField = JunBudget
TheLocation = NorthAmerica
CoffeeType = Decaffeinated

The NDS processing looks for instances of the variables in the file. Then, it replaces them before processing. The file contents effectively become:
 

.es.plugin.JdbcPlugIn
driverName=sun.jdbc.odbc.JdbcOdbcDriver
serverURLBase=jdbc\:odbc\:
ndsType=ODBC
includeColumnNames=true
columnSort=
dbName=Sunshine
variables=DataField,TheLocation,CoffeeType
defaultvalues=AllBudget,All,All
sql=select CostType, sum(JunBudget) as Cost from ProductionCosts\r\nwhere
location\=NorthAmerica and\r\nCaffeinated like ‘Decaffeinated’ and\r\nCostType
<> ‘All’ and Caffeinated <> ‘ALL’ group by CostType

The query is then run. The attribute defaultvalues provides defaults for the given variables that can be used if no variable values are provided as part of the reqest.

NOTE: Particular care needs to be taken if the cache is used in conjunction with NDS variables. Variables are applied to the .ndx file before the data is put into the cache, so if two requests are made, with different variables but the same cache key, the second request’s variables will not be applied.

NDS Process Modules

NDS Process modules operate on data after it is retrieved from the data source but before it is passed to a chart or a table. You can build your own process modules to perform calculations, apply custom business logic or simply format the data into a more appealing format for display.

Implementation

Each process module is a Java method that takes three arguments as input and then outputs an object array containing the modified data. The process module is inserted into the reporting process after data has been retrieved from the data store, but before any graphical or tabular rendering has begun. This allows the same process module to be applied to any back-end data store as well as allowing the modified data to be used for any presentation component (e.g. chart or table). Additionally, process modules may be “chained” together so that data fabricated by one process module invocation can be used by a subsequent process invocation.

All process modules must implement the NDSPostProcessorInterface, which has one method, processNDS(). The method processNDS() has three arguments: indata, params, and nth:

  • indata is an object array that represents the data coming from the data store or possibly from another process module.
  • params is a hashtable of all the configuration parameters passed to the NDS. If the process module requires runtime parameters, this hashtable can be used to supply them from the NDS file.
  • nth is an index that identifies how many times this process module has been called in this particular NDS instance. This is useful if the NDS file must contain multiple sets of parameters for each time the process module is called.

The following is an example of a process module that will calculate a mean on a given column:
 

package processmodules;
 
import java.util.Hashtable;
import netcharts.server.api.NDSPostProcessorInterface;
 
public class CalculateMean implements NDSPostProcessorInterface {
 
public Object[][] processNDS(Object indata[][], Hashtable params, int nth) {
 
if (indata == null || indata.length < 1 || indata[0]==null)
return indata;
 
// create an output structure with one additional column.
Object outdata[][] = new Object[indata.length][indata[0].length+1];
 
// get the index of the column that we are to calculate the mean for
String indexString = (String)params.get(“ColumnIndex”+nth);
 
// convert index string to int
int columnIndex = -1;
 
try {
columnIndex = Integer.parseInt(indexString);
} catch (Exception ex) {
columnIndex = -1;
}
// ensure we have a legal column index
if (columnIndex < 0 || columnIndex >= indata[0].length)
return indata;
 
// get sum and count for use in calculating mean
double sum = 0;
double dv = 0;
int count = 0;
 
// for each row in the target column
for (int row=0; row < indata.length; row++) {
if(indata[row]==null || columnIndex>=indata[row].length)
continue;
 
Object val = indata[row][columnIndex];
 
if (val instanceof Number) {
// the data object is a number
dv = ((Number)val).doubleValue();
} else {
// attempt to convert the data object to a number
try {
dv = Double.parseDouble(val.toString());
} catch (Exception ex) {
// non-numeric value; ignore and continue
continue;
}
}
 
sum += dv;
count++;
}
 
//calculate mean
double mean = 0;
if(count != 0)
mean = sum/count;
 
// copy input to output
for (int row=0; row < indata.length; row++) {
if(indata[row]!=null)
System.arraycopy(indata[row], 0, outdata[row], 0, indata[row].length);
}
 
// add mean column to output, header for the new Column is
// hardcoded as “Mean” but could be passed in as param
int meancol = outdata[0].length-1;
outdata[0][meancol] = “Mean”;
Double meanValue = new Double(mean);
 
for (int row=1; row < outdata.length; row++) {
outdata[row][meancol] = meanValue;
}
 
// return the output
return outdata;
}
}

 
Compiling and Deploying Process Modules

Compile the source and create a jar file called ProcessModules.jar. When compiling, the file ncs.jar will need to be included in the classpath. This file can be found in the \Server\common\lib directory in your NetCharts Server distribution. The compilation and jar commands will look something like:
 

javac -classpath “C:\Program Files\Visual Mining\NetCharts Server 6.0\Server\common\lib\ncs.jar;.” -d .\ processmodules\CalculateMean.java
 
jar -cf ProcessModules.jar processmodules\*.class

 
To make the Process Module available within NetCharts Server, you can either use the Import Jar Files link in the Administration Console, or copy ProcessModules.jar file to the NetCharts Server /Server/common\lib directory. Restart NetCharts Server.

Testing a Process Module in NetCharts Server

Go into the Development Console of the NetCharts Server and create a new project called PMTest.

Select New and create a new file called MeanTest and select the Data/NDS option and then Create.

When the wizard appears, set the type to HSQL, and enter the following into the SQL field:
 

SELECT * FROM pagehittable

Click on the Done button.

Now, edit the NDS file from the NetCharts Server’s Development Console by clicking on the Edit icon.

First, test the NDS by clicking on Test. Note that you only have four columns of data. Add the following code to the end of the NDX file:
 

processModules=processmodules.CalculateMean
 
ColumnIndex1=3

Select Test again. See that there is a new fifth column titled “Mean” and it contains an average of all of the Server 3 hits from column four.
 
NDS PlugIns

The following is a description of the NDS PlugIns provided by Visual Mining.

The JDBC PlugIn

The JDBC PlugIn is the main interface for accessing data from relational databases including ODBCenabled databases, Oracle, Sybase, and MySql. Developers with ODBC, Oracle, Sybase, or other JDBCaccessible databases should use the NDS creation form available in the Developer Console.

The properties used to control the JdbcPlugIn behavior include all the NDSPlugInAdapter properties, serverURLBase, driverName, dbName, sql, login, password, properties, skipNulls, nullString, stayAlive, simpleDateFormat, includeColumnNames and multipleResults.

Some examples:

Sample Meaning
serverURLBase= jdbc:odbc: The first part of the JDBC URL. This is concatenated with the dbName property to create the complete JDBC URL. The generated URL must be in the form that the given database driver expects.
driverName=sun.jdbc.odbc.JdbcOdbcDriver The path to the JDBC driver to use.
dbName=Sunshine Concatenated with the serverURLBase property. For ODBC the dbName represents the DSN.
sql=select * from ProductionCosts A valid Structured Query Language (SQL) statement.
login=foo If non-null, this parameter is sent with the password parameter when connecting to the database. If login is set, password must also be set (even if there is no password).
password=bar If non-null, this parameter is sent with the login parameter when connecting to the database. If no password is required, but a login is, password should be set to the empty string, “”
properties=user:foo&&password:bar Some JDBC drivers require parameters in addition to those already specified.
skipNulls=true If true, does not return any string or placeholder when a null is found in the dataset. If false, inserts the contents of nullString.
nullString=NoData Used to indicate a null value if skipNulls is set to false.
stayAlive=true If true, the database connection is kept open.
simpleDateFormat=dd/MM/yyyy Convert any date fields into the given format.
includeColumnNames=true If true, the database column names are written into the first row of the result.
multipleResults=true If true, all returned results from the database are processed. If set to false, the default, only the first result is processed. This property should be used when doing an update or insert as well as a select in the same SQL statement.

 
The CSV/Flat File PlugIn

The CSV/Flat File PlugIn is used to parse data from flat files.

The properties used to control the CsvPlugIn behavior include all the NDSPlugInAdapter properties, filename, fieldDelim, endOfLineDelim, strings and comments.

Some examples:

Sample Meaning
filename=mydatafile.dat Data file to read from. Must be in the project tree or available via URL.
fieldDelim=, The character used to delimit fields.
EndOfLineDelim=\n\r Characters used to delimit the end of a line.
Strings=” The character used to indicate the start and stop of a text string.
Comments=# The character used to indicate the beginning of a comment.
ignoreChars=%, Remove these characters from the output. Applied after other field processing.

 
The XML PlugIn

The XML PlugIn provides the ability to parse XML formatted files.

The properties used to control the XMLPlugIn behavior include all the NDSPlugInAdapter properties, filename, xmlString and parserClassName.

Some examples:

Sample Meaning
filename=mydatafile.xml XML data file to read from. Must be in the project tree or available via URL.
xmlString=<xml>…..</xml> Process the xml data passed directly in. This is an alternative to using the filename property.
parserClassName=myXMLParser.class Name of an XML parser that implements the netcharts.server.api.XMLParserInterface class.
Strings=” The character used to indicate the start and stop of a text string.
Comments=# The character used to indicate the beginning of a comment.
ignoreChars=%, Remove these characters from the output. Applied after other field processing.

 
The Cache PlugIn

The Cache PlugIn is used as a data store. It allows data to be put into the cache from outside sources. One data is placed into the NCS data cache, it can be accessed with the same filtering, aggregation, and sort capabilities as other NDS data.

The properties used to control the CachePlugIn behavior include all the NDSPlugInAdapter properties, and the defaultData parameter. The defaultData parameter is used when the CachePlugIn is invoked with no cacheKey, or when the cache does not contain the given cacheKey. An example of defaultData is shown below:

defaultData=1,2,3,4\n5,6,7,8\n9,10,11,12,13

An NDS Example

In order to explain how NDS works, this example describes the flow of an NDS request.
A bar chart template is requested. It has the following CDL parameters:
 


NdsParams = (“rowFilter”, “0”), (“columnFilter”, “3,6*”), (“orderBy”,
“ROW”);
LegendItems = NDS “BarChartData.ndx”;

NdsParams = (“rowFilter”, “2*”), (“columnFilter”, “0”);
BarLabels = NDS “BarChartData.ndx”;

NdsParams = (“rowFilter”, “2*”), (“columnFilter”, “3,6*”);
DataSet1 = NDS “BarChartData.ndx”;

The chart resource parses the CDL. It creates three NDS objects for the LegendItems, BarLabels, and DataSet1 parameters.

A thread is created to retrieve the LegendItems data by requesting NDS execution with the BarChartData.ndx property file. The runtime properties rowFilter=0, columnFilter=3,6*, orderBy=ROW are passed along to the NDS plugin.

Two other threads are created to get the BarLabels and DataSet1. Note that each thread has a unique set of runtime parameters to minimally identify the row and columns of data to use for the parameter.

The NDS resource processes the thread requests. It first reads the BarChartData.ndx property files. This file contains the following:
 

filename=EconomicCensusIns1997.csv
Class=chartworks.es.plugin.CsvPlugIn
ndsType=CSV
fieldDelim=,
lineDelim=
columnSort=3
skipFirstRowSort=true
cacheKey=ExamplesBarchart
cacheLifeSpan=3
ignoreChars=,”
strings=”
rawRowFilter=3-14
rawColumnFilter=2*

An instance of the chartworks.es.plugin.CsvPlugIn is created and started for the LegendItems parameter. The plugin superclass, NDSPlugInAdapter, is called and passed the properties in the .ndx property file and the runtime properties from the NdsParams CDL parameter.

The NDSPlugInAdapter first checks the cache to see if a cache entry named “ExamplesBarchart” is present. It is not, so it calls the CsvPlugIn’s retrieveData method.

The retrieveData opens the named data file and parses it. It returns a two-dimensional object array of data back to its superclass (NDSPlugInAdapter).

NDSPlugInAdapter applies the “raw” data filters, and then stores the array into the cache with a three second life span.
 
dev
 
NDSPlugInAdapter then passes the array to the post processing filters. The post-processing filters extract columns 3, 6 and up from row 0. The data is formatted automatically and returned to the chart data client.
 
dev
 
The BarLabels thread runs. It checks the cache for ExamplesBarchart and gets a cache hit. It passes along the data to the post processing filters to extract column 0 from rows 2 and up. The data is formatted automatically and returned to the chart data client.
 
dev
 
The DataSet1 thread runs. It checks the cache for ExamplesBarchart and gets a cache hit. It passes along the data to the post processing filters to extract columns 3, 6 and up from rows 2 and up. Since barchart data is one dimensional, each column will cause a new DataSet parameter to be automatically generated. The data is then formatted and returned to the chart data client.
 
dev
 
All NDS references have been resolved. The chart renders.

Configuring a Third-Party JDBC Driver for Use by NDS

In some situations, developers may need to use NDS to retrieve data from a JDBC data source not initially configured by NetCharts Server. This is an easy problem to solve, since the provided JdbcPlugIn already does most of the work. The steps are:
 

  • Install the database drivers into the NetCharts Server classpath.
  • Create the NDS
  • Specify the driver class path
  • Specify the appropriate URL base
  • Specify the appropriate data base name to append to the URL base
  • Specify any login or password information
  • Create a chart and table that uses this information.

As an example, consider the MySQL program, an open source database downloadable from http://www.mysql.com. Although NCS has a specific wizard for configuring MySQL NDS files, MySQL access can also be configured by using the generic JDBC connection wizard as follows:

  • Download the MySQL JDBC Drivers. The JDBC drivers for MySQL can be found at http://www.mysql.com. The JDBC drivers will be in a Java Archive (JAR) format with a .jar extension.
  • Use the Import Jar facility in the NetCharts Server Administration console to import the driver.
  • Restart NetCharts Server.
  • Create a project in NetCharts Server
  • Create a new NDS file in the project by selecting New, entering mysql for the name, and selecting Data/NDS as the Type.
  • Set the NDS Type to JDBC. The wizard will present text fields that prompt for properties required by a generic JDBC connection

Server URL Base: jdbc:mysql://
Drive name: com.mysql.jdbc.Driver
DB Name: localhost/test
Login: username
Password: password

The serverURL Base represents the static part of the URL that will be sent to the driver when it attempts to connect to the database. The DB Name parameter represents the dynamic part of the connection USL. For MySQL the URL connection strings look like: jdbc:mysql://127.0.0.1/test where 127.0.0.1 represents the IP address of the machine on which the database is loaded, and ‘test’ represents a database within MySQL.

The sql parameter can be set to any valid SQL statement for the selected database. For example,

sql=select price from shop

Once the NDS is created, you can verify that it works using the Test button. Once verified, this NDS will be able to perform all the same functions as the supported NDS types, including filtering, sorting, and populating charts and tables.

NOTE: Some JDBC drivers explicitly require that login and password information be passed as name=value style properties. If this is the case, this information can be passed by using the properties NDS parameter in the form:

properties=property1:value1&&property2:value2.

So, for example, if your database driver expected the login and password to be passed with the properties “user” and “password”, your parameter would look like: properties=user:foo&&password:bar

CDX Variables

NetCharts Server supports the use of dynamic variables for passing data into chart data clients. These variables are known as CDX variables. CDX is a quick, convenient way to pass data or other dynamic attributes into a chart data client.

CDX is a nickname for “Dynamic CDL,” where CDL refers to Visual Mining’s Chart Definition Language. CDX allows the chart to be dynamically modified at run time, just before the chart is rendered.

How CDX Works

A chart template file contains a number of Chart Definition Language (CDL) parameters. When a chart template file is requested either by a ToolKit or URL call, one or more parameters containing a variable name and value can be passed into the request. Then, when processing the chart template file (with a .cdx extension), the NetCharts Server takes the input variable-value parameter and searches the requested template file for instances of the variable. Each instance found in the template file will be replaced by the value of the parameter.

Consider the following CDL fragment for a pie chart’s slice data and slice labels.

SliceData = SLICEDATA;
SliceLabels = SLICELABELS;

If we pass in “10,20,30” for SLICEDATA and “oh,happy,day” for the SLICELABELS parameter, then the parameters will be replaced before rendering, showing a pie chart with three slices of values 10, 20, and 30, and with the respective labels “oh”, “happy”, and “day”
 
dev
 
Passing CDX Parameters

The NetCharts Server API Toolkit is the preferred method for requesting charts and passing CDX parameters. The following examples show how CDX parameters are passed for each of the supported toolkits and directly via a URL query string.

NOTE: Everything passed via URL must not contain special URL reserved characters. Those characters include: = ? & / %. If those characters are needed, you must URL-encode those characters by converting the character’s ASCII code into hexadecimal and prefixing with a percent sign.

Java Example
 

Hashtable chartParams = new Hashtable();
chartParams.put(“SLICEDATA”,”10,20,30″);
chartParams.put(“SLICELABELS”,”oh,happy,day”);
pie = toolKit.getChartAsImageAndMap(“/CDXExamples/piex.cdx”, chartParams, request, false);

ASP Example

<%
Dim toolKit, chartParams
Set toolKit = Server.CreateObject(“NetChartsServer.NSToolKit”)
Set chartParams = Server.CreateObject(“NetChartsServer.Hashtable”)
toolKit.setProject “Examples”
 
chartParams.putValue “type”, “PNG”
chartParams.putValue “SLICEDATA”, “10,20,30”
chartParams.putValue “SLICELABELS”,”oh,happy,day”;
%>
<%=toolKit.getChartAsImageAndMap(“/CDXExamples/piex.cdx”, myHashTable)%>

.NET Example

Dim toolKit as NetChartsServer.NSWebToolKit = new
NetChartsServer.NSWebToolKit(Examples”)
Dim hashTable as Hashtable = new Hashtable
 
myHashTable.Add(“type”,”PNG”)
myHashTable.Add(“SLICEDATA”,”10,20,30″)
myHashTable.Add(“SLICELABELS”,”oh,happy,day”)
 
Try
theChart.Text = toolkit.getChartAsImageAndMap(“/CDXExamples/piex.cdx”, hashTable, “”, False, Request.ApplicationPath & “/getncsimage.aspx”)
Catch nce As NetChartsServer.NSToolKitException
theChart.Text = nce.Message
End Try
clientScript.Text = toolkit.getRolloverJavaScript

URL Example

http://yourserver.yourcompany.com:8001/projects/Examples/CDXExamples/piex.cdx?type=
PNG&SLICEDATA=10,20,30&SLICELABELS=oh,happy,day

CDX Variables

A CDX variable need not be explicitly specified as a variable within the CDX file. Any contiguous alphanumeric text may be replaced anywhere in the CDL code. For example, assume the following Footer CDL parameter is in the CDX file:

Footer =(“Portfolio Distribution For USER”,black,”Helvetica”,12);

In this case, we could consider “USER” to be a variable. But we could also consider “Portfolio” to be a variable as well. If our request contained the CDX parameters

USER=James
Portfolio=Cash

then the Footer would look like the following before rendering:

Footer = (“Cash Distribution For James”,black,”Helvetica”,12);

CDX variables are not limited to text or data – they can also be used to change the chart appearance. Given our footer example above, we can change the color and font of the text by appending more CDX variables

USER=James
Portfolio=Cash
black=green
Helvetica=Courier New

resulting in a footer of:

Footer =(“Cash Distribution For James”,green,”Courier New”,12);

Finally, CDX variables are not just limited to changing the value of a parameter – they can change the parameter itself. If we append the parameters

Footer=Header

to the request we’ve created so far, the Footer parameter will change to a Header instead and will appear in the CDL as:

Header = (“Cash Distribution For James”,green,”Courier New”,12);

CDX Variables

The value component of the CDX parameter is extremely flexible. It can be a fragment of a CDL parameter, comma delimited lists, CDL tuples, the entire parameter, or multiple parameters. You can use comment lines in CDL and then uncomment them by replacing the comment parameter (‘#’) with a space.

Generally the only “rule” is that after replacement, the result must be valid CDL.