Automatic processing of results (or other) data using an extensible data exchange framework

Product(s): WaterGEMS
Version(s): 10.01.XX.XX
Area: Output and Reporting
Author: Wayne Hartell, Principle Software Engineer, Bentley Systems

Problems

Have you ever wondered whether it is possible to do the following things?

  • Using SCADAConnect to compute WaterGEMS every X minutes and automatically export results using custom logic.
  • Automatically post-process computed results to produce a report to custom specifications.
  • Convert computed results into a format that can be readily consumed by a 3rd party system such as Power BI or other web dash-boarding tool.
  • Execute some additional calculations then update model data based on those results.
  • Do a statistical analysis on computed results and trigger some kind of action based on the results.
  • Anything else that can be defined as acquiring some data, acting upon it, then producing some kind of output.

Solution

For those of you who are familiar with WaterObjects.NET you'll be aware that it is and has been possible for a long time to write .NET code against the WaterObjects object model to carry out tasks such as reading and writing model data and also reading model results. With regards to the kind of use cases described above, what has been lacking in the past is an easy way to write code that can be coupled with WaterGEMS in a way that can execute automatically and also the provision of generic support for doing the types of things that data processing tools might need to do, such as command line processing, interacting with files and directories, file archival, working with databases, providing configuration support and logging operations.

In the following series of documents you will find a description of a framework called the Haestad.DataIntegration Framework (HDI). Included in these documents are a description of several off-the-shelf functionalities (built on the framework) that enable automatic, configurable, conversion of WaterGEMS energy costs and hydraulics results into comma separated values (CSV) format. Such support could be useful if wanting to automatically publish results data for key assets to other systems such as Bentley AssetWise Operational Analytics, Power PI, OSI PI, or other real-time/dashboarding type system.

Furthermore, you will find documents that describe how using this framework it is possible to write software components that have the following traits:

  1. Able to convert data from one format to another. (e.g., model results to CSV, model spatial data to SQL server, demand data to model, etc.).
  2. Support user configurable behaviors.
  3. Conform to a standard specification, thus allowing for a standardized and robust boot-strapping process.
  4. Support different levels of logging (e.g., errors, warnings, info, debug etc).
  5. Easily extensible and re-usable.
  6. Can be executed automatically either out-of-process (by the HDI driver) or in-process (by a compatible WaterObjects application (e.g., SCADAConnect Simulator)).
  7. Only limited by your imagination in terms of custom capabilities.

Documents

The series of documents that describe this framework are listed below. The first document is a good place to start as it provides a bird's eye view of the framework, plus describes how the additional documents relate to the various pieces of the framework. For instance, if you are interested in publishing results to CSV format after a SCADAConnect Simulator auto-compute, then you may be able to jump right into the "WaterGEMS to CSV User Guide". If, however, you have more specific requirements that may require some kind of customization, you might be better off starting with the "Publishing Hydraulic Modeling Data to 3rd Party Systems" document. The last two documents talk specifically about how to hook custom post calculation logic into SCADAConnect Simulator, though this is not a requirement for utilizing the framework.

Note: to download these files, you will need to be signed in.

DataIntegration Framework Overview

Publishing Hydraulic Modeling Data to 3rd Party Systems

DataIntegration User Guide

DataIntegration User Guide (In Process)

WaterGEMS to CSV User Guide

WaterGEMS to CSV User Guide (Batch Processing)

SCADAConnect Post Calculation Batch File User Guide

SCADAConnect Post Calculation Data Exchanger User Guide

Sample Files

In addition to the above documents, several sample files are available as follows:

Watertown (7z File) - (Used with WaterGEMS to CSV User Guide)

BatchSample (7z File) - (Used with WaterGEMS to CSV User Guide (Batch Processing))

SampleDataExchanger (Zip File) - (Used with Publishing Hydraulic Modeling Data to 3rd Party Systems)

SCADAPostCalculationDataExchanger (Zip File) - (Used with SCADAConnect Post Calculation Data Exchanger User Guide)

Notes

Some of the functionality described in this document is available with WaterGEMS versions as early as WaterGEMS 10.01.XX.XX, however, some of the most recent functionality, including in-process data exchanger support in SCADAConnect Simulator, require the latest Update 2 of WaterGEMS (10.02.01.06) in addition to the latest patch set (Patch01 or higher - please contact Technical Support via the Forum or Service Request Manager).

See Also

SCADAConnect Simulator for WaterGEMS (V8i SELECTseries 6 and CONNECT edition)

WaterObjects.NET FAQ

Anonymous
Recommended
Related