Product(s):	APM Implementation and Performance Management
	Version(s):	7
	Environment:	N/A
	Area:	N/A
	Subarea:	N/A

Problem Description

An APM user with an ODC ON Prem installation suddenly experienced a total loss of the ODC service. The service had been turned-off as the error logging was taking taking up all of the server resources.

Solution

The ODC service itself was running without issue but significant server resource was required to process over a 1million pending indicator requests. The reading requests were erroring with the following type of message:

2023-12-11 06:25:45.6650Z Error servername01odc HOSTNAME\servername01$ Ivara.ServiceProvider.ODC.DataSourceWrapper [19] Failed to process ODC reading request EXCEPTION OCCURRED:Exception Invalid online indicator [Assetname Indicator name runhours]. ODC Tag name is blank at Ivara.ServiceProvider.ODC.IndicatorWrapper.ValidateForOnlineCollection()
at Ivara.ServiceProvider.ODC.DataSourceWrapper.CheckForPollingRequests_impl(EXPLinkSession session)

It was found that the user had accidentally updated a large block of ODC tag names by using the replace function. Unfortunately, the user had not properly filtered the table configuration before using using the replace function, breaking the ODC Tag Name for thousands of indicators, these indicators were also set at a high sampling frequency leading to a substantially amount of erroring indicators and pending reading requests which were filling-up the log files and taking a large amount of server resource.

The ODC indicator tags were updated and the 1million erroring pending reading requests were purged and the service restarted. ODC readings were then successfully received and processed. Note, the frequency was also updated as the polling frequency was set way too frequent for APM purposes, the APM ODC service is not meant to be a replication of a PI service of data historian, the frequency of data sampling for each PI/ODC tag should always be in keeping with the defined P-F interval for the particular failure being managed.