Hi Everyone - I wanted to put feelers out and see if anyone had encountered similar issues with their Orchestration Framework servers and scheduled/ad-hoc jobs and what they've done to resolve them.
Every couple of weeks, our job definitions we have set on a schedule randomly seem to stop distributing their files after renditions ran successfully. My typical steps to resolve this issue are listed below, but they've since stopped working. It eventually starts working again, but after a few days of rinsing and repeating the same steps.
1. Stopped the current jobs within Orchestration Framework that were running at the time of discovery.2. Stopped the ProjectWise Orchestration Frameworkservice.3. Delete all the Private Message queues pertaining to Bentley iCS for PDF.4. Deleted all files and folders possible fromC:\Windows\Temp.5. Deleted all subfolders within the iCS working directory (D:\ics_workingdir).6. Restarted the server hosting the Bentley iCS for PDF and Orchestration Framework Administrator.7. With all services running normally, started an incremental job on our most commonly ran job definition.
The current versions of our software are:
VM running Windows Server 2012 R2
ProjectWise InterPlot Organizer 10.00.01.22
ProjectWise Orchestration Framework CONNECT Edition 10.00.03.116
Bentley iCS for PDF 10.0.03.140
ProjectWise Explorer CONNECT Edition 10.0.03.140
ProjectWise Administrator 10.00.3140
MicroStation CONNECT Edition 10.10.0.23
MicroStation V8i (SELECTseries 3) 08.11.09.459
We added a Windows task to periodically do the following on the ICS server-
ICS on ProjectWise Update 3.2 has better management of Temp files. However, if you have any version pre 3.2, you need to clear the Temp files manually (or, using Windows Task Scheduler). The frequency of this job depends on the load of the ICS. Start with weekly clearing of files. You may increase the frequency to daily if needed.
Running a very similar build, only exception being an older version of v8i and newer version of PWE (3.167).We had the same issues for a while and resolved by changing the properties on the DistributionDispatcher so it force restarts itself after 100 messages.Similar situation for the iPlotProcessors we set to reset after 30-50 messages each.
After the above changed we had it running for about 6 months without issues until some recent windows updates/reboots caused some issues... (It always prefers a clean reboot after a Windows update reboot).
I still do your 7-step process and the steps mentioned below by Sukalyan. Since setting the Queue's to force restart themselves I don't need to do it often.
We tried the same thing for the ExitDispatcher but it makes it worse, so I don't recommend letting it restart.