RE: Job Queue Not reachable

CARBSE — Mon, 06 Dec 2021 06:24:59 GMT

Hi Sylvain, For a AT process, I am facing the same issues, Request you to help:

Let me paste you screenshot/Text of all 3 WSs in Cluster:

WorkStation1 (Master)

ContextCapture Center version 10.19.0.122 running << C:\Program Files\Bentley\ContextCapture Center\bin\CCEngine.exe >> from directory << C:\Program Files\Bentley\ContextCapture Center\bin >>
[2021-Dec-06 05:31:21 UTC] Starting CCEngine.exe on DELL@carbsews01

====================================================================
Welcome to ContextCapture Center Update 19 - v10.19.0.122 (Academic)
====================================================================
We detected 1 physical devices.
There are total 1 physical devices for application:
Device 0: NVIDIA TITAN RTX of NVIDIA: vulkan version 1.2.186, device memory: 24276 MB, host memory: 65384.4 MB

Workstation 2 (Worker)

Warning: Expired beta features
Processing the following task types: AI AT RasterProduction TileProduction
The Engine will profile jobs
The Engine will use multigpu option
[2021-Dec-06 11:01:54] Starting Engine on job queue "FILE://carbsews01/Users/DELL/Documents/Bentley/ContextCapture Center/Jobs"
[2021-Dec-06 11:02:19] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / 37174caf-b5ec-4702-a382-342ee3802122
[2021-Dec-06 11:06:04] Info: Task completed (returned : 0)
[2021-Dec-06 11:06:32] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:09:44] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:10:14] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:11:16] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:11:46] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:12:47] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:13:15] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:14:13] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:14:44] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:15:43] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:16:14] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:17:15] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:17:38] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:18:42] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:19:07] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:20:06] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:20:31] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:21:32] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:21:58] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:23:00] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:23:26] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:24:30] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:24:57] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:25:58] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:26:24] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:27:26] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:27:52] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:28:49] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:29:15] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:30:15] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:30:41] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:31:37] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:32:00] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:33:05] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:33:31] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:34:31] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:34:57] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:35:57] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:36:24] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:37:26] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:37:52] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:38:51] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:39:17] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:40:20] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:40:46] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:41:43] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:42:08] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:43:09] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:44:00] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:44:59] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:45:24] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:46:24] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:46:48] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:47:40] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:48:01] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:48:59] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:49:26] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:50:28] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:50:52] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:51:56] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:52:20] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972

Workstation 3 (Worker):

RE: Job Queue Not reachable

Sylvain Lotteau — Fri, 03 Dec 2021 10:02:05 GMT

Hello,

This error means that the engine was not able to communicate with the jobqueue because it was already accessed and locked by another process (master or engine). The engine retries automatically the connection a few second laters. this doesn't affect the processing performance. This error is only concerning if it appears every 15 seconds (or so) in the engine console.

RE: Job Queue Not reachable

CARBSE — Tue, 30 Nov 2021 04:34:05 GMT

Thank you for your answer,

Although most of problems have been solved now regarding this, but I still face this error frequently :

[2021-11-29 17:58:38.100][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database

I guess this is because the CC engine is IDLE?

Thanks

RE: Job Queue Not reachable

Sylvain Lotteau — Tue, 16 Nov 2021 13:35:22 GMT

Hello,

there were some known issues with the jobqueue database when doing cluster. Especially if the network is not stable or not so strong.

I highly recommend that you update ContextCapture to Update 19 that has just been released. Please let us know if the problem persists.

Thanks,

Sylvain