Job Queue Not reachable

Offline CARBSE over 1 year ago

HI All,

I am using a 3 Workstation Cluster setup with CC Centre.

Many times, I have been getting errors that Job Queue is not readable (error code: 287571968). Please check job queue Access. The screenshot is attached below.
The Job queue is the default location and the pathname are also short. But still, I am not able to resolve this.

The production that I am attempting is a ContextInsight production

I request you to please help me in the same

Top Replies

Offline Sylvain Lotteau Tue, Nov 16 2021 8:35 AM +1 verified

Hello, there were some known issues with the jobqueue database when doing cluster. Especially if the network is not stable or not so strong. I highly recommend that you update ContextCapture to Update…

Parents

+1 Offline Sylvain Lotteau Tue, Nov 16 2021 8:35 AM
Hello,

there were some known issues with the jobqueue database when doing cluster. Especially if the network is not stable or not so strong.

I highly recommend that you update ContextCapture to Update 19 that has just been released. Please let us know if the problem persists.

Thanks,

Sylvain
Answer Verified By: CARBSE
Cancel
Vote Up +1 Vote Down

Sign in to reply

Reject Answer

Cancel
0 Offline CARBSE Mon, Nov 29 2021 11:34 PM in reply to Sylvain Lotteau

Thank you for your answer,

Although most of problems have been solved now regarding this, but I still face this error frequently :

[2021-11-29 17:58:38.100][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database

I guess this is because the CC engine is IDLE?

Thanks
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel
0 Offline Sylvain Lotteau Fri, Dec 3 2021 5:02 AM in reply to CARBSE
Hello,

This error means that the engine was not able to communicate with the jobqueue because it was already accessed and locked by another process (master or engine). The engine retries automatically the connection a few second laters. this doesn't affect the processing performance. This error is only concerning if it appears every 15 seconds (or so) in the engine console.
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Reply

0 Offline Sylvain Lotteau Fri, Dec 3 2021 5:02 AM in reply to CARBSE
Hello,

This error means that the engine was not able to communicate with the jobqueue because it was already accessed and locked by another process (master or engine). The engine retries automatically the connection a few second laters. this doesn't affect the processing performance. This error is only concerning if it appears every 15 seconds (or so) in the engine console.
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Children

0 Offline CARBSE Mon, Dec 6 2021 1:24 AM in reply to Sylvain Lotteau

Hi Sylvain, For a AT process, I am facing the same issues, Request you to help:

Let me paste you screenshot/Text of all 3 WSs in Cluster:

WorkStation1 (Master)

ContextCapture Center version 10.19.0.122 running << C:\Program Files\Bentley\ContextCapture Center\bin\CCEngine.exe >> from directory << C:\Program Files\Bentley\ContextCapture Center\bin >>
[2021-Dec-06 05:31:21 UTC] Starting CCEngine.exe on DELL@carbsews01

====================================================================
Welcome to ContextCapture Center Update 19 - v10.19.0.122 (Academic)
====================================================================
We detected 1 physical devices.
There are total 1 physical devices for application:
Device 0: NVIDIA TITAN RTX of NVIDIA: vulkan version 1.2.186, device memory: 24276 MB, host memory: 65384.4 MB

Warning: Expired beta features
Processing the following task types: AI AT RasterProduction TileProduction
The Engine will profile jobs
The Engine will use multigpu option
[2021-Dec-06 11:01:22] Starting Engine on job queue "FILE:C:/Users/DELL/Documents/Bentley/ContextCapture Center/Jobs"
[2021-Dec-06 11:01:22] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / 7a2d4cdc-d25e-43da-a9da-cb6d383a361c
[2021-Dec-06 11:01:25] Info: Task completed (returned : 0)
[2021-Dec-06 11:01:25] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / 4cc093ed-11d0-46df-ac49-2bc30c871611
[2021-Dec-06 11:04:09] Info: Task completed (returned : 0)
[2021-12-06 11:51:45.285][error ][JobTaskMgrDB] cleanUpJobTasks: Could not lock the jobqueue database
[2021-12-06 11:51:51.503][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:52:07.083][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:52:23.034][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:52:39.111][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:52:55.066][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database

Workstation 2 (Worker)

ContextCapture Center version 10.19.0.122 running << C:\Program Files\Bentley\ContextCapture Center\bin\CCEngine.exe >> from directory << C:\Program Files\Bentley\ContextCapture Center\bin >>
[2021-Dec-06 05:31:53 UTC] Starting CCEngine.exe on DELL@carbsews02

====================================================================
Welcome to ContextCapture Center Update 19 - v10.19.0.122 (Academic)
====================================================================
We detected 1 physical devices.
There are total 1 physical devices for application:
Device 0: NVIDIA TITAN RTX of NVIDIA: vulkan version 1.2.186, device memory: 24276 MB, host memory: 65384.4 MB

Warning: Expired beta features
Processing the following task types: AI AT RasterProduction TileProduction
The Engine will profile jobs
The Engine will use multigpu option
[2021-Dec-06 11:01:54] Starting Engine on job queue "FILE://carbsews01/Users/DELL/Documents/Bentley/ContextCapture Center/Jobs"
[2021-Dec-06 11:02:19] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / 37174caf-b5ec-4702-a382-342ee3802122
[2021-Dec-06 11:06:04] Info: Task completed (returned : 0)
[2021-Dec-06 11:06:32] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:09:44] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:10:14] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:11:16] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:11:46] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:12:47] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:13:15] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:14:13] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:14:44] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:15:43] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:16:14] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:17:15] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:17:38] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:18:42] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:19:07] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:20:06] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:20:31] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:21:32] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:21:58] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:23:00] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:23:26] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:24:30] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:24:57] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:25:58] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:26:24] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:27:26] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:27:52] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:28:49] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:29:15] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:30:15] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:30:41] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:31:37] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:32:00] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:33:05] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:33:31] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:34:31] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:34:57] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:35:57] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:36:24] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:37:26] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:37:52] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:38:51] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:39:17] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:40:20] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:40:46] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:41:43] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:42:08] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:43:09] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:44:00] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:44:59] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:45:24] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:46:24] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:46:48] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:47:40] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:48:01] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:48:59] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:49:26] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:50:28] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:50:52] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
[2021-Dec-06 11:51:56] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
[2021-Dec-06 11:52:20] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972

Workstation 3 (Worker):

ContextCapture Center version 10.19.0.122 running << C:\Program Files\Bentley\ContextCapture Center\bin\CCEngine.exe >> from directory << C:\Program Files\Bentley\ContextCapture Center\bin >>
[2021-Dec-06 05:32:06 UTC] Starting CCEngine.exe on admin@carbsews03

====================================================================
Welcome to ContextCapture Center Update 19 - v10.19.0.122 (Academic)
====================================================================
We detected 1 physical devices.
There are total 1 physical devices for application:
Device 0: NVIDIA TITAN RTX of NVIDIA: vulkan version 1.2.186, device memory: 24276 MB, host memory: 65384.4 MB

Warning: Expired beta features
Processing the following task types: AI AT RasterProduction TileProduction
The Engine will profile jobs
The Engine will use multigpu option
[2021-Dec-06 11:02:08] Starting Engine on job queue "FILE://carbsews01/Users/DELL/Documents/Bentley/ContextCapture Center/Jobs"
[2021-12-06 11:02:13.599][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-Dec-06 11:02:53] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / 1ec93716-8ec6-4175-8857-bad0a1924113
[2021-Dec-06 11:05:41] Info: Task completed (returned : 0)
[2021-Dec-06 11:06:12] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / 5432cdfe-69f0-45f9-89f9-7aaa3144d12f
[2021-Dec-06 11:08:54] Info: Task completed (returned : 0)
[2021-12-06 11:10:09.344][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:11:41.336][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:13:12.318][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:14:38.385][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:16:08.313][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:17:39.339][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:19:07.321][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:20:31.286][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:21:57.391][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:23:24.341][error ][JobTaskMgrDB] cleanUpJobTasks: Could not lock the jobqueue database
[2021-12-06 11:23:29.405][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:24:55.325][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:26:23.362][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:27:50.330][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:29:14.360][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:30:40.338][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:32:02.302][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:33:30.322][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:34:56.291][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:36:22.367][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:37:50.361][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:39:16.333][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:40:45.349][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:42:08.351][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:43:34.422][error ][JobTaskMgrDB] cleanUpJobTasks: Could not lock the jobqueue database
[2021-12-06 11:43:39.462][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:43:55.293][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:45:24.336][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:46:48.291][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:48:05.323][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:49:24.318][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
[2021-12-06 11:50:53.336][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel