Job Queue Not reachable

HI All, 

I am using a 3 Workstation Cluster setup with CC Centre.

Many times, I have been getting errors that Job Queue is not readable (error code: 287571968). Please check job queue Access. The screenshot is attached below.
The Job queue is the default location and the pathname are also short. But still, I am not able to resolve this.

The production that I am attempting is a ContextInsight production

I request you to please help me in the same

  • Hello, 

    there were some known issues with the jobqueue database when doing cluster. Especially if the network is not stable or not so strong. 

    I highly recommend that you update ContextCapture to Update 19 that has just been released. Please let us know if the problem persists. 

    Thanks,

    Sylvain

    Answer Verified By: CARBSE 

  • Thank you for your answer, 

    Although most of problems have been solved now regarding this, but I still face this error frequently :

    [2021-11-29 17:58:38.100][error   ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database

     I guess this is because the CC engine is IDLE?

    Thanks 

  • Hello, 

    This error means that the engine was not able to communicate with the jobqueue because it was already accessed and locked by another process (master or engine). The engine retries automatically the connection a few second laters. this doesn't affect the processing performance. This error is only concerning if it appears every 15 seconds (or so) in the engine console. 

  • Hi Sylvain, For a AT process, I am facing the same issues, Request you to help:

    Let me paste you screenshot/Text of all 3 WSs in Cluster:

    WorkStation1 (Master)

    ContextCapture Center version 10.19.0.122 running << C:\Program Files\Bentley\ContextCapture Center\bin\CCEngine.exe >> from directory << C:\Program Files\Bentley\ContextCapture Center\bin >>
    [2021-Dec-06 05:31:21 UTC] Starting CCEngine.exe on DELL@carbsews01

    ====================================================================
    Welcome to ContextCapture Center Update 19 - v10.19.0.122 (Academic)
    ====================================================================
    We detected 1 physical devices.
    There are total 1 physical devices for application:
    Device 0: NVIDIA TITAN RTX of NVIDIA: vulkan version 1.2.186, device memory: 24276 MB, host memory: 65384.4 MB

    Warning: Expired beta features
    Processing the following task types: AI AT RasterProduction TileProduction
    The Engine will profile jobs
    The Engine will use multigpu option
    [2021-Dec-06 11:01:22] Starting Engine on job queue "FILE:C:/Users/DELL/Documents/Bentley/ContextCapture Center/Jobs"
    [2021-Dec-06 11:01:22] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / 7a2d4cdc-d25e-43da-a9da-cb6d383a361c
    [2021-Dec-06 11:01:25] Info: Task completed (returned : 0)
    [2021-Dec-06 11:01:25] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / 4cc093ed-11d0-46df-ac49-2bc30c871611
    [2021-Dec-06 11:04:09] Info: Task completed (returned : 0)
    [2021-12-06 11:51:45.285][error ][JobTaskMgrDB] cleanUpJobTasks: Could not lock the jobqueue database
    [2021-12-06 11:51:51.503][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:52:07.083][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:52:23.034][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:52:39.111][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:52:55.066][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database


    Workstation 2 (Worker)

    ContextCapture Center version 10.19.0.122 running << C:\Program Files\Bentley\ContextCapture Center\bin\CCEngine.exe >> from directory << C:\Program Files\Bentley\ContextCapture Center\bin >>
    [2021-Dec-06 05:31:53 UTC] Starting CCEngine.exe on DELL@carbsews02

    ====================================================================
    Welcome to ContextCapture Center Update 19 - v10.19.0.122 (Academic)
    ====================================================================
    We detected 1 physical devices.
    There are total 1 physical devices for application:
    Device 0: NVIDIA TITAN RTX of NVIDIA: vulkan version 1.2.186, device memory: 24276 MB, host memory: 65384.4 MB

    Warning: Expired beta features
    Processing the following task types: AI AT RasterProduction TileProduction
    The Engine will profile jobs
    The Engine will use multigpu option
    [2021-Dec-06 11:01:54] Starting Engine on job queue "FILE://carbsews01/Users/DELL/Documents/Bentley/ContextCapture Center/Jobs"
    [2021-Dec-06 11:02:19] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / 37174caf-b5ec-4702-a382-342ee3802122
    [2021-Dec-06 11:06:04] Info: Task completed (returned : 0)
    [2021-Dec-06 11:06:32] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:09:44] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:10:14] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:11:16] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:11:46] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:12:47] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:13:15] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:14:13] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:14:44] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:15:43] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:16:14] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:17:15] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:17:38] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:18:42] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:19:07] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:20:06] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:20:31] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:21:32] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:21:58] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:23:00] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:23:26] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:24:30] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:24:57] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:25:58] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:26:24] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:27:26] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:27:52] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:28:49] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:29:15] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:30:15] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:30:41] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:31:37] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:32:00] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:33:05] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:33:31] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:34:31] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:34:57] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:35:57] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:36:24] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:37:26] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:37:52] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:38:51] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:39:17] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:40:20] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:40:46] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:41:43] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:42:08] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:43:09] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:44:00] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:44:59] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:45:24] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:46:24] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:46:48] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:47:40] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:48:01] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:48:59] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:49:26] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:50:28] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:50:52] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972
    [2021-Dec-06 11:51:56] Task was interrupted (job queue not reachable: 287440896). Please check job queue access.
    [2021-Dec-06 11:52:20] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / a64d32ec-383f-40ec-86d4-181303454972

    Workstation 3 (Worker):

    ContextCapture Center version 10.19.0.122 running << C:\Program Files\Bentley\ContextCapture Center\bin\CCEngine.exe >> from directory << C:\Program Files\Bentley\ContextCapture Center\bin >>
    [2021-Dec-06 05:32:06 UTC] Starting CCEngine.exe on admin@carbsews03

    ====================================================================
    Welcome to ContextCapture Center Update 19 - v10.19.0.122 (Academic)
    ====================================================================
    We detected 1 physical devices.
    There are total 1 physical devices for application:
    Device 0: NVIDIA TITAN RTX of NVIDIA: vulkan version 1.2.186, device memory: 24276 MB, host memory: 65384.4 MB

    Warning: Expired beta features
    Processing the following task types: AI AT RasterProduction TileProduction
    The Engine will profile jobs
    The Engine will use multigpu option
    [2021-Dec-06 11:02:08] Starting Engine on job queue "FILE://carbsews01/Users/DELL/Documents/Bentley/ContextCapture Center/Jobs"
    [2021-12-06 11:02:13.599][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-Dec-06 11:02:53] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / 1ec93716-8ec6-4175-8857-bad0a1924113
    [2021-Dec-06 11:05:41] Info: Task completed (returned : 0)
    [2021-Dec-06 11:06:12] Starting Task job_20211206T053111.971929_T_P_23_11_2021_AT / 5432cdfe-69f0-45f9-89f9-7aaa3144d12f
    [2021-Dec-06 11:08:54] Info: Task completed (returned : 0)
    [2021-12-06 11:10:09.344][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:11:41.336][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:13:12.318][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:14:38.385][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:16:08.313][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:17:39.339][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:19:07.321][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:20:31.286][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:21:57.391][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:23:24.341][error ][JobTaskMgrDB] cleanUpJobTasks: Could not lock the jobqueue database
    [2021-12-06 11:23:29.405][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:24:55.325][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:26:23.362][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:27:50.330][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:29:14.360][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:30:40.338][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:32:02.302][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:33:30.322][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:34:56.291][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:36:22.367][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:37:50.361][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:39:16.333][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:40:45.349][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:42:08.351][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:43:34.422][error ][JobTaskMgrDB] cleanUpJobTasks: Could not lock the jobqueue database
    [2021-12-06 11:43:39.462][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:43:55.293][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:45:24.336][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:46:48.291][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:48:05.323][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:49:24.318][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database
    [2021-12-06 11:50:53.336][error ][JobTaskMgrDB] PopTask: Could not lock the jobqueue database