Multi-threading or Using Jobs

I used jobs a long time ago to do something similar, but i'm not currently doing it (not using the script currently). There were a bunch of tricks I had to figure out to get it to work.

One was to make the script that does the steps you want to do on each datasource a standalone script that accepts parameters. Then use a 2nd script to spawn the jobs with the appropriate parameters. Trying to do it with functions or code blocks made my brain hurt.

It was also tricky getting data back. If I remember correctly I basically saved the data into an array then processed all the returned data after the jobs were done. Having multiple jobs attempt to write to the same excel file was bad (even different sheets I believe).

Also I found it was a good idea to spawn a limited number of jobs in parallel and then watch the job queue for jobs to finish before starting new ones.

Let me see if I can find the actual code.

Oh yeah, scoping was a whole deal both on passing parameters and getting data back.

OK, I found my script but the way I was running it was due to our database server configuration. We have multiple database servers with multiple datasources on them rather than one big load-balanced instance. So I would launch my jobs so their was one job per database server and that job processed all the datasources on that server synchronously while other jobs did the same for other database servers. This meant that no more than one database statistics update was taking place per database server at a time.

Let me massage this a bit and I think it'll be more understandable (i also have to strip a bunch of junk out, i haven't touched these since 2021)

I figured you'd be the one to have done something with this before.  I have been able to get multiple jobs to run and return data, which totally made my brain hurt.. Now I am trying to figure out how to throttle the number of concurrent jobs running.   Thanks.

Ah, this is roughly how I did it (this code has never been run so it probably has errors): I have an array of items I want to process, and a maximum number i want to run at a time. As I launch jobs I add the job info to an array. When the size of the array is the maximum number of jobs, i switch to polling current jobs until they complete or crash. As they complete i save the results and then remove the job info from the array (which opens another slot to start a new job...)

$dsList = @(
	'server:dsource1'
	'server:dsource2'
	'server:dsource3'
	'server:dsource4'
	'server:dsource5'
	'server:dsource6'
)

$maxJobs = 3
$dsIdx = 0
$sleepSecs = 5
$jobList = [System.Collections.ArrayList]@()
$results = [System.Collections.ArrayList]@()
While ($dsIdx -le $dsList.GetUpperBound(0)) {
	# not enough jobs launched yet?
	if ($jobList.count -lt $maxJobs) {
		$ds = $dsList[$dsIdx]
		$jobSplat = @{
			Name         = $ds
			FilePath     = '\path\to\script.ps1'
			ArgumentList = "-datasource $($ds)"
		}
		Write-Host "Starting job $($jobList.count + 1): $ds"
		$jobList.Add((Start-Job @jobSplat))
		$dsIdx++
	} else {
		# Check job list for finished/crashed jobs
		ForEach ($j in $jobList) {
			$curState = (Get-Job -Id $j.ID).State
			Switch ($curState) {
				"Running" { break }              # break exits switch
				"Completed" {
					# add job results to array list
					[void]$results.Add((Remove-Job -Id $j.ID))
					# remove completed job from job list
					[void]$jobList.Remove($j)
					break
				}
				default {
					$err = Receive-Job -ID $j.ID
					Write-Error "Error occurred in job $($j.ID): $($err)"
					[void](Remove-Job -ID $j.ID)
					[void]$jobList.Remove($j)
					break
				}
			}
		}
	}
	# pause to give jobs time to run
	[void](Start-Sleep -Seconds $sleepSecs)
}