Problem with recursion when using Get-PWFolders in large directories

If you can, please downgrade the version of PWPS_DAB to version 23.2.0.0, at least to determine whether Get-PWFolders works better in that version.  Significant changes were made to Get-PWFolders last July.

Sorry for the inconvenience,

Mark Weisman | Bentley Systems 

I may be missing something obvious here, but why couldn't you just do the following:

$rootFolderPath = 'Your\Special\ProjectWise\Path\Goes\Here'

$folders = Get-PWFolders -FolderPath $rootFolderPath -PopulatePaths # this took just over 7 seconds to traverse the approx 12000 folders

foreach ($folder in $folders) {
	# get this folders documents, if any
	$folderDocuments = $folder.GetFolderDocuments()

	# loop folder documents
	foreach ($folderDocument in $folderDocuments) {
		# do something with document
	}
}

BTW, in your code on line 31, Get-PWFolders is returning ALL the folders under the path specified and then you're doing a where on that, which is extra processing IMHO.

If you really wanted to do recursion, I suppose you could do something like this:

1. Get-PwFolders -FolderPath SomePath -JustOne

2. Get-PwFoldersImmediateChildren passing folder from step 1

3. Loop them in a recursive function, getting documents and their immediate child folders all the way down.

Recursion is a handy trick and nice to have in the toolbox, but it can get tricky.

I hope this helps.

Cheers

I needed something similar, so I rewrote yours a bit to just loop through all the folders/files directly, you don't need to worry about unique folder/documents because it will touch each file exactly once.

# New-PWLogin
$start = Get-Date
$masterFolderPath = '\Companyname\Region\PS6_Garla-Pärnu\02_HCL\03_Deliverables\05_Master_Designs\ABC\'

# get just the sub-folders immediately under the master path
Write-Host "Retrieving sub-folders from $($masterFolderPath)"
$pwfList = Get-PWFoldersImmediateChildren -FolderPath $masterFolderPath | Sort Name

# if $attributeMat to always be in order, then make this an ordered hashtable as well
$attributeMap = [Ordered]@{
	"PROP:CreateDate" = "Creation Date"
	"PROP:DocumentCreatorName" = "Created By"
	"PROP:Name" = "File Name"
	"PROP:FolderPath" = "Folder Name"
	"PROP:ProjectID" = "Folder ID"
	"PROP:DocumentGUID" = "Document GUID"
	"PROP:WorkflowState" = "State"
	"PROP:Version" = "Version"
	"CA:dc_document_no" = "File Number"
	"CA:tb_title_line_1" = "Deliverable (Title)"
	"CA:pw_doc_description" = "Document Description"
	"CA:qa_doc_no_status" = "Doc_No_Status"
	"CA:pw_deliverable" = "Is Deliverable"
	"CA:a_attrno" = "Attribute Record ID"
}

$results = [System.Collections.ArrayList]@()
# Loop through each folder
ForEach ($pwf in $pwfList) {
	Write-Host "Processing folder $(Join-Path $masterFolderPath $pwf.Name). " -NoNewLine
	# Get all documents under the sub-folder (including documents in the sub-folders under current folder)
	$pwdList = Get-PWDocumentsBySearch -FolderID $pwf.ProjectID -Slow | Sort FullPath
	Write-Host "Found $($pwdList.count) documents"
	ForEach ($pwd in $pwdList) {
		Write-Host "Processing file $($pwd.FullPath)"
		$row = [ordered]@{}                                 # maintain key order
		ForEach ($k in $attributeMap.keys) {
			$colName = $attributeMap.$k
			($type,$attName) = $k.split(':',2)
			if ($type -eq 'PROP') {
				$row.$colName = $pwd.$attName
			} elseif ($type -eq 'CA') {
				$row.$colName = $pwd.CustomAttributes[$attName]
			} else {
				Write-Warning "Improper key in attribute map: $($k)"
				continue
			}
		}
		[void]$results.Add([PSCustomObject]$row)
	}
}

$results | Export-Excel -Path '.\output.xlsx' -WorkSheetName 'Documents' -AutoSize -FreezeTopRow

$runTime = ((Get-Date) - $start).TotalSeconds
Write-Host "Total Script Run Time: $($runTime) seconds"
Undo-PWLogin

It uses Get-PWFoldersImmediateChildren to get just the sub-folders under the specified folder. It then processes all the documents under that folder in one go.

Get-PWFolders without the -JustOne option, and Get-PWDocumentsBySearch without -JustThisFolder automatically recurses all sub-folders so you don't need to write code to recurse it again. I used Get-PWFoldersImmediateChildren to get around the issues with really big folder/document structures.

A few notes:

• If you save your script file in the UTF-8 with BOM (the with BOM is important) encoding you can embed unicode characters directly in your script without needing to use things like [char]0x00E4
  • In Notepad you can do this by doing a Save As... then changing the encoding next to the Save button
  • In Notepad++ this is in the Encoding menu at the top
  • Not sure where it is in the IDE
• If you want the columns to always be in order, make the attributeMap [Ordered] as well, otherwise when you add a key it may change the order of the keys when you do $attributeMap.keys.
• I changed the results to an ArrayList instead of a GenericList so that when new elements are added it doesn't create a whole new array, add the new entry, then delete the old.
• ($type,$attName) = $k.split(':',2) is a cute way to split a string into 2 parts and assign each part to a different variable. I split on the colon
• I added sorting by the folder paths, not sure that slows it down much or not. Probably depends on the number of folders & files you have

This solution still requires a huge amount of memory to hold all the documents in a single array. If this is a problem you could leverage the fact that Export-Excel appends to the end of an existing xlsx by changing the loop:

# Loop through each folder
ForEach ($pwf in $pwfList) {
	# reset results array for each folder
	$results = [System.Collections.ArrayList]@()

	Write-Host "Processing folder $(Join-Path $masterFolderPath $pwf.Name). " -NoNewLine
	# Get all documents under the sub-folder (including documents in the sub-folders under current folder)
	$pwdList = Get-PWDocumentsBySearch -FolderID $pwf.ProjectID -Slow | Sort FullPath
	Write-Host "Found $($pwdList.count) documents"
	ForEach ($pwd in $pwdList) {
		Write-Host "Processing file $($pwd.FullPath)"
		$row = [ordered]@{}                                 # maintain key order
		ForEach ($k in $attributeMap.keys) {
			$colName = $attributeMap.$k
			($type,$attName) = $k.split(':',2)
			if ($type -eq 'PROP') {
				$row.$colName = $pwd.$attName
			} elseif ($type -eq 'CA') {
				$row.$colName = $pwd.CustomAttributes[$attName]
			} else {
				Write-Warning "Improper key in attribute map: $($k)"
				continue
			}
		}
		[void]$results.Add([PSCustomObject]$row)
	}
	# Append results for current folder to end of workbook
	$results | Export-Excel -Path '.\output.xlsx' -WorkSheetName 'Documents' -AutoSize -FreezeTopRow
	# clear out array from memory
	$results = $null
}

Oh yeah, I assumed your $sw variable was a stopwatch. I prefer to do timings by just saving get-date at start and subtracting at the end. No real reason, i just find it easier to setup.

if this still has too many folders to process at once you could go another level down with Get-PWFoldersImmediateChildren:

$pwfList = Get-PWFoldersImmediateChildren -FolderPath $masterPath
ForEach ($pwf in $pwfList) {
    $pwfSubList = Get-PWFoldersImmediateChildren -FolderID $pwf.ProjectID
    ForEach ($subPwf in $pwfSubList) {
        $pwdList = Get-PWDocumentsBySearch -FolderID $subPwf.ProjectID
    }
}