Bentley Communities
Bentley Communities
  • Site
  • User
  • Site
  • Search
  • User
ProjectWise
  • Product Communities
ProjectWise
ProjectWise Design Integration Forum Bulk file uploading - best practice suggestions
    • Sign In

    • State Not Answered
    • Replies 3 replies
    • Subscribers 59 subscribers
    • Views 297 views
    • Users 0 members are here

    Bulk file uploading - best practice suggestions

    David Nicholson
    Offline David Nicholson 5 months ago

    I would greatly appreciate the collective wisdom and experiences of the Community on this topic - "How does one upload large volumes of windows file-based data into ProjectWise?" 

    The requirement is to upload large folder structures from a datacentre-based file server to a Bentley-hosted datasource.  The source structure consists of thousands of folders\subfolders containing hundreds of thousands of MS Office-based files, with the vast majority in the 1Mb-50Mb range.

    • In a PWE environment using "Drag & Drop", the process starts well but typically fails with a collection of errors at about the 3,000 file mark.  
    • In a PWE environment, using Excel import, uploads appear to stop after several thousand entries.  Log files reveal few clues.
    • In PW Web, the 365 project seems to time out before the upload process starts.  Less than 1000 files appear to be the limit.
    • I'm yet to try PW Drive; however Im not hopeful.

    We are increasingly told that data is "dark" and of low value unless we put it into a system that enables the information to be indexed, searched and potentially attributed. 
    How can we do this efficiently when the seemingly simple process of uploading swathes of dumb data proves so frustrating to do efficiently?

    • Sign in to reply
    • Cancel

    Top Replies

    • Robert McMillan
      Offline Robert McMillan Wed, Apr 5 2023 12:05 AM +2
      I'd use the PowerShell Import-PWDocuments cmdlet with the -MultiThreaded switch. You can also add switches to filter the data for example exclude file types like bak or tmp etc.
    • Robert McMillan
      0 Offline Robert McMillan Wed, Apr 5 2023 12:05 AM

      I'd use the PowerShell Import-PWDocuments cmdlet with the -MultiThreaded switch. You can also add switches to filter the data for example exclude file types like bak or tmp etc.

      • Cancel
      • Vote Up +2 Vote Down
      • Sign in to reply
      • Verify Answer
      • Cancel
    • David Nicholson
      0 Offline David Nicholson Wed, Apr 5 2023 12:08 AM in reply to Robert McMillan

      Thanks Robert,  Great suggestion.  Nevertheless, why should we resort to a programming solution to efficiently address a simple functionality requirement that should be part of the commercial offering?  After all, not everyone is skilled in Powershell.

      Regards,  David

      • Cancel
      • Vote Up 0 Vote Down
      • Sign in to reply
      • Verify Answer
      • Cancel
    • Kevin van Haaren
      0 Offline Kevin van Haaren Wed, Apr 5 2023 9:31 AM

      Have you talked to your Bentley hosting support people? When we migrate an existing datasource into PW we use GoodSync to upload the storage area to Azure and then they put it in the appropriate place.

      This is a bit different since it isn't a storage area and has to be imported into PW after uploading, but it should be faster at that point.

      Projectwise has a timeout on long file transfers (even multi-file transfers), this is adjustable in the server Config, we set ours to 30 minutes, Bentley can adjust this longer on hosting for you. Uploading it all in a single go probably isn't feasible but you should be able to make the blocks larger.

       

      • Cancel
      • Vote Up 0 Vote Down
      • Sign in to reply
      • Verify Answer
      • Cancel

    Communities
    • Home
    • Getting Started
    • Community Central
    • Products
    • Support
    • Secure File Upload
    • Feedback
    Support and Services
    • Home
    • Product Support
    • Downloads
    • Subscription Services Portal
    Training and Learning
    • Home
    • About Bentley Institute
    • My Learning History
    • Reference Books
    Social Media
    •    LinkedIn
    •    Facebook
    •    Twitter
    •    YouTube
    •    RSS Feed
    •    Email

    © 2023 Bentley Systems, Incorporated  |  Contact Us  |  Privacy |  Terms of Use  |  Cookies