Why VBA XML file parsing and element creation is slower in OCMCE than BMv8i?

Using Bentley Map v8i SS4 we had to move our XML file parsing and element creation code to VBA from C# due to very slow execution. Now we are migrating to OpenCities Map Connect Edition, and the same VBA code performance is about 1.7x slower for the very same input, configuration and machine. Why? Can we expect any performance tuning soon or later? Thanks!

DG

Parents
  • Hi,

    it seems to me there are two questions in your post:

    • XML file parsing: Can you provide more information what do you want to achieve?
      If it's about external XML file, I see no reason to use anything from MicroStation or BM/OCM API, because there have been always both in NET 3.5 and now in 4.6.2 better tools available.
    • VBA performance drop: VBA itself is a synonymum for slowness, but I guess only somebody from Bentley can provide better insight why VBA is now slower than it was. But you should provide more information, at least some code snippet, because now it's not clear what class / method you use under what conditions. My experience (from MStn CE, not OCM CE) is that VBA seems to be approximately the same as it was in V8i.

    With regards,

      Jan

  • Hi,

    I will separate xml and element creation benchmarking soon to determine if it is an overall VBA perfomance problem under OCMCE or only a Bentley API issue. I cannot provide the whole huge code, but here is some key fragments:

    ...
    Set xmldoc = CreateObject("Microsoft.XMLDOM")
    XMLFileName = strArgs(0)
    xmldoc.Load (XMLFileName)
    ...
    Set subnodes = TitleNodes(i).SelectNodes("line_string")
    ...
    Set detnodes = subnodes(0).SelectNodes("point")
    ...
    ReDim linePoints(detnodes.Length - 1)
    ...
    linePoints(k).X = CDbl(SetCommaStr(detnodes(k).getAttribute("x")))
    linePoints(k).Y = CDbl(SetCommaStr(detnodes(k).getAttribute("y")))
    ...
    Set newLine = CreateLineElement1(Nothing, linePoints)
    ...
    Set newXData = newLine.GetXData1("xxx")
    newXData.AppendXDatum msdXDatumTypeInt16, a
    newXData.AppendXDatum msdXDatumTypeInt32, b
    newLine.SetXData1 "xxx", newXData
    ...
    Set newLine.level = GetItemlevel(newlevel)
    Set newLine.LineStyle = GetItemlinestyle(newlinestyle)
    newLine.Color = newcolor
    newLine.LineWeight = newwidth
    newLine.GraphicGroup = groupid
    ActiveModelReference.AddElement newLine
    newLine.Redraw
    ...

    The code creates about 50000 elements (lines, shapes, ellipses, shared cells, texts) while parsing the XML file. For several items the performance drop is inconspicuous.

    Could you tell me who is the VBA specialist at Bentley? Thanks!

    DG

  • Hi,

    I will separate xml and element creation benchmarking soon

    It would be great, because I see (at least) three different sets of operations and it's not clear whether (A) everything is slower or (B) some operation causes e.g. 90% of all slowness. In fact, from this reason (performance benchamrks and available analysis tools) VBA is very bad tool to implement anything bigger, especially when performance is priority.

    • XML file parsing: It does not depend on Bentley API, maybe Microsoft XML code is slow in 64bit version?
    • Create element and add it to active model: I do not assume it slow, but it's worth to check. To add 50 000 elements one by one is probably slow anyway, maybe to add more elements at once (which other APIs allow) can be faster? But it depends how you want to control MicroStation transaction manager.
    • Working with XData: Oops, why XData? It's obsolete data structure, implemented only because of AutoCAD compatibility. I have no idea how fast it is, because it's quite forgotten not very often used part of MicroStation (and its API).

    Also the code benchmarking should be done using OCM and also using plain MicroStation, because in your code nothing is specifically OCM API, but OCM can be slower for the same functions because of more functionality at background.

    The code creates about 50000 elements (lines, shapes, ellipses, shared cells, texts) while parsing the XML file.

    It seems like task that never should be done using VBA.

    I do not know context, but my feeling is that to rewrite it to new NET API (I think some support for XDada exists in MStnPlatformNET) could be faster than to analyze the issue and to find solution / workaround.

    Could you tell me who is the VBA specialist at Bentley?

    I have no idea, but if this issue is so important for you, use standard way and create Service Ticket. You can share the ticket number here to link official (service ticket) and community (this forum) worlds. Usually answers VBA issues related MicroStation VBA. Because your code seems to use MicroStation API and not OCM API, maybe he will be able to help you.

    With regards,

      Jan

Reply
  • Hi,

    I will separate xml and element creation benchmarking soon

    It would be great, because I see (at least) three different sets of operations and it's not clear whether (A) everything is slower or (B) some operation causes e.g. 90% of all slowness. In fact, from this reason (performance benchamrks and available analysis tools) VBA is very bad tool to implement anything bigger, especially when performance is priority.

    • XML file parsing: It does not depend on Bentley API, maybe Microsoft XML code is slow in 64bit version?
    • Create element and add it to active model: I do not assume it slow, but it's worth to check. To add 50 000 elements one by one is probably slow anyway, maybe to add more elements at once (which other APIs allow) can be faster? But it depends how you want to control MicroStation transaction manager.
    • Working with XData: Oops, why XData? It's obsolete data structure, implemented only because of AutoCAD compatibility. I have no idea how fast it is, because it's quite forgotten not very often used part of MicroStation (and its API).

    Also the code benchmarking should be done using OCM and also using plain MicroStation, because in your code nothing is specifically OCM API, but OCM can be slower for the same functions because of more functionality at background.

    The code creates about 50000 elements (lines, shapes, ellipses, shared cells, texts) while parsing the XML file.

    It seems like task that never should be done using VBA.

    I do not know context, but my feeling is that to rewrite it to new NET API (I think some support for XDada exists in MStnPlatformNET) could be faster than to analyze the issue and to find solution / workaround.

    Could you tell me who is the VBA specialist at Bentley?

    I have no idea, but if this issue is so important for you, use standard way and create Service Ticket. You can share the ticket number here to link official (service ticket) and community (this forum) worlds. Usually answers VBA issues related MicroStation VBA. Because your code seems to use MicroStation API and not OCM API, maybe he will be able to help you.

    With regards,

      Jan

Children
  • Hi,

    Thank you for the fast reply and valuable comments. In the meantime I measured XML parsing and processing without any Bentley API calls, and even this is significantly (1.3x) slower in OCMCE than in BMv8i. So it seems to be a general OCMCE VBA (or pure VBA) problem. In BMv8i C# we added all elements at once to the model, of course, because ii is much more faster. However, when I checked OCMCE C# performance I got exception for that method... Anyway, it is not so important for a while because the slower OCMCE VBA is even faster than any C# in this case. XData: as far as I remember in C# it was the fastest way to add custom data. We still use C# as main code, just some parts were reimplemented in VBA due to slow performance compared to the old mjava implementation.

    I know very well that the real solution for performance problems is native code C++ or new .NET API, but we need more time to migrate.

    Thank you again!

    DG

  • Hi,

    So it seems to be a general OCMCE VBA (or pure VBA) problem.

    The speed is always an issue in VBA because of COM overhead. How big it is depends on many things, but I would expect there will be no substantial difference between 32bit and 64bit COM or VBA. But I am not expert in this area and fortunately I have been able to stay away from these technologies in the most of my projects even in V8 era (and to use native C/C++, C# for managed and C++/CLI to join these two worlds).

    In BMv8i C# we added all elements at once to the model, of course, because ii is much more faster.

    I have not done any benchamrking, but it makes sense, because of to add more elements at once is still one transaction. It would be interesting to also measure difference between plain MicroStation and Bentley Map, because I assume BM B8 / OCM CE adds extra overhead.

    However, when I checked OCMCE C# performance I got exception for that method

    For what method? When you receive an exception for any method and it worked in V8i, you should report it as bug plus optionally to discuss it in this forum (or in MicroStation Programming when the problem exists in plain MicroStation also).

    It's not clear from you description how much from OCM API do you use, or you just use OCM but use standard MicroStation API without OCM features, so it's hard to make any recommendation, but you should be aware that anything like "V8 C#", which is based on Interop, can be still used and is fully supported, but there is much better, faster and richer NET API available.

    Anyway, it is not so important for a while because the slower OCMCE VBA is even faster than any C# in this case.

    I do not believe it, but it's not clear what parts of APIs and what code do you use for such evalation.

    There are a huge amount of XML parses available for NET, providing different set of features. Even the standard ones, delivered with NET Framework API (e.g. XmlReader) are not bad.

    Also, I do not believer that new NET API, which is typically implemented as "fast wrapper to native API" can be slower than VBA.

    XData: as far as I remember in C# it was the fastest way to add custom data.

    I do not know your application context and its history, but XData were originally designed to be used solely for AutoCAD compatibility. I found Don H. Fu this note MicroStation users may only need it for DWG round trip purpose.  In rare occasions if someone ever needs it, he/she presumably knows XDATA perhaps because he/she has used it in his/her apps on AutoCAD, therefore he/she does not need much assistance for that to work on MicroStation.

    Bentley published XData API, so technically it's fine to use it even in DGN, but strategically it's in my opinion quite bad idea to use it anywhere else than for AutoCAD compatibility. XData are special animals, maintained because it's required, but it does not follow Bentley concepts and priorities (XAttributes and EC data in MStn or XFM in BM).

    To use something because "it's fastest way" can easily become nightmare pain in future.

    MicroStation has offered a wide selection of how to add custom data to elements, where XAttributes are probably even older than XData, whereas EC Framework API is available from I guess V8i SS3 (so still many years).

    When C# is not enough (and it was the reality in V8, but not anymore in CE), standard P/Invoke can be used to use native API (both Microsoft and Bentley recommended way how to solve situation when anything is missing in managed API). Or C++/CLI can be used to implement own wrapper (more complex, but very efficient).

    I know very well that the real solution for performance problems is native code C++ or new .NET API, but we need more time to migrate.

    I both agree and disagree ;-)

    To learn new API requires a lot of time, but after 3 years of CE existence there is available good knowledge base (in a form of SDK examples, not too many blogs and plenty of discussions) focusing typical operations and the most common issues, so the learning complexity has been decreased a bit.

    Because new (NET) and old (Interop) APIs can coexist in parallel in the same code (despite of direct sharing data is not usually possible), I know about companies who replace any code that does not work (often because of a bug in product or API) by newly written code. The results are not nice, it can be more demanding to maintain them until everything old will be removed, but often they found out it was faster / cheaper than to try to find workarounds and to analyze / report / wait for fix in old APIs.

    But every case and every migration is different with own priorities and issues.

    With regards,

      Jan

  • Hi,

    Regarding the following subjects, I did not express myself well:

    - XData: I mean it was the way providing the best performance in (old) C#. After switching to new API we will return to a standard custom data handling, of course (assuming no performance drop).

    - We need more time to migrate: Considering requirements and resources, first we need a working solution ASAP to prove OCMCE can provide at least the same functionality and speed as BMv8i using with our system. Then we will have more time to rewrite our C# + VBA + MDL code using only one new API having the best performance.

    Fortunately, making some minimal changes, formal updates and rebuilds, now we have a working solution, only this unexpected VBA slowness issue remained (as far as the code concerned).

    By the way, this COM call causes exception (in OCMCE only):

    Bentley.MstnPlatformNET.InteropServices.Utilities.ComApp.ActiveModelReference.AddElements(...);

    I did not reported yet, because I had no time to investigate whether it is a previously tolerated misuse or really a bug in OCMCE COM API.

    I will try your idea and check if I can replace only element creation and adding parts with new C# API to gain suitable performance without rewriting everything.

    Thanks,

    DG

  • Hi,

    thanks for further details.

    XData: I mean it was the way providing the best performance in (old) C#.

    I am not sure and I have not done any scuh testing. Fortunately I read explanation "XData are for DWG compatibility, don't use it for anything else" earlier than I had to work with custom data in C#. It's true that manu custom data options from native API were not available in C#, but when performance is a priority, it's alwyas wise to use C/C++ to implement critical functioanlity and use C++/CLI for wrapper to publish it to NET.

    We need more time to migrate

    It's a general experience of a majority of people working on code migration from V8 to CE.

    first we need a working solution ASAP to prove OCMCE can provide at least the same functionality and speed as BMv8i using with our system

    Despite of I agree completely with such aim, a question is whether it can be achieved. I do not test OCMCE APIs yet (because more requirements are for plain MicroStation now), but performance is still issue. And my personal feeling is that new APIs (especially native, but also managed) are fixed and optimized fater than Interop.

    Fortunately, making some minimal changes, formal updates and rebuilds, now we have a working solution

    Congratulations! :-)

    I did not reported yet, because I had no time to investigate whether it is a previously tolerated misuse

    I think it has not been missuse. On the other hand BM / OCMCE has always tried to do things internally differently than MicroStation does, so even simple element adding represent dynamic conversion to feature, which requires extra code, so I guess it's bug in implementation.

    I will try your idea and check if I can replace only element creation and adding parts with new C# API to gain suitable performance without rewriting everything.

    New API, when you understand its structure and philosophy (which is based on native API fortunately, not own one as was VBA / COM API), it's much better and following OOP principles. But be aware you cannot share new and Interop APIs, so Element is not Element and a functionality has to be implemented completely in one or in second API (of course runtime changes like change color or attach cell library can be done in both).

    Regards,

      Jan

  • Hi,

    Issue has been filed as Defect 1069771 recently according my service request and test case. Thanks for your help!

    With regards,

    DG