[CONNECT C#] Faster method to extract all element Item Type Property values?

Hi Mark,

I do not know a structure of your files, but code similar to this one should work. I had no time to test it in detail, but it lists "something" when used with simple test DGN.

DgnModel dgnModel = Session.Instance.GetActiveDgnModel();
ModelElementsCollection elements = dgnModel.GetGraphicElements();

foreach (Element element in elements)
{
    CustomItemHost host = new CustomItemHost(element, false);

    IList<IDgnECInstance> items = host.CustomItems;

    foreach (IDgnECInstance item in items)
    {
        IEnumerable<IECProperty> properties = item.ClassDefinition.Properties(false);

        StringBuilder sb = new StringBuilder();

        foreach (IECProperty property in properties)
        {
            IECPropertyValue value = item.GetPropertyValue(property.Name);

            if (value is null) { continue; }

            sb.AppendLine($"Property '{property.DisplayLabel}' = {value.StringValue} ");
        }

        string msg = $"Element id {element.ElementId} / Item {item.ClassDefinition.DisplayLabel}";
        string details = sb.ToString();
        NotifyMessageDetails message = new NotifyMessageDetails(OutputMessagePriority.Information, msg, details, NotifyTextAttributes.None, OutputMessageAlert.None);
        NotificationManager.OutputMessage(message);
    }
}

There is a space for improvements, like to use value formatters, plus this snippet does not support arrays.

With regards,

  Jan

Thanks Jan, that looks to be much more cleaner than what i had and works a bit better.

it is a bit faster, but still much more excessive than expected - it seems like the Item Type expressions are being recalculated as the item types are being read into my code. The ones which have if statements involving numerical calcs run slower than the rest and some based on Expressions with a State value of "NotYetCalculated" take even longer.

I am only really interested in extracting the Item Type text and numerical values per element for an export addin i am putting together. I'll be trying Manesh's suggestion to see if it helps. There looks to be something strange happening with these Expressions...

Jan Šlegr said:

Maybe to ensure Expression are written in efficient way is bettersolution. Although I alwyas tell and write that Expressions represents overhed and it is not "an aid for everything and anything", I do not recall realy issue with Expressions. And what I remember from a presentation about Item Types new features in ORD 10.11, it was mentioned that even hundreds thousands of attache items, with expressions defined, does not cause any substatnial slow down.

Or maybe it is the issue of 10.09 / 10.10? Did you try it with 10.11?

Funny you say that, the aim of this code is to resolve an issue with IFC export of ORD 10.11. In both 10.10 and 10.11 it takes us hours (3-4) to write an IFC that includes Item Type data, and when no Item Type data is included it takes 5-10 minutes. It is looking like this issue is the culprit. Unfortunately we can't simplify our Item Types any more to get the automation we are looking for, but need to find a faster way to extract data out of our models. Even exporting Item Type data using the Excel Export Instances tool takes a very long time - something our users have started to do to reimport "static" data just for export.

There have been performance gains in ORD 10.11 in this regard, but nothing substantial enough to save hours in this regard unfortunately.

Mark Shamoun said:

In both 10.10 and 10.11 it takes us hours (3-4) to write an IFC that includes Item Type data, and when no Item Type data is included it takes 5-10 minutes.

It tells more than IFC export is implemented in a wrong way than Item Types are slow.

Mark Shamoun said:

Unfortunately we can't simplify our Item Types any more to get the automation we are looking for

I think you moved the discussion focus: Now it seems that it's not about "fast Item Type exctaction", because in general, there is no slow done and is really fast, but about to analyze why the evaluation of EC Expressions, that you use, seems to slow down the whole process.

But, because nobody saw the expressions, it's hard to even guess... And I would assume such analysis requires a knowledge of ORD internals.

With regards,

  Jan

True, but the speed in extracting Item Type values i am finding here in isolation correlates to the speed it is taking to export to IFC, eg. 1350 elements taking 7 seconds to read Item Types per element = 2.6 hours. Noting that the majority of values are Excel lookup string values, but there is one creative GetRelatedInstance expression that may be exacerbating the whole thing.

I'll keep experimenting and see if i find something. Thanks for all the help, it is very good information.

Mark Shamoun said:

but there is one creative GetRelatedInstance expression that may be exacerbating the whole thing.

Personally I do not think the usage of related instance can cause any substantial slow down: the relation between EC classes is a part of native EC data, so to read the relation definition to obtain access to target instance should be equally fast as to read normal EC data. Of course it depends how GetRelatedInstance method is implemented, but at least, it is about access inside DGN.

Mark Shamoun said:

Noting that the majority of values are Excel lookup string values

I have no experience with lookup tables in Excel, when used from Item Types, but I can imagine it can be slow, because external file must be opened and read.

It would be great if somebody from Bentley is able to share best practices and what to check, when external tables are used so masivelly.

With regards,

  Jan

Interestingly, using Interop Property Handler to read the Property values is exponentially faster. I'm thinking this may not be re-evauating the same way as EC direct methods. A little more messy than i was after, but SEEMS to work well...

 List<PropertySetIfc> properties = new List<PropertySetIfc>();
                            CustomItemHost host = new CustomItemHost(meshHeader, false);
                            IList<IDgnECInstance> items = host.CustomItems;

                            BIM.Element eleInter = ElementOperation.ConvertToInteropEle(meshHeader);
                            BIM.PropertyHandler oProps = BMI.Utilities.ComApp.CreatePropertyHandler(eleInter);

                            foreach (IDgnECInstance item in items)
                            {
                                PropertySetIfc ps = new PropertySetIfc(item.ClassDefinition.Name, "");
                                IEnumerable<IECProperty> itemProperties = item.ClassDefinition.Properties(false);

                                foreach (IECProperty itemProperty in itemProperties)
                                {
                                    string itemPropName = itemProperty.Name;
                                    MessageCenter.Instance.ShowMessage(MessageType.TempRight, "Scanning active model...Scanning Mesh " + (csCount + 1).ToString() + " of " + meshCount.ToString() + " > reading Item Type: " + itemPropName, "", MessageAlert.None);
                                    string unitName = "";
                                    object propValOut = null;

                                    try
                                    {

                                        oProps.GetAccessStrings();
                                        if (oProps.SelectByAccessString(itemPropName))
                                        {
                                            object propVal = oProps.GetDisplayString();
                                            propValOut = propVal;

Mark Shamoun said:

Interestingly, using Interop Property Handler to read the Property values is exponentially faster.

Wow, it is something I did not expect

Mangesh.Shelar Is there any reason for the behavior Mark described?

Regards,

  Jan

CpropertyHandler is a COM wrapper over native ECInstance. When you call GetDisplayString or GetValue, we are directly returning a native value. It doesn't involve managed-to-native marshalling and no intermediate callbacks. Moreover, when you try Civil elements at managed layer, I am doubting that they may have many other handlers or callbacks which might make things slower. 

Mark, please raise this to Civil.

Thanks guys, will do

Hi all,

We think we are closer to tracking down the issue. It seems that calculated values using IIF statements is slowing down the extraction of these properties substantially. Why would this be the case?

Attached are the expressions we are using. Note that this was made for Civil GetRelatedInstance expressions, hence all references to corridors, etc. but we have confirmed that the GetRelatedInstance expression has minimal effects on performance.

TfNSW_DE_Data_Arcadis.xlsx

Hi Mark,

Mark Shamoun said:

Why would this be the case?

I think it's hard to guess without knowing how individual functions are implemented.

Mark Shamoun said:

but we have confirmed that the GetRelatedInstance expression has minimal effects on performance.

To reference related instance is efficient, because it is already existing relationship, so (I guess) the only difference is that one more extra EC instance must be read.

Mark Shamoun said:

It seems that calculated values using IIF statements is slowing down the extraction of these properties substantially.

I do not think IIF itself is slow (although as I wrote, nobody know how exactly it is implemented ;-). But the used conditions, based on string operations, can be expensive (generally, in software development, string operations are one from the most expensive, because when not used carefully, require plenty of memory allocations and data iteration and comparing).

I think, when complete data is available, it would be possible (at least for NET code) to do some benchmarks, to see what code is (A) called the most often and (B) what method in code consumes the most time.

With regards,

  Jan

Hi Mark,

Mark Shamoun said:

Why would this be the case?

I think it's hard to guess without knowing how individual functions are implemented.

Mark Shamoun said:

but we have confirmed that the GetRelatedInstance expression has minimal effects on performance.

To reference related instance is efficient, because it is already existing relationship, so (I guess) the only difference is that one more extra EC instance must be read.

Mark Shamoun said:

It seems that calculated values using IIF statements is slowing down the extraction of these properties substantially.

I do not think IIF itself is slow (although as I wrote, nobody know how exactly it is implemented ;-). But the used conditions, based on string operations, can be expensive (generally, in software development, string operations are one from the most expensive, because when not used carefully, require plenty of memory allocations and data iteration and comparing).

I think, when complete data is available, it would be possible (at least for NET code) to do some benchmarks, to see what code is (A) called the most often and (B) what method in code consumes the most time.

With regards,

  Jan

Hi Jan,

Looking at this further, it looks like every time any IIF statement is trying to access an Item Type that contains an expression, that expression is reevaluated. Based on the nature of EC expressions that behaviour makes sense, however is extremely expensive performance wise for any practical use of a value derived from an Expression elsewhere.

Having the ability to access these values "read-only" surely would aid in general performance increases for operations on Expressions globally surely? I'm seeing that reevaluation looks to be also happening with data from read-only sources like external references.