Import Overwrite Options Issue

I am working with a template provided by another person. I create a new project using that template. All looks empty and pretty standard. I do an import of my source data using the "Named Fields" option,  and get the following log with the indicated Warnings

95 Records imported.

PROJECT: 1 records OK
POINT: 1 records OK
LAB SPECIMEN: 10 records OK
ATTERBERG: 4 records OK
SIEVE: 3 records OK
WC DENSITY: 10 records OK
ATTB READINGS: 12 records OK
SV READINGS: 54 records OK

Warnings:

The following tables allow duplicate keys: ATTERBERG,SIEVE,WC DENSITY.
It is not possible to update specific records in these tables.
gINT has imported data only to those data sets that were initially empty.
If you wish to replace existing data use the 'Overwrite Data Sets' option for these tables.
Existing datasets will also be overwritten if the 'Overwrite Records' option is chosen for the entire import.


The following tables have hidden counters: ATTB READINGS.
It is not possible to update specific records in these tables.
gINT has imported data only to those data sets that were initially empty.
If you wish to replace existing data use the 'Overwrite Data Sets' option for these tables.
Existing datasets will also be overwritten if the 'Overwrite Records' option is chosen for the entire import.

I look over the data and it is there and visually looks fine.

Now, at times  I get into a situation where I need to re-import the source because some values have changed (eg. change in a tare weight)

I do the import and now get the same message, BUT as noted in the error message " ...... gINT has imported data only to those data sets that were initially empty." The bad data has not been replaced by the good data in my revised source file.

I have run into this before on other tempates, and I have used the Overwrite Data Sets option as indicated in the Warning Message. When I do this, all is cleared out and the entire source is brought into the target. I do NOT get the "The following tables have hidden counters: " message either.

On this particular project, however, created with the template I mentioned above, if do the import, using the Overwrite Option of "Data Sets, I get a slew of these messages. 

WC DENSITY Table:
The changes you requested to the table were not successful because they would create duplicate values in the index, primary key, or relationship. Change the data in the field or fields that contain duplicate data, remove the index, or redefine the index to permit duplicate entries and try again 

I do some more testing, and find that even if I delete all the boreholes, and have what looks like a clean blank file, and do an Overwrite Option of Data Sets again, I get the same slew of messages. It is like it is NOT clearing everything out.

Again, with previous templates using the Overwrite Data Sets, things "look like" it has cleaned out all the data, and prevented that "The following tables have hidden counters: ATTB READINGS. message on a newly created project, and also prevented either message if used on an existing project that had data in it already. 

I have see an earlier response here that seems to be in the same ball field and may be spot on, but my ignorance is keeping me from discerning the "fix". What I do discern form the post is that all the "Data" is NOT cleared out, with an Overwrite Data Sets option, and that although the user interface is blank, there is still some hidden structural placeholders attached to the file (Counters/keysets, etc??)

https://communities.bentley.com/products/geotechnical1/f/geotechnical-forum/71792/import-unc-data-over-writing-previously-imported-data/189945#189945

What I Want:

a) to understand better what is going on.

b) to get things set up so that if I revise the underlying data in the source, I can import it using the Named Fields option, and it will actually bring in the Named field source data, even into the non-empty fields, thus updating their data.

Ideas??



2022_11_10 EDIT/SUPPLEMENT


I did a database compare. Could the "GintVersion" difference as shown below in the "Database Property Differences:"  have something to do with it??

DB #1 is the DB that seems to give the problem with the slew of messages on the imports

DB #2 is the DB that does not throw the slew of messages on the imports

Property[GintVersion]
1: [4.15.002]
2: [8.30.004]

Parents
  • Don't know if I know the answer but I will add some comments/sugestions that might help. This is my understanding and I would encourage others to correct/ add to my discussion

    For your question a.  what is happening is that the tables in question are set to allow duplicate record keys. For example, if the keyset for a table is PoitID, depth you are allowed to enter more than one record with the same pointID and depth. Suppose your table for atterburg had the following fields

    PointID, depth, LL, PL

    if you had 2 results at the same depth, you could enter

    B-1, 10.3, 32, 23

    B-1, 10.3, 67, 42

    This would normally not be permitted because you have 2 records with the same key but checking the allow duplicate keys option in the data structure causes gINT/Access to add a hidden index to keep the duplicate records distinct.  The problem comes when you try to reimport revised data. In this case gINT doesn't know which record to assign the changed record to and refuses to do it unless you overwrite the entire dataset. this essentially deletes the existing records and imports the correct records.  it also deletes parent records and any other data you may have added to the data set in gINT as well so if you have made changes to those in gint they will be lost when you reimport.

    For your question b. in my limited testing i was able to import records with duplicate keys into tables that allowed duplicate keys and it comes in fine with both records intact.  I do get the warning message.  If I try to reimport revised data it refuses to import the revised data unless I use the overwrite data sets option as described above. this wipes out any other changes I may have made in gINT as expected. not sure why you are not able to do the same. could be a version issue as you noted so i would try opening the data template (.gdt) in your current gint and resaving it. then create a new project using the resaved data template and see if it functions as expected.  Ultimate solution is to either not allow duplicate keys or to not maintain 2 completely different data sets while allowing changes to be made to either one. sort of like having 2 copies of the same report in 2 word documents and allowing 2 people to work on their copy independently. When it comes time to assemble the final report you have a hot mess on your hands as the same section may have been modified in 2 different ways. It requires that you carefully track what changes you have made in each document, when those changes were made and only merge carefully selected parts of the data 

  • I have done some more digging, and it seems that some of the relations in the offending template are “off”.

    Specifically these table relations:

    Table Lab Specimen and Table Atterberg
    Table Lab Specimen and Table Sieve
    Table Lab Specimen and Table Wc Density

    By off, for example, take the Table Lab Specimen Table Atterberg relation. (Image attached) In the offending template, the relationship-line termination symbol in the MSAccess display of Table Relations, shows a one-to-many relation, whereas on clicking the relationship line, and then right clicking to Edit Relationship, the Edit Relationships dialog says one-to-one.


    I took a look at the “gint std us lab.gdt”, and the relations symbols there match the Edit Relations Dialog. The MS Access Table Relations graphic says one-to-one on the Lab Specimen to Atterberg relation, and the Edit Relations dialog also says one-to-one. (See attached screen grabs.)

    Imports go fine against projects created from the “gint std us lab.gdt”, and also go fine against projects created from other templates from other companies, but not this one client, and I suspect that somehow their template got mis-configured, and that they do not do any imports that fire the issue.

    I note that on projects made off the offending template, not only does the “Data Sets” overwrite option make things go crazy, but that on doing an import, and then editing (for example, editing a Wc Wet Weight), and then doing a re-import, using the Named Fields option, the changed data does not get replaced by the source data, but stays the same.

    If one does the same import against a “gint std us lab.gdt” project, all works as expected, and the source data overwrites that changed Wc Wet Weight.

    Go figure

    In summary it looks to me like somehow their data structures got laid up incorrectly, or that somehow the template got corrupted. BTW- a DB repair compact does not resolve the issue.

    My Solution: I deleted the existing relation between:

    Table Lab Specimen and Table Atterberg
    Table Lab Specimen and Table Sieve
    Table Lab Specimen and Table Wc Density

    and then recreated them. On the creation they get laid in properly as one-to-one, both as to the line termination icons and in the Edit Relations dialog. After this the import works as expected.

    Go Figure!

Reply
  • I have done some more digging, and it seems that some of the relations in the offending template are “off”.

    Specifically these table relations:

    Table Lab Specimen and Table Atterberg
    Table Lab Specimen and Table Sieve
    Table Lab Specimen and Table Wc Density

    By off, for example, take the Table Lab Specimen Table Atterberg relation. (Image attached) In the offending template, the relationship-line termination symbol in the MSAccess display of Table Relations, shows a one-to-many relation, whereas on clicking the relationship line, and then right clicking to Edit Relationship, the Edit Relationships dialog says one-to-one.


    I took a look at the “gint std us lab.gdt”, and the relations symbols there match the Edit Relations Dialog. The MS Access Table Relations graphic says one-to-one on the Lab Specimen to Atterberg relation, and the Edit Relations dialog also says one-to-one. (See attached screen grabs.)

    Imports go fine against projects created from the “gint std us lab.gdt”, and also go fine against projects created from other templates from other companies, but not this one client, and I suspect that somehow their template got mis-configured, and that they do not do any imports that fire the issue.

    I note that on projects made off the offending template, not only does the “Data Sets” overwrite option make things go crazy, but that on doing an import, and then editing (for example, editing a Wc Wet Weight), and then doing a re-import, using the Named Fields option, the changed data does not get replaced by the source data, but stays the same.

    If one does the same import against a “gint std us lab.gdt” project, all works as expected, and the source data overwrites that changed Wc Wet Weight.

    Go figure

    In summary it looks to me like somehow their data structures got laid up incorrectly, or that somehow the template got corrupted. BTW- a DB repair compact does not resolve the issue.

    My Solution: I deleted the existing relation between:

    Table Lab Specimen and Table Atterberg
    Table Lab Specimen and Table Sieve
    Table Lab Specimen and Table Wc Density

    and then recreated them. On the creation they get laid in properly as one-to-one, both as to the line termination icons and in the Edit Relations dialog. After this the import works as expected.

    Go Figure!

Children
No Data