Import Overwrite Options Issue

I am working with a template provided by another person. I create a new project using that template. All looks empty and pretty standard. I do an import of my source data using the "Named Fields" option,  and get the following log with the indicated Warnings

95 Records imported.

PROJECT: 1 records OK
POINT: 1 records OK
LAB SPECIMEN: 10 records OK
ATTERBERG: 4 records OK
SIEVE: 3 records OK
WC DENSITY: 10 records OK
ATTB READINGS: 12 records OK
SV READINGS: 54 records OK

Warnings:

The following tables allow duplicate keys: ATTERBERG,SIEVE,WC DENSITY.
It is not possible to update specific records in these tables.
gINT has imported data only to those data sets that were initially empty.
If you wish to replace existing data use the 'Overwrite Data Sets' option for these tables.
Existing datasets will also be overwritten if the 'Overwrite Records' option is chosen for the entire import.


The following tables have hidden counters: ATTB READINGS.
It is not possible to update specific records in these tables.
gINT has imported data only to those data sets that were initially empty.
If you wish to replace existing data use the 'Overwrite Data Sets' option for these tables.
Existing datasets will also be overwritten if the 'Overwrite Records' option is chosen for the entire import.

I look over the data and it is there and visually looks fine.

Now, at times  I get into a situation where I need to re-import the source because some values have changed (eg. change in a tare weight)

I do the import and now get the same message, BUT as noted in the error message " ...... gINT has imported data only to those data sets that were initially empty." The bad data has not been replaced by the good data in my revised source file.

I have run into this before on other tempates, and I have used the Overwrite Data Sets option as indicated in the Warning Message. When I do this, all is cleared out and the entire source is brought into the target. I do NOT get the "The following tables have hidden counters: " message either.

On this particular project, however, created with the template I mentioned above, if do the import, using the Overwrite Option of "Data Sets, I get a slew of these messages. 

WC DENSITY Table:
The changes you requested to the table were not successful because they would create duplicate values in the index, primary key, or relationship. Change the data in the field or fields that contain duplicate data, remove the index, or redefine the index to permit duplicate entries and try again 

I do some more testing, and find that even if I delete all the boreholes, and have what looks like a clean blank file, and do an Overwrite Option of Data Sets again, I get the same slew of messages. It is like it is NOT clearing everything out.

Again, with previous templates using the Overwrite Data Sets, things "look like" it has cleaned out all the data, and prevented that "The following tables have hidden counters: ATTB READINGS. message on a newly created project, and also prevented either message if used on an existing project that had data in it already. 

I have see an earlier response here that seems to be in the same ball field and may be spot on, but my ignorance is keeping me from discerning the "fix". What I do discern form the post is that all the "Data" is NOT cleared out, with an Overwrite Data Sets option, and that although the user interface is blank, there is still some hidden structural placeholders attached to the file (Counters/keysets, etc??)

https://communities.bentley.com/products/geotechnical1/f/geotechnical-forum/71792/import-unc-data-over-writing-previously-imported-data/189945#189945

What I Want:

a) to understand better what is going on.

b) to get things set up so that if I revise the underlying data in the source, I can import it using the Named Fields option, and it will actually bring in the Named field source data, even into the non-empty fields, thus updating their data.

Ideas??



2022_11_10 EDIT/SUPPLEMENT


I did a database compare. Could the "GintVersion" difference as shown below in the "Database Property Differences:"  have something to do with it??

DB #1 is the DB that seems to give the problem with the slew of messages on the imports

DB #2 is the DB that does not throw the slew of messages on the imports

Property[GintVersion]
1: [4.15.002]
2: [8.30.004]

Parents
  • Don't know if I know the answer but I will add some comments/sugestions that might help. This is my understanding and I would encourage others to correct/ add to my discussion

    For your question a.  what is happening is that the tables in question are set to allow duplicate record keys. For example, if the keyset for a table is PoitID, depth you are allowed to enter more than one record with the same pointID and depth. Suppose your table for atterburg had the following fields

    PointID, depth, LL, PL

    if you had 2 results at the same depth, you could enter

    B-1, 10.3, 32, 23

    B-1, 10.3, 67, 42

    This would normally not be permitted because you have 2 records with the same key but checking the allow duplicate keys option in the data structure causes gINT/Access to add a hidden index to keep the duplicate records distinct.  The problem comes when you try to reimport revised data. In this case gINT doesn't know which record to assign the changed record to and refuses to do it unless you overwrite the entire dataset. this essentially deletes the existing records and imports the correct records.  it also deletes parent records and any other data you may have added to the data set in gINT as well so if you have made changes to those in gint they will be lost when you reimport.

    For your question b. in my limited testing i was able to import records with duplicate keys into tables that allowed duplicate keys and it comes in fine with both records intact.  I do get the warning message.  If I try to reimport revised data it refuses to import the revised data unless I use the overwrite data sets option as described above. this wipes out any other changes I may have made in gINT as expected. not sure why you are not able to do the same. could be a version issue as you noted so i would try opening the data template (.gdt) in your current gint and resaving it. then create a new project using the resaved data template and see if it functions as expected.  Ultimate solution is to either not allow duplicate keys or to not maintain 2 completely different data sets while allowing changes to be made to either one. sort of like having 2 copies of the same report in 2 word documents and allowing 2 people to work on their copy independently. When it comes time to assemble the final report you have a hot mess on your hands as the same section may have been modified in 2 different ways. It requires that you carefully track what changes you have made in each document, when those changes were made and only merge carefully selected parts of the data 

  • YES -- My testing indicate this is on the right track. I have done some more sleuthing and will have follow-up with results in a bit.

    In the meantime, you say... 

     but checking the allow duplicate keys option in the data structure causes gINT/Access to add a hidden index to keep the duplicate records distinct.

    Re: That hidden index - I am seeming to remember somewhere reading that the underlying source database of gINT (the .gpj file itself) is MS Access, and that when the database connection is made, that gINT creates a temporary database, and that the temporary database is what gINT is working with on a primary level. With this, I am thinking that on reading the underlying .gpj file into the temporary database, that these hidden counters are added. i.e., That the counters are added into the database that gINT is using as the intermediary.

    #1   Is this correct about the creation of  temporary database, and also about the insertion of the "hidden" counters into that database?? On inspection in Access, these "hidden counters" are not in the underlying .gpj ( MS Access) file.

    #2   When the file is saved, and the original access database did NOT have the Indexed property set to  "duplicates allowed" ( when you look at it in MS Access), on the save, AND if in gINT one selects the "Allow Duplicates" option,  does gINT modify that "Indexed" setting in the foundational .gpj? ie if one looks at the underlying original access DB, will they find that the settings for the Indexed Property of the field(s) have changed.

    #3  If it does modify the indexed setting, will it modify the underlying "Join(s)"

Reply
  • YES -- My testing indicate this is on the right track. I have done some more sleuthing and will have follow-up with results in a bit.

    In the meantime, you say... 

     but checking the allow duplicate keys option in the data structure causes gINT/Access to add a hidden index to keep the duplicate records distinct.

    Re: That hidden index - I am seeming to remember somewhere reading that the underlying source database of gINT (the .gpj file itself) is MS Access, and that when the database connection is made, that gINT creates a temporary database, and that the temporary database is what gINT is working with on a primary level. With this, I am thinking that on reading the underlying .gpj file into the temporary database, that these hidden counters are added. i.e., That the counters are added into the database that gINT is using as the intermediary.

    #1   Is this correct about the creation of  temporary database, and also about the insertion of the "hidden" counters into that database?? On inspection in Access, these "hidden counters" are not in the underlying .gpj ( MS Access) file.

    #2   When the file is saved, and the original access database did NOT have the Indexed property set to  "duplicates allowed" ( when you look at it in MS Access), on the save, AND if in gINT one selects the "Allow Duplicates" option,  does gINT modify that "Indexed" setting in the foundational .gpj? ie if one looks at the underlying original access DB, will they find that the settings for the Indexed Property of the field(s) have changed.

    #3  If it does modify the indexed setting, will it modify the underlying "Join(s)"

Children
No Data