Slow Import

[John Gerhardy]

Hello,

I am importing large DIAdem data files (289MB +) into FlexPro 8. For each day there is one file. Since the whole test goes over 91 days, there are 91 files to load. After loading three files or so, the computer gets very slugisch. I have noticed that the swap memory grows with each imported file. At about 2 GB the system starts to get slow. At present, I am loading three files at a time, saving the databank, closing Flexpro and then re-starting.
1) The Databank is openen in split files
2) Flexpro opens the DB directly (no temp files)
3) The undo function is turned off.

This works, but is there an easier way to get around this problem?

Is there a way to automate the import of all the files alá start import in the evenings and come back the next day with all data imported (something like a batch import)?

regards,
John

[John Gerhardy]

Hello,

I am importing large DIAdem data files (289MB +) into FlexPro 8. For each day there is one file. Since the whole test goes over 91 days, there are 91 files to load. After loading three files or so, the computer gets very slugisch. I have noticed that the swap memory grows with each imported file. At about 2 GB the system starts to get slow. At present, I am loading three files at a time, saving the databank, closing Flexpro and then re-starting.
1) The Databank is openen in split files
2) Flexpro opens the DB directly (no temp files)
3) The undo function is turned off.

This works, but is there an easier way to get around this problem?

Is there a way to automate the import of all the files alá start import in the evenings and come back the next day with all data imported (something like a batch import)?

regards,
John

[Bernhard Kantz]

Please read the article Optimizing FlexPro for Large Volumes of Data in the FlexPro Help (Data Management|Project Database|Working with Project Databases).

If you want to import the complete file(s), check the option Fast import of complete files and select the (91) files you would like to import.

support@weisang.com

[John Gerhardy]

Hello,

I had not tried selecting more than one file when importing and this makes things easier. Thanks.

I have read the section about optimizing for large amounts of data and followed the suggestions there. However, this does not change the fact that memory is used for each imported file and at some point the memory is all used up. I can imagine that the swap memory is then used which really slows things down. Saving the DB does not reduce the memory used, only closing the program. I have attached pictures of the system stats before and after importing three files. You can see how the available memory drops from 1.15GB to 0.32GB after the import. In the first picture you can see how the swap file is reduced in size after the program has been exited. The save before exiting did not redurce the file.

The section on optimization also suggests only importing the data set which I need. Good idea. All 91 files have the same format, but with 98 channels, I do not want to search out the 51 channels I need 91 times. Do you have something like a template function where I can select the 51 channels I need 1 time, save the template and then use it to import the other 90 files?

Regards,
John

[Bernhard Kantz]

Unfortunately, this is not possible at the moment. In this case you have to program your own import filter using VBA.

support@weisang.com

[John Gerhardy]

That is a pity 🙁

Unfortunatel I use the standard version which does not have the VBA environment.

Maybe you could add the idea of an import template to the wish list!?

Thanks for your responses.

[Author]

Posts