Originally Posted by: maverick77

Hi, I am converting CSV files of several thousand stocks in the Downloader (Metastock 16) to MS Local Files. If the destination folder points to an existing MS Local file it is really very slow. It doesn't matter if I tick the overwrite option or not.

If the destination folder points to an emtpy folder, it is much faster.

My hard drive is fully defragmented. What I believe is happening under the hood is that the Metastock Downloader / Converter shifts data around if newer stock data is inserted which just takes a lot of time since the MS Local files are greater than 1 GB.

It would be so great, if you would release the details of the MS Local file format so that we could resolve such issues ourselves.

Hi,

Thank you for your feedback here. To clarify a couple points, when you mention converting to an empty folder, do you mean a completely empty folder (no price repository inside it) or just a folder with securities but no data? Does the progress seem to get slower the longer the conversion process goes, or does it seem to convert at a fairly stable rate througout the entire process?

With regards to your CSV files, are these files with full ticker history, or are you just converting for example 1 day's worth of data for each security? How much total history would you estimate is being converted?

We can try to reproduce the behavior you are seeing but some of the above questions should help us to more accurately produce the scenario you are working in. Thank you!

Originally Posted by: maverick77

Yes, only when the destination folder is completely empty the conversion is very fast. If there is already an MS Local file in there (PriceRepository.MSPR01) it becomes very slow. For example, when converting approx. 8000 CSVs with EOD data ranging back from 2010 until today the conversion takes more than an hour, if the destination folder is not empty (resulting in a MS Local price repository file which is about 1.2 GB).

If the destination folder is empty, then it only takes a few minutes.

By the way it doesn't matter if I check the overwrite option or not in the Converter tool.

Thank you again for the feedback. I will pass this along to the development team for review on our end. There may still be some ways to improve the performance right now, depending on how you are typically performing conversions. For example, it was brought up that if you are converting into an existing price repository and the source file you are converting only has 1-2 records (i.e. 1-2 most recent dates) this would probably be faster than converting a file that goes back to the beginning of the history in the price repository.

This could improve independently of the 'Overwrite destination data' option since there is still more file checking that needs to be done vs. simply checking the most recent 1-2 data points. We will do some additional testing here to see if we can accurately capture the experience you are having with this.

Originally Posted by: maverick77

Hi, I have modified my EoD downloader program accordingly and just wanted to confirm that the conversion process is significantly faster if the .csv files only contain a few records and not the full range of records covering multiple years. Thanks for pointing this out.

That is great to hear. This was feedback from the developer who mentioned the logic checking is more extensive based on the date range in the source file.