logo
Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Options
Go to last post Go to first unread
maverick77  
#1 Posted : Friday, April 26, 2019 2:33:39 AM(UTC)
maverick77

Rank: Newbie

Groups: Registered, Registered Users
Joined: 3/17/2011(UTC)
Posts: 4

Was thanked: 1 time(s) in 1 post(s)

Hi, I am converting CSV files of several thousand stocks in the Downloader (Metastock 16) to MS Local Files. If the destination folder points to an existing MS Local file it is really very slow. It doesn't matter if I tick the overwrite option or not.

If the destination folder points to an emtpy folder, it is much faster.

My hard drive is fully defragmented. What I believe is happening under the hood is that the Metastock Downloader / Converter shifts data around if newer stock data is inserted which just takes a lot of time since the MS Local files are greater than 1 GB.

It would be so great, if you would release the details of the MS Local file format so that we could resolve such issues ourselves.

MS Support  
#2 Posted : Friday, April 26, 2019 7:55:18 PM(UTC)
MS Support

Rank: Advanced Member

Groups: Moderators, Registered, Registered Users, Subscribers
Joined: 10/8/2010(UTC)
Posts: 1,928

Thanks: 85 times
Was thanked: 154 time(s) in 150 post(s)
Originally Posted by: maverick77 Go to Quoted Post

Hi, I am converting CSV files of several thousand stocks in the Downloader (Metastock 16) to MS Local Files. If the destination folder points to an existing MS Local file it is really very slow. It doesn't matter if I tick the overwrite option or not.

If the destination folder points to an emtpy folder, it is much faster.

My hard drive is fully defragmented. What I believe is happening under the hood is that the Metastock Downloader / Converter shifts data around if newer stock data is inserted which just takes a lot of time since the MS Local files are greater than 1 GB.

It would be so great, if you would release the details of the MS Local file format so that we could resolve such issues ourselves.

Hi,

Thank you for your feedback here. To clarify a couple points, when you mention converting to an empty folder, do you mean a completely empty folder (no price repository inside it) or just a folder with securities but no data? Does the progress seem to get slower the longer the conversion process goes, or does it seem to convert at a fairly stable rate througout the entire process?

With regards to your CSV files, are these files with full ticker history, or are you just converting for example 1 day's worth of data for each security? How much total history would you estimate is being converted?

We can try to reproduce the behavior you are seeing but some of the above questions should help us to more accurately produce the scenario you are working in. Thank you!

maverick77  
#3 Posted : Friday, April 26, 2019 8:16:02 PM(UTC)
maverick77

Rank: Newbie

Groups: Registered, Registered Users
Joined: 3/17/2011(UTC)
Posts: 4

Was thanked: 1 time(s) in 1 post(s)

Yes, only when the destination folder is completely empty the conversion is very fast. If there is already an MS Local file in there (PriceRepository.MSPR01) it becomes very slow. For example, when converting approx. 8000 CSVs with EOD data ranging back from 2010 until today the conversion takes more than an hour, if the destination folder is not empty (resulting in a MS Local price repository file which is about 1.2 GB).

If the destination folder is empty, then it only takes a few minutes.

By the way it doesn't matter if I check the overwrite option or not in the Converter tool.

MS Support  
#4 Posted : Tuesday, April 30, 2019 2:42:33 PM(UTC)
MS Support

Rank: Advanced Member

Groups: Moderators, Registered, Registered Users, Subscribers
Joined: 10/8/2010(UTC)
Posts: 1,928

Thanks: 85 times
Was thanked: 154 time(s) in 150 post(s)
Originally Posted by: maverick77 Go to Quoted Post

Yes, only when the destination folder is completely empty the conversion is very fast. If there is already an MS Local file in there (PriceRepository.MSPR01) it becomes very slow. For example, when converting approx. 8000 CSVs with EOD data ranging back from 2010 until today the conversion takes more than an hour, if the destination folder is not empty (resulting in a MS Local price repository file which is about 1.2 GB).

If the destination folder is empty, then it only takes a few minutes.

By the way it doesn't matter if I check the overwrite option or not in the Converter tool.

Thank you again for the feedback. I will pass this along to the development team for review on our end. There may still be some ways to improve the performance right now, depending on how you are typically performing conversions. For example, it was brought up that if you are converting into an existing price repository and the source file you are converting only has 1-2 records (i.e. 1-2 most recent dates) this would probably be faster than converting a file that goes back to the beginning of the history in the price repository.

This could improve independently of the 'Overwrite destination data' option since there is still more file checking that needs to be done vs. simply checking the most recent 1-2 data points. We will do some additional testing here to see if we can accurately capture the experience you are having with this.

maverick77  
#5 Posted : Saturday, May 11, 2019 7:36:39 PM(UTC)
maverick77

Rank: Newbie

Groups: Registered, Registered Users
Joined: 3/17/2011(UTC)
Posts: 4

Was thanked: 1 time(s) in 1 post(s)

Hi, I have modified my EoD downloader program accordingly and just wanted to confirm that the conversion process is significantly faster if the .csv files only contain a few records and not the full range of records covering multiple years. Thanks for pointing this out.

thanks 1 user thanked maverick77 for this useful post.
MS Support on 5/14/2019(UTC)
MS Support  
#6 Posted : Tuesday, May 14, 2019 2:40:28 PM(UTC)
MS Support

Rank: Advanced Member

Groups: Moderators, Registered, Registered Users, Subscribers
Joined: 10/8/2010(UTC)
Posts: 1,928

Thanks: 85 times
Was thanked: 154 time(s) in 150 post(s)
Originally Posted by: maverick77 Go to Quoted Post

Hi, I have modified my EoD downloader program accordingly and just wanted to confirm that the conversion process is significantly faster if the .csv files only contain a few records and not the full range of records covering multiple years. Thanks for pointing this out.

That is great to hear. This was feedback from the developer who mentioned the logic checking is more extensive based on the date range in the source file.

Users browsing this topic
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.