Rank: Newbie
Groups: Registered, Registered Users
Joined: 12/28/2007(UTC)
Posts: 2
|
My data provider giving me updates in ASCII format and i need to convert it to Metastock format using downloader. The issue is it take long time to convert all the securities in the text file.
Now i have 1 year worth of data to be converted but it take 24hours to do 1 month of conversion,, if i need to do the whole year it might take few months !!
is there any thing i can do to speed up the process..
|
|
|
|
|
|
Rank: Advanced Member
Groups: Registered, Registered Users, Subscribers
Joined: 4/5/2006(UTC)
Posts: 129
Location: Norgate Data
|
It will go much quicker if you can consolidate/sort all of your daily files into a single file and then sort them by code.
Let's say you have 5000 securities to convert for each day.
If you have daily files, it will process them one day at a time, opening and closing each of the 5000 securities for each day.
If you consolidate them into one file, sort them, then convert, you'll find that it will open each security only once, dramatically increasing the speed.
e.g. I was able to convert over 18000 securities each with around 20 years of history securities in about 2 hours on my pretty quick PC.
Another way of speeding things up is to exclude both your ASCII and MetaStock folders from your real-time virus checking. This typically results in a 50% reduction in time on my PC to read/write files in this area.
The alternative is to subscribe to a data vendor that can automatically provide your data in MetaStock format with no conversion required.
The other thing to watch out for is that daily ASCII files have no way of providing any corporate actions (such as stock splits, code change and delistings) so your database will rapidly become untidy unless you really keep on top of the maintenance.
Cheers,
Richard.
|
|
|
|
|
|
|
Rank: Newbie
Groups: Registered, Registered Users
Joined: 12/28/2007(UTC)
Posts: 2
|
Hi Richard
Thank you for the valuable infor.. I have turned off the virus checker and it improve the speed dramatically.
I have 1 year of ascii data each day is a separate file that has around 2300 securities. IS that what you mean by grouping. I didnt follow your argument.
With regards to maintaining the database, My provider is the one giving the ASCII files so i assume its already done by the data provider
cheers
|
|
|
|
|
|
Rank: Advanced Member
Groups: Registered, Registered Users, Unverified Users
Joined: 9/13/2004(UTC)
Posts: 673
Location: Salt Lake City, UT
|
Security programs can definitely slow the process down as you noted. What Richard was suggesting is that all the data and all the securities reside in a single file so that The DownLoader only has to open one file to convert everything.
|
|
|
|
|
|
| Users browsing this topic |
|
Guest (Hidden)
|
Forum Jump
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.
