Rank: Member
Groups: Registered, Registered Users
Joined: 6/6/2011(UTC)
Posts: 27
Thanks: 2 times
|
Hi MS Support,
I convert data from weekly text files using Downloader 11.0. The text files are about 23,000 lines long. Downloader puts the converted files into a Data folder on my computer. That folder currently contains 29,326 files spread over 63 folders - a total of 256 MB.
The process used to take 15 minutes or so but recently has become slower and slower. The process now takes several hours!
Any thoughts?
(I've also noticed that charts are taking longer to load.)
Thanks,
Tim
|
|
|
|
|
|
Rank: Advanced Member
Groups: Moderators, Registered, Registered Users, Subscribers
Joined: 10/8/2010(UTC)
Posts: 1,960
Thanks: 92 times
Was thanked: 155 time(s) in 150 post(s)
|
It's hard to say assuming the folder locations did not change. Most modern operating systems should keep the file system defragmented on its own. Does the File-->Open dialogue take a long time to open? We have seen sometimes where large compressed files sitting on the Desktop can cause slowness here, and moving these files to a subfolder instead (i.e. My Documents / My Downloads) might speed this up.
Have any other data folders been created / added to the process recently? Do you typically append to the existing files or do you create new files each time you convert? Are either of the folder locations on a network drive?
|
|
|
|
|
|
Rank: Member
Groups: Registered, Registered Users
Joined: 6/6/2011(UTC)
Posts: 27
Thanks: 2 times
|
Thanks for your reply.
It's hard to say assuming the folder locations did not change................. The folder locations have not been changed.
Does the File-->Open dialogue take a long time to open? ............................That process takes 2" - 5" (so quite long) But a layout can take up to 25 seconds to fully open
We have seen sometimes where large compressed files sitting on the Desktop can cause slowness here, and moving these files to a subfolder instead (i.e. My Documents / My Downloads) might speed this up...................................The files I am converting are simple, uncompressed end-of-week prices Text files located in a folder on the Desktop. Downloader then converts the Text file nto Metastock format puts the data into a Data folder located at E:|Metastockdata\DATA
Have any other data folders been created / added to the process recently? ..............................No
Do you typically append to the existing files or do you create new files each time you convert?.................. The new data is appended to the existing files,
Are either of the folder locations on a network drive?........................ No
So two main problems - both recent:
1. Downloader is taking several hours to convert and append weekly prices around 29,000 securities
2. Metastock is taking up to 25 seconds to load a layout.
Thanks, Tim
|
|
|
|
|
|
Rank: Advanced Member
Groups: Moderators, Registered, Registered Users, Subscribers
Joined: 10/8/2010(UTC)
Posts: 1,960
Thanks: 92 times
Was thanked: 155 time(s) in 150 post(s)
|
The speed issue in relation to large compressed files has more to do with other files that might reside on the Desktop, not the files used by MetaStock. I would personally look for any large files being stored directly on the Desktop and try to move them into a subfolder.
You may also want to try moving the source / destination text files / MetaStock data to subfolders on the local drive, although I am not sure this would help much. You do also want to be careful when storing the data in subfolders that you do not nest data and subfolders together (i.e. storing data in a root folder and then also having a subfolder in that same folder that also contains MetaStock data files) as problems in the parent folder can then cascade into the subfolders.
You could try opening items in another location on the drive to see if you notice speed issues there as well, such as opening something inside C:\MetaStock Data\Sample. MetaStock file format is very fast to read, so slowness here typically involves slowness in the Windows Explorer responsiveness (since we have to access the Windows Explorer when we access our files).
|
 1 user thanked MS Support for this useful post.
|
|
|
|
Rank: Member
Groups: Registered, Registered Users
Joined: 6/6/2011(UTC)
Posts: 27
Thanks: 2 times
|
A little more on this...
I have tried moving folder locations but this makes little difference to the time that Downloader takes to convert 26,000 text files into MetaStock files - in excess of one hour.
Any idea how long that ought to take?
Other behaviour:
1. The File > Open dialogue takes 3 - 5 seconds.
2. Opening a 3-window layout takes about 10 seconds.
Any thoughts?
Thanks, Tim
|
|
|
|
|
|
Rank: Advanced Member
Groups: Moderators, Registered, Registered Users, Subscribers
Joined: 10/8/2010(UTC)
Posts: 1,960
Thanks: 92 times
Was thanked: 155 time(s) in 150 post(s)
|
Hi Tim,
Some other considerations that could have an impact on the overall speed/performance.
Within Tools-->Convert-->Options (Source tab), if you set a specific date range, it could be converting a larger amount of data than previous.
Within Tools-->Convert-->Options (Destination tab), if you are using the 'Replace existing file' option instead of 'Append data to end of file' this might slow down the process as well.
One other thing you may want to try is Tools-->Test to test the integrity of each folder. The problem with this is you can't do the parent folder and test all the subfolders, so you would have to individually test each of the 63 folders.
|
|
|
|
|
|
Rank: Member
Groups: Registered, Registered Users
Joined: 6/6/2011(UTC)
Posts: 27
Thanks: 2 times
|
Thanks for your reply.
1. I am using the same date in the First and Last boxes
2. I am using 'Append Data....'
3 Results of test as follows:
Several folders have Control Record Error #1133
Several folders have Invalid Character Error #1109 or Error #1035
One folder has Error #1080
Any thoughts?
Thanks, Tim
|
|
|
|
|
|
Rank: Advanced Member
Groups: Moderators, Registered, Registered Users, Subscribers
Joined: 10/8/2010(UTC)
Posts: 1,960
Thanks: 92 times
Was thanked: 155 time(s) in 150 post(s)
|
Here's those particular errors in the Help topic:
Error #1035. EMASTER: Invalid character(s) in EMASTER name
A character other than a normal alphanumeric character is stored where the name should be.
Error #1080. No records in DAT file
This is a warning more than an error. The message occurs if the DAT file contains no records.
Error #1109. XMASTER: Invalid characters(s) in name
A character other than a normal alphanumeric character is stored where the name should be.
Error #1133. XMASTER: Invalid version
The version of the XMASTER file is not recognized.
The 1080 "no records" error is not necessarily a concern although it may be there in relation to the other errors. Errors with any of the MASTER files do concern me a bit as it would imply there is at least some corruption / out-of-sync type issues in some of the folders. You could try using the Tools-->Copy command to make a copy of the folders with errors (Then you can test the copies and replace the original folders with the copies if the copies appear to be working). By making a copy of the folder, this can effectively rebuild the MASTER files which may eliminate any issues inside the folders. Edited by user Monday, July 27, 2015 2:37:49 PM(UTC)
| Reason: Not specified
|
|
|
|
|
|
Rank: Member
Groups: Registered, Registered Users
Joined: 6/6/2011(UTC)
Posts: 27
Thanks: 2 times
|
Thanks.
I tried copying a folder and re-testing but the error #1133 persists. Is there any way to get rid of that?
Tim
|
|
|
|
|
|
Rank: Advanced Member
Groups: Moderators, Registered, Registered Users, Subscribers
Joined: 10/8/2010(UTC)
Posts: 1,960
Thanks: 92 times
Was thanked: 155 time(s) in 150 post(s)
|
Unfortunately in that case those particular folders may be corrupted beyond salvaging. You could look inside these particular folders through Windows explorer. If you see any files with a .TMP extension, this could be a problem. For example
XMASTER
XMASTER.TMP
EMASTER
EMASTER.TMP
MASTER
MASTER.TMP
We would not want any TMP files to be inside your data folders. However, you don't necessarily want to delete the TMP file. If you have a file situation as described above, you typically want to keep the larger of the two files. If that happens to be the TMP version, you would want to move or delete the smaller file, and then rename the .TMP by removing the .TMP extension on the file.
If the above does not describe your situation, it may be that you will need to rebuild these particular folders. Edited by user Tuesday, July 28, 2015 2:06:03 PM(UTC)
| Reason: Not specified
|
|
|
|
|
|
| Users browsing this topic |
|
Guest (Hidden)
|
Forum Jump
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.
