The thing about a utility like this, is that I would assume most people don't know that their stock lists (local data) may be missing stocks etc or have not been updated since who knows when. This utility would gain large suppport if people knew of the short comings of the downloader.

I only became aware because I am running an explorer that is searching through all my local data and I have downloader set to update all 16000+ stocks each night. When I sent the report to a PDF to analyze in Excel, it was 130 pages long.

I guess it is a bit of a catch 22, whereas you don't want to tell the customers the short comings of your software, but the interest will not be there for it unless you do.

Dean

I updated the utility in the files location.

http://forum.equis.com/files/3015/metastock_files/entry22426.aspx

I made it able to recursively delete so you can specify the main folder and it will go through all the sub folders of all the sub folders and delete unwanted securities.

I forgot, it would probably be good to mention that my first test of the recursive delete was on the data CD with somewhere around 26,000 securities to delete. It took a couple minutes and the program is single threaded, so the program isn't responsive while it's deleting.

So, just keep that in mind if you have a lot of securities to delete, it will take a bit and the program will appear frozen, but it reallly is still working.

Just tried the Jan 12th file on my 9.1EOD database. Because I'm paranoid, I set it to delete securities not traded for 2000 days. As expected, it reported deleting 0 symbols.

But it didn't crash or report any errors and my database seems intact. So far, so good! I think it'll be a handy utility. As development progresses, of course, it would be nice to see some sort of progress indication to confirm it's working and help us estimate remaining processing time. I have a WD Raptor RAID 0 drive setup, so it's pretty fast, but it would be useful to have that feedback.

I just thought of a useful (to me, anyway) addition to the program - instead of deleting files, how about the option to scan and prepare a report (printable, savable to file) of symbols meeting the input parameters? Sometimes I have a "stuck" datafile that hasn't been updated in a long time due to a symbol change, typo etc. and this function would help spot those so I can attempt a fix.

I use ML Downloader with Yahoo data, and it catches most of these, but if the symbol isn't in MLD for some reason it won't be picked up.

I should have been a little clearer; generating a progress bar would require more overhead than it's worth.

If you could show file location, symbol name and the number of symbols processed so far (assuming updating this wouldn't cost too much processing time) it would be very useful. After running it once or twice, we'll know approximately how many symbols it will process in total and can estimate overall progress.

If updating this display eats up too much time, maybe updating it once a second, or every 5 seconds would suffice. Certainly doesn't have to be realtime or updated for every single symbol unless the overhead cost is trivial.

Thanks for plugging away at it.

Brandon: Thanks for writing this. I am going to download it now. Are there instructions on how to use it? Harry

Right now, it doesn't really matter what version of MetaStock you have. It will read your Types and Groups from your symbol database if you have version 9.2 or later, but doesn't yet do anything with that information. I only put a copy up that does that much with the database because I need people testing it and making sure it does what it should. Since you have version 8, your Type and Group shouldn't fill in. When all the functions in the utility are done, it probably will only work with 9.2 and above. After I finish, I'll see how much more free time I have to do older versions. The only function that currently really does anything is the delete securites X or more days old. To use that feature, all you have to do is specify the location of the securities you want to clean up (delete old ones from) and check the recursive box if you want it to search all the sub folders too. Then you enter in a number of days that you want to check for as not traded. If the security has that many days that it hasn't traded, it will be deleted when you click delete. It will show you a progress of what folder and security it is currently processing and at the end, it will tell you how many securities it deleted. This utility seems to be working fine with the little testing it has received. Just remember that not all the functions are done (the ones that aren't done are disabled) and that it is compeletly unsupported.

Hi Brandan

Having read your post and checked out your utility out like to add a few more usability comments. I download my end of day data using Reuters which notifies me of symbol changes, symbols that are not updated, and other errors. My data files are spread across several folders with symbols appearing multiple times across those. At present the method of deleting symbols that may be in several folders is cumbersome and very time-consuming, more especially since this needs to be done on a daily basis. Likewise the method of changing symbols have been renamed it is also very time-consuming. Occasionally I need to move a security from one folder to another and again this is very time-consuming.

In my humble opinion the downloader is woefully inadequate carrying out these daily housekeeping tasks since maintaining the data files is such an essential task. Using a symbol list provided by metastock, is useful but only a starting step. I would therefore like to suggest that a facility to simply find and delete sets of symbols, find a move set of symbols, and change of symbol tickers will be a great addition to the utility. It seems to me that development of the downloader into a truly useful facility has been set aside for many years at the expense of improving the charting side of metastock, adding ever more indicators and so on. There is little point in having all these one of all indicators if the data on which it is based is so hard to keep up-to-date.

I hope you'll given more time to expand and improve on your facility and I am definitely champing at the bit to try anything that makes maintaining data easier. slipperx

Hey Branden.....

Are you still working on this project?

A feature, which we may have already discussed, that would be extremely helpful, would be to review the collection report, process the "renames" and go out and rename the stocks.

Thanks

Dean

Hi Branden, nice utility to remove symbols from your local DB that have not been updated in a while. I just downloaded it and really appreciate your hard work! I have a set of custom folders setup nothing fancy (Amex, Nasdaq1, Nasdaq2, Mutual Funds...under my Metastock Data folder..),but it does have a folder named Mutual Funds. I assume it will just ignore these symbols when looking for stocks to delete? Please clarify.

Also I use the Downloader to receive all the stocks/mutual fund data on the exchanges and average about 20 symbol changes daily. I do these manually and I assume others do the same. Are there any plans to create a utility to help with this? I pull about a ten year history of prices from stocks and mutual funds and it helps me to have this quasi custom folder list mentioned before instead of the list Equis offers, which if I understand would allow me to just update new stocks symbols but then I would have to download the 10 year history for all stocks/mutual funds as it purges the old list out. May be too much time involvled in a new download. I guess my question is have you heard of any one trying to solve this by creating an automatic system to keep your data up to date?

If not what's the most efficient way of keeping up to date with symbol changes?