Discussions
»
Community Forums
»
General Chat
»
Symbol Database Utility - Need everyone's feedback
Rank: Advanced Member
Groups: Registered, Registered Users Joined: 11/28/2005(UTC) Posts: 276 Location: Salt Lake City, UT
|
I'm thinking about doing a utility in my personal free time to give to people on the forum. I have to get it approved by Equis since it isn't something someone outside of Equis could do and so on. So, I'm not making any promises that it will get done.
Having said that, I would really like to know how many people would be interested in the utility I have in mind.
It would be a program that you run after updating your symbol database. It would allow you to pick a group or securites (All, NYSE, S&P 500, etc ) just like it is in the symbol database. Then you pick a folder on your hard drive. It does all the rest of the work for you. It would create any security file that doesn't already exist in the specified folder or a sub folder of it.
That means you could use this to either create a whole local database or just use it to update an existing one when you get a new database file.
It would create the folders needed. You would just specify a main folder such as C:\MetaStock Data\NASDAQ. It would either look in or create folders as needed to make all the securites in the group you selected.
Is this something that you would be interested in?
|
|
|
|
Rank: Newbie
Groups: Registered, Registered Users Joined: 9/14/2005(UTC) Posts: 5
|
Hi Branden,
I personally am interested in a tool the performs the above tasks.
I have subscribed to MS Quotecenter EMEA region data since Oct 05. The symbols in the markets I use are not updated regularly, hence a lot of manual work on my part.
regards, Fahad
|
|
|
|
Rank: Advanced Member
Groups: Registered, Registered Users Joined: 11/28/2005(UTC) Posts: 276 Location: Salt Lake City, UT
|
This wouldn't change how often symbols in your area are updated.
This would just get rid of the manual work of creating symbols from the symbol database, whether it be a new symbol database or a new group that you want to look at.
|
|
|
|
Rank: Member
Groups: Registered, Registered Users, Subscribers Joined: 3/22/2006(UTC) Posts: 18
|
Brandon:
Anything would be better that what is now available. What is really needed may be beyond your plan, but all of the stock datafiles should be in one master list, with pointers to the actual file from any/all groups of securities that are created.
In the late 1990's, thee was a Metastock competitor, Window on Wallstreet. All stocks were in one master list. To create a group, all you had to do was select a name, and "check" off the stocks you wanted included from the master list. The user never had to worry about where the actual datafile was located, and there was only one datafile for each stock, no matter how many groups or sectors the stock was part of. I have attached WOW1.png and WOW2.png, to give you an idea of what the WOW group creation screens looked like.
Harry
|
|
|
|
Rank: Member
Groups: Registered, Registered Users, Subscribers Joined: 3/22/2006(UTC) Posts: 18
|
Brandon:
I can't figure out how to upload a file on this new forum, so nothing is attached to my earlier note. Sorry.
Harry
|
|
|
|
Rank: Advanced Member
Groups: Registered, Registered Users Joined: 1/19/2005(UTC) Posts: 1,065 Location: Koh Pha-Ngan, Earth
Was thanked: 2 time(s) in 2 post(s)
|
bex1210 wrote:...all of the stock datafiles should be in one master list, with pointers to the actual file from any/all groups of securities that are created.
Having all data in one single file may not be a good idea, for at least two reasons:
1) If the single composite data file becomes corrupted (e.g., not closed properly), the complete database may be lost. Having said that, the current master.dat format is also somewhat vulnerable to file corruption.
2) Updating a single large 100MB+ data file may result in a much slower data updating process. The Downloader would have to open the file, search within for the security to be updated, and close it, thousands of times for each complete data update.
jose '-)
|
|
|
|
Rank: Advanced Member
Groups: Registered, Registered Users Joined: 11/28/2005(UTC) Posts: 276 Location: Salt Lake City, UT
|
Jose, he didn't say all data was in one file, he is saying what I want MetaStock to do, but won't be able to quite get to.
He said he wants there to be one file that MetaStock looks at and that file knows the location of all the other needed security files.
That would make it so you aren't limited to how many securities you have because you could span multiple folders without knowing it.
With what I intend to do, you won't have to worry as much about where your securities are.
DownLoader lets you add multiple folders to your download at once and you only have to do it once.
With this utility I want to write, it would create all the securities and folders needed based on the group you select from the symbol database.
This utility gets you much closer to what you describe. You wouldn't get individual security selection, for that, you'd have to do it manually in the DownLoader, but you can create a whole group at a time.
|
|
|
|
Rank: Advanced Member
Groups: Registered, Registered Users Joined: 1/19/2005(UTC) Posts: 1,065 Location: Koh Pha-Ngan, Earth
Was thanked: 2 time(s) in 2 post(s)
|
I see...
So, what you envisage would be something like a Security() or data query redirection tool, no?
You could have securities X/Y/Z in data folders A/B C, and then use the redirection tool to find them regardless of their data folder location:
x:=ExtFml("Redirect",X,C);
x:=ExtFml("Redirect",Y,C);
x:=ExtFml("Redirect",Z,C);
Branden, would this be what you have in mind?
And if so, how would it deal with duplicate symbols?
jose '-)
|
|
|
|
Rank: Advanced Member
Groups: Registered, Registered Users Joined: 11/28/2005(UTC) Posts: 276 Location: Salt Lake City, UT
|
No. The only thing I am planning to do with this utility is to create symbols from a symbol database automatically for you.
You select a group and specify a main folder and the utility will look at any existing subfolders and securities and create any securities from the group that don't already exist.
This way everytime you get a new symbol database, whether it is your first or your 100th, it will be easy to get all the symbols from any group. You just select it from a list, it does all the checking and creation.
No more manually selecting the next 6,000 to put in a folder.
|
|
|
|
Rank: Advanced Member
Groups: Registered, Registered Users Joined: 1/19/2005(UTC) Posts: 1,065 Location: Koh Pha-Ngan, Earth
Was thanked: 2 time(s) in 2 post(s)
|
Isn't this a function already included in The Downloader?
The Downloader -> File -> New -> Security -> Browse to new folder -> Lookup -> choose symbol.sym -> Select All -> Ok -> etc...
j '-)
|
|
|
|
Rank: Advanced Member
Groups: Registered, Registered Users Joined: 11/28/2005(UTC) Posts: 276 Location: Salt Lake City, UT
|
Jose wrote:Isn't this a function already included in The Downloader? The Downloader -> File -> New -> Security -> Browse to new folder -> Lookup -> choose symbol.sym -> Select All -> Ok -> etc... j '-)
Yes, you can do that in DownLoader, but that isn't what I'm talking about doing.
I'm saying this program would keep you up to date. It would add any securities you don't have and the folders needed to keep them in so you don't have to manually create folders or worry about the 6,000 security per folder max.
|
|
|
|
Rank: Member
Groups: Registered, Registered Users, Subscribers Joined: 3/22/2006(UTC) Posts: 18
|
Branden Russell wrote:
He said he wants there to be one file that MetaStock looks at and that file knows the location of all the other needed security files.
That would make it so you aren't limited to how many securities you have because you could span multiple folders without knowing it.
Brandon:
That is essentially what I said. The user should not have to know where a security file is located on the hard drive. He just just have one list of all securities there, and then could check which belong in which group/sector that he wants. Compters are good at that, whereas the Metastock way takes lots of manual work.
If I could figure out how to attach a file to this post, you could see what I mean.
If I understand it, what you are proposing goes in this direction, but will still require a fair amount of manual setup, at least initially.
Harry
|
|
|
|
Rank: Member
Groups: Registered, Registered Users, Subscribers Joined: 5/1/2006(UTC) Posts: 26
|
It would create the folders needed. You would just specify a main folder such as C:\MetaStock Data\NASDAQ. It would either look in or create folders as needed to make all the securites in the group you selected.
Is this something that you would be interested in? Most definately would be interested. I have come up with a kludge way to do some of what your going to do. Howver it currently involes "printing" the error log from the downloader to a PDF, then copy/paste into an excel sheet that list what symbols have changed, and what are invalid, but I stiull need to find the file format for the SymbolDatabase10.dta file .... Yours would be SO SO much better. One other feature would be to be able to send it a list of securities to remove, say in a .SYM file I would like ot offer any assistance I can, Thanks Dean
|
|
|
|
Rank: Advanced Member
Groups: Registered, Registered Users Joined: 11/28/2005(UTC) Posts: 276 Location: Salt Lake City, UT
|
You're not supposed to know the format of the SymbolDatabase.dta file.
It's good to hear interest in this project. I dropped the idea after the response I got when I posted it. My idea for the program was also to have an option to remove symbols. I was trying to decide if I should have it based on not trading for x days or by it not being in the current symbol database.
|
|
|
|
Rank: Member
Groups: Registered, Registered Users, Subscribers Joined: 5/1/2006(UTC) Posts: 26
|
Branden Russell wrote:
You're not supposed to know the format of the SymbolDatabase.dta file.
OK, but WHY. Its just a list of symbols and associated to exchanges. Oh well, will just have to spend some times with a hex editor to decode it. Just wasn't wanting to go that route.
Branden Russell wrote:
It's good to hear interest in this project. I dropped the idea after the response I got when I posted it. My idea for the program was also to have an option to remove symbols. I was trying to decide if I should have it based on not trading for x days or by it not being in the current symbol database.
To me, it would be worth atleast $50 (lifetime updates free) if we can do all that has been discussed.
If you could read the report produced by the downloader, it says which are invalid etc (as I am sure you already know), that would be the most effective. But perhaps that is a proprietary file format we aren't suppoed to know either :)
Dean
|
|
|
|
Rank: Advanced Member
Groups: Registered, Registered Users Joined: 11/28/2005(UTC) Posts: 276 Location: Salt Lake City, UT
|
It is against the license agreement to decode the symbol database or any other file that comes with the software. It is that way with pretty much any software you purchase from any company.
I hadn't thought about the download log as a possibility. If it is saved to the drive at all, my guess is it's just a plain text file, but I don't remember. Even if it's not, I'm allowed to know the format. It's you that isn't supposed to ;).
|
|
|
|
Rank: Member
Groups: Registered, Registered Users, Subscribers Joined: 5/1/2006(UTC) Posts: 26
|
Branden Russell wrote:
It is against the license agreement to decode the symbol database or any other file that comes with the software. It is that way with pretty much any software you purchase from any company.
Right :) The license agreement! OK, I won't do that then. However, I thought that was related to teh executable. I was unaware that it related to the data.
Branden Russell wrote:
I hadn't thought about the download log as a possibility. If it is saved to the drive at all, my guess is it's just a plain text file, but I don't remember. Even if it's not, I'm allowed to know the format. It's you that isn't supposed to ;).
I just found it. another one of those danm .DTA files I am not supposed to know anything about.
Its COLREPT.DTA in the c:\program files\equis he Downloader\ directory.
Is there "any" way to get the format for this file, say a "developer" kit?
Let me know if I can be of any help in the development of this util
Dean
|
|
|
|
Rank: Advanced Member
Groups: Registered, Registered Users Joined: 11/28/2005(UTC) Posts: 276 Location: Salt Lake City, UT
|
Ah, you're right. That is the file. No, there isn't anyway for you to access that file legally (well, other than through the DownLoader). The developer's kit will give you the ability to read data files, but not the .dta files.
Actually, you already have helped this project more than you may realize. I had dropped the idea when it seemed nobody was interested in me spending my time doing it. After your post, I'm seeing about being alloted time to work on it. Can't promise anything, but it looks like it just might happen now.
|
|
|
|
Rank: Member
Groups: Registered, Registered Users, Subscribers Joined: 5/1/2006(UTC) Posts: 26
|
Branden Russell wrote:
Ah, you're right. That is the file. No, there isn't anyway for you to access that file legally (well, other than through the DownLoader).
At first this ticked me off, but after a bit rationalization I can see why. Its to do with some dude gettting the format and writing software that can remove your companies ability to make money, whereas I have been clearly focused on doing this to save ME time.No interest in prodcuing software for others either pay or free or GNU. It would have been JUST for me.
Branden Russell wrote:
Actually, you already have helped this project more than you may realize. I had dropped the idea when it seemed nobody was interested in me spending my time doing it. After your post, I'm seeing about being alloted time to work on it. Can't promise anything, but it looks like it just might happen now.
Sweet! If you need a tester, drop me a message (or better an email) and I would be happy to help.
Dean
|
|
|
|
Rank: Advanced Member
Groups: Registered, Registered Users Joined: 11/28/2005(UTC) Posts: 276 Location: Salt Lake City, UT
|
I'm glad you understand and I am sorry that you're frustrated. Please know we do care about our customers and we'd be very happy to hear what your suggestions are to make our software better. Submit your suggustions in the suggustion area of the forum or let support know. You never know, there may be a way to do what you want or it just might get done because you ask.
Thank you for you offer on testing. I might take you up on that.
|
|
|
|
Users browsing this topic |
Guest (Hidden)
|
Discussions
»
Community Forums
»
General Chat
»
Symbol Database Utility - Need everyone's feedback
Forum Jump
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.