October 8, 2011
· Lookup
· Setup
This program requires a recent version of Windows to run. It is a collection management tool. It looks up multiple md5, ffp, and st5 checksum sets for flac and shn files in a local database on your computer. It reports all matching ids and also any checksums missing for a found id. So you can tell what you have and what is missing. Currently there is a database of checksums for bob dylan (from losslessbob) and another for the grateful dead related (from etree).
Many times I see a torrent description that provides checksums. I copy the block of checksums from my browser to the clipboard and then click on this program’s “look up from clipboard” button, and see if the checksums are in the database and if so which id they are from and whether they are complete. Occasionally someone makes a mistake when specifying the id of a torrent, so this program can be used to verify the id.
Similarly, I may have a file containing checksums, a folder containing multiple files that contain checksums, or a group of folders such as on a dvdr. I can drag and drop the files or folders into the program’s listbox. The files and folders can also be fetched into the listbox using a file dialog box. Next I click on this program’s “look up from listbox” button, and see if the checksums in the files are in the database and if so which id they are from and whether they are complete. This is useful to verify the id I think I have is correct. Further there is an option to rename folders missing an id in the folder name so that the new folder name contains the id. That can be useful for those who want to mark a folder with an id on a hard drive or before burning to dvdr.
The lookup result is displayed in summary and detail grids. The detail grid indicates for each checksum whether it has a match and also shows all duplicate matches if the checksum belongs to multiple ids. If a checksum matches to an id in the database, a reverse lookup is done for that id to show any checksums which are in the database but missing from the checksums being looked up. From this one can find out how complete the set is. From a matched checksum, you can double click on the grid and go to the web page representing the id that was matched. For the gd etree database only, if the checksum is not matched locally like if it is from a newer release than the local database, there is an option to try to retrieve the id across the internet from the web gd etree database using one of its checksums.
The program was created for personal use to make checksum lookup and folder renaming easier and more efficient. It is being freely shared so that it may do the same for others.
Sometimes a torrenter in their torrent description, either on the web or by email, will list the md5 and sometimes the ffp or st5 checksums that match their torrent. So when I see those, I select all on either the web page or email with CTRL-A and then copy to the clipboard with the right click context menu copy option. I then click on this programs “Lookup From Clipboard” button and see in the summary grid how well those checksums matched to the database. If there are problems like missing checksums or multiple ids found, I may review the detail grid. If an id is found, I may double click on a row and go to that found ids web page for further information. If I am on the gd etree database and if I think the torrent contents may be more recent than the database, I may double click on a row without a found id to see if that checksum is in etree database and try to see its web page.
If there are no md5 or ffp or st5 checksums displayed in the torrent description, I may look to see it any of those files were provided in the torrent and just download them. When the download completes, I drag the folder and drop it in the listbox. I then click on “Lookup from Listbox” and see how they matched as if from the clipboard.
If I have a folder or multiple folders on my hard drive or dvdr, I drag and drop them into the listbox and click on “Lookup from Listbox” and see how they matched as if from the clipboard. I have a lot of folders from before I was placing and id in the folder name so this easily identifies them. If I am listening to them and want to add a comment, I can double click and try to go to the web page to enter the comment. If I want I can double click on any file in the listbox to show it in notepad.
I have a hard drive full of many folders. Some already have ids and others do not and some may have an id that has a typo in it or was just previously misnamed. I select a group of folders to check and drag and drop them into the listbox and then click on “Lookup from Listbox” and see how they matched. I go to the rename folders tab and see which folders are ready for rename. If there are some folders with a failure reason, I may research that folder on the detail grid by clicking on the folders row on the rename grid and then right clicking and selecting the context menu item “jump to detail for folder group”. There on the detail grid, if I can resolve which id it is for example if multiple matches are given, I click on a detail grid row with that id and then right click to display the context menu and select “calculate new folder name from id of current row”. I then return to the rename tab and resolve the other anomalies. When I am satisfied with the folders to rename, I click on the rename button to rename all the folders which are checked to rename.
There are further options available on the gd etree database because etree has a method for looking up a checksum on the web. This can help when the id is not found in the local database because it is more recent than the current local database or from other bands not in the database. I can select a rename grid row by clicking on it and then right click and select the grid context menu “lookup last checksum at etree” and if an id is found it will be used to rename the folder. Other times I am preparing a batch of folders that are recent downloads to archive either to dvdr or the hard drive and want them first renamed with an id. For that I click on the “Lookup All "no id found" at etree” button and confirm to submit them all at once for sequential lookup at etree. Those found get set to be renamed with the found id.
In order for this program to run, it requires Microsoft Windows .Net Framework 3.5 SP1 and Microsoft SQL Server Compact 3.5 SP1 runtime. Those both require a version of windows like windows XP or later. Both are available for free download from Microsoft if you do not already have them. Also each checksum database is currently less than100meg but may grow in future releases.
System Requirements for Microsoft .Net Framework 3.5 SP1
(I have only tested this with Windows XP and briefly in Windows 7 and copying to the clipboard from Internet Explorer 8, but I assume it will also work with Windows Vista and other browsers)
If you don’t know whether you have the required Microsoft software installed already, you can look in Control Panel’s Add Remove Programs. If you have windows 7, .NET Framework 3.5 sp1 comes pre-installed and you may have to turn it on in “Turn Windows Features On or Off.”
If it is no longer at that location, search in your browser for “Download details: .NET Framework 3.5 Service Pack 1” and you should find a Microsoft site with a web address beginning with http://www.microsoft.com/downloads. (google found it but bing did not); . (Make sure you are downloading this from Microsoft).
It is large and can take some time to download and install it as it will install a lot of precursor .NET software if needed. Follow the installation instructions
Most will want to download this option: SSCERuntime-ENU-x86.msi
If it is no longer at that location, search in your browser for “Download details: Microsoft SQL server compact 3.5 SP1” and you should find a Microsoft site with a web address beginning with http://www.microsoft.com/downloads. (Make sure you are downloading this from Microsoft).
It downloads and installs quickly. Follow the installation instructions
It is suggested you place this new folder in one of the User folders like My Documents. But you can put it where you want. However, if you are a Windows 7 or Vista user, do not put it in Program Files or other system oriented folders or you will have security access problems running the program.
Place all the files in the chesksum_lookup download set in that folder. You should then have checksum_lookup.exe, checksum_lookup_help.htm, and one or both database files named checksum_lookup_lb_compact.sdf or checksum_lookup_gd_compact.sdf. You may want to create a program shortcut to checksum_lookup.exe on your desktop or in a favorite location to make it easier to run.
This button will search through the text on the clipboard for any checksum and its associated file name and then display them in the grid along with all matching checksums and filenames found in the database. If a match is found, any missing checksums and filenames for the id are also displayed. (note: on a torrent web page with checksums, you can do a Ctrl-A in IE8 to select the entire web page and then copy that to the clipboard as this buttons action will parse through all the captured text to find just the checksum portion)
This button will search through all the files in the listbox for all checksums and their associated file names and then display them in the grid along with all matching checksums and filenames found in the database. If a match is found, any missing checksum and filenames for the id is also displayed. The checksums are grouped and processed by their parent folder and duplicate checksums found in the files are removed.
This button clears the listbox to make it ready for another drop or selection.
You can drag and drop filenames and folders into the listbox. Multiple filenames and folders can be dropped at once. The listbox will accept and list any filenames with extensions of txt, md5, ffp, and st5 that were dropped or were in a folder that was dropped. When processing a folder, the listbox will only accept files within that folder and within one level of subfolders. If someone created a folder with more than one level of subfolders, they made it more difficult for everyone. On encountering more folder sublevels, the program will issue a warning message and those must be checked manually. Preferably restructure the folder or you can manually add the desired files to check from the lower sublevels.
The contents of the listbox can be managed by right clicking on it and displaying a context menu from which you can clear the listbox, clear selected items in the listbox, get files, get folders, or get all subfolders. You can expand the listbox for easier viewing when there are a lot of files or files with long filenames. You can restore the listbox back to its original size. You can view the currently selected files in notepad by selecting “show in notepad”. You can also double click on a listbox file to display it in notepad. If you have more than one database, you will have a context menu option to switch databases.
The listbox context menu item option “filter listbox to _mychecksums.txt only” can be set or unset by clicking on it. It is set on when it is checked. When it is on, only files ending in “_mychecksums.txt” are allowed into the listbox. These files are generated by the MyChecksums software program. They contain the checksums that were generated in that program for the folder and represent what you have. Other files in the folder may contain checksums that do not match what you have such as old checksums or those from a larger set. This option is useful when trying to lookup large batches of folders for determining validity or for renaming purposes where you want to exclude from the lookup the checksums that do not match to what you have in the folder. For more information on MyCheckusms program go to http://tech.groups.yahoo.com/group/MyChecksums/
Files dropped or selected into the listbox are grouped by their parent folder. The group divider is indicated by the 2 extra slashes in the displayed filename in the listbox. All files with the same left portion from the divider slashes are in the same lookup group. Files are grouped to avoid repetition of checksums within the group from the different files. For example the same checksum may be repeated in the txt, ffp, and st5 file and the duplicates are filtered out in the lookup process. In the detail grid, each group will have a row header and trailer to distinguish it from other groups that may be being looked up at the same time. Each group will also have one row in the rename grid.
There are some software programs that output very nonstandard checksum files, and this program may be unable to find the checksums in those types of files.
The summary grid summarizes what is in the detail grid. The type column indicates the totals for md5 and for ffp/st5 type checksums. The given column indicates the total unique checksums found either from the clipboard or the files in the listbox. The matched column indicates how many of the given checksums were matched to the database. The xref column indicates how many given checksums were matched to an xref number off of an id. The missing column indicates the total checksums found for the id that were not given from the clipboard or the files. Missing or unmatched checksums should be considered a starting point for further research. Note that the databases on your computer are a snapshot in time and do not grow like the ones online. There also may be data entry mistakes in the databases or other anomalies.
The detail grid lists a row for each checksum found either from the clipboard or listbox, its associated filename, and type of f=ffp, s=st5, or m=md5. If a match to the checksum is found in the database, then that matched checksum, filename, type, id, and xref are displayed on the same row. If the checksum matches more than one id in the database, those matches are also displayed in a following row and highlighted. All matches to an id xref are also highlighted. For each id found, a reverse lookup and comparison on the ids checksum to the checksums given is done and any missing checksums are displayed in a following row and highlighted. If the checksum lookup was from the listbox, then all the checksum rows within a group of files will be surrounded by a start and end row indicating the group highlighted in light blue.
Double clicking on a row with an id will attempt to open a browser page of that id from the internet. You can also do the same by right clicking and choosing the context menu option “display web page for id in browser”. For the gd etree database, if an id was not found in the database for the checksum, the double click will do a search at etree for the checksum in the grid row.
The context menu item “calculate new folder name from id of current row” is a more advanced feature used in folder renaming. If the lookup from listbox was unable to determine an id with a failure reason on the rename tab of either “incomplete”, “extra checksums”, or “multiple found”, you may want to check out the detail grid to try to resolve whether your folder name may be worthy of an id. If you think so, you can set the new calculated folder name to include an id by selecting a checksum row with the id you choose and making it the current row and then selecting this menu item. Remember to run the rename function on the rename tab.
The context menu item “jump to rename folders tab”, lets you jump to the rename folders tab without accidentally clicking on “lookup from clipboard” and losing any ongoing renaming work in that grid.
Grey Blue – group start and end row
Light Orange – checksum not found in database
Light Turquoise Blue – checksum was matched to more than one id in the database
Pink – found id has additional checksums in the database that were missing in the lookup set
Light Green – found id has an xref number
In the losslessbob database, an id may have multiple sets of checksums associated with it and the extra sets are assigned an xref number. Those with a non-zero xref are usually different more flawed cdr rips of the original id that have been torrented.
In the gd etree database, the xref is used to differentiate among the various checksum files that might be listed for a shnid. Files named “shn-md5”, “st5, “dvd-md5” and “dvda-md5” always get assigned an xref of 0. If a “flac-md5” or “ffp” file is also present when there is a “shn-md5” file, they get assigned an xref of 1. Otherwise, without a “shn-md5” file present, a “flac-md5” and “ffp” will get assigned an xref of 0. Any of the above listed files with a prefix of “orig-“ will be assigned an xref of 2. All files named something else get assigned an xref number starting with 11 where the number minus 10 represents the sequence that they are displayed on the web page. Unfortunately for this program, there was no standard in naming checksum files in the etree database to help in matching them well into an xref. There are just some common frequently used ones that could be matched in the program, so this is not exact all the time, and may require the user to do more research on the web site when encountering an xref. Some of the common ones that are also assigned to xref 0 as a shn type are "d1", "d2", "d3", "d4", "shn", "d1-shn", "d2-shn", "d3-shn", "d4-shn", "shn.md5","shn_md5", "d1shn", "d2shn", "d3shn", "d4shn", "all", "both", "only", "md5-all", "md5.all", "md5_all", "md5 all", "shnall", "only.shn", "d3shns", "shns", "shns-all", "shns_all", "all.shn", "d1 shn", "d1fixed", "d2fixed", "d3fixed","md5 all", and "d1-md5". Some of the common ones that are also assigned to xref 0 or 1 as a flac type are "flac", "flac d1", "flac d2", "flac d3", "ffp.all", "ffp-all", "flac.ffp", "d1-ffp", "d2_ffp", "d3-ffp","all_ffp", "flac fingerprint", "only-flac", and “md5". There are also more like these as new common ones are found with each new band imported.
Following a Lookup Listbox action, this grid will be populated one row for each file group looked up. The row is based on the parent folder for the group. The columns are a select for rename checkbox, the folder name to rename, the calculated new folder name, the id found in the lookup, the xref found in the lookup, the reason given why no new calculated folder name is present, and totals of md5, st5, and ffp counts for the groups given, matched, xref, and missing occurrences (see the summary grid description for what these totals mean).
After the lookup, the only rows with a checked checkbox and a calculated new folder name are those where an id was found clear and complete and not already specified in the folder name. You can manually change the checkbox or the new folder name. If you make manual changes to the new name, make sure you check the checkbox also if you want it applied during the rename process.
The format of the calculated new folder name is different for each database. For both, the id is inserted at the end just before common folder extensions. For the losslessbob database, the format for inserting the id is “–LB####” and if there is an xref, then “–LB####-xref####”. For the etree database the format for the inserted id is “.#######” and any “.xxxxxx” type strings are removed. In calculating the new name, the filename is searched for existing ids already in the name so that a warning can be issued if different or so the rename can be skipped if the id is already present. Sometimes the existing folder name includes a number that is not an id but is in the same format as an id, and those must be dealt with manually. The calculation of the new folder name works best when etree naming convention standards have been followed in naming the current folder name. Although the calculation is flexible and may work with some other folder naming standards, it will not work will all. It needs the folder name to have a date before any existing id so it can differentiate between them or else you may not like the calculated new name and need to manually adjust it.
If a folder has a reason of "folder has different id", the rename to folder can be calculated from that folder name and found id by using the context menu on the rename folders grid. You can click on “calculate new folder name from id of current row – replace” to replace the last number in the folder name with the found id or “calculate new folder name from id of current row – append” to append the found id to the folder name
Right clicking on this grid brings up a context menu which operates on the current selected row. You must first left click on a row for it to be the current row and the current row is indicated on the far left of the grid. The “display web page for id in browser” menu item will try to open the web browser page associated with that id on the internet. The “jump to detail for folder group” menu item will return you to Lookup tab page and position the detail grid to show that groups detail. If you are on the etree database, you will have the option of “lookup last checksum at etree” menu item which will try to find the id on the web using the last ffp and md5 checksum for the group and if found set the folder to be renamed.
Double clicking on the reason column for a row will jump to the detail on the lookup tab for that folder name similar to the context menu option. Double clicking on all other columns for a row will still attempt to display the web page for the row in the browser
This will attempt to rename all folders with the select column checked and a new folder name. It is suggested that you proceed slowly at first to make sure this rename function works as you expect it to.
You will have this option only if you are on the gd etree database. It will attempt to lookup an id by checksum for each row that has a reason column of “no id found”. This lookup will be across the internet at the etree web site using the last md5 and/or ffp/st5 checksum found for the folder. You will not get verification of all the checksums but just one to match. If a checksum is matched, then the select column will be checked and the calculated new folder name will be set with the found id. This feature is useful if you have folders without an id which are more recent than the database. There is a 5 second delay between web lookups to better share the web resource.
You will have this option only if you are on the gd etree database. Since the Lookup All "no id found" at etree feature can take a while depending how busy etree is and how many requests there are to look up in the batch, by clicking this button the batch lookup will stop after the current row lookup is completed.
This is used to set a folder from which to start the file or folder dialog for selection into the listbox. Otherwise the dialogs start from the desktop.
If both databases are detected, you must choose one before doing a lookup. The database selected is displayed in the program’s title bar. If you need to lookup from both, you can switch back and forth using this option or run 2 instances of the program each set on a different database. The databases may also be switched from the listbox context menu.
The LosslessBob database is derived from the lbdir files on that site.
The gd etree database is derived from the set of md5, st5, and ffp files available at the site. An attempt was made to include all bands that the major gd band members performed in during their careers. There are initially shnids from 39 bands.
It is unknown whether or how often future snapshot updates of the databases will be created and distributed.
To allow for possible future or alternate locations of the LosslessBob web pages, the web address to go to on a grid row doubleclick can be re-fashioned around the LB number using these fields. For example to get to the current location, the fields would be (without the quotes) “http://jokerman.org.uk/lb/detail/LB-”, “00000”, “.html”.
· Save forms positions and sizes when saving (this setting is not saved)
If you want to save the position and size of the open forms for the next time you open the program, check this checkbox and then save the settings.
This will save any changes and they will be available for this instance’s session and all future instances, but not concurrently running instances
This resets all settings back to their last saved settings
This will save any settings changes made and store them in the file “\Checksum_Lookup\settings\ Checksum_Lookup.ini”. This provides a backup for your settings. It is possible on future program updates or a reinstall that your settings will be lost. They may then be reloaded from this file. Also you may wish to transfer your settings to another computer or temporarily try other settings knowing you can reload the saved ones.
If you have previously saved your settings to a file, this will reload your settings from “Checksum_Lookup\settings\ Checksum_Lookup.ini”
The version number of the program
This displays the last id in the databases. Ids are generally issued in ascending order so you can determine how current your local database is
Displays this Help file
You need to make sure “Microsoft Windows .Net Framework 3.5 Service Pack 1” is installed according to the installation instructions
You need to make sure the database files are installed in the correct directory with the correct name according to the installation instructions
You need to make sure “Microsoft SQL server compact 3.5 SP1” is installed according to the installation instructions
Not all checksums are in the database. If you have more than one database installed, make sure you are searching the correct one for your checksum.
· Start Position of Form is on a monitor that no longer exists
Add this parameter to the end of the command line that launches the program so it is like (keep your old folder location):
checksum_lookup.exe -ignore_start_positions
If you run this program from a shortcut, right click on the shortcut and bring up the properties panel and add “ -ignore_start_positions” to the end of the target field without the quotes and a leading space, Click Apply button and then the Ok button. And then run it and save the new start positions. Then shut it down and go back and remove the new parameter you added.
· a yahoo user support group is at
http://tech.groups.yahoo.com/group/Checksum_Lookup/
· Version 1.06 – September 18, 2011
o New program icon
o Able to save form position and size for future starts
o Able to override saved form position with “ -ignore_start_positions” startup parameter
o For gd database users, adapt to a change at etree.org on how the lookup id is returned so the folder name is back to being renamed properly
o For gd database users, if multiple matches are found for a checksum but those matches all have the same id but just another alternate xref, then allow for that folder name to be updated with the id
· Version 1.07 – October 8, 2011
o For gd database users, relax restrictions on setting the folder renaming id so only have to match either all ffp or all md5 but not both. This makes it easier to rename folders where etree only had the md5 or the ffp but not both
o Filter the listbox on the Lookup Tab to only files ending in “_mychecksums.txt”. Set this filter on to interface with output from MyChecksums software program. The filter turns on or off through the context menu on the listbox. Using this filter helps in folder renaming when the folder has old or extra checksums in it that interfere with the lookup getting a clean match. With this filter on, you only match to the checksums of the files you have when the “_mychecksums.txt” was generated. For more information go to http://tech.groups.yahoo.com/group/MyChecksums/
o Add the ability on the Rename Folder tab rename folders grid to set the rename to folder to the found folder id when have a folder that appears to have a different id. This is done from the context menu on the rename folders grid. You can replace the last number in the folder name with the found id or append the found id to the folder name.
o For folder renaming purposes, improve the matching when have multiple ids matching the checksums for a folder. Now the first id to get a complete match to the checksums in the folder is selected and available for renaming where before just got a “multiple ids” message and had to manually select which id to use in the rename.
o On the Rename Folders Tab, double clicking on the reason column for a row will now jump to the detail on the lookup tab for that folder name. Double clicking on all other columns for a row will still attempt to display the web page for the row in the browser.
· Version 1.08 – October 17, 2011
o When inserting an id for renaming a folder, prior versions did not insert the id before the folder extension when the folder extension included a capital letter. This is corrected.