CybTekSol

My LATEST Universal SnapFiles.com template: ****PLEASE TEST it and provide suggestions for improvement if possible! ****NOTE: This template uses the application's DOWNLOAD page URL (NOT the information page URL)! In other words, go to the application's info page URL at SnapFiles in your web browser and click the 'Download' button, cancel the download and copy the URL from your browser's address bar for THAT page. 1) There are two user defined global variables used, download_url={download_url} (used as an alternative to {url}) and root_folder=YOUR Download Destination ROOT Folder (for example C:\Latest Downloads). 2) The latest version of Ketarin (.0.9.9.22) released January 18, 2009 or later is required as this revision uses the new 'placeholder' feature implemented by Florian. 3) The template follows: <?xml version="1.0" encoding="utf-16"?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <DownloadBeta>Avoid</DownloadBeta> <DownloadDate xsi:nil="true" /> <VariableChangeIndicator>{version}</VariableChangeIndicator> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <HttpReferer /> <Variables> <item> <key> <string>version</string> </key> <value> <UrlVariable> <VariableType>RegularExpression</VariableType> <Regex><title>.*?(\d[.].*?)[ ].*?</title></Regex> <Url><placeholder name="Application Download Page URL from SnapFiles" /></Url> <Name>version</Name> </UrlVariable> </value> </item> <item> <key> <string>download_url</string> </key> <value> <UrlVariable> <VariableType>RegularExpression</VariableType> <Regex>url=(.*?)"</Regex> <Url><placeholder name="Application Download Page URL from SnapFiles" /></Url> <Name>download_url</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand /> <Category>[iNSERT CATEGORY HERE]</Category> <SourceType>FixedUrl</SourceType> <DeletePreviousFile>false</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <LastUpdated xsi:nil="true" /> <TargetPath>{root_folder}\{category}\{appname:replace: :_}_v{version:replace: :_}.{url:ext}</TargetPath> <FixedDownloadUrl>{download_url}</FixedDownloadUrl> <Name><placeholder name="Application Name" /></Name> </ApplicationJob> </Jobs> To use it: 1) Define the user defined global variables under 'Settings' in Ketarin to YOUR preferences 2) Edit the template externally to reflect these modifications (if you choose not to use mine) and change the <TargetPath> line to YOUR preferences 3) Then: a) Import it (Obviously) Enter the information needed as requested as a result of the 'placeholder' prompts... it will ask for the application's DOWNLOAD page URL (NOT the information page URL) and the application's name c) Modify the Category to your preferences d) Change the 'Advanced Settings' to your preferences e) VERIFY...VERIFY...VERIFY that ALL regex matches WORK and MAKE ADJUSTMENTS to them as necessary! ****NOTE: IF YOU WISH TO CREATE A TEMPLATE from one of YOUR XML exports: 1)It is important that the Guid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" string be removed from the XML export to use it as a template, otherwise you'll just keep overwriting the previously saved application entry that was created using the imported template 2) The <DownloadDate> line must be changed to <DownloadDate xsi:nil="true" /> 3) The <LastUpdated> line must be changed to <LastUpdated xsi:nil="true" /> 4) Remove the entire <PreviousLocation> line or replace it with <PreviousLocation /> 5) Create 'placeholder' entries as needed (see THIS thread) I hope this helps someone... CybTekSol

I still use it... simply make small adjustments to the regex if needed. Thanks for labeling me a power user! I try hard. I understand and thanks for considering it.

Hey Florian, What about coloring the BACKGROUND of the entire regex match?

I'm afraid it will be near impossible to develop a consistent template for Softpedia for the very reasons that Kerberos has so eloquently described. It did not take long for me to realize that there are too many variations in the way they list and serve applications! Even with if...then...else statements it would be quite an undertaking and above my skill level. Yikes! HOWEVER, on a brighter note, the template that I have developed for SnapFiles and still testing is working WELL!

Florian... any response to my reply above or am I off base?

I developed one yesterday and I'm still testing it... I'll post it when it has been thoroughly vetted.

Not that I can glean while actively modifying the regex in the 'Edit Variables' dialog window... and the offending line-feed is not displayed in red obviously. I am unaware of its presence until I attempt to update (download) the application and I receive 'The specified target path is not valid' error due to the illegal character. Is there a feature that can be implemented to discover this while in the 'Edit Variables' dialog window?

Florian... now I feel like a failure. A solution to this escapes me at the moment, so I've included a 'disclaimer' in the LATEST revision above that takes advantage of the 'placeholder' feature you implemented! If only I were as gifted.

You're welcome. Just don't forget to do it in the code above Stalker, I've modified the code above to reflect my latest changes... and provided a 'disclaimer' since Florian found a flaw in my regex...

The actual regex match... I like your sense of humor.

I just finished a global search and replace of my XML exports replacing occurrences of {url} and associated tags with {download_url}... hope this solves the issue. Thanks!

So this will be a problem until Flo adds the new variable {ext} or {extension} to Ketarin... I noticed that it doesn't seem to be implemented in version .0.9.9.22 so we'll just have to wait, eeh?

I know you're probably tired of my 'right-click' fetish Florian, please forgive me as I have another. 1) The ability to 'right-click' copy to clipboard the highlighted text in the 'html code' display of the 'Edit Variables' window for externally creating and modifying regex scrapes. This is a minor issue for me as I currently use control-c... I can wait, it would only improve my productivity slightly. More importantly: 2) For troubleshooting failed scrapes, some method of pasting the scrape results to the clipboard or displaying the entire scrape result including non-standard characters, line feeds, etc.. I'm currently having a problem developing a 'universal scrape' for a template as it is picking up these but not displaying them highlighted in color (obviously). Is there some way around this? Maybe a 'Paste scrape results to clipboard' button or right-click option... Does this make sense to you? I'm a little puzzled as to how to overcome this.

Can you clarify this for me as I'm not experiencing any problems with this. Am I missing something or just slightly confused (has happened before)?

I may have to delay this as they have changed their search engine's posted html and I am pulling my hair out trying to refine a regular expression scrape that will work consistently.

Noted... will do...

If you have successfully created an app entry in Ketarin that has the potential to work 'universally' for other apps hosted by the particular site the app is downloaded from (utilizing the power of regular expressions within Ketarin), simply export that single entry to an XML file named considering the site's name (i.e. MajorGeeks-Template.xml) and edit it externally using NotePad2 (freeware) as Florian suggests if my memory serves me correctly... ****NOTE: IF YOU WISH TO CREATE A TEMPLATE from one of YOUR XML exports: 1)It is important that the Guid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" string be removed from the XML export to use it as a template, otherwise you'll just keep overwriting the previously saved application entry that was created using the imported template 2) The <DownloadDate> line must be changed to <DownloadDate xsi:nil="true" /> 3) The <LastUpdated> line must be changed to <LastUpdated xsi:nil="true" /> 4) Remove the entire <PreviousLocation> line or replace it with <PreviousLocation /> 5) Create 'placeholder' entries as needed (see THIS thread) This works with the latest version of Ketarin (0.9.9.22) released January 18, 2009... If Florian changes the XML tag structure that Ketarin writes and utilizes this WILL HAVE TO BE MODIFIED obviously. Other entries can be customized to suit your tastes, but be careful and verify all is well with your creation by importing it... modifying it... then saving it as a new app entry for at least two different apps. If you fail to remove the 'Guid=' entry as in 1) above, the first app WILL be overwritten and lost. GOOD LUCK! I hope this helps someone in their quest for generic templates... CybTekSol

My LATEST Universal MajorGeeks.com (32-bit Apps Only) template: 1) There are three user defined global variables used, redir1={redir1} (short for re-direct 1) and download_url={download_url} (used as an alternative to {url}) and root_folder=Insert YOUR download destination ROOT folder here (for example C:\Latest-Apps). 2) The latest version of Ketarin (.0.9.9.22) released January 18, 2009 or later is required as this revision uses the new 'placeholder' feature implemented by Florian. 3) The template follows: <?xml version="1.0" encoding="utf-16"?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <DownloadBeta>Avoid</DownloadBeta> <DownloadDate xsi:nil="true" /> <VariableChangeIndicator>{version}</VariableChangeIndicator> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <HttpReferer /> <Variables> <item> <key> <string>version</string> </key> <value> <UrlVariable> <VariableType>RegularExpression</VariableType> <Regex>((?:\d+?\.\d+?.*?|\w+?\s\d\d\,\s\d{4})(?=</title>)|(?<=Date:.*?>)(?:\d{4}-\d{2}-\d{2}).*?(?=<))</Regex> <Url><placeholder name="Application Info Page URL from Major Geeks" /></Url> <Name>version</Name> </UrlVariable> </value> </item> <item> <key> <string>redir1</string> </key> <value> <UrlVariable> <VariableType>RegularExpression</VariableType> <Regex><A href="(downloadget.php\?id=[\d]+&file=(?:10|9)&[a-z]+=\w+)"></Regex> <Url><placeholder name="Application Info Page URL from Major Geeks" /></Url> <Name>redir1</Name> </UrlVariable> </value> </item> <item> <key> <string>download_url</string> </key> <value> <UrlVariable> <VariableType>RegularExpression</VariableType> <Regex><META HTTP-EQUIV="Refresh".*?URL=(.*?)"></Regex> <Url>http://majorgeeks.com/{redir1}</Url> <Name>download_url</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand /> <Category>[iNSERT CATEGORY HERE]</Category> <SourceType>FixedUrl</SourceType> <DeletePreviousFile>false</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <LastUpdated xsi:nil="true" /> <TargetPath>{root_folder}\{category}\{appname:replace: :_}_v{version:replace: :_}.{url:ext}</TargetPath> <FixedDownloadUrl>{download_url}</FixedDownloadUrl> <Name><placeholder name="Application Name" /></Name> </ApplicationJob> </Jobs> Note** I use PSPad to edit my XML exports... Florian suggests NotePad2 if my memory serves me correctly... both are freeware and I like both. NotePad2 is lighter and quicker in my tests, so I will probably switch. To use it: 1) Define the user defined global variables under 'Settings' in Ketarin to YOUR preferences 2) Edit the template externally to reflect these modifications (if you choose not to use mine) and change the <TargetPath> line to YOUR preferences 3) Then: a) Import it (Obviously) Enter the information needed as requested as a result of the 'placeholder' prompts... it will ask for the application's information page URL and the application's name c) Modify the Category to your preferences d) Change the 'Advanced Settings' to your preferences e) VERIFY...VERIFY...VERIFY that ALL regex matches WORK and MAKE ADJUSTMENTS to them as necessary! ****NOTE: IF YOU WISH TO CREATE A TEMPLATE from one of YOUR XML exports: 1)It is important that the Guid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" string be removed from the XML export to use it as a template, otherwise you'll just keep overwriting the previously saved application entry that was created using the imported template 2) The <DownloadDate> line must be changed to <DownloadDate xsi:nil="true" /> 3) The <LastUpdated> line must be changed to <LastUpdated xsi:nil="true" /> 4) Remove the entire <PreviousLocation> line or replace it with <PreviousLocation /> 5) Create 'placeholder' entries as needed (see THIS thread) ****ADDENDUM: For 32-bit Applications ONLY! The servers that are most reliable for me (Servers 10 & 9) are selected with the file=(?:10|9) portion of the {redir1} regex. These can be adjusted to use the servers that are most reliable for you. ****CREDITS: MANY THANKS to FranciscoR for dramatically improving the {version} scrape which FranciscoR has graciously allowed me to use in this updated version of my template! It appears that FranciscoR's MajorGeeks.com template will handle BOTH 32-bit and 64-bit apps from their site... so you may want to give it a try! I hope this helps someone... CybTekSol

@MadDog, I'll try to get my 'BetaNews FileForum template' that uses their search engine posted in the new 'Templates' sub-forum that Flo created within a couple of days.

THANKS Florian!! And created within minutes... now that's service!

Ketarin users wishing to share custom templates that they have developed and tested to work with software distribution or developer sites (especially the complex and/or difficult sites) are invited to share them here. Hopefully sharing them will aid everyone in their usage of Ketarin and showcase its power and flexibility! Please include a brief textual description and overview of each template with the pasted XML export contents. Also, Florian has requested that we use [ code ] tags to produce a 'code box' (SEE BBCode HELP) when posting the templates (which I should have done from the start...duh. ENJOY!!

@appyface, I use an asterisk (*) in front of the name entry in the templates that I import which keeps them at the top of the sort until the templates are modified and completed... works for me... your mileage may vary!

@MadDog, I'm glad I was able to help. I use the {redir1} variable to form the 'Contents from URL:' (hxxp://majorgeeks.com/{redir1}) that the app download URL is scraped from... take a look at the url entry in the 'edit variables' dialog after importing the template. Without this, my template fails to scrape this URL for download unless I'm missing something obvious (has happened before). As for the 'Share Your Templates' thread... I discussed this with Florian before when I suggested the 'Tips and Tutorials' section last month and if Flo doesn't have a problem with creating another section for this purpose under the 'Resources' category of the forum, that would probably be the best way to accomplish it. I'm in favor of it, so, Florian if you are monitoring this thread, please give us your thoughts. Otherwise... we can start a thread under 'Tips and Tutorials'... MadDog, would you be willing to help with moderation of a separate 'Share Your Templates' section if Flo chooses to go that route? Obviously either way the best that could be done is to post the XMLs as text (which may wrap at posting) and not the files themselves unless Flo has a different solution.

@Stalker, I agree completely! Using Ketarin, I'm able to fetch the latest versions of nearly 500 apps I use to build, customize and maintain a wide variety of PCs... I can do in minutes what use to take hours! Ketarin (and Flo) are a blessing to me indeed!! That's one heck of a WOW FACTOR for me!!

@MadDog I apologize for the delay in getting back to you regarding my MajorGeeks template. I got sidetracked last night testing my new template for BetaNews FileForum... I have successfully created a template that uses their search engine... all you have to feed it is the app name or if you get more than one return to scrape, the author and app name. It seems to work consistently. I'll try to find the time to share that one later if anyone is interested. This is my LATEST MajorGeeks template: 1) There are three user defined global variables used, redir1={redir1} (short for re-direct 1) and download_url={download_url} (used as an alternative to {url}) and root_folder={Insert YOUR download destination ROOT folder here} (for example C:\Latest Downloads). 2) The latest version of Ketarin (.0.9.9.22) released January 18, 2009 or later is required as this revision uses the new 'placeholder' feature implemented by Florian. 3) The template follows: <?xml version="1.0" encoding="utf-16"?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <DownloadBeta>Avoid</DownloadBeta> <DownloadDate xsi:nil="true" /> <VariableChangeIndicator>{version}</VariableChangeIndicator> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <HttpReferer /> <Variables> <item> <key> <string>version</string> </key> <value> <UrlVariable> <VariableType>RegularExpression</VariableType> <Regex><title>\|MG\| [a-z| ]+([\d|.| ||-||_|a-z]+)</title></Regex> <Url><placeholder name="Application Info Page URL from Major Geeks" /></Url> <Name>version</Name> </UrlVariable> </value> </item> <item> <key> <string>redir1</string> </key> <value> <UrlVariable> <VariableType>RegularExpression</VariableType> <Regex><A href="(downloadget.php\?id=[\d]+&file=[\d][\d]&[a-z]+=[\d|a-z]+)"></Regex> <Url><placeholder name="Application Info Page URL from Major Geeks" /></Url> <Name>redir1</Name> </UrlVariable> </value> </item> <item> <key> <string>download_url</string> </key> <value> <UrlVariable> <VariableType>RegularExpression</VariableType> <Regex>URL=(http://files[\d].majorgeeks.com/files/[\d|a-z]+/[a-z]+/[a-z|\d|_|.]+)"></Regex> <Url>http://majorgeeks.com/{redir1}</Url> <Name>download_url</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand /> <Category>[iNSERT CATEGORY HERE]</Category> <SourceType>FixedUrl</SourceType> <DeletePreviousFile>false</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <LastUpdated xsi:nil="true" /> <TargetPath>{root_folder}\{category}\{appname:replace: :_}_v{version:replace: :_}.{url:ext}</TargetPath> <FixedDownloadUrl>{download_url}</FixedDownloadUrl> <Name><placeholder name="Application Name" /></Name> </ApplicationJob> </Jobs> Note** I use PSPad to edit my XML exports... Florian suggests NotePad2 if my memory serves me correctly... both are freeware and I like both. NotePad2 is lighter and quicker in my tests, so I will probably switch. To use it: 1) Define the user defined global variables under 'Settings' in Ketarin to YOUR preferences 2) Edit the template externally to reflect these modifications (if you chose not to use mine) and change the <TargetPath> line to YOUR preferences 3) Then: a) Import it (Obviously) Enter the information needed as requested as a result of the 'placeholder' prompts... it will ask for the application's information page URL and the application's name c) Modify the Category to your preferences d) Change the 'Advanced Settings' to your preferences NOTE**** IF YOU WISH TO CREATE A TEMPLATE from one of YOUR XML exports: 1)It is important that the Guid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" string be removed from the XML export to use it as a template, otherwise you'll just keep overwriting the previously saved application entry that was created using the imported template 2) The <DownloadDate> line must be changed to <DownloadDate xsi:nil="true" /> 3) The <LastUpdated> line must be changed to <LastUpdated xsi:nil="true" /> 4) Remove the entire <PreviousLocation> line 5) Create 'placeholder' entries as needed (see THIS thread) ADDENDUM**** Florian has found an application hosted by MajorGeeks (Process Lasso - 64bit Version) that this template will NOT work with... unfortunately, I do not see an easy solution to this problem. This is the nature of a 'universal' template... I DO NOT present this template as a perfect creation (far from it). I offer this in an attempt to decrease repetitive entries for an application downloaded from MajorGeeks and to stir the creative juices in Ketarin users... your mileage may vary. It WILL work with most applications at MajorGeeks for the moment... when you encounter one in which it does not... tweak the regular expressions as needed. I hope this helps... CybTekSol