CybTekSol Posted February 20, 2009 Report Share Posted February 20, 2009 My LATEST Universal Softpedia.com Template (TEST): ****PLEASE TEST it and give me some feedback on this one if you refine it or have significant problems citing specific examples. ****NOTE: This template uses the application's INFO page URL. 1) There are two user defined global variables used, download_url={download_url} (used as an alternative to {url}) and root_folder=Insert YOUR download destination ROOT folder here (for example C:\Latest-Apps). 2) The latest version of Ketarin (1.0.3) released February 24, 2009 or later is required to overcome their site changes for 'user-agent'. 3) The template follows: <?xml version="1.0" encoding="utf-16"?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <DownloadBeta>Avoid</DownloadBeta> <DownloadDate xsi:nil="true" /> <VariableChangeIndicator /> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <HttpReferer>{redir1}</HttpReferer> <Variables> <item> <key> <string>version</string> </key> <value> <UrlVariable> <VariableType>RegularExpression</VariableType> <Regex><title>Download.*?(\d.*?|\d.*?[ ]Build[ ]\d.*?|\d.*?[ ]Beta[ ]\d.*?)(?:\s\/\s|\s-\s)</Regex> <Url><placeholder name="Application Info Page URL from Softpedia" /></Url> <Name>version</Name> </UrlVariable> </value> </item> <item> <key> <string>redir1</string> </key> <value> <UrlVariable> <VariableType>RegularExpression</VariableType> <Regex>(?<=download_bkg.gif.*?<a href=").*?(?=")</Regex> <Url><placeholder name="Application Info Page URL from Softpedia" /></Url> <Name>redir1</Name> </UrlVariable> </value> </item> <item> <key> <string>download_url</string> </key> <value> <UrlVariable> <VariableType>RegularExpression</VariableType> <Regex>(?<=DOWNLOAD LOCATIONS.*?<table width=".*?<a href=").*?(?=")</Regex> <Url>{redir1}</Url> <Name>download_url</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand /> <Category>[iNSERT CATEGORY HERE]</Category> <SourceType>FixedUrl</SourceType> <DeletePreviousFile>false</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <LastUpdated xsi:nil="true" /> <TargetPath>{root_folder}\{category}\{appname:replace: :_}_v{version:replace: :_}.{url:ext}</TargetPath> <FixedDownloadUrl>{download_url}</FixedDownloadUrl> <Name><placeholder name="Application Name" /></Name> </ApplicationJob> </Jobs> To use it: 1) Define the user defined global variables under 'Settings' in Ketarin to YOUR preferences 2) Edit the template externally to reflect these modifications (if you choose not to use mine) and change the <TargetPath> line to YOUR preferences 3) Then: a) Import it (Obviously) Enter the information needed as requested as a result of the 'placeholder' prompts... it will ask for the application's DOWNLOAD page URL (NOT the information page URL) and the application's name c) Modify the Category to your preferences d) Change the 'Advanced Settings' to your preferences e) VERIFY...VERIFY...VERIFY that ALL regex matches WORK and MAKE ADJUSTMENTS to them as necessary! I have had a 80-90% success rate with this template; in instances where it failed, I simply tweak the regular expression. CybTekSol Link to comment Share on other sites More sharing options...
FranciscoR Posted February 20, 2009 Report Share Posted February 20, 2009 Already in my to-do list =D... but you have <HttpReferer> with {redir} and only a {redir1} variable ? Link to comment Share on other sites More sharing options...
CybTekSol Posted February 20, 2009 Author Report Share Posted February 20, 2009 but you have <HttpReferer> with {redir} and only a {redir1} variable ?A typo during an external edit... corrected now... THANKS for catching it! Link to comment Share on other sites More sharing options...
FranciscoR Posted February 20, 2009 Report Share Posted February 20, 2009 (edited) Nice indeed, 20-02-2009 12:32:34: Everest: Replacing {redir1} in '{redir1}' with 'http://www.softpedia.com/progDownload/EVEREST-Ultimate-Edition-Download-30373.html' 20-02-2009 12:34:04: Update started with 1 application(s) 20-02-2009 12:34:08: Everest: Replacing {redir1} in '{redir1}' with 'http://www.softpedia.com/progDownload/EVEREST-Ultimate-Edition-Download-30373.html' 20-02-2009 12:34:10: Everest: Replacing {download_url} in '{download_url}' with 'http://download.softpedia.com/dl/69b0c84876b9d5f6361db8e5f5e3095e/499ea33d/100030373/software/system/info/everestultimate500.zip' 20-02-2009 12:34:11: Everest: Checking if update is required... 20-02-2009 12:34:11: Everest: Update required, 'D:\Programas\Test\everestultimate500.zip' does not yet exist 20-02-2009 12:34:13: Everest: Replacing {version} in '{version}' with '5.00.1664 Beta / 5.00.1650 Free Trial' 20-02-2009 12:34:13: Update finished Will I actually DL Beta or Trial ? Edited February 20, 2009 by FranciscoR Link to comment Share on other sites More sharing options...
FranciscoR Posted February 20, 2009 Report Share Posted February 20, 2009 Trial. {version} could be adjusted... ? Link to comment Share on other sites More sharing options...
CybTekSol Posted February 20, 2009 Author Report Share Posted February 20, 2009 Trial. {version} could be adjusted... ? Go for it! Suggestions? Link to comment Share on other sites More sharing options...
FranciscoR Posted February 20, 2009 Report Share Posted February 20, 2009 Not as speaking... =D maybe later ? Nice template ! Link to comment Share on other sites More sharing options...
CybTekSol Posted February 20, 2009 Author Report Share Posted February 20, 2009 @FranciscoR, This is an example of one of the Softpedia inconsistencies we discussed in the past. This particular example would require some major regex tweaking... OBVIOUSLY! The 80-90% success rate is still better than starting from scratch with every app there though... know what I mean? I don't ever see an infallible template for their site. Link to comment Share on other sites More sharing options...
FranciscoR Posted February 20, 2009 Report Share Posted February 20, 2009 {download_url} seems to be working just fine... I'll be back with some suggestions for {version}. Link to comment Share on other sites More sharing options...
CybTekSol Posted February 20, 2009 Author Report Share Posted February 20, 2009 Made a small temporary tweak to the {version} regex... looking forward to your input. Link to comment Share on other sites More sharing options...
FranciscoR Posted February 20, 2009 Report Share Posted February 20, 2009 Nope, now I have beta version with a trial DL 20-02-2009 13:28:38: Update started with 1 application(s) 20-02-2009 13:28:40: Everest: Replacing {redir1} in '{redir1}' with 'http://www.softpedia.com/progDownload/EVEREST-Ultimate-Edition-Download-30373.html' 20-02-2009 13:28:42: Everest: Replacing {download_url} in '{download_url}' with 'http://download.softpedia.com/dl/9850f0f1396405d88f2e6437aadbc0b7/499eb006/100030373/software/system/info/everestultimate500.zip' 20-02-2009 13:28:54: Everest: Checking if update is required... 20-02-2009 13:28:54: Everest: Update required, 'D:\Programas\Test\everestultimate500.zip' does not yet exist 20-02-2009 13:28:57: Everest: Replacing {version} in '{version}' with '5.00.1664 Beta' 20-02-2009 13:28:57: Update finished 3dsMax {version} is not 100% either 20-02-2009 13:30:43: Update started with 1 application(s) 20-02-2009 13:30:46: 3dsMax: Replacing {redir1} in '{redir1}' with 'http://www.softpedia.com/progDownload/3D-Studio-Max-Download-15976.html' 20-02-2009 13:30:48: 3dsMax: Replacing {download_url} in '{download_url}' with 'http://download.softpedia.ro/dl/aeb999d1855cbaa2dc62e88aa7863a2f/499eb084/100015976/software/MULTIMEDIA/GRAPHIC/Autodesk3dsMax2009_ENU_TrialDownload.exe' 20-02-2009 13:30:50: 3dsMax: Checking if update is required... 20-02-2009 13:30:50: 3dsMax: Update required, 'D:\Programas\Test\Autodesk3dsMax2009_ENU_TrialDownload.exe' does not yet exist 20-02-2009 13:30:51: 3dsMax: Replacing {version} in '{version}' with '3ds Max 2009 11.0 Free Trial' 20-02-2009 13:30:51: Update finished It needs more time... no pressure from me. I'll be back after some testing on {version}. =) Link to comment Share on other sites More sharing options...
FranciscoR Posted February 20, 2009 Report Share Posted February 20, 2009 As is, it's a very, very, very good template and I must say it already is one of my personal favorites. {version} is good enought to track updates, and I am amazed on how easily Softpedia allowed you to manage the page redirection... I don't have such help on my VersionTracker template, but {version} is so much easy... there's no perfect web design(er) ! Link to comment Share on other sites More sharing options...
CybTekSol Posted February 20, 2009 Author Report Share Posted February 20, 2009 Nope, now I have beta version with a trial DLI just took a quick stab at it when I made that small tweak as my time is limited... didn't have a chance to verify the match between version and DL... I should have known they would be reversed! Link to comment Share on other sites More sharing options...
FranciscoR Posted February 20, 2009 Report Share Posted February 20, 2009 Me2, running out of time right now. But I do understand the SIZE of your problem... I think that for the moment I will be using 'Last Updated:' row, e.g. 'November 3rd, 2008, 14:48 GMT', properly customized, to track versions. Easy to-do and less unpredictable. P.S. You're using HTTP referer for any specific reason ? Seems to work fine without it. Link to comment Share on other sites More sharing options...
CybTekSol Posted February 20, 2009 Author Report Share Posted February 20, 2009 P.S. You're using HTTP referer for any specific reason ? Seems to work fine without it.If my memory serves me correctly, it was an issue for a couple of the apps I get from there for some freaky reason, so I left it in the template... can't remember the specifics. Link to comment Share on other sites More sharing options...
FranciscoR Posted February 20, 2009 Report Share Posted February 20, 2009 This is just my personal preference, but until they include a somewhat standardized version format, I will be using ([a-z]+?\s\d{1,2}[a-z]{2}\,\s\d{4}\,\s\d{2}\:\d{2})(?:\sGMT)? . It works for me with every Adobe and Autodesk app and plugin I tested with, and allows me to solve problems like Adobe Photoshop Lightroom 2.3 RC Build 534370 / 2.2 Build 523352. 20-02-2009 16:49:31: Update started with 1 application(s) 20-02-2009 16:49:33: Everest: Replacing {redir1} in '{redir1}' with 'http://www.softpedia.com/progDownload/EVEREST-Ultimate-Edition-Download-30373.html' 20-02-2009 16:49:35: Everest: Replacing {download_url} in '{download_url}' with 'http://download.softpedia.com/dl/fb8af7afa01229399789c3ff567275fc/499edf1b/100030373/software/system/info/everestultimate500.zip' 20-02-2009 16:49:36: Everest: Checking if update is required... 20-02-2009 16:49:36: Everest: Update required, 'D:\Programas\Test\everestultimate500.zip' does not yet exist 20-02-2009 16:49:37: Everest: Replacing {version} in '{version}' with 'February 20th, 2009, 08:12' 20-02-2009 16:49:37: Update finished I'm quite happy with template, thanks a lot CybTekSol. =) Link to comment Share on other sites More sharing options...
CybTekSol Posted February 21, 2009 Author Report Share Posted February 21, 2009 This is just my personal preference, but until they include a somewhat standardized version format, I will be using ([a-z]+?\s\d{1,2}[a-z]{2}\,\s\d{4}\,\s\d{2}\:\d{2})(?:\sGMT)? . It works for me with every Adobe and Autodesk app and plugin I tested with, and allows me to solve problems like Adobe Photoshop Lightroom 2.3 RC Build 534370 / 2.2 Build 523352. I'm quite happy with template, thanks a lot CybTekSol. =) I'm glad you could use it. Would you please take my template, make changes that you have implemented to suit your needs and post it to this same thread to give other users better coverage of Softpedia apps? Thanks in advance... Link to comment Share on other sites More sharing options...
FranciscoR Posted February 21, 2009 Report Share Posted February 21, 2009 Like I said, it's *only* my personal preference. If a user wants to use last updated date instead of file version, all he/she has to do is to copy that regex into the {version} variable. But your template is working great, let users experience it and decide, if a request is forward I will do that. Still, a community input would be most useful to improve it... Let it rest now, no worries, we cannot be too demanding with DL sites such as this. Well done again. Link to comment Share on other sites More sharing options...
CybTekSol Posted February 21, 2009 Author Report Share Posted February 21, 2009 If a user wants to use last updated date instead of file version, all he/she has to do is to copy that regex into the {version} variable.Agreed... I think the average user will not have any problems doing that. Users can create two different versions of the template on their own with a simple copy/paste replacement of the {version} regex with this: ([a-z]+?\s\d{1,2}[a-z]{2}\,\s\d{4}\,\s\d{2}\:\d{2})(?:\sGMT)? either within an external editor, since there are no converted characters (new users see more on this below) or within Ketarin and exporting it as template. When confronted with a situation like this at Softpedia (where multiple versions and download locations are combined), they can import the alternative template which captures the 'Last updated' date and then tweak the {download_url} by inserting an additional anchor from the app's download page for the download location displayed ABOVE the location they actually want to use, such as Softpedia Secure Download \(RO\).*? inserted which would change the {download_url} scrape from: (?<=DOWNLOAD LOCATIONS.*?<table width=".*?<a href=").*?(?=") to: (?<=DOWNLOAD LOCATIONS.*?<table width=".*?Softpedia Secure Download \(RO\).*?<a href=").*?(?=") and in the case of "Everest Ultimate Edition" would capture the third download url (the Beta version). I must caution new users regarding pasting text directly into a template with an external editor as certain characters are converted on export from Ketarin. For example: If you paste the regular expression above into Ketarin it would be exported to a template as: (?<=DOWNLOAD LOCATIONS.*?<table width=".*?Softpedia Secure Download \(RO\).*?<a href=").*?(?=") Notice the < character is converted to < (its hypertext equivalent). This is the case for several other characters as well and can result in template import errors if you are not careful. Just 'food for thought' to give new users a head start! If you are unsure, the safest method is to paste the expression into Ketarin first, then export, and work with the template in an external editor afterwards. CybTekSol Link to comment Share on other sites More sharing options...
CybTekSol Posted February 24, 2009 Author Report Share Posted February 24, 2009 CHANGES at Softpedia on 2-24-09 have apparently broken this template. Link to comment Share on other sites More sharing options...
CybTekSol Posted February 24, 2009 Author Report Share Posted February 24, 2009 CHANGES at Softpedia on 2-24-09 have apparently broken this template. The latest version of Ketarin (1.0.3) released February 24, 2009 solves the issue of their site changes for 'user-agent'. Flo had a QUICK fix for this! Link to comment Share on other sites More sharing options...
FranciscoR Posted May 2, 2009 Report Share Posted May 2, 2009 (edited) Now using <Regex>(??<=\<title\>Download.*?<placeholder name="Stable release ?" value="\/.*?" />\s)(\d+[\.\d+]+?.+?)(?=\s[\-\/\[]<placeholder name="Version format is 1.x.x ?" value="?" />)<placeholder name="Capture date ?" value="|(?:[a-z]+?\s\d{1,2}[a-z]{2}\,\s\d{4}\,\s\d{2}\:\d{2})(?:\sGMT)?" />)</Regex> to capture version, based on my SF template. Works with all my SP jobs, probably will do the same with the {download_url} variable. Or not. =) ----------------------- Edited May 2, 2009 by FranciscoR Link to comment Share on other sites More sharing options...
CybTekSol Posted May 2, 2009 Author Report Share Posted May 2, 2009 Works with all my SP jobs, probably will do the same with the {download_url} variable. Or not. =) Looks promising FranciscoR... I'll do a little experimenting when I have the time. Link to comment Share on other sites More sharing options...
FranciscoR Posted May 2, 2009 Report Share Posted May 2, 2009 (edited) I have a restricted universe here, that's no miracle for sure. 'Stable release' should be used only when you are presented multiple versions in a format like version #1/version #2 (#2 is usually stable, whereas #1 is beta). If #3 and #4 exist, adjust regex. 'Version format' excludes 'free trials' stuff from capture, but remenber using it will also exclude 'builds', 'betas', etc. from capture (i'm using it for stable releases). As for date, I could just remove it but maybe tomorow i need it, so i prefer to delete it upon pop-up. =) ------------------------- Edited May 3, 2009 by FranciscoR Link to comment Share on other sites More sharing options...
FranciscoR Posted May 3, 2009 Report Share Posted May 3, 2009 (edited) Try this one instead. You need to escape ( and ) with the "Paste link text" placeholder: 'Softpedia Secure Download \(US\)' or 'Softpedia Secure Download \(RO\)'. <?xml version="1.0" encoding="utf-16"?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <DownloadBeta>Default</DownloadBeta> <DownloadDate xsi:nil="true" /> <VariableChangeIndicator /> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <ExclusiveDownload>false</ExclusiveDownload> <HttpReferer><placeholder name="Download Page URL" /></HttpReferer> <Variables> <item> <key> <string>version</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>(??<=\<title\>Download.*?<placeholder name="Stable release ?" value="\/.*?" />\s)(\d+[\.\d+]+?.*?)(?=\s[\-\/\[]<placeholder name="Version format is 1.x.x ?" value="?" />)<placeholder name="Capture date ?" value="|(?:[a-z]+?\s\d{1,2}[a-z]{2}\,\s\d{4}\,\s\d{2}\:\d{2})(?:\sGMT)?" />)</Regex> <Url><placeholder name="Info Page URL" /></Url> <Name>version</Name> </UrlVariable> </value> </item> <item> <key> <string>download_url</string> </key> <value> <UrlVariable> <RegexRightToLeft>true</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>(?<=href\=")(\w+\:\/\/[\w][\w\-?\w.:]+\/?[\w\.?=%&=\-/,\)\(]*)(?=".*?<placeholder name="Paste link text" />)</Regex> <Url><placeholder name="Download Page URL" /></Url> <Name>download_url</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand>{rem} copy /y {file} {shared}</ExecuteCommand> <ExecutePreCommand>{rem} del /q {shared}\{appname}*.*</ExecutePreCommand> <Category><placeholder name="Category" value="Backup-Benchmark-Clean-Compression-Grep-Internet-Multimedia-Misc-Office-PDF-Recovery-Registry-Scrapbook-Security-Test-Virtualization" /></Category> <SourceType>FixedUrl</SourceType> <DeletePreviousFile>false</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <LastUpdated xsi:nil="true" /> <TargetPath><placeholder name="Save to" value="{target}\{category}\" /><placeholder name="Class" value="Diagramming\DL_Manager\Extensions\Graphics\Network\Notepad\Plugins\ScreenCapture\Test\Uninstall\USB\"/><placeholder name="Rename" value="{appname}_{version}.{url:ext}" /></TargetPath> <FixedDownloadUrl>{download_url}</FixedDownloadUrl> <Name><placeholder name="AppName" /></Name> </ApplicationJob> </Jobs> Edited May 3, 2009 by FranciscoR Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now