Jump to content
Ketarin forum

Shawn's Microsoft template: revised 2011-07-08


shawn
 Share

Recommended Posts

This is a search for MS downloads. It does NOT work for downloads that require validation, but everything else (like the powerpoint viewer, games for windows live, IE of various flavors) works fine. When importing you're prompted for a search term. This is used because there are often several results in the MS downloads list for the same application, based on the release date or version. This uses that search term, by date descending, to obtain the correct URL key (mskey) and then parses it for the download.

 

You can "find" the right search term to use here.

 

Let me know if you have any issues or feedback.

 

Revised 2011/07/08 to use only Google for searching for results and to allow it to work with either form of download addresses (familyid and download id). Last week I also added a Guid to ensure that you'll be able to use this template to update any apps that use it. Sadly, you WILL have to recreate every app using this template in order to make it happen (no retroactive assignment). This is because the Guid wasn't there first. That was a horrible lesson to learn. :)

 

<?xml version='1.0' encoding='utf-8'?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="debb1deb-1024-1975-abcd-aa0000000008" >
   <Name><placeholder name="Application Name?" value="" /></Name>
   <Category><placeholder name="Application Category?" value="Unassigned" /></Category>
   <SourceType>FixedUrl</SourceType>
   <DeletePreviousFile><placeholder name="Delete Previous Version?" value="true" options="false|true" /></DeletePreviousFile>
   <WebsiteUrl />
   <UserNotes />
   <IgnoreFileInformation>false</IgnoreFileInformation>
   <DownloadBeta>Default</DownloadBeta>
   <DownloadDate xsi:nil="true" />
   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator />
   <CanBeShared>true</CanBeShared>
   <ShareApplication>false</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <HttpReferer />
   <Variables>
     <item>
       <key>
         <string>mssearch</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent><placeholder name="What is the search term on MS/Downloads?" value="" /></TextualContent>
           <Name>mssearch</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>mskey</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>"(http://www.microsoft.com/download/en/details.aspx.id=\d+|http://www.microsoft.com/downloads/details.aspx?familyid=[a-z\d\-]+)"</Regex>
           <Url>http://www.google.com/search?hl=&cat=&meta=&num=100&ie=utf-8&q=site%3Amicrosoft.com%2Fdownload+%22{mssearch}%22#q=site:microsoft.com/download+%22{mssearch}%22&hl=en&tbo=1&tbs=qdr:y,sbd:1&prmd=ivfd&source=lnt&sa=X</Url>
           <Name>mskey</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>version</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>Version\:</strong></td>\s*<td[^>]*>(?:<span[^<>]*>)?([^<>]+)(?:</span>)?</td></Regex>
           <Url>{mskey}</Url>
           <Name>version</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>sdownload</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>fileurl="([^'"]+download[^'"]+)['"]</Regex>
           <Url>{mskey}</Url>
           <TextualContent>onclick="openDownloadWindow\('([^'"]+download[^'"]+)['"]</TextualContent>
           <Name>sdownload</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>swebsite</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>{mskey}</TextualContent>
           <Name>swebsite</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>schangelog</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent />
           <Name>schangelog</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>snotes</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <Name>snotes</Name>
         </UrlVariable>
       </value>
     </item>
   </Variables>
   <ExecuteCommand />
   <ExecutePreCommand />
   <Enabled>true</Enabled>
   <FileHippoId />
   <TargetPath>.\{category}\{appname:regexreplace:([\s\t\r\n\-\\&]+):_}-{version}.{url:ext}</TargetPath>
   <FixedDownloadUrl>{sdownload}</FixedDownloadUrl>
 </ApplicationJob>
</Jobs>

Edited by shawn
Link to comment
Share on other sites

Hi shawn,

 

Good job.

I discovered some pages with multiple downloads are a bit troublesome, for instance http://www.microsoft.com/downloads/en/details.aspx?FamilyID=02be8aee-a3b6-4d94-b1c9-4b1989e0900c also http://www.microsoft.com/downloads/en/details.aspx?FamilyID=24da89e9-b581-47b0-b45e-492dd6da2971

Another thing, what are the swebsite, schangelog and snotes variables for?

 

Best regards!

Link to comment
Share on other sites

what are the swebsite, schangelog and snotes variables for?

Sorry, those are for how I use Ketarin. I maintain a list of updated software and security apps and publish it for my clients, and this helps me automate much of that. :)

 

For apps with multiple downloads, I use Syncaid:

http://saferpc.info/syncaid/

 

I started writing that a few years ago specifically for that purpose, and then I found Ketarin, and haven't needed to maintain Syncaid since. It does have a couple nifty features lacking in Ketarin (most significantly multiple downloads), but is nowhere near as polished as Ketarin. It would be so nice to have the ability to tell Ketarin to download "myapp.*" (or "myapp\.(exe|pl|hlp|ini|chm|bin|xml)") from a given URL and have it download all matching files (see a great example of "why" here: http://live.sysinternals.com/). The single-result matching thingy is very limiting, IMHO.

 

Still, I wouldn't trade Ketarin for anything.

Link to comment
Share on other sites

Also, if the MS DL search sucks (as unfortunately, they often do), you can try a Google search by replacing the search URL from:

http://www.microsoft.com/downloads/en/results.aspx?freetext=%22{mssearch}%22&displaylang=en&stype=ss_rr&nr=10&sortCriteria=Date&sortOrder=Descending

to:

http://www.google.com/search?hl=&cat=&meta=&num=100&ie=utf-8&q=site%3Amicrosoft.com%2Fdownloads+%22{mssearch}%22#q=site:microsoft.com/downloads+%22{mssearch}%22&hl=en&tbo=1&tbs=qdr:y,sbd:1&prmd=ivfd&source=lnt&sa=X

 

This will change it to a Google search of all links within the last year, sorted by date, descending (newest on top). That can help if the MS search functionality keeps returning totally useless crap.

Link to comment
Share on other sites

Hi all,

 

What is "multiple downloads" actually? Alternative mirrors?

 

I meant pages where +1 download is available --> for instance MSE @ http://www.microsoft.com/downloads/en/details.aspx?displayLang=en&FamilyID=e1605e70-9649-4a87-8532-33d813687a7f or the 2 other examples in my post above.

 

For apps with multiple downloads, I use Syncaid:

http://saferpc.info/syncaid/

 

I started writing that a few years ago specifically for that purpose, and then I found Ketarin, and haven't needed to maintain Syncaid since. It does have a couple nifty features lacking in Ketarin (most significantly multiple downloads), but is nowhere near as polished as Ketarin. It would be so nice to have the ability to tell Ketarin to download "myapp.*" (or "myapp\.(exe|pl|hlp|ini|chm|bin|xml)") from a given URL and have it download all matching files (see a great example of "why" here: http://live.sysinternals.com/). The single-result matching thingy is very limiting, IMHO.

 

Still, I wouldn't trade Ketarin for anything.

 

Ketarin works fine for me, what i did was to replace regex from sdownload variable with something more specific like

 

href=["'](http://[^"']+_x86\.exe)["']

 

No doubt you or Cybteksol or some other knowleadgeble Ketarin user can find something +elegant. BTW i haven't used this forum section material before but this specific template is very handy for me. :)

 

Also, if the MS DL search sucks (as unfortunately, they often do), you can try a Google search by replacing the search URL from:

http://www.microsoft.com/downloads/en/results.aspx?freetext=%22{mssearch}%22&displaylang=en&stype=ss_rr&nr=10&sortCriteria=Date&sortOrder=Descending

to:

http://www.google.com/search?hl=&cat=&meta=&num=100&ie=utf-8&q=site%3Amicrosoft.com%2Fdownloads+%22{mssearch}%22#q=site:microsoft.com/downloads+%22{mssearch}%22&hl=en&tbo=1&tbs=qdr:y,sbd:1&prmd=ivfd&source=lnt&sa=X

 

This will change it to a Google search of all links within the last year, sorted by date, descending (newest on top). That can help if the MS search functionality keeps returning totally useless crap.

 

Thank you for this suggestion it can be handy. So far i have only found 1 application that doesn't go to correct page, workaround for me was to add some adtional keywords from source page.

Edited by josh
Link to comment
Share on other sites

Hey shawn... I'm going to play around with SyncAid if you don't mind. I may have an occasion to use it. Do you have any other configuration files for 'unrepentant technogeeks' like me? BTW... I use to capture all Nirsoft and Sysinternals apps until I found 'Windows System Control Center' from KLS Soft. It is a portable gui for them and keeps them updated for you. It has become one of my favorites... give it a try. ;)

Link to comment
Share on other sites

What is "multiple downloads" actually? Alternative mirrors?

 

No. It's exactly what it describes: multiple files that all related to the same package. If you hit http://live.sysinternals.com/ you'll see that all the SysInternals tools are there, already extracted, but there are at least two files for each "package". Sure, I could setup two separate jobs to ensure that I get both the EXE and the CHM file (and another job for each other dependency, when necessary), but it would be so much easier to be able to assign a pattern. If you look at the syntax options for Syncaid it'll make a lot more sense (I hope):

http://saferpc.info/syncaid/#extensions

It's literally designed to capture the contents of *every* link by default, so you can either pattern match, or use other filters to reduce the number of results. I actually push specific categories from this page to Syncaid when I'm onsite for a client and they need a certain type of removal tool/utility:

http://saferpc.info/syncaid.asp

 

 

CybTekSol - thanks! I'll definitely check it out.

 

The most common problem I have is, I sometimes end up onsite without my USB key or am remoting into a computer and just need certain files (like all the current common updaters). I can either individually go find 'em or keep a list of the links somewhere (as above), or I can use Syncaid to "go get 'em" for me, and not have to worry about it, freeing me up to work on something else ad interim. This is especially important since I'm doing more and more of the once-a-month updates for remote clients, where I login to their computers with LogMeIn, and install the current spate of updates. On 99% of computers out there, I can either run UpdateChecker (remotely) and then individually download each one, or I can just run Syncaid (which downloads all the most common updates for me in the background) while I run UpdateChecker to find out the "uncommon" updates that need applied, and queue Windows Update and Apple Updater, too. Then, when Syncaid is done I can start running those. Very streamlined already, but always looking for any tips. :)

 

If anyone is interested, I can provide a list of my shortlinks for the common updaters.

Link to comment
Share on other sites

  • 1 month later...

Shawn: you'd mentioned that this doesn't work on downloads that need to be authenticated. What I used to do in the days of quazi-legal was to search for the filename at http://findfiles.com/ which led me to the direct download. In case you're unfamiliar, FindFiles has links generated by what GetRight users download through that download manager. Maybe scraping the filename and then searching FindFiles for "http://\S*microsoft.com\S+{executable}\S+?" or something similar is just what ya need in that case. Thoughts?

Link to comment
Share on other sites

  • 1 month later...
  • 3 weeks later...

Last few days i've been getting the following error with all the apps that use this template as base.Has MS changed it's website or what gives?

DirectX Official    Invalid URI: The format of the URI could not be determined. (/downloads/info.aspx?na=41&SrcFamilyId=3B170B25-ABAB-4BC3-AE91-50CEB6D8FA8D&SrcDisplayLang=en&u=http%3a%2f%2fdownload.microsoft.com%2fdownload%2f8%2f4%2fA%2f84A35BF1-DAFE-4AE8-82AF-AD2AE20B6B14%2fdirectx_Jun2010_redist.exe)

Link to comment
Share on other sites

You need to change your fixed URL to the following. Template should be updated to reflect this.

 

http://www.microsoft.com{sdownload:multireplace:,:%2F,%3A,amp;:/,\\:,}

 

Won't work for me any other way

 

Out of curiosity why doesn't Ketarin handle HTML encoding anyway? I have a few other apps that require me to manually replace encoding. Isn't this something Ketarin should do by default?

Link to comment
Share on other sites

  • 2 weeks later...
  • 4 months later...

Well it looks like you have to re-do part of your template Shawn.

Microsoft has been rolling out a new desing for the past few weeks already so {sdownload} is broken in the current template.

Gotta love redesings eh? :D

 

Honestly i don't know whose bright idea it is to do a redesing of a webpage right before or even in the middle of the vacation period.You'd think the designers are slipping margaritas on a white sand beach somewhere near indian ocean but noo - they're busy causing havoc and general mayhem by redesigning whole interfaces :P

First Microsoft and now Google too...

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.