A simple SourceForge template

[ste123]

After playing around a bit with Ketarin, I wanted a simple way to download the latest versions offered by SourceForge. I know there's many templates available, but either they're too difficult for me to understand (e.g. FranciscoR's), required use of external tools (e.g. Gozi's), or didn't work for me in its presented form (e.g. Shawn's). I'm not knocking any of these templates - what they've done shows the time and brainpower invested into making such great templates - but they just aren't for me (at this present time!).

 

Mine is simple in terms of design as well required user input. I've used Shawn's template as a base for this one.

 

When accessing the 'Files' page, SourceForge presents the user with a download link to the newest version, seemingly assisted by the browser's User-Agent value. This template emulates IE8 on Windows to return a download link preferred for Windows. Also, due to no consistency regarding versioning on SF, this template sets the {version} variable to the text presented in the link e.g. KeePass-2.13-Setup.exe. Personally I don't think it's a huge deal, as any updates would be determined by a change in this value since last check. As long as SF keeps this consistent, there shouldn't be a problem.

 

Due of this template's simplicity, several drawbacks arise:

 

• you don't get to choose the specific file type or version to download -- only what SF deems the latest available, and in the format it deems you would want
  
• as {version} is not just the traditional version number, it's not ideal to use this field in the resulting filename -- I suggest using {url:basefile}.{url:ext} in this instance.
  
• the desired SF mirror needs to be manually selected/entered -- a list of which can be found here, and its default value can be changed within the template.
  

 

If these aren't deal-breakers for you, I suggest giving this template a go.

 

This is my first attempt at Ketarin templates so there's going to be bugs or ways to do things better. Your thoughts are appreciated!

 

<?xml version='1.0' encoding='utf-8'?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
   <Name><placeholder name="Application Name" value="" /></Name>
   <Category><placeholder name="Application Category" value="Unassigned" /></Category>
   <SourceType>FixedUrl</SourceType>
   <DeletePreviousFile><placeholder name="Delete Previous Version?" value="true" options="false|true" /></DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />
   <WebsiteUrl>http://sourceforge.net/projects/{sftag}/</WebsiteUrl>
   <IgnoreFileInformation>false</IgnoreFileInformation>
   <DownloadBeta>Default</DownloadBeta>
   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator>{version}</VariableChangeIndicator>
   <CanBeShared>true</CanBeShared>
   <ShareApplication>false</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <HttpReferer>http://sourceforge.net/projects/{sftag}/files/{filenamepath}/download</HttpReferer>
   <Variables>
     <item>
       <key>
         <string>sftag</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <TextualContent><placeholder name="SF application tag" value="" /></TextualContent>
           <Name>sftag</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>version</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex><span>Download (.+?) \([\d\.]{1,7} (?:M|K|G|T)?B\)</span></Regex>
           <Url>http://sourceforge.net/projects/{sftag}/files/</Url>
           <Name>version</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>sfmirror</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <TextualContent><placeholder name="'Short name' of desired SourceForge mirror" value="internode" /></TextualContent>
           <Name>sfmirror</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>filenamepath</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>"http://sourceforge.net/projects/{sftag}/files/([^"'=]+?)/download"</Regex>
           <Url>http://sourceforge.net/projects/{sftag}/files/</Url>
           <Name>filenamepath</Name>
         </UrlVariable>
       </value>
     </item>
   </Variables>
   <TargetPath>.\{category}\{url:basefile}.{url:ext}</TargetPath>
   <FixedDownloadUrl>http://{sfmirror}.dl.sourceforge.net/project/{sftag}/{filenamepath}</FixedDownloadUrl>
   <UserAgent>Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)</UserAgent>
 </ApplicationJob>
</Jobs>

 

Thanks.

[Omniferum]

If you'd prefer this one extracts the path from the 'get the latest' and travels there and there you can further specify the file you want. I've just made it so the first exe zip rar 7z msi is found. You can edit it easily.

 

You don't need to manually specify your mirror.

 

As for version definitions, i'm not sure why people are so gung ho on setting this variable. I've only ever had to do it for one app because otherwise it just kept downloading. Ketarin has done a perfect job so far of determining by itself whether or not the download is new.

 

With my template you can add the URL from any page that at least has "Looking for the latest version?" which is practically all of them and automatically names the application. Even the front page has it, just doesn't say "looking for latest version?"

 

<?xml version='1.0' encoding='utf-8'?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
   <WebsiteUrl />
   <UserNotes />
   <IgnoreFileInformation>false</IgnoreFileInformation>
   <DownloadBeta>Default</DownloadBeta>
   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator />
   <CanBeShared>true</CanBeShared>
   <ShareApplication>false</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <UserAgent>(!)</UserAgent>
   <HttpReferer></HttpReferer>
   <Variables>
     <item>
       <key>
         <string>download</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>([^"']+projects[^"']+files[^"']+(\.exe|\.msi|\.zip|\.rar|\.7z))</Regex>
           <Url>{refer}</Url>
           <Name>download</Name>
         </UrlVariable>
       </value>
     </item>
           <item>
       <key>
         <string>refer</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>([^"']+files[^"']+/)[^"']+\.\w\w\w</Regex>
           <Url><placeholder name="Application Page URL"/></Url>
           <Name>refer</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>name</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex><title>(.*?) </Regex>
           <Url><placeholder name="Application Page URL"/></Url>
           <Name>name</Name>
         </UrlVariable>
       </value>
     </item>     
   </Variables>
   <ExecuteCommand>/*This is a comment line, the line below is the C# script. It will run the instructions found in the Setup Tab*/
app.Install(null);</ExecuteCommand>
   <ExecutePreCommand />
   <ExecuteCommandType>CS</ExecuteCommandType>
   <ExecutePreCommandType>Batch</ExecutePreCommandType>
   <SetupInstructions />
   <Category><placeholder name="Category" /></Category>
   <SourceType>FixedUrl</SourceType>
   <DeletePreviousFile>true</DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />
   <TargetPath>{downloadpath}{category}\</TargetPath>
   <FixedDownloadUrl>{download}</FixedDownloadUrl>
   <Name><placeholder name="Name of application" variable="name" /></Name>
 </ApplicationJob>
</Jobs>

Edited December 28, 2010 by Omniferum

[shawn]

As for version definitions, i'm not sure why people are so gung ho on setting this variable.

 

I absolutely MUST have version numbers. I do a lot of onsite computer jobs, and it's important that while I'm there I install the updates. We're in a resort community where only a small fraction can get broadband, so many of my clients are on dialup or satellite (not a lot of difference between them if you ask me). Going to an onsite job and then manually trying to figure out what updates I've previously installed and copied over adds time to the job unnecessarily. To that end, I've got a few batch scripts I wrote and have tuned over the last few years to automate most of this, including only copying over an app package if it doesn't already exist and in some cases is already installed but isn't the current version. The only way this works is if the apps are named consistently, including the version numbers.

 

Surely most people don't require that level of, well, obsession, but I find that it is far more professional to simply get it done the right way in as little time as possible, so I can get on with everything else that needs done.

 

I've worked to revise my SF template quite a bit over the last month (since they implemented the new "files" structures), but it's still lacking a couple things. Once it's fixed I'll be re-releasing it up here.

[Omniferum]

Fair enough shawn, I remember being a roving network engineer so I can understand that necessity.

 

I'm merely saying for general Ketarin use it does the job rather well on its own, specifying version variables for me has only been needed for the one app that keep downloading itself because Ketarin couldn't determine it correctly.

 

I just see a lot of people new to the forums saying they've specified it, when it isn't specifically necessary.

[ste123]

@Omniferum: Many thanks for providing your template! Much simpler than mine, I'll I'll probably end up taking parts of yours and blending it into what I have already.

 

A quick question. You don't have tag VariableChangeIndicator set - is this because it will default to the last filename downloaded as a comparison?

 

@shawn,Omniferum: Thank you both for replying. Without your help I would be still scratching my head with all of this!

[shawn]

My pleasure.

 

It's not the filename alone but the file profile that's used to determine if an update is necessary. This is based on the timestamp and filesize the server includes within headers in the reply.

[UksusoFF]

Today all progs that uses this template give 403 error

[UksusoFF]

Heeey! Anybody have worked SourceForge template?

Or it's not worked only for me? =)

And anyway how replace template for many programs without editing each them?

[shawn]

Today all progs that uses this template give 403 error

 

403 is "forbidden", so it's likely either the referer (change this under "advanced settings" to "http://sourceforge.net/"), or the useragent (change this under "advanced settings" to "pycurl/7.18.2"), or your IP address was banned from the host you're connecting to. Make these two changes and you'll know for sure if it's just you, or if it's the settings.

[shawn]

Anybody have worked SourceForge template?

 

Mine works fine for me with over 40 apps.

 

And anyway how replace template for many programs without editing each them?

 

Sadly, for this behavior to work (automatic template update), the app profile has to have been created with a template that used a Guid value (see my Sourceforge.net template for an example). As long as it has a Guid attribute ON THE TEMPLATE when you import it, it'll prompt you to update any other templates based on that template the next time you import it if a change is detected.

[UksusoFF]

or your IP address was banned from the host you're connecting to.

Where i can check this?

I have access via browser normally, but all sf prog via ketarin give 403 error...

I changed referer and useragent as you say but it's 403 again.

I have no highlighted in "version" and not correctly code in "filenamepath". Look at this:

 

 

 

On screens this project.

[UksusoFF]

[shawn]

I don't know what's up with the template, but the way it's selecting the filename is wrong. Each of those errors are not because your IP is banned, but because they're trying to download a folder (not the "/" at the end of the URL). SF url's can be obtained from the project homepage or the project "files" directory - which is a recursive request (one for the version, another for the file), but in any case, whatever this template is using to grab the value is wrong.

For 7zcracker, this would be grabbing the version from here:

http://sourceforge.net/projects/sevenzcracker/files/

Then getting the most recent filename from here:

http://sourceforge.net/projects/sevenzcracker/files/V0.01/

The second URL is based on the first.

While there are ways to get the SF promoted download on other pages, that pattern isn't always trustworthy as it's based on browser detection, so it really shouldn't be trusted alone.

[UksusoFF]

{filenamepath} need change to:

"http://sourceforge.net/projects/{sftag}/files/([^"'=]+?)/download.?source=files" 

and it works again

 

but i can't understand how to get {version}

[shawn]

This should work:

"http://sourceforge.net/projects/{sftag}/files/([vV]?[\d\.]+)/

[UksusoFF]

This should work:

"http://sourceforge.net/projects/{sftag}/files/([vV]?[\d\.]+)/

works only for 7z cracker, not for other progs... i try your template.