Jump to content
Ketarin forum

SourceForge.net Template (Test)


FranciscoR
 Share

Recommended Posts

  • 3 weeks later...
  • Replies 50
  • Created
  • Last Reply

Top Posters In This Topic

I managed to develop what I believe is a consistent successful download path for SF apps.

 

It requires parsing the target app based on whatever you want to consume. This works fine for most apps with a parse like this (saved to "dl"):

URL: http://sourceforge.net/projects/tuxtype/files/

Pattern: [t'"]/projects/([^"']+)/download['"]

 

The captured portion then gets passed to the download as:

http://downloads.sourceforge.net{dl:replace:/files/:/}?download

 

The captured portion in bold up there can be modified to help limit the version/type/style of the download (for example, x64, msi, 7z and so on).

Link to comment
Share on other sites

  • 2 weeks later...
Sigh.

 

SourceForge changed their link structure, breaking over 30 of my apps. This is infuriating.

Yes indeed shawn, ALL of mine are broken as well. It appears you MUST have a referrer now for ALL downloads from SF. :( I have figured it out, but it will take some time to adjust all of the effected apps and change my personal template as it is a major tweak. I am capturing portions of the download URL and piecing it together to form the referrer. Such as:

hxxp://sourceforge.net/projects/{refer1}/files/{refer2}/download

Where:

{refer1} is the alphanumeric section just after hxxp://downloads.sourceforge.net/project/ in the download URL

and

{refer2} is the remainder of the download URL before ?use_mirror

to form referrers such as:

hxxp://sourceforge.net/projects/akelpad/files/AkelPad%204/x.x.x/AkelPad-x.x.x-bin-eng.zip/download

Hope this helps!

Link to comment
Share on other sites

Hello everyone. I'm new here, so forgive me if I say something ridiculously wrong. :)

All my SF applications are broken too, but I solved the problem reading this referrer tip here, from you.

Although I don't really know what it means, I 've just used http://sourceforge.net as the "Spoof HTTP Referrer" for the applications I'm downloading, no variables, no parsing.

And it worked. ;)

Cheers!

Link to comment
Share on other sites

Thanks Wagner. To save myself some time with your fix I used the SQLite Manage addon for Firefox https://addons.mozilla.org/en-US/firefox/addon/5817/ to modify the database. I then used the following SQL statement to update all of my sourceforge entries:

 

<code>

UPDATE jobs

SET HttpReferer='http://sourceforge.net'

WHERE FixedDownloadURL LIKE '%sourceforge%';

</code>

 

Maybe others do this already, but I thought I would share in case I could save someone some time.

 

Beavel

Link to comment
Share on other sites

Thanks for the tip, but I don't understand. Is it even possible to set a dynamic referer in Ketarin? I've tried using variables in the spoof referer box without luck (for another site, but with similar requirements).

Yes Shawn... variables within the referrer is one of the things we lobbied Flo for and it was added in February to Ketarin's feature-set. See this post: http://ketarin.canneverbe.com/forum/viewtopic.php?pid=3294#p3294

Link to comment
Share on other sites

  • 1 month later...

Is there a currently working template for SF now? The 'sticky' thread at the top of this forum gives up on it over a year ago.

 

If there is, would someone please post the full template AND full instructions on how to use it (or point to full and complete instructions if they already exist?). I have never used templates so I need instructions from the very beginning :-) I'm good to go with regex syntax BUT haven't used the Ketarin variables version of it much so a little explanation for any of those in the template would be helpful too.

 

Thanks and regards,

--appyface

Link to comment
Share on other sites

@appyface,

Good to see you back... hope you are well. ;) As for a SF template... there are so many site inconsistencies, the best template I've ever been able to develop is at best 80-90% without tweaking. I avoid SF like the plague unless there are no other options!

 

The reason I have never posted the template here is that every time FranciscoR and I posted/discussed it here, they would globally replace html code and break the template... coincidence or not... I'll never know. This post of mine for ffdshow (x64) will give you an idea of my template's design which relies on what I term 'anchors'.

 

If this looks like something you can work with, I will e-mail the full template to you directly. All you have to do is import the template, supply information as requested by the 'placholder' prompts (they are described when prompted) and then tweak the regex as required and test. The template will at the very least get you 'in the ballpark'... so to speak.

 

I will do the same for others as well if they post back here...

 

Regards,

CybTekSol

Link to comment
Share on other sites

Unfortunately, there isn't a standard location (that I've found) to identify the version numbers for SourceForge projects, so you end up with a custom version-check every single time. Once you have that, however, you can build the download URL very easily. I have about 40 SF apps in my database, and using a referrer of "http://sourceforge.net/" and the following URL pattern **always** works:

 

hxxp://downloads.sourceforge.net/{appstub}/{path}?download

 

{appstub} is the specific application stub. For example, for phpMyAdmin, it's "phpmyadmin". Basically, it's whatever the is within the project URL, as so:

hxxp://sourceforge.net/projects/phpmyadmin

hxxp://sourceforge.net/projects/dqsd

hxxp://sourceforge.net/projects/synergy2

 

{path} is the parsed result of this URL:

hxxp://sourceforge.net/projects/phpmyadmin/files/

hxxp://sourceforge.net/projects/dqsd/files/

hxxp://sourceforge.net/projects/synergy2/files/

...using this RegEx pattern:

[t'"]/projects?/([^"']+{filepattern})[\?'"&]

 

{filepattern} will often need changed for each package. It is how the filenames are composed and desired under that specific release. For example, for phpMyAdmin I always want the "all languages" zip file for the current {version}. So the filepattern would be "phpMyAdmin-{version}-all-languages.zip". Here it is in the code view as above:

[t'"]/projects?/([^"']+phpMyAdmin-{version}-all-languages.zip)[\?'"&]

 

Then in the URL box you use:

http://downloads.sourceforge.net{download:replace:/files/:/}?download

Link to comment
Share on other sites

@shawn Thank you. This is similar to what I'm currently doing. One difference with my approach is I go straight to the mirror instead of letting SF provide one. It works 99.44% of the time. The only problems I've encountered is when the mirror I'm using is not available or doesn't carry the file.

 

My approach picks up the first version matching my scrape. Typically that's the one I want because the most recent version is presented first. However, with some authors that can be a beta. If I don't want the beta I have to modify the app a bit to skip it.

 

I'm interested in a more generic approach, if that is even possible. But our approaches do work!

 

Thank you for posting your approach and details, I'm sure this info will be of help to many people.

 

 

@CybTekSol Thank you as always for your help, and for the link and info on your template. I have some free time coming up in a couple of weeks, I'll get back to you then. I'm interested in trying out your latest template :-)

 

Thanks again,

--appyface

Link to comment
Share on other sites

In my case I've had to change the mirror only once and only for two apps, so it's something I'm quite happy to live with so far. In my setup I don't have to make any pattern changes to set up an app, except in the case of avoiding betas that are listed first. So I'll take far less mucking over 100% reliability any day :-)

 

--appyface

Link to comment
Share on other sites

It is rare for me to like a program so much that I actually sign up to forums, so kudos to the brilliant developer of this app. Last 2 days solid has somewhat consumed me with trying to get everything in working order :P

 

In saying that though I have two peculiar issues, one to do specifically with downloading Thunderbird Portable and Firefox Portable. Even just using the direct link with referer I always download a corrupted file, which drives me insane. It is always smaller than the actual file is, i've tried a different mirror direct and same problem. Tried different referer's and a whole bunch of stuff. I still get the same corrupted, but all my other portableapps.com downloads are perfectly fine.

 

File has the right filename and version number... it's just bloody strange.

 

And thanks to whoever made that PortableApps template! I just needed to add the http://sourceforge.net referer and it worked like a charm for everything else.

 

Second problem is to do with commands, specifically execute after downloading. It seems like you can have multiple commands, which is great. Because some you need to unzip twice or uniextract, delete superfluous folders and move them around.

 

Trouble is I can run the unzip then uniextract fine, both programs will run sequentially (yay). I can't however delete/erase/rmdir or move anything, the only command that seems to function is xcopy. I can even dir /a /s /b {root}\*.exe > c.txt, which I use to extract filename/paths to a text file if I want to parse it to uniextract or 7zip (which i'm not trying to do here, I just wanted to see if it would).

 

Am I missing something here? I just assumed that the command entry window was essentially treated LIKE a .bat file. But some important commands aren't working or perhaps the {appname} functions aren't being passed on.

 

I could just write a .bat file for each one but I would rather it stay with Ketarin for 'neatness' sake. Plus parsing {file:filename} and the like to 7zip/uniextract is bloody helpful and I wouldn't be able to get the same clean result just using a .bat file.

 

Oh, and as for templates. There isn't really a section for this but i'm happy to share my "Install" methods when i've perfected them. haha, first project i've ever wanted to contribute to :P

 

There are several types of install methods (I'm a portable app junkie)

Downloading setup and uniextract, which is great but requires cleanup afterwards

Binaries with no subdirectory, extracting to {appname} will give you what you want

Binaries with a subdirectory that is, thankfully, {appname} to begin with

Standalones, files that are just straight executables and not zipped at all anyway. i.e. utorrent

Paf.Exe, these are portableapps.com packages. There is an AHK script in this forum that helps out, I have never used AHK and two of my packages (foxit/openoffice) don't follow its format. If you just take out the blockinput code you can just click on what you need to make it progress (i.e. a licence agreement) and it will pick up after that.

 

Binaries with different subdirectories (i.e. notepad++ has ansi/unicode) - this also requires cleanup stuff similar to uniextract

And fiddly bastards, things that are zipped and the file in that need to be uniextracted

 

All the above I know how to do with .bat files, would just rather do them all in Ketarin.

 

The ONLY thing I have that isn't portable is the obvious like java/flash/codecs/antivirus (not because it isn't available, I just like nod32)

 

Once I can get this down pat i'll be a happy bloody pumpkin!

 

 

tl;dr

 

-Thunderbird Portable and Firefox Portable are always corrupted downloads but have the correct filename, why? sourceforge is where they are downloaded from

 

-Why does ketarin not like move/erase/rmdir commands? and it doesn't seem to like ;for /F %%c in (c.txt) do set c=%%~fc

 

-What does ketarin use for that command window anyway? Can I just queue up a list of commands and it will perform them sequentially a la bat file or what? I assume it just uses cli to perform all those extra commands anyway.

Link to comment
Share on other sites

OK, command update.

 

The rmdir and the like commands DO work but it seems that Ketarin doesn't do sequential commands. I just thought it treated the window like a batch file.

 

So I queued a few commands up, while they work in .bat form perfectly fine, ketarin seems treat it all as commands to send in parallel. So for those I will just call my batch files, which is fine. Just not ideal as such because

 

*UPDATE*

 

I'm a freaking moron, didn't see how Ketarin queued up commands. Had to hunt through forums, wee.

 

Given my delicious romp through stupid land, i'd be happy to write up a quick-start guide or FAQ. There are multiple sources to reference for this wonderful program, just needs to be a bit more centralized.

 

Only thing left is the mystery of the thunderbird/firefox portable corrupted downloads

Edited by Omniferum
Link to comment
Share on other sites

@shawn

 

Would you please give detailed step-by-step instructions on your method, for how to pick up Keepass 2.12 installer file (KeePass-2.12-Setup.exe) from here:

 

http://sourceforge.net/projects/keepass/files/

 

This one can on occasion be problematic for my download method, so I thought I'd try your approach on it. But I must be doing something wrong, because all I can get with your method is the infamous 1.2M bogus exe file that is actually HTML.

 

Thanks in advance for details, I'm sure it will not only help me but help others to clearly understand how your approach works.

 

--appyface

Link to comment
Share on other sites

@appyface,

 

I'm obviously not Shawn... but these are exports based on my SF template which is slow but reliable...

 

First... the ZIP version:

<?xml version='1.0' encoding='utf-8'?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
   <WebsiteUrl />
   <UserNotes />
   <IgnoreFileInformation>false</IgnoreFileInformation>
   <DownloadBeta>Default</DownloadBeta>
   <DownloadDate xsi:nil="true" />
   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator />
   <CanBeShared>true</CanBeShared>
   <ShareApplication>false</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <HttpReferer>http://sourceforge.net</HttpReferer>
   <Variables>
     <item>
       <key>
         <string>ProjectPageURL</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>http://sourceforge.net/projects/keepass/</TextualContent>
           <Name>ProjectPageURL</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>ProjectName</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>{ProjectPageURL:replace:http\\://sourceforge.net/projects:}</TextualContent>
           <Name>ProjectName</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>anchor1</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>KeePass-</TextualContent>
           <Name>anchor1</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>anchor2</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>zip</TextualContent>
           <Name>anchor2</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>ver_scrape</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>(?:['|"|;]http://downloads.sourceforge.net/project{ProjectName}[\w\s_\.?=%&=\-@/$,]*(?:_v|%20v|[a-z]v|[a-z]-|[a-z]_|/|%20|\s))(\d[\w\s_\.?=%&=\-@/$,]*)(?:/{anchor1}\d[\w\s_\.?=%&=\-@/$,]*\d\.{anchor2})(?:['|"|\?])</Regex>
           <Url>{ProjectPageURL}files/</Url>
           <Name>ver_scrape</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>truncated_url</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>(?:['|"|;]http://downloads.sourceforge.net/project)({ProjectName}[\w\s_\.?=%&=\-@/$,]*{ver_scrape}/{anchor1}\d[\w\s_\.?=%&=\-@/$,]*\d\.{anchor2})(?:['|"|\?])</Regex>
           <Url>{ProjectPageURL}files/</Url>
           <Name>truncated_url</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>SF_mirror</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>hivelocity</TextualContent>
           <Name>SF_mirror</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>download_url</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>http://{SF_mirror}.dl.sourceforge.net/project{truncated_url}</TextualContent>
           <Name>download_url</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>version</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>{ver_scrape:replace:%20:_}</TextualContent>
           <Name>version</Name>
         </UrlVariable>
       </value>
     </item>
   </Variables>
   <ExecuteCommand />
   <ExecutePreCommand />
   <Category>Security\Password Managers</Category>
   <SourceType>FixedUrl</SourceType>
   <DeletePreviousFile>true</DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />
   <LastUpdated xsi:nil="true" />
   <TargetPath>{target}\{category}\KeePass_v{version:replace: :_}.{url:ext}</TargetPath>
   <FixedDownloadUrl>{download_url}</FixedDownloadUrl>
   <Name>KeePass Password Safe (ZIP)</Name>
 </ApplicationJob>
</Jobs>

Next... the SETUP version...

<?xml version='1.0' encoding='utf-8'?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
   <WebsiteUrl />
   <UserNotes />
   <IgnoreFileInformation>false</IgnoreFileInformation>
   <DownloadBeta>Default</DownloadBeta>
   <DownloadDate xsi:nil="true" />
   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator />
   <CanBeShared>true</CanBeShared>
   <ShareApplication>false</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <HttpReferer>http://sourceforge.net</HttpReferer>
   <Variables>
     <item>
       <key>
         <string>ProjectPageURL</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>http://sourceforge.net/projects/keepass/</TextualContent>
           <Name>ProjectPageURL</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>ProjectName</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>{ProjectPageURL:replace:http\\://sourceforge.net/projects:}</TextualContent>
           <Name>ProjectName</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>anchor1</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>KeePass-</TextualContent>
           <Name>anchor1</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>anchor2</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>-Setup</TextualContent>
           <Name>anchor2</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>anchor3</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>exe</TextualContent>
           <Name>anchor3</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>ver_scrape</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>(?:['|"|;]http://downloads.sourceforge.net/project{ProjectName}[\w\s_\.?=%&=\-@/$,]*(?:_v|%20v|[a-z]v|[a-z]-|[a-z]_|/|%20|\s))(\d[\w\s_\.?=%&=\-@/$,]*)(?:/{anchor1}\d[\w\s_\.?=%&=\-@/$,]*{anchor2}\.{anchor3})(?:['|"|\?])</Regex>
           <Url>{ProjectPageURL}files/</Url>
           <Name>ver_scrape</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>truncated_url</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>(?:['|"|;]http://downloads.sourceforge.net/project)({ProjectName}[\w\s_\.?=%&=\-@/$,]*{ver_scrape}/{anchor1}\d[\w\s_\.?=%&=\-@/$,]*{anchor2}\.{anchor3})(?:['|"|\?])</Regex>
           <Url>{ProjectPageURL}files/</Url>
           <Name>truncated_url</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>SF_mirror</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>softlayer</TextualContent>
           <Name>SF_mirror</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>download_url</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>http://{SF_mirror}.dl.sourceforge.net/project{truncated_url}</TextualContent>
           <Name>download_url</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>version</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>{ver_scrape:replace:%20:_}</TextualContent>
           <Name>version</Name>
         </UrlVariable>
       </value>
     </item>
   </Variables>
   <ExecuteCommand />
   <ExecutePreCommand />
   <Category>Security\Password Managers</Category>
   <SourceType>FixedUrl</SourceType>
   <DeletePreviousFile>true</DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />
   <LastUpdated xsi:nil="true" />
   <TargetPath>{target}\{category}\KeePass_setup_v{version:replace: :_}.{url:ext}</TargetPath>
   <FixedDownloadUrl>{download_url}</FixedDownloadUrl>
   <Name>KeePass Password Safe (SETUP)</Name>
 </ApplicationJob>
</Jobs>

Link to comment
Share on other sites

Actually, Ketarin should pretty much "emulate" a bat file. If something does not work, please post the script and the result, so that I can actually look at something. I don't have much time, so I rather not waste an hour on just finding a reproduction case.

Preferably, you just post the whole app.

 

Happy to read that you like it so much though :)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share


×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.