Download via CGI script/POST data

[GaronneBC]

I got a problem adding some applications I wanted to integrate into Ketarin to download that I registered.

 

My problem is, I need to add data into a POST form, which is sent to a cgi script which in turn supplies the file itself (not a link to it).

 

So, adding POST data for variable evaluation doesn't help me as the variable itself will contain the entire file to be downloaded, not a link pointing to it. I can't add the POST data to the download url though, as there seems to be no option to do that.

 

The applications in question are tools from JGSoftware:

 

One example page would be the download page for AceText.

 

Download CGI:   http://www.acetext.com/cgi-bin/download

 

POST data:

 

lang=en

app=act

email={Registered eMail Address}

id={licence code}

vp=

 

vp can contain various values, in my case it is empty.

 

The CGI script results in the final file, there's no URL generated. It directly returns the exe file containing my registration data.

 

("SetupAceText314[MyName].exe" as of today)

[shawn]

Currently Ketarin does not provide the ability to POST to the download URL. You can get around this by passing your form through a GET-to-POST converter. I have used this one in the past:

http://convert-get-proxy.appspot.com/

It works reasonably well.

[GaronneBC]

Ok, yeah, guessed as much as POSTing to the download URL is not yet part of Ketarin.

Ok, so, how does that convert-proxy-thing work? My japanese (or what ever language that page is in) isn't as good as my english, so... No idea how to use that proxy-thingy.

 

Ok, I added a feature request to extend the httpx-protocol to allow for submitting POSTs to urls. (Link)

[shawn]

Click the "English" button on the right side and it'll translate automatically. Enter the target URL in the top field. Enter the parameter name and value pairs in the next two boxes, clicking 'add query field' for each new parameter set.

 

Finally, click 'compute' and it'll generate the complete URL. Use the "server side" type URL as the DL link in Ketarin.

 

Be aware that since this service is public and simple HTTP, any text or data you send (including email address and license codes) can be cached and stored anywhere between you and the POST URL, including at appspot.com and the code may be using it to harvest request information. Tread carefully.

[GaronneBC]

Hmmm... Quess in that case I'll hope for flo to extend the httpx protocol to allow for POST data. And just let Ketarin CHECK for newer versions and download them manually.

[shawn]

If you're going that route you can use WGET to download the files with POST in the pre-download command. Use something like this:

wget -N --post-data="param1=value1&param2=value2&submit=Return" https://reliableanswers.com/asp/asptest.asp

That page (reliableanswers.com) is one of my sites and you can use it to debug the specific query string and POST data you're playing with. Open it in a browser to see exactly how it operates. Note that from a WGet or other command-line interface it's CRITICAL that you include the parameter "submit=Return", otherwise it will simply ignore the passed data. Use generic text while testing, then change it to your user & license key and the destination URLs to what you need. You'll probably want to use something like this in the final result:

wget -N
 --output-document="mybinaryfile.exe" 
 --no-check-certificate 
 --http-user="myusername" --http-password="mypassword" 
 --post-data="param1=value1&param2=value2"
 --referer=http://example.com/download.html
 http://example.com/download.zip
 1>>"error.log" 2>>"error.log"

The various parameters allow you to set the output document filename, the ability to ignore SSL certificates (sometimes necessary unless you recompile WGet yourself), the ability to pass HTTP username & password, the actual post data, a referrer URL (sometimes necessary), and of course the actual download URL. The last line allows you to log the errors for later debugging use.

 

Good luck!

[GaronneBC]

Ok, unless someone is stupid enough to have the license data sent to your server I guess using WGET sounds reasonably safe.

 

Guess I might give it a try.

 

...

 

At least as long as Ketarin doesn't support POST data for the actual download url.

 

Task 1: Get WGET...

 

---

 

EDIT 1:

Does anybody know if WinWGet from PortableApps.com does work the same way?

I'm definitely able to download my files using the GUI of WinWGet but can't seem to find a Users Guide.

 

EDIT 2:

Ok, WinWGet is just a GUI for WGet which is stored in a subfolder.

 

EDIT 3:

Hmmm, can't seem to get Ketarin to download the file with WGet.

Seems to ignore my pre-download commands (wget + exit 3) and tries to download using it's internal downloader (as well as stored download url)

The pre-download commands don't seem to get triggered...

 

EDIT 4:

Verified: The commands AREN'T triggered. No matter what I try, they are completely ignored.

 

EDIT 5:

Doesn't process either pre-update nor post-update commands, even if I remove prior downloads so a file SHOULD be tried to download anyways... But nothing's happening.

 

EDIT 6:

Ok, unless httpx will support https I will need POST data for downloads at some point anyways... As the autor of the JGSoft Tools just stated to me he might at some time - not that far into the future - require the form POST to be transmitted via https returning the binary file, meaning the download-url will be https AND a POST form.

Edited April 14, 2015 by GaronneBC

[shawn]

Are you sure you don't have "check only" enabled? This will prevent "pre-update" since the update will, of course, not be queued. Within your actual pre-update command, set the last line as:

 

EXIT 3

 

More here:

http://wiki.ketarin.org/index.php/Commands#Pre-update_commands

[GaronneBC]

Nope. Neither "Download independantly" nor "Don't download, just check" are selected.

Neither no download uri nor some fake data worked.

 

Actually I bundled exit 2 and exit 3

wget [...] && exit 3  // exit 3 if wget succeded
exit 2  // exit 2 otherwise

but it didn't help either.

Actually added some

echo Test>\test.txt

at the beginning to check if ANY command is running at all...

Nope, the echo command isn't processed either.

[shawn]

What's the value of the download URL? This has to be populated, too.

 

If you continue to have issues, please post the app profile (select it in the main Ketarin window, CTRL+C, paste it here).

[GaronneBC]

Ok, can anybody explain, why I have to supply a VALID download uri I don't have to use an external downloader?

Using a download uri for an entirely different (but binary) file finally allows me to use WGet to download the file I need.

 

That's kind of nonsense, isn't it?

 

Now I need to figure out what's wrong now with my WGet call... Worked before (from the command line) now it doesn't...

 

Ok... Case closed anyways... For some reason you need a valid download uri to a binary file to use predownload-commands for external downloaders.

 

How about a feature request for a more direct support of external download tools? (Like an additional tab with a similar commands section and a checkbox to enable the external downloader (and disable the internal one - in conjunction with the check if the download uri is a 'valid' one).)

[GaronneBC]

Ok, for anyone who wants to check it out:

<?xml version='1.0' encoding='utf-8'?>
<Jobs>
  <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="ffc3edf6-0f51-41c6-861f-21e7588bfbab">
    <WebsiteUrl />
    <UserAgent />
    <UserNotes />
    <LastFileSize>6585968</LastFileSize>
    <LastFileDate>2014-10-02T06:20:33+02:00</LastFileDate>
    <IgnoreFileInformation>true</IgnoreFileInformation>
    <DownloadBeta>Default</DownloadBeta>
    <DownloadDate xsi:nil="true" />
    <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
    <VariableChangeIndicator>Version</VariableChangeIndicator>
    <CanBeShared>true</CanBeShared>
    <ShareApplication>false</ShareApplication>
    <ExclusiveDownload>false</ExclusiveDownload>
    <HttpReferer />
    <SetupInstructions />
    <Variables>
      <item>
        <key>
          <string>Version</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>StartEnd</VariableType>
            <Regex />
            <Url>http://www.editpadpro.com/download.html</Url>
            <StartText> Version </StartText>
            <EndText>, released </EndText>
            <Name>Version</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>url</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>RegularExpression</VariableType>
            <Regex><A HREF="(http://download.jgsoft.com/acetext/SetupAceTextDemo.exe)"></Regex>
            <Url>http://www.editpadpro.com/download.html</Url>
            <Name>url</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>POST</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>lang=en&app=epp&email=REMOVED&id=REMOVED</TextualContent>
            <Name>POST</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>POSTurl</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>http://www.editpadpro.com/cgi-bin/download</TextualContent>
            <Name>POSTurl</Name>
          </UrlVariable>
        </value>
      </item>
    </Variables>
    <ExecuteCommand />
    <ExecutePreCommand>start "" /wait {wget} "{POSTurl}" -P "\Downloads" -c -O "\Installer\JGSoft\Setup {appname} {Version} [NAME REMOVED].exe" --post-data="{POST}" && exit 3
exit 2</ExecutePreCommand>
    <ExecuteCommandType>Batch</ExecuteCommandType>
    <ExecutePreCommandType>Batch</ExecutePreCommandType>
    <Category>JGSoft</Category>
    <SourceType>FixedUrl</SourceType>
    <DeletePreviousFile>false</DeletePreviousFile>
    <Enabled>true</Enabled>
    <FileHippoId />
    <LastUpdated>2015-04-16T17:44:51.7100135+02:00</LastUpdated>
    <TargetPath>\Installer\JGSoft\{appname} {Version} [NAME REMOVED].exe</TargetPath>
    <FixedDownloadUrl>http://download.jgsoft.com/acetext/SetupAceTextDemo.exe</FixedDownloadUrl>
    <Name>EditPad Pro</Name>
  </ApplicationJob>
</Jobs>

I removed the license data nessisary for downloading the file but anyone owning JGSoft-Apps can easily modify this code to download their versions of those files.

[shawn]

Try unchecking the box "ignore file information" on the advanced settings tab, or press CTRL+F5 ("force download") instead of simply running "update". Looks like you've got it set to only download when the version changes. This is a great option when you've got it working perfectly and you're NOT using an external downloader, but it greatly increases the difficulty of debugging.

[GaronneBC]

The files always differ (personal license info is patched into the files on the fly, so hashes change every time (based on Jan Goyvaerts himself)) if I uncheck "ignore file information" and Ketarin constantly tries to download the files again and again. Seems like my WGet got spoiled somehow... Reimported the command line from WinWGet. Now it works as long as I keep "ignore file information" checked. I need to "force download" the files once, but probably because Ketarin didn't have any means to check if the version changed or not and assumes I already have a current version... Anyways, "force download" works.

 

Still, the only strange thing (at least to me) is, why do I need a valid download uri? That was giving me headaches... Pointed the uri to a demo of one of their tools (not all tools have shareware versions) but as long as I have a valid uri WGet runs and is able to download the files.

 

And exit 3 works fine to opt out before Ketarin tries to download the demo version.

 

Well, the above code is the working code I used for "EditPad Pro", one of said tools.

 

As mentioned above I don't mind if anyone uses it for their own benefit.

 

The variables "Version", "POST" and "POSTurl" need to be changed if you want to use it for the different tools, but should be easy to change now.

I might change "Version" to a RegEx based system so only the url needs to be corrected to be working with the different tools.

 

Guess I might try to create a parameter-based template from it.. But probably not in the next couple of days...

[appyface]

I have same issue, also not able to download registered versions from Just Great Software, mine is EditPadPro and RegExBuddy. Haven't been able to download either one using Ketarin since Jan G. changed his site some months back.  Using Ketarin now only to check for new version.

[shawn]

Since you're using the version number in the filename you could use a file exists check to perform a pre-check for the version so you can enable "ignore previous file" and manually process version data before downloading. Here's a vapor-code sample:

 

 

if exist "\Installer\JGSoft\{appname} {Version} [NAME REMOVED].exe" exit 2
wget ...

 

This will exit before attempting to download if the version-specific download file already exists. If it doesn't then it will proceed to the download.