Jump to content
Ketarin forum
  • Announcements

appyface

POST form help needed

Recommended Posts

Yes Tomorrow I know what you meant. The fact that you prefer to avoid POST and get a file another way is not relevant to my request.

 

ANY tutorial that can teach how to evaluate a webform's download method is POST and then shows how to construct the POST request in Ketarin that successfully retrieves the file is a useful tutorial. It is of no importance that there are other ways of getting the file because this request is not about getting the file, it is about teaching the method.

 

If you prefer to write a tutorial for the synaptics site that's fine, I'll use it on the Neosmart site and see if it works.

Share this post


Link to post
Share on other sites
I don't understand what you're saying, Flo? This web form has POST specified -- doesn't that mean it uses POST?

 

Yes and no. The form sends its data both per POST (form fields) and GET (target URL specified with the "action" attribute), but in this case, the server side download script does not evaluate the POST content to decide whether or not to start the download. It only checks if the GET parameter "download" is specified. If the owner wanted to, he could check if the POST data is empty and deny the download, but this is not done here.

 

Can you take us through the webpage step-by-step and show us how you discovered it specifies POST but doesn't actually do that?

 

It does use POST, but whether or not the server side script requires the presence of POST data can only be determined by trial and error. It's the same as with user agents: You don't know whether the server side requires any particular browser, unless you make some tests.

 

Flo if you're making a tutorial make it about this: http://www.synaptics.com/resources/drivers

 

Even this is not suitable. If you check, you will notice that you only have to agree to the terms once, no matter which file you download. That hints at the fact that cookies are being used here. And indeed, when you accept the terms, a cookie with the name "_syn_terms" and the value "accept" is being created (you can see this in the Firefox cookie list for the domain synaptics.com). You can then rather easily download them with Ketarin, using the HTTPX protocol:

 

httpx://drivers.synaptics.com/Synaptics_v15_2_20_C_XP32_Vista32_Win7-32_Signed_Marketing_SGS94_UI-Scrybe.exe?cookie:_syn_terms=accept

Share this post


Link to post
Share on other sites
httpx://drivers.synaptics.com/Synaptics_v15_2_20_C_XP32_Vista32_Win7-32_Signed_Marketing_SGS94_UI-Scrybe.exe?cookie:_syn_terms=accept

 

Can this be used for some other things using cookies too?

Share this post


Link to post
Share on other sites

Appy, POST is NOT required for this scenario:

http://neosmart.net/download.php?download&id=1

 

Thus, trying to force it into using POST is an exercise in futility. Further, even *IF* it were required, the current version of Ketarin doesn't have the ability to POST for the download URL. This is simply not an option, so no matter of munging THIS FORM would enable you to both POST the content AND download the file. It just isn't necessary, and it isn't even possible.

 

The POST capabilities within Ketarin are necessary for other situations, however, such as obtaining download server names, obtaining a specific download key ot gathering other information which is USUALLY used in a GET (normal URL) operation. In some situations (again, NOT THIS ONE), the actual download link is a POST operation. Ketarin simply doesn't, yet, have the ability to address these situations, so a third party service like the one used in my Codeplex template is necessary.

Share this post


Link to post
Share on other sites

@shawn You keep using 'not required', which says to me that POST is possible, just not a must. I think you must really mean to say 'not possible to use POST on this form with Ketarin', instead of 'not required'?

 

I'll state my request again: I'm looking for a tutorial on how to evaluate that a form does use POST and then how to use POST in Ketarin to get the file.

 

Equally valid tutorial is what I asked Flo up above: If the form shows POST (as this one does) but Ketarin really can't do POST with it, what is the process I would go through to learn that is the case?

 

I will write both tutorials myself IF I can just get some solid and basic step-by-step information here so I can learn what to do with these things. But so far none of that coming in this thread, just bashing me over the head about things that are 'pointless' and 'not required', none of which is helpful to those goals.

 

How about we now make some positive progress toward both situations?

Share this post


Link to post
Share on other sites
I will write both tutorials myself IF I can just get some solid and basic step-by-step information here so I can learn what to do with these things. But so far none of that coming in this thread, just bashing me over the head about things that are 'pointless' and 'not required', none of which is helpful to those goals.

 

Please keep in mind that we are not trying to bash you or make you look stupid. I'd really like to help, but I cannot properly explain the process of downloading from a website through a POST form, if we don't have a suitable example. I believe we once had one in the forum, but I can't remember which topic it is. Otherwise it's like showing me a car with a flat tire and asking me for an example of how to fix a broken windshield.

 

If the form shows POST (as this one does) but Ketarin really can't do POST with it, what is the process I would go through to learn that is the case?

 

It's not that Ketarin can't do it*. You can make POST requests to whatever URL you like, but you need to check the server side behaviour first. There is no simple procedure, same as with any other download job you contruct for Ketarin.

 

The basic idea is that you create a variable with the URL that is target of the form ("action" attribute), then add POST data to it, using the POST data button and adding the key value pairs that can be extracted from the HTML source ("input" tags, "name" attribute is key and content of the textbox or "value" attribute is the value). If you then load this URL, you would see the result of submitting the form, which should either be another HTML page which contains the real download URL or binary content, which allows extracting the URL from the HTTP headers of that request.

 

As we tried with Java though, this is not a straight forward process. In particular, if there are autogenerated form fields because of Java or ASP.NET generated websites, it is hard to predict or emulate the form behaviour, so that a lot of effort is required to understand how the forms work.

 

Again, with proper examples it would be somewhat easier to explain.

 

 

* Though in this case it does indeed not work, as there is an HTTP redirection to the binary file when submitting the form, which makes Ketarin do a second POST request to the actual download URL, which results in a 403 HTTP error, since the server apparently does not allow a POST request to a binary file. There'd need to be a workaround for such a situation, but so far, no one had a need for it.

Share this post


Link to post
Share on other sites

I don't feel anyone is trying to make me look stupid, nor do I care if they do. I'm just trying to keep all respondents ON TOPIC, which is frustratingly difficult in this forum sometimes.

 

No one here has asked you to make a tutorial to fix a windshield by showing you a flat tire. I asked about a tutorial using POST with this particular site. And, after some pain it was finally revealed that Ketarin cannot be configured to use POST with that form successfully, I asked that we use THAT case for a tutorial. To me it is just as useful to be able to evaluate what cannot work, as well as evaluate what can.

 

The working example in the forum that you recall, is the very same one I referred to in my opening post in this thread -- and sadly it was the Java JDE and JDK before Sun-now-Oracle changed the site around to the present and confounding methods of serving files. However, I believe Tomorrow gave you one a few posts back -- have a look?

 

Thank you for the time you put into your post and you have given some good information. But it will take me some effort to work through it and translate it into the beginnings of some step-by-step tutorials. And, I have no expectation for generalized one-size-fits-all tutorials, I believe there is value in documenting ANY example. It can always be prefaced with the caveat that there are many possible scenarios of which this tutorial covers only one, and is meant to illustrate some idea of how one might go about looking at this stuff and setting it up in Ketarin.

Share this post


Link to post
Share on other sites
The working example in the forum that you recall, is the very same one I referred to in my opening post in this thread -- and sadly it was the Java JDE and JDK before Sun-now-Oracle changed the site around to the present and confounding methods of serving files. However, I believe Tomorrow gave you one a few posts back -- have a look?

Nope apparently that one could be solved trough httpx and specifying a cookie in the url.

 

What about codeplex?.Like shawn said he uses 3rd party service in his template for codeplex apps.

Perhaps a worthy challenge?

Share this post


Link to post
Share on other sites

@Tomorrow Thanks I see now that POST won't work there either...

 

I don't know offhand of another example that might actually work with POST in Ketarin, but I'll look around for one I'm interested in. If the codeplex stuff of shawn's that you mention qualifies, let's use that.

Edited by appyface

Share this post


Link to post
Share on other sites

Let me repeat my Teamspeak 3.x download as simple example. Actually I don't know if the POST method is needed here but it works for me.

 

To get the download url it is necessary to submit the generated mirror url and accept the license agreement (in this case both via POST variables).

 

<?xml version='1.0' encoding='utf-8'?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="c19976b4-c8e0-4ccc-9602-10b1f8e300b8">
   <WebsiteUrl>http://www.teamspeak.com/?page=downloads</WebsiteUrl>
   <UserAgent />
   <UserNotes>Information for variable {url}
----------------------------------------------------------------------------------------
POST data to send with URL:

1. 
name = license
value = eula

2. 
name = mirror
value = {mirror}</UserNotes>
   <LastFileSize>29461184</LastFileSize>
   <LastFileDate>2011-06-17T22:39:18.7391087</LastFileDate>
   <IgnoreFileInformation>false</IgnoreFileInformation>
   <DownloadBeta>Default</DownloadBeta>
   <DownloadDate xsi:nil="true" />
   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator />
   <CanBeShared>true</CanBeShared>
   <ShareApplication>true</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <HttpReferer>http://www.teamspeak.com/?page=downloads</HttpReferer>
   <SetupInstructions />
   <Variables>
     <item>
       <key>
         <string>appcat</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>{appname:regexreplace: \(.*\):}</TextualContent>
           <Name>appcat</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>appfile</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent>{appname:multireplace:||)| :|_|_}</TextualContent>
           <Name>appfile</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>version</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>(?<=Windows.*?Client\s32-bit.*?)[0-9.\-aAbBcCeErRtT]{2,}(?=\<)</Regex>
           <Url>http://www.teamspeak.com/?page=downloads</Url>
           <Name>version</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>url</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>(?<=PAID\sFOR\sTHIS\sLICENSE.*?href\=\").+?(?=\"\>I agree)</Regex>
           <PostData>license=eula&mirror=%7bmirror%7d</PostData>
           <Url>http://www.teamspeak.com/?page=downloads</Url>
           <Name>url</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>mirror</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>(?<=Windows.*?Client\s32-bit.*?option value\=\").+?(?=\")</Regex>
           <Url>http://www.teamspeak.com/?page=downloads</Url>
           <Name>mirror</Name>
         </UrlVariable>
       </value>
     </item>
   </Variables>
   <ExecuteCommand>echo %date% - %time% // {appname} v{version} >> "{file:directory}"\history.txt</ExecuteCommand>
   <ExecutePreCommand />
   <ExecuteCommandType>Batch</ExecuteCommandType>
   <ExecutePreCommandType>Batch</ExecutePreCommandType>
   <Category>Chat</Category>
   <SourceType>FixedUrl</SourceType>
   <PreviousLocation>I:\!Ketarin\Downloads\Chat\TeamSpeak\TeamSpeak_Client_Setup_32-bit_v3.0.0-rc2.exe</PreviousLocation>
   <DeletePreviousFile>true</DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />
   <LastUpdated>2011-06-17T22:39:18.7391087</LastUpdated>
   <TargetPath>{downloads}\{category}\{appcat}\{appfile}v{version}.{url:ext}</TargetPath>
   <FixedDownloadUrl>{url}</FixedDownloadUrl>
   <Name>TeamSpeak (Client Setup 32-bit)</Name>
 </ApplicationJob>
</Jobs>

 

mirror98su.jpg

 

urlbkvy.jpg

 

post1ku6.jpg

Share this post


Link to post
Share on other sites

If you actually would like to see a webpage where POST data is required, take a look at JGSoft.com and check the registered downloads pages...

 

That's my current problem, as there's a form where I need to enter my license data and for sending the data I get the binary download file in return.

 

Just in case you misread:

 

I get the binary file for sending the formular... I'm not getting a website or link to klick and download.

 

I can't download this via ketarin right now (at least I don't know how) since the post url of the form would be the download url and I can't use POST with the download url only with the variables. But it doesn't make any sense to have the downloaded file contained in a variable, does it?

Share this post


Link to post
Share on other sites

Perhaps floele can say something about this. Maybe it's a feature bug ;)? It is hard to test it without license data and I can't find any other example using POST generated downloads...

Share this post


Link to post
Share on other sites

Well... Actually this isn't a problem anymore. I switched to using WGet as external downloader... Not having any problems downloading these files now.

Threre are still some things I wouldn't mind floele taking a look at about using external downloaders in Ketarin right now...

 

But I don't see any reasons to repeat it in another thread (Link)...

Share this post


Link to post
Share on other sites

Old thread of mine ... but I still have no idea about using POST with Ketarin, nor how to tell if it is really being leveraged in the webpage to bother trying with it.

Is this a good example to use?

https://getonetastic.com/downloadFile&fv=0&file=OnetasticInstaller.x86.zip

Go to the above link, agree to the license terms, the click the download button.

Everything I've tried so far gives me a message from Ketarin "file is not binary..."

If this is a good example, please give step by step on how to evaluate and construct the Ketarin entry?

 

 

Share this post


Link to post
Share on other sites

Some background info:

Two commonly used methods for request/response communication between clients and servers are GET and POST. GET requests data and POST submits data. (For a more detailed explanation: W3Schools - HTTP Methods)

Regarding Onetastic: In this case, you can reach your goal in two different ways.

Method 1:

Add &agree=1 to your download link and that's it! :D

For the 32-bit version:
https://getonetastic.com/downloadFile&fv=0&file=OnetasticInstaller.x86.zip&agree=1
For the 64-bit version:
https://getonetastic.com/downloadFile&fv=0&file=OnetasticInstaller.x64.zip&agree=1

Method 2:

Currently, downloading via ketarin only works via method 1, because there are probably also on this page problems with the TLS 1.2 protocol. I hope, floele can fix this issue. But it would work like this:

First of all, we need to check which POST variables must be passed in order for the download to start. There are two ways to this. You can take a look at the source code of the web site or you can use web developer tools (like the Firefox Developer Tools, as described here).

The first way:

0xmuea.jpg

We need the web page to which the variable is passed and, of course, the variables. For this we look in the source code for the form with the variable to be passed. In this case, our required data is in line 37 and 38. In line 37 we have the page to which the variables are sent and in line 38 the variable itself (in this case by using checkboxes, which means checked=1 unchecked=0). They called the variable agree. So the value for the checked checkbox called 'agree' will be 1.

In the next step we use this data in Ketarin.

105u95.jpg

2jwu0r.jpg

You can also use the main downloadpage https://getonetastic.com/downloadFile as download url to pass all the other variables instead. Which looks like this: 6gxuxs.jpg But in this case there is no need to do this. This is just another variation for this website.

The second way:

The source code (and much more) can also be analyzed with the Firefox developer tools. We need the referer, and all the POST variables submitted. Here is a short video of how to do this for Onetastic.

 

 

 

 

 

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×