Jump to content
Ketarin forum

Scrape help wanted

appyface

By appyface,
in General discussion

 Share

Recommended Posts

appyface Newbie

appyface

Posted
Posted

I have spent more than a little time trying to set up Ketarin to download this app, and have come up empty.

 

Please note there are other places where I can download from with Ketarin, so getting the file is not my issue :) I'm peeved at not being able to solve the challenge :-)

 

So don't feel like you have to spend a bunch of time looking at this, unless are also motivated by the challenge. :) (Or maybe it is not a challenge for you, in which case I feel really stupid LOL.)

 

 

Here's the app: The latest released version of JavaSE JDK *from the official Sun website*.

 

Here's the manual path to download the app:

 

1. Visit here: http://java.sun.com/javase/downloads/index.jsp

 

2. Click the 'download' button for the second listing (JDK 6 Update 12, at time of this writing)

 

3. Receive this SSL page: https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/ViewProductDetail-Start?ProductRef=jdk-6u12-oth-JPR@CDS-CDS_Developer

 

4. I select "Windows" for platform, mark the box to agree to Sun's license, and click the 'Continue' button.

 

5. Receive this SSL page: https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/ViewFilteredProducts-SingleVariationTypeFilter

 

6. I right-click the download filename (jdk-6u12-windows-i586-p.exe) and choose "save target as". From there I download the file.

 

 

Ignoring for a moment, the fact that all these links except the first download page, are hardcoded to a specific version of the app, I can't even get an entry in Ketarin to download the hard-coded one! This just bugs me :)

 

Of course the real goal would be to have Ketarin download whatever is the current version.

 

If anyone is interested in taking up the challenge and figuring out how to configure Ketarin to get the app *from here*, I'm interested in your ideas.

 

Good hunting!

--appyface

Link to comment
Share on other sites
somerandomhash Newbie

somerandomhash

Posted
Posted (edited)

 

Of course the real goal would be to have Ketarin download whatever is the current version.

 

If anyone is interested in taking up the challenge and figuring out how to configure Ketarin to get the app *from here*, I'm interested in your ideas.

 

Good hunting!

--appyface

 

Challenge Accepted :-)

 

From the source I can extract a direct link which I imagine you have already tried, I have just tried to work something out and am not entirely sure what I came up with here but are you able to to get it into Ketarin at all, I have just experimented with the 'hardcoded links' from the source and entered them as 'textual content'

 

While this is not ideal I can confirm I managed to download via Ketarin however I could NOT appear to 'manage start & end' the Load content from url "http://java.sun.com/javase/downloads/index.jsp" or "https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/ViewFilteredProducts-SingleVariationTypeFilter" why I am not sure??

 

Caveat, while here I removed the encrypted variables it would be interesting to see if some of these are static despite the request for example {ProductID} maybe even {OrderID} although I expect that value i unique per request.

 

Also due to the problem I encountered with 'manage start & end' from url the platform and version are also hard coded so in reality all it's doing is downloading via Ketarin I have not got as far as working out how it would update but if the start and end point can be read from the downloads url for {version} & {platform} there is no reason it would not work with updates?

 

10/02/2009 23:35:41: Update started with 1 application(s)
10/02/2009 23:35:41: SunJRETest: Replacing {sdm} in 'http://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/VerifyItem-Start/SunJRETest-{platform}.exe?sdm={sdm}&BundledLineItemUUID={BundledLineItemUUID}&OrderID={OrderID}&ProductID={ProductID}&FileName=/{version}-{platform}.exe' with '[b]xxxxxxxxxxxxxx[/b]'
10/02/2009 23:35:41: SunJRETest: Replacing {BundledLineItemUUID} in 'http://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/VerifyItem-Start/SunJRETest-{platform}.exe?sdm=[b]xxxxxxxxxxxxxx[/b]&BundledLineItemUUID={BundledLineItemUUID}&OrderID={OrderID}&ProductID={ProductID}&FileName=/{version}-{platform}.exe' with '[b]xxxxxxxxxxxxxx[/b]'
10/02/2009 23:35:41: SunJRETest: Replacing {OrderID} in 'http://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/VerifyItem-Start/SunJRETest-{platform}.exe?sdm=[b]xxxxxxxxxxxxxx[/b]&BundledLineItemUUID=[b]xxxxxxxxxxxxxx[/b]&OrderID={OrderID}&ProductID={ProductID}&FileName=/{version}-{platform}.exe' with '[b]xxxxxxxxxxxxxx[/b]'
10/02/2009 23:35:41: SunJRETest: Replacing {ProductID} in 'http://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/VerifyItem-Start/SunJRETest-{platform}.exe?sdm=[b]xxxxxxxxxxxxxx[/b]&BundledLineItemUUID=[b]xxxxxxxxxxxxxx[/b]&OrderID=[b]xxxxxxxxxxxxxx[/b]&ProductID={ProductID}&FileName=/{version}-{platform}.exe' with '[b]xxxxxxxxxxxxxx[/b]'
10/02/2009 23:35:41: SunJRETest: Replacing {version} in 'http://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/VerifyItem-Start/SunJRETest-{platform}.exe?sdm=[b]xxxxxxxxxxxxxx[/b]&BundledLineItemUUID=[b]xxxxxxxxxxxxxx[/b]&OrderID=[b]xxxxxxxxxxxxxx[/b]&ProductID=[b]xxxxxxxxxxxxxx[/b]&FileName=/{version}-{platform}.exe' with 'jre-6u12'
10/02/2009 23:35:41: SunJRETest: Replacing {platform} in 'http://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/VerifyItem-Start/SunJRETest-{platform}.exe?sdm=[b]xxxxxxxxxxxxxx[/b]&BundledLineItemUUID=[b]xxxxxxxxxxxxxx[/b]&OrderID=[b]xxxxxxxxxxxxxx[/b]&ProductID=[b]xxxxxxxxxxxxxx[/b]&FileName=/jre-6u12-{platform}.exe' with 'windows-i586-p'
10/02/2009 23:35:42: SunJRETest: Checking if update is required...10/02/2009 23:41:02: SunJRETest: Update required, 'C:\Browsers and Plugins\jre-6u12-windows-i586-p.exe' does not yet exist
10/02/2009 23:41:16: SunJRETest: Replacing {version} in '{version}' with 'jre-6u12'
10/02/2009 23:41:16: Update finished

 

10/02/2009 23:42:44: Update started with 1 application(s)
10/02/2009 23:42:45: SunJRETest: Checking if update is required...
10/02/2009 23:42:45: SunJRETest: Update not required
10/02/2009 23:42:45: SunJRETest: Replacing {version} in '{version}' with 'jre-6u12'
10/02/2009 23:42:45: Update finished

 

Not ideal but worthwhile 1/2 hour pent learning a little more.

 

P.S I would be interested to know what value you get for {ProductID} {OrderID} if you try this (assume you use the same files) you can email me via forum.

 

P.P.S It would be MUCH easier if FileHippo added this lol

 

UPDATE:

 

I deleted the file and have downloaded with the same params a few times so it's a headstart BTW I am using 1.01.184

 

Example Link: (USD could be currency but not sure)

 

http://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/{locale}/-/USD/VerifyItem-Start/{appname}-{platform}.exe?sdm={sdm}&BundledLineItemUUID={BundledLineItemUUID}&OrderID={OrderID}&ProductID={ProductID}&FileName=/{version}-{platform}.exe

 

Lets know your results :-0

 

PS I just realised that you were looking at JavaSE JDK and this is JRE but it should still work

Edited by somerandomhash
Link to comment
Share on other sites
appyface Newbie

appyface

Posted
  • Author
Posted

@somerandomhash

 

Sorry for delay in getting back here.

 

No, Sun does not handle JDK same as JRE. I already get the latest JRE updates easily from Sun's site. It's the JDK challenge that I'm interested in solving :)

 

Have you tried the JDK? If you try it and get it to work, please post back :)

 

Thanks!

--appyface

Link to comment
Share on other sites
  • 1 month later...
FranciscoR Newbie

FranciscoR

Posted
Posted (edited)

Not sure if this will help, but I have been looking at this one (JDK) again. I see no reference to Verisign in your comments above, but, since the *first* input @ http://java.sun.com/javase/downloads/index.jsp, there are 3 POST requests to their server @ http://ocsp.verisign.com/. (not sure if I am getting a proper encoding ?)

 

1.

0q0o0M0K0I0 +ÎÉË4/DÁ¬JÈû³m×Ý^¾Ï\ãÜÒòí¸¬ùúh·êª Ñq`O¾¨ÂÈ¢00 +0 0 +0

2.

0q0o0M0K0I0 +ÎÉË4/DÁ¬JÈû³m×Ý^¾Ï\ãÜÒòí¸¬ùúU~Dc|+írùJ<ÌW¢00 +0 0 +0

3.

0q0o0M0K0I0 +ÎÉË4/DÁ¬JÈû³m×Ý^¾Ï\ãÜÒòí¸¬ùútèçø1¥%nêu/ë6¢00 +0 0 +0

 

In the end *only* you have the final POST request @ https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/ViewFilteredProducts-SingleVariationTypeFilter, (in the Select Platform and Language page) with the following parameters and values:

 

4.

submitName1=VVA_KylIohiUumSjApgfWEOPzGxZ

submitName2=VVA_xfqhKzkIkArSAZZmOxradlbb

submitName1Type=Platform

submitName2Type=Language

VVA_KylIohiUumSjApgfWEOPzGxZ=Windows

VVA_xfqhKzkIkArSAZZmOxradlbb=Multi-language

ProductUUID=epRIBe.puk4AAAEezTlTv_an

ProductRegistration=0

LicenseUUID=

LICENSE-epRIBe.puk4AAAEezTlTv_an=on

LoginForm_Login=Continue+%C2%BB

 

 

This should now work (using #4 alone ?) but I didn't bother trying - too troublesome for me.

Edited by FranciscoR
Link to comment
Share on other sites
floele Newbie

floele

Posted
Posted

So...assuming you have the latest beta, you can do this:

 

dljebl.jpg

 

The following XML will do the job:

 

<?xml version="1.0" encoding="utf-16"?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="3800febc-881d-4302-bc66-f4fd0412bbd5">
   <DownloadBeta>Default</DownloadBeta>
   <DownloadDate xsi:nil="true" />
   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator />
   <CanBeShared>true</CanBeShared>
   <ShareApplication>false</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <HttpReferer />
   <Variables>
     <item>
       <key>
         <string>UrlOfForm</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>Java SE Runtime Environment \(JRE\).+?"(https://.+?)"</Regex>
           <Url>http://java.sun.com/javase/downloads/index.jsp</Url>
           <Name>UrlOfForm</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>ActionUrl</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>action="(.+?)"</Regex>
           <Url>{UrlOfForm}</Url>
           <Name>ActionUrl</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>FinalUrl</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>value="(http.+?)"</Regex>
           <PostData>VVA_CGpcrYAOYRSioFdZOIihIJJJ=Windows&VVA_EZgJuGXIecpyhSdemJBAOFSk=Multi-language&ProductUUID=%7bProductUUID%7d&LICENSE-%7bProductUUID%7d=on</PostData>
           <Url>{ActionUrl}</Url>
           <Name>FinalUrl</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>ProductUUID</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>name="ProductUUID" value="(.+?)"/></Regex>
           <Url>{UrlOfForm}</Url>
           <Name>ProductUUID</Name>
         </UrlVariable>
       </value>
     </item>
   </Variables>
   <ExecuteCommand />
   <ExecutePreCommand />
   <Category />
   <SourceType>FixedUrl</SourceType>
   <DeletePreviousFile>true</DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />
   <LastUpdated xsi:nil="true" />
   <TargetPath>c:\temp\</TargetPath>
   <FixedDownloadUrl>{FinalUrl}</FixedDownloadUrl>
   <Name>Java (JRE)</Name>
 </ApplicationJob>
</Jobs>

 

PS: Did you know, that you don't have to export an application in order to get the XML? Just use Ctrl+C in Ketarin and then Ctrl+V in any text editor.

Link to comment
Share on other sites
floele Newbie

floele

Posted
Posted

Sure, use this one:

 

<?xml version="1.0" encoding="utf-16"?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="3801febc-881d-4302-bc66-f4fd0412bbd5">
   <DownloadBeta>Default</DownloadBeta>
   <DownloadDate xsi:nil="true" />
   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator />
   <CanBeShared>true</CanBeShared>
   <ShareApplication>false</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <HttpReferer />
   <Variables>
     <item>
       <key>
         <string>UrlOfForm</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>Java SE Development Kit \(JDK\).+?"(https://.+?)"</Regex>
           <Url>http://java.sun.com/javase/downloads/index.jsp</Url>
           <Name>UrlOfForm</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>ActionUrl</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>action="(.+?)"</Regex>
           <Url>{UrlOfForm}</Url>
           <Name>ActionUrl</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>FinalUrl</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>value="(http.+?)"</Regex>
           <PostData>VVA_KylIohiUumSjApgfWEOPzGxZ=Windows&VVA_xfqhKzkIkArSAZZmOxradlbb=Multi-language&ProductUUID=%7bProductUUID%7d&LICENSE-%7bProductUUID%7d=on</PostData>
           <Url>{ActionUrl}</Url>
           <Name>FinalUrl</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>ProductUUID</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>name="ProductUUID" value="(.+?)"/></Regex>
           <Url>{UrlOfForm}</Url>
           <Name>ProductUUID</Name>
         </UrlVariable>
       </value>
     </item>
   </Variables>
   <ExecuteCommand />
   <ExecutePreCommand />
   <Category />
   <SourceType>FixedUrl</SourceType>
   <DeletePreviousFile>true</DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />
   <LastUpdated xsi:nil="true" />
   <TargetPath>c:\temp\</TargetPath>
   <FixedDownloadUrl>{FinalUrl}</FixedDownloadUrl>
   <Name>Java (JDK)</Name>
 </ApplicationJob>
</Jobs>

Link to comment
Share on other sites
FranciscoR Newbie

FranciscoR

Posted
Posted (edited)

PS: Did you know, that you don't have to export an application in order to get the XML? Just use Ctrl+C in Ketarin and then Ctrl+V in any text editor.

This alone makes the whole process a lot faster, thanks. The preview is also nice. ;)

 

I was thinking: what about a XML direct edit inside Ketarin ? With a shortcut/ right-click option ? E.g.: I push F11, Ketarin opens Notepad2, Pspad... I edit XML, save/close it, and job is updated... this would make the process even more straightforward. We could tell Ketarin how to locate preferred Notepad with a new predifined variable, {notepad}...

 

P.S. Thanks for XML Flo, this is an excellent example on how to use POST data ! =D But why didn't you forward ALL data being posted ? How can you tell ?

 

scgqit.png

(red=forwarded)

Edited by FranciscoR
Link to comment
Share on other sites
CybTekSol Newbie

CybTekSol

Posted
Posted

A note for those wishing to import and test the two XML exports that Flo posted today for Java (JRE) and (JDK)... they both have the same 'Guid' assignment...

Guid="3800febc-881d-4302-bc66-f4fd0412bbd5"

which means either one will overwrite the first one imported. Just remove this info to avoid this. ;)

 

Flo, this beta is VERY impressive indeed! :D I'm a little confused about how you decided which POST data was needed as I believe FranciscoR is also... can you clarify this for us?

Link to comment
Share on other sites
floele Newbie

floele

Posted
Posted

Well, I suppose it comes down to trial and error. Usually it's easy to guess, in this case however, there is a whole lot of cryptic POST data being created (some Java based form manager probably). You can start with submitting all post data, and then remove one at a time until it doesn't work anymore and you have found the minimum of data you have to submit.

 

@FranciscoR: Why do you actually want to edit the XML that often?

Link to comment
Share on other sites
FranciscoR Newbie

FranciscoR

Posted
Posted (edited)

Yeah, lots of cryptic post data out there. Excellent job ! ;)

 

Well, seems to me editing XML directly is the fastest way to do it. When I'm creating a template I do a lot of exports-imports, and this would help speeding things up. And it's kind'a logic, IMO we don't need an integrated editor, just a HDD link to our favorite one. =)

 

--------------------

 

Edit: example:

 

#1 - pasting raw post data into XML:

 

<PostData>submitName1=VVA_KylIohiUumSjApgfWEOPzGxZ&submitName2=VVA_xfqhKzkIkArSAZZmOxradlbb&submitName1Type=Platform&submitName2Type=Language&VVA_KylIohiUumSjApgfWEOPzGxZ=Windows&VVA_xfqhKzkIkArSAZZmOxradlbb=Multi-language&ProductUUID=epRIBe.puk4AAAEezTlTv_an&ProductRegistration=0&LicenseUUID=&LoginForm_Login=Continue+%C2%BB</PostData>

 

#2 - changing data into Ketarin pairs, saving/updating DB:

 

<PostData>submitName1=VVA_KylIohiUumSjApgfWEOPzGxZ&submitName2=VVA_xfqhKzkIkArSAZZmOxradlbb&submitName1Type=Platform&submitName2Type=Language&VVA_KylIohiUumSjApgfWEOPzGxZ=Windows&VVA_xfqhKzkIkArSAZZmOxradlbb=Multi-language&ProductUUID=epRIBe.puk4AAAEezTlTv_an&ProductRegistration=0&LicenseUUID=&LoginForm_Login=Continue+%C2%BB</PostData>

 

#3,4,5,6...? - after testing removing unnecessary data, saving/updating DB:

 

<PostData>VVA_KylIohiUumSjApgfWEOPzGxZ=Windows&VVA_xfqhKzkIkArSAZZmOxradlbb=Multi-language&ProductUUID=epRIBe.puk4AAAEezTlTv_an</PostData>

 

#7 - inserting some variables, saving/updating DB:

 

<PostData>%7bPlatform%7d=Windows&VVA_xfqhKzkIkArSAZZmOxradlbb=Multi-language&ProductUUID=%7bProductUUID%7d</PostData>

 

 

PS: This is just an idea, I'm quite happy with the latest MAJOR changes you introduced ! ;)

Edited by FranciscoR
Link to comment
Share on other sites
CybTekSol Newbie

CybTekSol

Posted
Posted
A note for those wishing to import and test the two XML exports that Flo posted today for Java (JRE) and (JDK)... they both have the same 'Guid' assignment...

Guid="3800febc-881d-4302-bc66-f4fd0412bbd5"

which means either one will overwrite the first one imported. Just remove this info to avoid this. ;)

Flo has corrected this issue manually, so please disregard this previous post of mine.

Link to comment
Share on other sites
appyface Newbie

appyface

Posted
  • Author
Posted
Hi Flo,

 

Looking good!

 

Your example shows the JavaSE (which Ketarin could already download). My request is for the JavaJDK.

 

Can you show example with the JDK I'm asking about?

 

--appyface

With Sun's latest updates (JRE and JDK to 6u13), these two XML examples give error now. Unfortunately my day job is requiring a lot of OT so I didn't have the time to fully understand how these worked. I just put them in and enjoyed the benefits... now that they give error I don't know how to fix them. :( :( :(

 

If anyone else here uses these particular examples and would like to share how these were fixed up to work with 6u13, I would appreciate the help. Otherwise I can get along with good old fashioned manual downloads until I can look at this myself (a couple of months from now :( ).

 

TIA,

--appyface

Link to comment
Share on other sites
CybTekSol Newbie

CybTekSol

Posted
Posted
If anyone else here uses these particular examples and would like to share how these were fixed up to work with 6u13, I would appreciate the help.

I just played with the POST data changes for nearly an hour and couldn't get it going for JDK 6u13... I think Flo will have to look at this one if time allows as that was all the time I could devote to it for now. Sorry appyface, I tried. :(

Link to comment
Share on other sites
FranciscoR Newbie

FranciscoR

Posted
Posted (edited)

Kinda weird. {finalurl} is not loaded properly into Ketarin, because of session cookies ?

 

<p>Login failed. You must enable cookies in your browser and then click on the Reload button to continue.</p>

 

 

Edit:

 

An alternative DL is Softpedia, http://www.softpedia.com/get/Programming/SDK-DDK/Sun-Java-JDK.shtml , but you have to adjust CybTekSol regex to DL

 

Softpedia Secure Download (US) - x86 - http://download.softpedia.com/dl/fc154a09305d674bcc1246e3530769d6/49cbdbd0/100082226/software/programming/jdk-6u13-windows-i586-p.exe

 

Otherwise you will be downloading the app downloader (68 KB).

Edited by FranciscoR
Link to comment
Share on other sites
appyface Newbie

appyface

Posted
  • Author
Posted

Thanks, both of you, for taking a look at this one. I wish I had taken the time to screen print and document the scrapes, step-by-step, back when it worked. Then I could compare and see what has changed.

 

Yes the {FinalUrl} variable is empty -- I am guessing because the {UrlOfForm} loaded something quite different in the past, the regex to scrape into {FinalUrl} found what it needed.

 

What that was, though, I don't have a clue. One of those problems when your target is a moving target...

 

Thanks again for looking into it.

--appyface

Link to comment
Share on other sites
FranciscoR Newbie

FranciscoR

Posted
Posted

No {FinalUrl} is not empty, scroll down (or use find: cookie) and you will see the message I posted. I updated all the post fields - like CybTekSol I suppose - without success but I believe the problem is not Flo's XML. Currently {FinalUrl} regex won't match a thing (you're not logged as before), hence the error.

Link to comment
Share on other sites
floele Newbie

floele

Posted
Posted

Well, there is a small problem. This, for example, could work:

 

<?xml version="1.0" encoding="utf-16"?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="3800febc-881d-4302-bc66-f4fd0412bbd5">
   <DownloadBeta>Default</DownloadBeta>
   <DownloadDate xsi:nil="true" />
   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator />
   <CanBeShared>true</CanBeShared>
   <ShareApplication>true</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <HttpReferer />
   <Variables>
     <item>
       <key>
         <string>UrlOfForm</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>Java SE Development Kit \(JDK\).+?"(https://.+?)"</Regex>
           <Url>http://java.sun.com/javase/downloads/index.jsp</Url>
           <Name>UrlOfForm</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>ActionUrl</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>action="(.+?)"</Regex>
           <Url>{UrlOfForm}</Url>
           <Name>ActionUrl</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>FinalUrl</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>value="(http.+?)"</Regex>
           <PostData>%7bsubmitName1%7d=Windows&%7bsubmitName2%7d=Multi-language&ProductUUID=%7bProductUUID%7d&LICENSE-%7bProductUUID%7d=on</PostData>
           <Url>{ActionUrl}</Url>
           <Name>FinalUrl</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>ProductUUID</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>name="ProductUUID" value="(.+?)"/></Regex>
           <Url>{UrlOfForm}</Url>
           <Name>ProductUUID</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>submitName1</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>submitName1.+?value="(.+?)"</Regex>
           <Url>{UrlOfForm}</Url>
           <Name>submitName1</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>submitName2</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>submitName2.+?value="(.+?)"</Regex>
           <Url>{UrlOfForm}</Url>
           <Name>submitName2</Name>
         </UrlVariable>
       </value>
     </item>
   </Variables>
   <ExecuteCommand />
   <ExecutePreCommand />
   <Category />
   <SourceType>FixedUrl</SourceType>
   <DeletePreviousFile>true</DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />
   <LastUpdated xsi:nil="true" />
   <TargetPath>c:\temp\</TargetPath>
   <FixedDownloadUrl>{FinalUrl}</FixedDownloadUrl>
   <Name>Java Development Kit (JDK)</Name>
 </ApplicationJob>
</Jobs>

 

However, the page contents when submitName1 and submitName2 are fetched may actually differ each time so that you rarely get two "matching" submit names. Weired.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.

  I accept