Download from MajorGeeks

[MadDog]

[h]Edit:

With version 0.9.9.22 and template importing (see Template section of the forum) this how-to is more than obsolete. Admin, Mod, feel free to erase this thread. [/h]

 

 

 

I don't know when Ketarin learned to use functions and variables in the "Edit variables" window, but this opens endless possibilities. Like downloading from MajorGeeks. Yay! I thought I write a quick how-to, as it isn't that obvious.

 

Improvements are more than welcome!

 

Example:

 

I want to download Realtek Drivers for Vista.

 

 

[h]Global Variable "majorgeeks"[/h]

 

First, go to File/Settings and create a global variable majorgeeks

 

Enter file=9

 

This selects the MajorGeeks mirror to be used. Use the download mirror you usually use.

 

file=15 Majorgeeks mirror FL - USA

file=10 ... TX-USA

file=11 ... TX-USA

file=9 ... FL-USA

 

(Do not enable "share this application with others" if you use a global variable. If you want to share this application with others, hardcode the file=.. string in the URL variable.)

 

 

 

[h]Variable "version"[/h]

 

Add a new application and click on Variables.

 

Create a variable version and load the URL you get from Majorgeeks:

 

URL: http://www.majorgeeks.com/Realtek_High_Definition_Audio_for_Vista_d5513.html

 

Select the version number using "use selection as start / end".

 

The result should be: 2.14

 

 

 

[h]Variable "dl_help"[/h]

 

Create a second variable dl_help.

 

Select Content from URL (Regular Expression)

 

Use the Majorgeeks URL: http://www.majorgeeks.com/Realtek_High_Definition_Audio_for_Vista_d5513.html

 

Regular expression: (?<=downloadget.php\?).+?(?=")

 

Result should be something like: id=5513&file=1&evp=9b4a172533b10c552baab558f5d72a2a

 

 

 

[h]Variable "dl"[/h]

 

Now finally create variable dl

 

Select Content from URL (Regular Expression)

 

URL: hxxp://www.majorgeeks.com/downloadget.php?{dl_help:split:&:0}&{majorgeeks}&{dl_help:split:&:2} (note: replace hxxp with http)

Regular Expression: (?<=URL=).+?(?=")

 

Result should be: hxxp://files1.majorgeeks.com/files/30ee748d38e21392de740e2f9dc686b6/drivers/Vista_R214.exe

 

 

 

Well, that's it. Click OK, right-click into the "Download Source URL:" field and select {dl}.

Edited January 20, 2009 by MadDog

[Stalker]

I would actually suggest against redefining any of built-in variables except {version}. This could, and probably would, show problematic when trying to use something like {url:ext}. There is absolutely no reason why you shouldn't just enter the contents of {url} variable directly to the link field.

Edited January 13, 2009 by Stalker

[MadDog]

True, didn't think of it.

 

> There is absolutely no reason why you shouldn't just enter the contents of {url} variable directly to the link field.

 

MajorGeeks kinda generates the download url...

Edited January 13, 2009 by MadDog

[Stalker]

Oh, I didn't notice that it was actually getting something I thought it was just a composite variable with no function rather then to collect all other variables into one string.

[CybTekSol]

I used a similar technique to create a 'Generic MajorGeeks Template' that I can import each time I add an app from MajorGeeks. After importing the template, I modify the name and the save file location... then feed the first two of three variables the apps page URL from MajorGeeks and that's it. Works like a charm! Got to love Ketarin, eeh? I've had to modify my regular expression for one app from MajorGeeks as it is not available on all file servers. Strange that the app is not available on all. It's simply a matter of removing four characters in the regular expression. All of the others work with the first file server scraped.

[MadDog]

Your solution sounds easier to use. :cool: Do you mind sharing your template? I'd love to see it.

[Kerberos]

Major Geeks was giving me a bit of a hard time, but I wasn't as perceptive as you were. I didn't think to just craft the final download URL from scrapes. I just noticed that it was redirecting to a page with an "HTTP-EQUIV Refresh" element pointing to the file I wanted, something I've seen a lot of download sites do, so I figured I'd give Ketarin the ability to automatically follow the refresh link. It was quite annoying at first, but I finally figured it out. Ketarin's structure can be quite intriguing sometimes.

 

If anyone modifies and compiles their own version I'd be happy to share how I got it to follow the refresh link. It's not as easy as I thought it would be. The recursion almost got to me. Or Flo, if you would like to implement it as a permanent feature at some point, let me know. It's quite stable, no problems yet. I'm going to throw some more sites at it though and see how it does.

[CybTekSol]

@MadDog - Yes I'd planned to share my templates next week when I do my tutorial on the subject... but, since you've asked I'll post the template for MajorGeeks with a brief overview and relevant variables either when I return home late tonight or tomorrow morning.

 

@Kerberos - You are right-on the money, even many app developers' sites use the "HTTP-EQUIV Refresh" element and the ability to process that is what I had begun to tackle... I'm ecstatic that you have conquered it. I can't wait to see it in action! Congrats!

 

I'll be back later...

[CybTekSol]

@MadDog

 

I apologize for the delay in getting back to you regarding my MajorGeeks template. I got sidetracked last night testing my new template for BetaNews FileForum... I have successfully created a template that uses their search engine... all you have to feed it is the app name or if you get more than one return to scrape, the author and app name. It seems to work consistently. I'll try to find the time to share that one later if anyone is interested.

 

This is my LATEST MajorGeeks template:

 

1) There are three user defined global variables used, redir1={redir1} (short for re-direct 1)

and download_url={download_url} (used as an alternative to {url}) and root_folder={Insert YOUR download destination ROOT folder here} (for example C:\Latest Downloads).

 

2) The latest version of Ketarin (.0.9.9.22) released January 18, 2009 or later is required

as this revision uses the new 'placeholder' feature implemented by Florian.

 

3) The template follows:

<?xml version="1.0" encoding="utf-16"?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
   <DownloadBeta>Avoid</DownloadBeta>
   <DownloadDate xsi:nil="true" />
   <VariableChangeIndicator>{version}</VariableChangeIndicator>
   <CanBeShared>true</CanBeShared>
   <ShareApplication>false</ShareApplication>
   <HttpReferer />
   <Variables>
     <item>
       <key>
         <string>version</string>
       </key>
       <value>
         <UrlVariable>
           <VariableType>RegularExpression</VariableType>
           <Regex><title>\|MG\| [a-z| ]+([\d|.| ||-||_|a-z]+)</title></Regex>
           <Url><placeholder name="Application Info Page URL from Major Geeks" /></Url>
           <Name>version</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>redir1</string>
       </key>
       <value>
         <UrlVariable>
           <VariableType>RegularExpression</VariableType>
           <Regex><A href="(downloadget.php\?id=[\d]+&file=[\d][\d]&[a-z]+=[\d|a-z]+)"></Regex>
           <Url><placeholder name="Application Info Page URL from Major Geeks" /></Url>
           <Name>redir1</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>download_url</string>
       </key>
       <value>
         <UrlVariable>
           <VariableType>RegularExpression</VariableType>
           <Regex>URL=(http://files[\d].majorgeeks.com/files/[\d|a-z]+/[a-z]+/[a-z|\d|_|.]+)"></Regex>
           <Url>http://majorgeeks.com/{redir1}</Url>
           <Name>download_url</Name>
         </UrlVariable>
       </value>
     </item>
   </Variables>
   <ExecuteCommand />
   <Category>[iNSERT CATEGORY HERE]</Category>
   <SourceType>FixedUrl</SourceType>
   <DeletePreviousFile>false</DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />
   <LastUpdated xsi:nil="true" />
   <TargetPath>{root_folder}\{category}\{appname:replace: :_}_v{version:replace: :_}.{url:ext}</TargetPath>
   <FixedDownloadUrl>{download_url}</FixedDownloadUrl>
   <Name><placeholder name="Application Name" /></Name>
 </ApplicationJob>
</Jobs>

Note** I use PSPad to edit my XML exports... Florian suggests NotePad2 if my memory

serves me correctly... both are freeware and I like both. NotePad2 is lighter and

quicker in my tests, so I will probably switch.

 

To use it:

 

1) Define the user defined global variables under 'Settings' in Ketarin to YOUR preferences

2) Edit the template externally to reflect these modifications (if you chose not to use mine)

and change the <TargetPath> line to YOUR preferences

3) Then:

a) Import it (Obviously)

Enter the information needed as requested as a result of the 'placeholder' prompts...

it will ask for the application's information page URL and the application's name

c) Modify the Category to your preferences

d) Change the 'Advanced Settings' to your preferences

 

 

NOTE****

 

IF YOU WISH TO CREATE A TEMPLATE from one of YOUR XML exports:

 

1)It is important that the Guid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" string be

removed from the XML export to use it as a template, otherwise you'll just keep

overwriting the previously saved application entry that was created using the

imported template

2) The <DownloadDate> line must be changed to <DownloadDate xsi:nil="true" />

3) The <LastUpdated> line must be changed to <LastUpdated xsi:nil="true" />

4) Remove the entire <PreviousLocation> line

5) Create 'placeholder' entries as needed (see THIS thread)

 

ADDENDUM****

 

Florian has found an application hosted by MajorGeeks (Process Lasso - 64bit Version) that this template will NOT work with... unfortunately, I do not see an easy solution to this problem. This is the nature of a 'universal' template... I DO NOT present this template as a perfect creation (far from it). I offer this in an attempt to decrease repetitive entries for an application downloaded from MajorGeeks and to stir the creative juices in Ketarin users... your mileage may vary. It WILL work with most applications at MajorGeeks for the moment... when you encounter one in which it does not... tweak the regular expressions as needed.

 

I hope this helps...

 

CybTekSol

[MadDog]

Thanks for sharing.

 

This is very much the same I did, but while I copy/pasted the regular expression from appyface's tutorial, you wrote own ones. The one getting the version number is very neat.

 

Using a template is of course nicer than copy/pasting. Thanks also for the info how to turn an export into a template.

 

 

I've a question about the global variable redir1. What is it doing? The download works without it just fine.

 

 

> new template for BetaNews FileForum...

 

Sounds interesting. I'd like to see it, of course. Perhaps create a new thread "Share your templates"... ?

[CybTekSol]

@MadDog,

 

I'm glad I was able to help.

 

I use the {redir1} variable to form the 'Contents from URL:' (hxxp://majorgeeks.com/{redir1}) that the app download URL is scraped from... take a look at the url entry in the 'edit variables' dialog after importing the template. Without this, my template fails to scrape this URL for download unless I'm missing something obvious (has happened before).

 

As for the 'Share Your Templates' thread... I discussed this with Florian before when I suggested the 'Tips and Tutorials' section last month and if Flo doesn't have a problem with creating another section for this purpose under the 'Resources' category of the forum, that would probably be the best way to accomplish it. I'm in favor of it, so, Florian if you are monitoring this thread, please give us your thoughts. Otherwise... we can start a thread under 'Tips and Tutorials'... MadDog, would you be willing to help with moderation of a separate 'Share Your Templates' section if Flo chooses to go that route? Obviously either way the best that could be done is to post the XMLs as text (which may wrap at posting) and not the files themselves unless Flo has a different solution.

[floele]

I'll create a new sub-form.

 

Also, future versions of Ketarin will replace global variables before uploading them to the online database. This does not alter your own application though.

[CybTekSol]

THANKS Florian!! And created within minutes... now that's service!

[CybTekSol]

@MadDog,

 

I'll try to get my 'BetaNews FileForum template' that uses their search engine posted in the new 'Templates' sub-forum that Flo created within a couple of days.

[CybTekSol]

@MadDog,

 

I'll try to get my 'BetaNews FileForum template' that uses their search engine posted in the new 'Templates' sub-forum that Flo created within a couple of days.

I may have to delay this as they have changed their search engine's posted html and I am pulling my hair out trying to refine a regular expression scrape that will work consistently.

[MadDog]

Hi, sorry to hear your template stopped working. I hope you'll get it working again. It's always nice to see what power users can do with Ketarin and regex.

 

Being a mod... thanks for the offer, but no, sorry.

[CybTekSol]

Hi, sorry to hear your template stopped working. I hope you'll get it working again. It's always nice to see what power users can do with Ketarin and regex.

I still use it... simply make small adjustments to the regex if needed. Thanks for labeling me a power user! I try hard.

Being a mod... thanks for the offer, but no, sorry.

I understand and thanks for considering it.