Download from dynamic links (e.g. Real Temp from Techpowerup)

[jusseppe]

Hello everybody,

any pointers on how to make downloads work for multiple servers / dynamic links such as Real Temp from techpowerup?

e.g. https://www.techpowerup.com/download/techpowerup-real-temp/

 

thank you very much!

 

p.s. another example would be Cheat Engine from Cloudfront:

https://www.cheatengine.org/downloads.php

 

with this one I get "405: method not allowed" error...

 

thanks again!

[shawn]

For TechPowerUp you can use the download page (the one you linked to) then collect the hidden "id" value from it. That's 164 for this app. I name this field "downloadid". With that parsed the URL to download becomes:

https://www.techpowerup.com/download/techpowerup-real-temp/?id={sdownloadid}

The site uses POST in the source code, but I tested way back in 2015 or so when I created a download profile for NVcleaninstall and you can use GET and the download will work fine. If that wasn't possible (if they actually checked for POST vs GET) then the next best option would be to perform a POST operation (using the "POST data" button) to get the raw file URL. 

 

For Cheat Engine you'll need to use a different method. The download URLs generated by cloudfront are unique and evil and rely on HTML5 javascript behaviors that you can't perform with Ketarin alone without massive effort. For these you need to use a browser puppeteer - a scriptable full browser that you can control externally - there's a flavor built for us here by @Ambimind called Kuppet, get it here on the forum.

One of the methods cloudfront uses to "prove" the visitor is human is to insist on a delay before rendering all of the content. For this you'll need to use the "redirection-delay" option. I'm not in a hurry so my redirection delay for any sites that impose one is set to 45 seconds. It feels like a long time (it's not), but it guarantees that sites that impose this delay behavior will work. I'd rather have it work reliably than stall out because I tried to push the envelope.

Just don't go too large because some sites have an idle timer, too, which either refreshes the page or redirects you to their home page after a longer delay, usually 5-15 minutes of inactivity. These webmasters suck because you don't always finish the article before you're thrown to a different page or place on the page when it triggers. Grrr. /soapbox

Please try it and post your XML if it's still not behaving.

[jusseppe]

thanks for your support!

unfortunately I couldn't get either of them to work

for Real Temp, I end up with either a 404 or 200 error... here is my XML.

Regarding Cheat Engine, I wanted to make sure that Kuppet was set correctly first but I can't seem to get the Logitech Options xml to work (the one downloaded from the kuppet page you linked. I get this error (impossible to connect to the remote server):

I think I got kuppet correctly setup:

unless I am using a (very) old version of it and the chrome that came with it?

 

thanks again!

Real Temp.xml Logitech_Options.xml

[shawn]

So the reason why the "downloadid" was failing is because you set the POST data within it. The downloadid should be used to parse *for* the ID value. Then the download_url will work correctly. If the ID remains consistent between all builds then you could probably hard-code it, but I would rather just parse for it to be sure.

Here's the fixed XML:

<?xml version='1.0' encoding='utf-8'?>
<Jobs>
  <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="bd752d5c-099d-4187-83e7-2ddb9cf3d546">
    <Category>System Maintenance &amp; Security</Category>
    <WebsiteUrl>http://www.techpowerup.com/realtemp/</WebsiteUrl>
    <UserAgent>Mozilla/5.0</UserAgent>
    <UserNotes>[Description]
Real Temp is a temperature monitoring program designed for all Intel single Core, Dual Core, Quad Core and Core i7 processors.

Each core on these processors has a digital thermal sensor (DTS) that reports temperature data relative to TJMax which is the safe maximum operating core temperature for the CPU. As your CPU heats up, your Distance to TJMax will decrease. If it reaches zero, your processor will start to thermal throttle or slow down so maximizing your distance away from TJMax will help your computer run at full speed and more reliably too.


[Official Download Sources]
http://www.techpowerup.com/downloads/SysInfo/Real_Temp/


[Changelog]
http://www.techpowerup.com/realtemp/



----------
corrections and improvements are welcome - @DHT (Ketarin forums)</UserNotes>
    <LastFileSize>330853</LastFileSize>
    <LastFileDate>2024-03-01T09:34:05.8630984</LastFileDate>
    <IgnoreFileInformation>false</IgnoreFileInformation>
    <DownloadBeta>Default</DownloadBeta>
    <DownloadDate xsi:nil="true" />
    <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
    <VariableChangeIndicator />
    <HashVariable />
    <HashType>None</HashType>
    <CanBeShared>true</CanBeShared>
    <ShareApplication>false</ShareApplication>
    <ExclusiveDownload>false</ExclusiveDownload>
    <HttpReferer />
    <SetupInstructions>
      <SetupInstruction xsi:type="StartProcessInstruction">
        <EnvironmentVariables />
        <FileName>"{file}"</FileName>
        <Parameters />
        <WaitForExit>true</WaitForExit>
      </SetupInstruction>
    </SetupInstructions>
    <Variables>
      <item>
        <key>
          <string>version</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>StartEnd</VariableType>
            <Regex>(?&lt;=temp )[\d.]+</Regex>
            <Url>https://www.techpowerup.com/download/techpowerup-real-temp/</Url>
            <StartText>&lt;title&gt;Real Temp </StartText>
            <EndText> Download</EndText>
            <TextualContent />
            <Name>version</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>download_url</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <Url />
            <TextualContent>https://www.techpowerup.com/download/techpowerup-real-temp/?id={downloadid}</TextualContent>
            <Name>download_url</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>previous</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <PostData>3.70</PostData>
            <TextualContent>3.70</TextualContent>
            <Name>previous</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>downloadid</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>RegularExpression</VariableType>
            <Regex>name="id" value="(\d+)"</Regex>
            <PostData />
            <Url>https://www.techpowerup.com/download/techpowerup-real-temp/</Url>
            <TextualContent />
            <Name>downloadid</Name>
          </UrlVariable>
        </value>
      </item>
    </Variables>
    <ExecuteCommand>"{winrar}\winrar.exe" x -ad -ibck -o+ "{file}" "{file:directory}"</ExecuteCommand>
    <ExecutePreCommand />
    <ExecuteCommandType>Batch</ExecuteCommandType>
    <ExecutePreCommandType>Batch</ExecutePreCommandType>
    <SourceType>FixedUrl</SourceType>
    <DeletePreviousFile>true</DeletePreviousFile>
    <Enabled>true</Enabled>
    <FileHippoId />
    <LastUpdated>2024-03-01T09:34:05.8630984</LastUpdated>
    <TargetPath>{root}{apps}\{category}\{appname}\</TargetPath>
    <FixedDownloadUrl>{download_url}</FixedDownloadUrl>
    <Name>_ Real Temp (Portable)</Name>
  </ApplicationJob>
</Jobs>

 

The Logitech Options app profile I posted a while back doesn't work now. Logitech has changed their site so many times recently that this particular code doesn't work anymore, and I don't know if/when I'll be inclined to pursue making one that works again. 

 

Kuppet requires you to create a firewall rule in order for the local server to run. Assuming you're using the native Windows Firewall, you could whitelist the application or the port. It looks like you're using port 9000 so you can do it with this:

netsh advfirewall firewall add rule name="Kuppet" dir=in protocol=TCP localport=9000 action=allow enable=yes profile=any localip=any remoteip=any interfacetype=any

To later remove it use:

netsh advfirewall firewall delete rule "Kuppet"

This will ensure that the firewall allows Kuppet to operate as a standalone server on your computer.

I also recommend you change the definition for the "run_kuppet" variable to:

if(!(PS kuppet -ea 0)) {START -WindowsStyle hidden BIN\kuppet 9000}

This will open Kuppet interactively - what you'll see is basically just a powershell window showing the log of actions. Use this to see whether it is loading correctly, and whether the requests are being processed correctly. There's a bit of a learning curve.

One issue with Kuppet is that by default it uses the "HeadlessChrome" user-agent, which many web servers block by default. You'll need to configure a kuppet.config file with the user-agent you wish to use. I show how to automate this process here. 

 

[jusseppe]

Thanks again!

I imported your xml for Real Temp but still get this error:

 

As for cheat engine, I will try to play with Kuppet some more

 

Update: no luck with Kuppet.. if you can share a working example .xml that uses kuppet, from which I can start building one for Cheat Engine, I would be grateful!

[shawn]

Looks like TechPowerUp is now using manual server selection, so clicking the button alone doesn't actually download the file, it just shows a page with the server selection. Parse that for "server_id" then use POST to download the file. Since Ketarin can't post to the download URL you'll have to shell it out in a powershell variable or use the pre-download command using something that can (Wget or Curl). This is usually pretty simply once you understand the syntax:

curl.exe -v -X POST -L -o "file.ext" -d "id=164&server_id=12" -4 "https://www.techpowerup.com/download/techpowerup-real-temp/"

This performs a verbose POST request, following redirects, saves the output to "file.exe", and POSTs the data "id=164&server_id=12" using IPv4 to the URL. 

This doesn't actually download the file. There's an nginx proxy there that's refusing to actually allow the download. It's either blocking the UA or the referrer tree. It doesn't matter - the verbose response means that it's still showing you the "real" download URL which you could capture and drop into the download URL in Ketarin and that will actually work. 

If you wrap this in a powershell variable you can capture the download URL. I don't have time today to demonstrate but I will try to come back in the next couple days to show you if you haven't figured it out.

 

I've attached the app profile for SetDefaultBrowser which uses Kuppet (though it looks like it's now EOL and being replaced by a more powerful tool). 

SetDefaultBrowser.xml

[jusseppe]

thanks so much for your support and thank you for the detailed explanation!

I think this is getting a little bit out of hand for me

If you will have the time to make it work, I can wait no problem otherwise, thanks anyway!

Regarding the SetDefaultBrowser.xml, just to make sure the issue is on my side, if you try to update it, your result is different from this, right?

I also tried it with firewall disabled, same result

[shawn]

Here's a working Real Temp that uses a PowerShell script to read the Location header from the POST response.

Real Temp.xml

 

SetDefaultBrowser works fine for me. Make sure you're not interfering with it! Kuppet has a built-in timeout that's higher than Ketarin alone, but this specific app uses an imposed 45 second delay *after the last activity*! This is important for anything that needs to bypass CloudFlare or Cloudfront CAPTCHA delays. This is the output I get in Kuppet with logging enabled:

---: Ketarin is playing with her puppet > Port:8080 Time: 3/13/2024 7:28:22 PM

===: KUPPET PARAMETERS
===: port            : 8080
===: kuppetTimeout   : 360
===: pageLoadTimeout : 90
===: userAgent       : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36
===: verboseLogging  : true
===: logging         : true
===: chromeVisible   : false
===: -----------------

---: R01 | REQUEST  : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/
===: R01 | Post Data String: redirection-delay=45
===: Chrome instance starting
===: The BROWSER user agent is    : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/100.0.4889.0 Safari/537.36
===: Opening new page
===: Set page-load timeout        : 90 seconds
===: Setting PAGE user agent      : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36
===: Set extra headers
===: Visiting : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/
===: Rendering the javascript
===: Redirection-Delay for: 45 seconds
===: Rendering the Shadow DOM
===: Returning the rendered HTML - everything between <HTML>...</HTML>
===: 'Beautify' the HTML
===: Closing browser
---: R01 | RESPONSE : Cached:false : Duration:50.22 seconds
---: R02 | REQUEST  : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/
===: R02 | Post Data String: redirection-delay=45
---: R02 | RESPONSE : Cached:true  : Duration:0.004 seconds
---: R03 | REQUEST  : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/
===: R03 | Post Data String: redirection-delay=45
---: R03 | RESPONSE : Cached:true  : Duration:0.003 seconds
---: R04 | REQUEST  : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/
===: R04 | Post Data String: redirection-delay=45
---: R04 | RESPONSE : Cached:true  : Duration:0.003 seconds
---: R05 | REQUEST  : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/
===: R05 | Post Data String: redirection-delay=45
---: R05 | RESPONSE : Cached:true  : Duration:0.005 seconds
---: R06 | REQUEST  : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/
===: R06 | Post Data String: redirection-delay=45
---: R06 | RESPONSE : Cached:true  : Duration:0.003 seconds

 

[shawn]

One more thing about Kuppet. It has a heavy-duty cache so it doesn't re-request pages unnecessarily, even if you think it should. If you ran it, cancelled, then ran it again you might have issues. Kill Kuppet and try again to make sure it purges it's own cache.

[jusseppe]

Real Temp now works, thank you very much!

regarding Kuppet, by looking at your output, I checked the config file included in the 0.8.7 release available on the forum and realized it needed setting up, so I modified that to make it look something similar to yours (see attached) and now I get a similar output:

 

although Ketarin still times out before Kuppet can finish its job.

What could be still missing here?

thank you once again for your support!

kuppet.config

[shawn]

I recommend removing the older user-agents from the userAgentArray variable. Firefox/65 and Chrome/100 are both *very* old and many security features on websites now block them by default. 

The next troubleshooting step is to change the chromeVisible value in your config to true. This will allow you to watch as Kuppet operates, which might be enough to see what is failing.

[jusseppe]

hmm, I removed the old UAs and manually updated chromium and the string to match the latest v124. With Browser visible, this is what I see:

but it still times out..

thank you all for the support

kuppet.config

[shawn]

Is it detecting the version number correctly? Or is it just the download that's failing?

You might also need to change the global user-agent (File, Settings, Connection, Default user agent) or at least assign one within the app profile at Advanced Settings, User-agent. By default the user-agent indicates it's Ketarin and might be getting blocked. 

[jusseppe]

Yes, my global UA is the default one: Mozilla/4.0 (compatible; Ketarin; +https://ketarin.org/)

is it a good practice to change it to something else? 

In any case, I copied the same UA, that I put in the kuppet.config file, in the app profile, which is the exact version stated in the browser About page:

 

I tried to check the versionstub variable, but the problem seems to be that it cannot load the page? although I see it in the Chromium browser (it loads correctly after the 3 second Anti-Crawler IP checker page): https://imgur.com/a/Kz9g0Bv

 

[shawn]

if you don't have a lot of requests or apps then changing the UA globally is unlikely to help. if you do, then you might want to change it and keep it relatively current.

the format for the version string in Chrome UA is "Chrome/major.0.0.0" - it never includes the minor or build numbers. This is similar to all other modern browsers - they don't include anything but the major version dot zero, and sometimes an additional couple dot-zeroes. Change it to:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36

That alone might help.

If you haven't set the Version number column to display in the main interface you might want to do that first. This ensures that you can see when new versions are detected and might help you troubleshoot this one. Since this site is one of the ones that is really aggressive about spiders you might want to test with a different one. I've attached another one (aria2) that uses Kuppet.

aria2.xml

[jusseppe]

thanks for the pointers!

Aria works! https://imgur.com/a/FmsH6yC

I have the version number set, along with previous (See also link above).

At this point, have no idea why the other one isn't working.. but I have a working example, so thank you for that!

EDIT: I noticed that aria works even without kuppet so I don't think it means that kuppet works..

In any case, even with kuppet, I still get the 405 error with Cheat Engine.

At this point, I guess it would be best to stop bothering you guys, thank you for the support so far

Cheat Engine.xml

[shawn]

Interesting issue with the SetDefaultBrowser and SetUserFTA programs: Microsoft broke them with a recent registry service "fix" to control the browser behavior and enforce "user choice". Evil bastards.