Jump to content
Ketarin forum

shawn

Moderators
  • Posts

    1,183
  • Joined

  • Last visited

2 Followers

About shawn

Profile Information

  • Gender
    Male

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

shawn's Achievements

  1. And these are the full Reader EXE installer URLs (for English): x64 https://ardownload3.adobe.com/pub/adobe/acrobat/win/AcrobatDC/{dirtyversion}/AcroRdrDCx64{dirtyversion}_en_US.exe x86 https://ardownload3.adobe.com/pub/adobe/reader/win/AcrobatDC/{dirtyversion}/AcroRdrDC{dirtyversion}_en_US.exe
  2. With those URLs and the JSON response I would recommend just pulling the version number directly with (as {version}): "version":"([\d\.]+)" This avoids manually fiddling with numbers that might have issues. Today they describe 2024 as the major version and use 24 internally, but who knows what they'll do tomorrow.
  3. Currently (Ketarin 1.8.11.0) there are four separate Event Commands available: 1) Before updating an application 2) After updating an application 3) After updating all applications 4) When application update fails These are great for pre-download validation and post-update & post-check failure documentation, but they don't address every scenario. I would like one more Event Command available: Before checking for updates for an application. When I have parallel downloads set to 1 then it's a safe bet that the next one in the list is responsible. When I have parallel set to 2 or 3 (or more) then it's a whole lot of guessing. This new event would trigger before any variables were parsed (other than the name) so that I can log the progress of "update all" or "check for updates" on a selection. This would allow me to write the pre-parse progress to a permanent log file so I can better troubleshoot errors. Currently, when Ketarin crashes (which isn't often, but is often enough that it's frustrating) I have to guess which application is responsible and which variable or URL it was within the application that might be causing the problem. I can work around some of this with special variables and the ":ps" function, but more difficult than it could be to figure out which one is responsible. Alternatively (or in addition to) there would be an option to write to a log file that mirrors the activity in the real-time log (CTRL+L). This could either be one massive file (ketarin.log), or based on the date, or based on variables (like "ketarin-{yyyy}{MM}{dd}{hh}{mm}.log" or "ketarin-{category}.log" or even based on the "save as" destination with an additional ".log" appended: "..\{category}\{appname:regexreplace:([\s\t\r\n\-\&]+):_}-{version}.{url:ext}.log" where as-yet-unparsed variables would be replaced with dashes or underscores). Each application should have the log file name preserved for the duration of the specific application, so if the log file name were "ketarin-{yyyy}{MM}{dd}T{hh}{mm}.log" at 20240414T1345 and it took 4 minutes to process the application, all log records relating to that application update would be written to that log even though the time were actually 20240414T1349 by the time it was done.
  4. Interesting issue with the SetDefaultBrowser and SetUserFTA programs: Microsoft broke them with a recent registry service "fix" to control the browser behavior and enforce "user choice". Evil bastards.
  5. if you don't have a lot of requests or apps then changing the UA globally is unlikely to help. if you do, then you might want to change it and keep it relatively current. the format for the version string in Chrome UA is "Chrome/major.0.0.0" - it never includes the minor or build numbers. This is similar to all other modern browsers - they don't include anything but the major version dot zero, and sometimes an additional couple dot-zeroes. Change it to: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 That alone might help. If you haven't set the Version number column to display in the main interface you might want to do that first. This ensures that you can see when new versions are detected and might help you troubleshoot this one. Since this site is one of the ones that is really aggressive about spiders you might want to test with a different one. I've attached another one (aria2) that uses Kuppet. aria2.xml
  6. Is it detecting the version number correctly? Or is it just the download that's failing? You might also need to change the global user-agent (File, Settings, Connection, Default user agent) or at least assign one within the app profile at Advanced Settings, User-agent. By default the user-agent indicates it's Ketarin and might be getting blocked.
  7. I recommend removing the older user-agents from the userAgentArray variable. Firefox/65 and Chrome/100 are both *very* old and many security features on websites now block them by default. The next troubleshooting step is to change the chromeVisible value in your config to true. This will allow you to watch as Kuppet operates, which might be enough to see what is failing.
  8. One more thing about Kuppet. It has a heavy-duty cache so it doesn't re-request pages unnecessarily, even if you think it should. If you ran it, cancelled, then ran it again you might have issues. Kill Kuppet and try again to make sure it purges it's own cache.
  9. Here's a working Real Temp that uses a PowerShell script to read the Location header from the POST response. Real Temp.xml SetDefaultBrowser works fine for me. Make sure you're not interfering with it! Kuppet has a built-in timeout that's higher than Ketarin alone, but this specific app uses an imposed 45 second delay *after the last activity*! This is important for anything that needs to bypass CloudFlare or Cloudfront CAPTCHA delays. This is the output I get in Kuppet with logging enabled: ---: Ketarin is playing with her puppet > Port:8080 Time: 3/13/2024 7:28:22 PM ===: KUPPET PARAMETERS ===: port : 8080 ===: kuppetTimeout : 360 ===: pageLoadTimeout : 90 ===: userAgent : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 ===: verboseLogging : true ===: logging : true ===: chromeVisible : false ===: ----------------- ---: R01 | REQUEST : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/ ===: R01 | Post Data String: redirection-delay=45 ===: Chrome instance starting ===: The BROWSER user agent is : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/100.0.4889.0 Safari/537.36 ===: Opening new page ===: Set page-load timeout : 90 seconds ===: Setting PAGE user agent : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 ===: Set extra headers ===: Visiting : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/ ===: Rendering the javascript ===: Redirection-Delay for: 45 seconds ===: Rendering the Shadow DOM ===: Returning the rendered HTML - everything between <HTML>...</HTML> ===: 'Beautify' the HTML ===: Closing browser ---: R01 | RESPONSE : Cached:false : Duration:50.22 seconds ---: R02 | REQUEST : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/ ===: R02 | Post Data String: redirection-delay=45 ---: R02 | RESPONSE : Cached:true : Duration:0.004 seconds ---: R03 | REQUEST : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/ ===: R03 | Post Data String: redirection-delay=45 ---: R03 | RESPONSE : Cached:true : Duration:0.003 seconds ---: R04 | REQUEST : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/ ===: R04 | Post Data String: redirection-delay=45 ---: R04 | RESPONSE : Cached:true : Duration:0.003 seconds ---: R05 | REQUEST : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/ ===: R05 | Post Data String: redirection-delay=45 ---: R05 | RESPONSE : Cached:true : Duration:0.005 seconds ---: R06 | REQUEST : https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/ ===: R06 | Post Data String: redirection-delay=45 ---: R06 | RESPONSE : Cached:true : Duration:0.003 seconds
  10. Looks like TechPowerUp is now using manual server selection, so clicking the button alone doesn't actually download the file, it just shows a page with the server selection. Parse that for "server_id" then use POST to download the file. Since Ketarin can't post to the download URL you'll have to shell it out in a powershell variable or use the pre-download command using something that can (Wget or Curl). This is usually pretty simply once you understand the syntax: curl.exe -v -X POST -L -o "file.ext" -d "id=164&server_id=12" -4 "https://www.techpowerup.com/download/techpowerup-real-temp/" This performs a verbose POST request, following redirects, saves the output to "file.exe", and POSTs the data "id=164&server_id=12" using IPv4 to the URL. This doesn't actually download the file. There's an nginx proxy there that's refusing to actually allow the download. It's either blocking the UA or the referrer tree. It doesn't matter - the verbose response means that it's still showing you the "real" download URL which you could capture and drop into the download URL in Ketarin and that will actually work. If you wrap this in a powershell variable you can capture the download URL. I don't have time today to demonstrate but I will try to come back in the next couple days to show you if you haven't figured it out. I've attached the app profile for SetDefaultBrowser which uses Kuppet (though it looks like it's now EOL and being replaced by a more powerful tool). SetDefaultBrowser.xml
  11. Welcome aboard @Mundee. This is demonstrated pretty well in this thread. While Ambimind uses variables for the parts of the calling function, if you already know the name, type and form of the variable, then you could use the abbreviated syntax. For Start/End and RegEx you use: $x = $app.variables.MyVersionVariable.CachedContent For Textual Content use: $x = $app.variables.MyVersionVariable.TextualContent In both samples "MyVersionVariable" is the name of the variable you'll be using instead of {MyVersionVariable}.
  12. So the reason why the "downloadid" was failing is because you set the POST data within it. The downloadid should be used to parse *for* the ID value. Then the download_url will work correctly. If the ID remains consistent between all builds then you could probably hard-code it, but I would rather just parse for it to be sure. Here's the fixed XML: <?xml version='1.0' encoding='utf-8'?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="bd752d5c-099d-4187-83e7-2ddb9cf3d546"> <Category>System Maintenance &amp; Security</Category> <WebsiteUrl>http://www.techpowerup.com/realtemp/</WebsiteUrl> <UserAgent>Mozilla/5.0</UserAgent> <UserNotes>[Description] Real Temp is a temperature monitoring program designed for all Intel single Core, Dual Core, Quad Core and Core i7 processors. Each core on these processors has a digital thermal sensor (DTS) that reports temperature data relative to TJMax which is the safe maximum operating core temperature for the CPU. As your CPU heats up, your Distance to TJMax will decrease. If it reaches zero, your processor will start to thermal throttle or slow down so maximizing your distance away from TJMax will help your computer run at full speed and more reliably too. [Official Download Sources] http://www.techpowerup.com/downloads/SysInfo/Real_Temp/ [Changelog] http://www.techpowerup.com/realtemp/ ---------- corrections and improvements are welcome - @DHT (Ketarin forums)</UserNotes> <LastFileSize>330853</LastFileSize> <LastFileDate>2024-03-01T09:34:05.8630984</LastFileDate> <IgnoreFileInformation>false</IgnoreFileInformation> <DownloadBeta>Default</DownloadBeta> <DownloadDate xsi:nil="true" /> <CheckForUpdatesOnly>false</CheckForUpdatesOnly> <VariableChangeIndicator /> <HashVariable /> <HashType>None</HashType> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <ExclusiveDownload>false</ExclusiveDownload> <HttpReferer /> <SetupInstructions> <SetupInstruction xsi:type="StartProcessInstruction"> <EnvironmentVariables /> <FileName>"{file}"</FileName> <Parameters /> <WaitForExit>true</WaitForExit> </SetupInstruction> </SetupInstructions> <Variables> <item> <key> <string>version</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>StartEnd</VariableType> <Regex>(?&lt;=temp )[\d.]+</Regex> <Url>https://www.techpowerup.com/download/techpowerup-real-temp/</Url> <StartText>&lt;title&gt;Real Temp </StartText> <EndText> Download</EndText> <TextualContent /> <Name>version</Name> </UrlVariable> </value> </item> <item> <key> <string>download_url</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <Url /> <TextualContent>https://www.techpowerup.com/download/techpowerup-real-temp/?id={downloadid}</TextualContent> <Name>download_url</Name> </UrlVariable> </value> </item> <item> <key> <string>previous</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <PostData>3.70</PostData> <TextualContent>3.70</TextualContent> <Name>previous</Name> </UrlVariable> </value> </item> <item> <key> <string>downloadid</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>name="id" value="(\d+)"</Regex> <PostData /> <Url>https://www.techpowerup.com/download/techpowerup-real-temp/</Url> <TextualContent /> <Name>downloadid</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand>"{winrar}\winrar.exe" x -ad -ibck -o+ "{file}" "{file:directory}"</ExecuteCommand> <ExecutePreCommand /> <ExecuteCommandType>Batch</ExecuteCommandType> <ExecutePreCommandType>Batch</ExecutePreCommandType> <SourceType>FixedUrl</SourceType> <DeletePreviousFile>true</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <LastUpdated>2024-03-01T09:34:05.8630984</LastUpdated> <TargetPath>{root}{apps}\{category}\{appname}\</TargetPath> <FixedDownloadUrl>{download_url}</FixedDownloadUrl> <Name>_ Real Temp (Portable)</Name> </ApplicationJob> </Jobs> The Logitech Options app profile I posted a while back doesn't work now. Logitech has changed their site so many times recently that this particular code doesn't work anymore, and I don't know if/when I'll be inclined to pursue making one that works again. Kuppet requires you to create a firewall rule in order for the local server to run. Assuming you're using the native Windows Firewall, you could whitelist the application or the port. It looks like you're using port 9000 so you can do it with this: netsh advfirewall firewall add rule name="Kuppet" dir=in protocol=TCP localport=9000 action=allow enable=yes profile=any localip=any remoteip=any interfacetype=any To later remove it use: netsh advfirewall firewall delete rule "Kuppet" This will ensure that the firewall allows Kuppet to operate as a standalone server on your computer. I also recommend you change the definition for the "run_kuppet" variable to: if(!(PS kuppet -ea 0)) {START -WindowsStyle hidden BIN\kuppet 9000} This will open Kuppet interactively - what you'll see is basically just a powershell window showing the log of actions. Use this to see whether it is loading correctly, and whether the requests are being processed correctly. There's a bit of a learning curve. One issue with Kuppet is that by default it uses the "HeadlessChrome" user-agent, which many web servers block by default. You'll need to configure a kuppet.config file with the user-agent you wish to use. I show how to automate this process here.
  13. For TechPowerUp you can use the download page (the one you linked to) then collect the hidden "id" value from it. That's 164 for this app. I name this field "downloadid". With that parsed the URL to download becomes: https://www.techpowerup.com/download/techpowerup-real-temp/?id={sdownloadid} The site uses POST in the source code, but I tested way back in 2015 or so when I created a download profile for NVcleaninstall and you can use GET and the download will work fine. If that wasn't possible (if they actually checked for POST vs GET) then the next best option would be to perform a POST operation (using the "POST data" button) to get the raw file URL. For Cheat Engine you'll need to use a different method. The download URLs generated by cloudfront are unique and evil and rely on HTML5 javascript behaviors that you can't perform with Ketarin alone without massive effort. For these you need to use a browser puppeteer - a scriptable full browser that you can control externally - there's a flavor built for us here by @Ambimind called Kuppet, get it here on the forum. One of the methods cloudfront uses to "prove" the visitor is human is to insist on a delay before rendering all of the content. For this you'll need to use the "redirection-delay" option. I'm not in a hurry so my redirection delay for any sites that impose one is set to 45 seconds. It feels like a long time (it's not), but it guarantees that sites that impose this delay behavior will work. I'd rather have it work reliably than stall out because I tried to push the envelope. Just don't go too large because some sites have an idle timer, too, which either refreshes the page or redirects you to their home page after a longer delay, usually 5-15 minutes of inactivity. These webmasters suck because you don't always finish the article before you're thrown to a different page or place on the page when it triggers. Grrr. /soapbox Please try it and post your XML if it's still not behaving.
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.