Jump to content
Ketarin forum

shawn

Moderators
  • Posts

    1,179
  • Joined

  • Last visited

Everything posted by shawn

  1. The ability to replace the user-agent string has become essential. The Cloudfront servers that are responsible for Logitech, SnagIt and others are now completely blocking the HeadlessChrome user-agent.
  2. Or maybe a configuration file that it will parse to populate these variables.
  3. The cache appears to only survive about 10 seconds - so on a large download or on a slow site, it may not keep the cache long enough for the post-update scripts to run without re-downloading the variable webpages. I've changed the command used to extend the timeout to: if(!(PS kuppet -ea 0)) {START -WindowsStyle hidden BIN\kuppet "8008 30000"} This increases the timeout to 30 seconds and resolves my immediate problem, but it will not accept any value above 30000 (30s). I suspect there is a hard limit in place to ensure that Kuppet doesn't stay in memory forever. Unfortunately, this forces some applications to re-request data in order to perform my post-update scripts. I guarantee on some sites (just experienced it with SnagIt) this will become a problem. Re-requesting pages can get you throttled with a recaptcha or cloudflare stall. Grrrr. Please increase the hard limit to 5m or more, and/or allow us to configure caching on an execution basis. Maybe allow Kuppet to run without a hard limit (timeout=0) - and use the timeout only in relation to the cache duration. I leave Ketarin open 24/7 anyway so there's no reason I can think of to close Kuppet any more frequently than I reboot (once per month or so). Something like this: kuppet.exe [port] [cache-timeout-ms] [hard-limit-ms] [custom-user-agent] Thanks, again, @Ambimind - this has already resolved problems I've been having with about a dozen Applications.
  4. It is caching - yay! I didn't read the log very carefully. It looks like the parameters are: kuppet.exe [port] [timeout-ms] It disconnects if idle for 30s or more no matter what. It's using the following user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/93.0.4577.63 Safari/537.36 Can you make it possible to change the user-agent? Some sites block HeadlessChrome by default.
  5. One more feature request: can I get the complete webpage contents and not just the body element?
  6. @Ambimind - this is fantastic. Thank you! Can I request a feature? It looks like (according to the log) that it is re-requesting the same URLs repeatedly with each instance of a variable that makes a request. Is it possible to make Kuppet cache the results temporarily until it has moved on to the next Application? Since I gather a lot of content from some pages this could save me a LOT of repeat requests. Are there other switches than the port that it runs on?
  7. Hi, @jusseppe! I've had a chance now to try this myself and the biggest issue was that the alternate port I attempted to use would not work until I closed and reopened Ketarin. This made all the difference for me. I've pasted the exact text you'll want to use in the "global variables" feature below to make it easier to ensure it is correct. For the variable named "run_kuppet": if(!(PS kuppet -ea 0)) {START -WindowsStyle hidden BIN\kuppet 8008} Or, alternatively, use this to show the log as it happens: if(!(PS kuppet -ea 0)) {START BIN\kuppet 8008} For the variable named ">": {run_kuppet:ps}http://localhost:8008/
  8. Note that modern browsers don't allow including an XSL file from file URLs, so you'll either need to put it on a website or load it with script.
  9. I recently made a change to the structure I use for all my variables and needed to publish it to my server. Since Ketarin doesn't (yet) support commands for non-updates I needed to get the data in a structured format from Ketarin to the database (I'll be passing it through Excel). Anyway... this is what I came up with. It uses Kryogenix sorttable, and some styling and stuff from a few other projects. You can use the aCols array to control which columns are hidden by default. For your own custom variables assign the variable name in one of these Variables/item[key/string='myVariableName']/value/UrlVariable/TextualContent Since parsed variables (like version) aren't cached automatically I setup temporary variables (LastVersion and LastUpdate) to store the version whenever it's updated so it can be accessed through the XML. This uses the script from @Ambimind here. Save the code below to a new file "ketarin.xsl". When you export your XML file you'll need to add this XSL rule above the <Jobs> line: <?xml-stylesheet type="text/xsl" href="ketarin.xsl" ?> I'm sure I'm forgetting stuff, but you are welcome to ask questions here and I'll reply. ketarin.xsl <?xml version="1.0" encoding="UTF-8"?> <html xsl:version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <head> <title>Ketarin.xml</title> <script type="text/javascript" src="sorttable.js" note="https://www.kryogenix.org/code/browser/sorttable/"></script> <script type="text/javascript"> var toggleDisplay = function (element) { element.style.display = (element.style.display === 'none')?'table-cell':'none'; }; function toggleCol(){ // Slug, Updated, Size, ? aCols=[2,11,12,13,15,16]; for(var lCol = 0; lCol &lt; aCols.length; lCol++){ document.querySelectorAll('th:nth-child('+aCols[lCol]+'), td:nth-child('+aCols[lCol] + ')').forEach(toggleDisplay); } }; </script> <style type="text/css"> body{ font-family:Verdana; font-size:12px; background-color:#ffffff; } table { border: none !important; background-color: #eee; width: 100%; text-align: left; border-collapse: collapse; } table th, table th tr { text-align: center; border: 0; padding: 0; margin: 0; } table td { border: 1px solid #CCCCCC; padding: 3px 2px; vertical-align: middle; } table tbody td { font-size: 11px; vertical-align: middle; } table tr:nth-child(even) { background: #D0F5E4; } table thead { background: #16a085; background: -moz-linear-gradient(top, #258b62 0%, #32bd7c 50%, #16a085 100%); background: -webkit-linear-gradient(top, #258b62 0%, #32bd7c 50%, #16a085 100%); background: linear-gradient(to bottom, #258b62 0%, #32bd7c 50%, #16a085 100%); border: 0 0 0 0; margin: 0 0 2px 0; } table thead tr, table thead tr th { font-size: 14px; font-weight: bold; color: #eee; border: 0 0 0 0 !important; padding: 2px 5px; border-radius: 16px; } table thead th:first-child { border-left: none; } table tfoot { font-size: 12px; font-weight: bold; color: #eee; background: #D0E4F5; background: -moz-linear-gradient(top, #dcebf7 0%, #d4e6f6 66%, #D0E4F5 100%); background: -webkit-linear-gradient(top, #dcebf7 0%, #d4e6f6 66%, #D0E4F5 100%); background: linear-gradient(to bottom, #dcebf7 0%, #d4e6f6 66%, #D0E4F5 100%); border-top: 2px solid #444444; } table tfoot td { font-size: 12px; } table tfoot .links { text-align: right; } table tfoot .links a{ display: inline-block; background: #1C6EA4; color: #FFFFFF; padding: 1px 5px; border-radius: 5px; } th { position: sticky; background: #16a085; top: 0; /* Don't forget this, required for the stickiness */ } table tr td img{width:12px; height:12px; margin:0 6px 0; padding:0 0 0 0; } #platforms img{width:18px; height:18px; margin:0 10px 0 10px; border:0;} .nowrap{ white-space: nowrap; } /* responsive columns */ .col-toggle { -webkit-appearance: none; -webkit-tap-highlight-color: transparent; position: relative; border: 0; outline: 0; cursor: pointer; margin: 10px; } /* To create surface of toggle button */ .col-toggle:after { content: ''; width: 45px; height: 14px; display: inline-block; background: rgba(196, 195, 195, 0.55); border-radius: 18px; clear: both; } /* Contents before checkbox to create toggle handle */ .col-toggle:before { content: ''; width: 16px; height: 16px; display: block; position: absolute; left: 0; top: -2px; border-radius: 50%; background: rgb(255, 255, 255); box-shadow: 1px 1px 3px rgba(0, 0, 0, 0.6); } /* Shift the handle to left on check event */ .col-toggle:checked:before { left: 31px; box-shadow: -1px 1px 3px rgba(0, 0, 0, 0.6); } /* Background color when toggle button will be active */ .col-toggle:checked:after { background: #16a085; } .cm-toggle, .cm-toggle:before, .cm-toggle:after, .cm-toggle:checked:before, .cm-toggle:checked:after { transition: ease .5s; -webkit-transition: ease .5s; -moz-transition: ease .5s; -o-transition: ease .5s; } </style> </head> <body onload="toggleCol();"> <h2>Ketarin</h2> <div id="platforms" style="float:right; text-align:right;"> <input type="checkbox" name="checkbox" class="col-toggle" onclick="toggleCol()" /> </div> <table border="1" class="sortable"> <thead> <tr> <th class="sorttable_alpha">Guid</th> <th class="sorttable_alpha">Enabled</th> <th class="sorttable_alpha">Vendor</th> <th class="sorttable_alpha">Name</th> <th class="sorttable_alpha">Version</th> <th class="sorttable_alpha">Slug</th> <th class="sorttable_alpha">Category</th> <th class="sorttable_alpha">Bits</th> <th class="sorttable_numeric">Size</th> <th class="sorttable_alpha">Platform</th> <th class="sorttable_date">Updated</th> <th class="sorttable_date">Filetime</th> <th class="sorttable_alpha">Website0</th> <th class="sorttable_alpha">Website1</th> <th class="sorttable_alpha">Changelog</th> <th class="sorttable_nosort">Notes</th> </tr> </thead> <tbody> <xsl:for-each select="Jobs/ApplicationJob"> <tr> <td class="nowrap"><xsl:value-of select="./@Guid"/></td> <td><xsl:value-of select="Enabled"/></td> <td><xsl:value-of select="Variables/item[key/string='svendor']/value/UrlVariable/TextualContent" /></td> <td><xsl:value-of select="Name"/></td> <td><xsl:value-of select="Variables/item[key/string='LastVersion']/value/UrlVariable/TextualContent" /></td> <td><xsl:value-of select="Variables/item[key/string='spc']/value/UrlVariable/TextualContent" /></td> <td><xsl:value-of select="Category"/></td> <td><xsl:value-of select="Variables/item[key/string='sbits']/value/UrlVariable/TextualContent" /></td> <td> <xsl:attribute name="sorttable_customkey"><xsl:value-of select="LastFileSize"/></xsl:attribute> <xsl:variable name="thesize" select="LastFileSize"/> <xsl:choose> <xsl:when test="$thesize &gt;= 1073741824"> <xsl:value-of select="format-number($thesize div 1073741824,'#,###.00')"/> <xsl:text> GB</xsl:text> </xsl:when> <xsl:when test="$thesize &gt;= 1048576"> <xsl:value-of select="format-number($thesize div 1048576,'#,###.0')"/> <xsl:text> MB</xsl:text> </xsl:when> <xsl:when test="$thesize &gt;= 1024"> <xsl:value-of select="format-number($thesize div 1024,'#,###')"/> <xsl:text> KB</xsl:text> </xsl:when> <xsl:when test="$thesize &gt; 0 and $thesize &lt; 1024"> <xsl:value-of select="format-number($thesize div 0,'#,###')"/> <xsl:text> Bytes</xsl:text> </xsl:when> </xsl:choose> </td> <td><xsl:value-of select="Variables/item[key/string='splatform']/value/UrlVariable/TextualContent" /></td> <td class="nowrap"><xsl:value-of select="LastUpdated"/></td> <td class="nowrap"><xsl:value-of select="LastFileDate"/></td> <td><xsl:value-of select="WebsiteUrl"/></td> <td><xsl:value-of select="Variables/item[key/string='swebsite']/value/UrlVariable/TextualContent" /></td> <td><xsl:value-of select="Variables/item[key/string='schangelog']/value/UrlVariable/TextualContent" /></td> <td><xsl:value-of select="Variables/item[key/string='snotes']/value/UrlVariable/TextualContent" /></td> </tr> </xsl:for-each> </tbody> </table> </body> </html>
  10. Thank you, @Ambimind! This looks like it could be very useful.
  11. I know this is years late, but I was looking for something else and saw that one of the issues in this wasn't addressed. The problem with your troubleshooting line is most likely that you didn't quote the URL. This means that your script will crash before it actually echoes anything to the file if there's an ampersand (&) or certain other characters in the {preupdate-url} variable. Try this instead: echo TEST "{preupdate-url}" >> T:\xx.txt
  12. Hi, @Etz! They're using the Akamai Captcha, which is a javascript+cookie-based method of preventing exactly what you're trying to do. It looks like it should be possible to harvest sufficient information from the javascript and build the cookies manually, but it looks like it's going to be a nightmare to setup initially and a single change on their side will break it, so I wouldn't bother with it.
  13. "Automatically" in the sense that it happens on a regular basis, without user interaction, or via some form of switch? As I'm sure you know, /export includes the app commands as well as all other app-specific properties. I suspect you might actually be talking about the global commands. If so, they're included in your jobs.db backup, but you can export those to an external file using Sqlite, DB4S, powershell with an sqlite assembly, or your favorite SQLite editor: SELECT `SettingPath`,`SettingValue` FROM `settings` WHERE `SettingPath` IN ( 'PostUpdateCommand','PostUpdateCommandType', 'PreUpdateCommand','PreUpdateCommandType', 'UpdateFailedCommand','UpdateFailedCommandType' );
  14. Yes. Anything you want to execute can be performed in the pre-update command. You can use this to do something like email yourself, or pipe to a text file, or with the /log parameter. If done correctly, you can use just about any information you like (anything you've stored in a variable) in order to send it to yourself. There is a great example of creating your own changelog here using the "after updating an application" global command. I leave Ketarin open 24/7 and pipe the results of every download to a log file in a global after-update command, and within the pre-update command for a few that use pre-update to perform the downloads or are only used to watch sites. I then use Notepad++ with the logfile open 24/7 so as updates are performed I can review the log and perform any actions based on what's new, such as reading changelogs for web packages like Joomla and installing any that are security updates, or triggering Chrome updates on all client devices and so on.
  15. If you're just looking for an Nvidia app profile, here's mine. Be aware that they no longer provide 32-bit drivers, so this is for the latest US-English 64-bit WHQL GeForce driver. It would only take a couple variable changes to modify the language/location, and other details. They're all stuffed into URL parameters so it's pretty easy to fix. <?xml version='1.0' encoding='utf-8'?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="db7445b0-3205-40e7-9c43-3502db1382f1"> <Category>Drivers</Category> <WebsiteUrl>https://www.nvidia.com/Download/index.aspx?lang=en-us</WebsiteUrl> <UserAgent /> <UserNotes>The Nvidia driver package is designed for Nvidia video cards.</UserNotes> <LastFileSize>600971672</LastFileSize> <LastFileDate>2021-04-14T21:38:24.0492945</LastFileDate> <IgnoreFileInformation>false</IgnoreFileInformation> <DownloadBeta>Default</DownloadBeta> <DownloadDate xsi:nil="true" /> <CheckForUpdatesOnly>false</CheckForUpdatesOnly> <VariableChangeIndicator>version</VariableChangeIndicator> <HashVariable /> <HashType>None</HashType> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <ExclusiveDownload>false</ExclusiveDownload> <HttpReferer /> <SetupInstructions /> <Variables> <item> <key> <string>version</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>gridItem"&gt;([\d\.]+)&lt;/td&gt;</Regex> <Url>https://www.nvidia.com/Download/processFind.aspx?osid=19&amp;whql=1&amp;lang=en-us&amp;ptid=1&amp;psid=101</Url> <Name>version</Name> </UrlVariable> </value> </item> <item> <key> <string>dl1</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>['"]([^'']*//[^'']*/en-us)['"]&gt;.*?[^&gt;]*&gt;{version}</Regex> <Url>https://www.nvidia.com/Download/processFind.aspx?osid=19&amp;whql=1&amp;lang=en-us&amp;ptid=1&amp;psid=101</Url> <Name>dl1</Name> </UrlVariable> </value> </item> <item> <key> <string>dl2</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>url=([^&amp;"']+)</Regex> <Url>{dl1}</Url> <Name>dl2</Name> </UrlVariable> </value> </item> <item> <key> <string>snotes</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent /> <Name>snotes</Name> </UrlVariable> </value> </item> <item> <key> <string>swebsite</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>https://www.nvidia.com/Download/index.aspx?lang=en-us</TextualContent> <Name>swebsite</Name> </UrlVariable> </value> </item> <item> <key> <string>schangelog</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex>"([^"]+\.pdf)"&gt;Release Notes</Regex> <Url>{dl1}</Url> <TextualContent>{dl1}</TextualContent> <Name>schangelog</Name> </UrlVariable> </value> </item> <item> <key> <string>spc</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>nvidia</TextualContent> <Name>spc</Name> </UrlVariable> </value> </item> <item> <key> <string>sdownload</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>https://us.download.nvidia.com{dl2}</TextualContent> <Name>sdownload</Name> </UrlVariable> </value> </item> <item> <key> <string>splatform</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>windows</TextualContent> <Name>splatform</Name> </UrlVariable> </value> </item> <item> <key> <string>sbits</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>64</TextualContent> <Name>sbits</Name> </UrlVariable> </value> </item> <item> <key> <string>LastUpdate</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>20210414T2138</TextualContent> <Name>LastUpdate</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand /> <ExecutePreCommand /> <ExecuteCommandType>Batch</ExecuteCommandType> <ExecutePreCommandType>Batch</ExecutePreCommandType> <SourceType>FixedUrl</SourceType> <PreviousLocation>K:\_Ketarin\Ketarin\..\Drivers\nVidia_(x64)-466.11.exe</PreviousLocation> <DeletePreviousFile>true</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <LastUpdated>2021-04-14T21:38:24.0492945</LastUpdated> <TargetPath>..\{category}\{appname:regexreplace:([\s\t\r\n\-\&amp;]+):_}-{version}.{url:ext}</TargetPath> <FixedDownloadUrl>{sdownload}</FixedDownloadUrl> <Name>Nvidia</Name> </ApplicationJob> </Jobs>
  16. You're confusing two languages: batch and Ketarin. Ketarin variables are the native variables as well as those you create yourself on the variables page. The Ketarin variables are, of course, processed before batch variables since the batch script is generated *from* Ketarin. This means that you will never use a "{%whatever%}" syntax, but might very well use a "%{whatever}%" syntax, depending on what you're putting in your variables. You could, for example, use a global Ketarin variable to specify what path you would like to extract to and use that in place of the {whatever} and have it processed by the batch script. "{file}" is the proper way to wrap the filename if it has spaces or other special characters (like ampersands). There are MANY EXAMPLES on this forum of using 7-zip to extract files to specific directories. As for the directory thing - you'll want to look at the functions page on the wiki for stuff like {file:directory}: 7z.exe e -o"{file:directory}" "{file}" *.exe
  17. There are a couple examples using the pre-update command to perform the actual download via an external program. I use this process with several apps that don't play well with Ketarin or otherwise require additional handling (like torrent-based downloads).
  18. This is how start/end works. If you create a new one and play with it you'll see how it works.
  19. I'm afraid some sites don't play well with Ketarin or any other automated means of analysis. This is one of them.
  20. There is a difference between "save in folder" and "save to file", and this is why you can select only one of those options. "Save in folder" allows you to specify the folder path that you want to save the downloaded file to. You can use a relative folder or explicit folder and you can use variables in the folder name. The file that is downloaded will be saved with the filename assigned by the server. You'll end up with something like ".\your\path\setup.exe" or "your\path\tails.iso". Notably, if the original filename does not include the version number or language, and you want to include these details in the resulting filename, it will not be included. "Save to file" allows you to specify the complete path including filename. Used with functions you can parse the original filename pattern to ensure you're assigning the right extension, then you can store much more detail within the filename. Speaking of functions, yes, you can use many in series or as part of a collection. Moreover, you can use the variables feature to assemble more or in a multi-step process to format variables how you want to use them based on the need.
  21. This is the reason I haven't switched over. I'm excited about the new feature (keep x versions) but even starting with a clean database it crashes on me more often than not.
  22. One option is to Ping localhost, which inserts a delay of roughly one second per echo. This will insert a 5 second delay: ping 127.0.0.1 -n 5 Or you could call the powershell Start-Sleep cmdlet: Start-Sleep -s 5
  23. The mailing function in the forum works fine as long as you're not blocking it. I received notification about your comment within 10 minutes of you posting it. Consider whitelisting 136.243.175.196, 138.201.250.173, and 138.201.207.69. These are the three IP addresses used by the Ketarin forum in the last couple years.
  24. We both gave you answers to accomplish your request using advanced options. Perhaps we need to show you the simplest method: Click the "Add new application" button. In the resulting New Application window: 1) Select either "save to file" or "save in folder" as the default behavior. 2) Enter the path or filename ({variables} OK!) 3) Click "Save as default" Every new application you create after this point will use these settings. Any other aspects you assign before clicking "Save as default" will also be assigned to new applications, such as category, new default variables, download source/URL and so on.
  25. There have been several issues with FileHippo support over the years. It would not surprise me at all that FileHippo has changed something and it doesn't work again. I don't use FileHippo anymore since it's been unreliable, so you really have two options: configure your downloads to use a different source, or wait until a new release resolves whatever is going on with FileHippo. @floele ?
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.