Jump to content
Ketarin forum

shawn

Moderators
  • Posts

    1,128
  • Joined

  • Last visited

1 Follower

About shawn

Profile Information

  • Gender
    Male

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

shawn's Achievements

  1. You can use an external download in the pre-update script and use something like Wget. Example here.
  2. Created an app to monitor this page for updates. It requires MEGA-Get from MEGA-CMD. <?xml version='1.0' encoding='utf-8'?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="2ca7df65-be33-47ec-860d-71425204d9f4"> <Category>Tools</Category> <WebsiteUrl /> <UserAgent /> <UserNotes /> <LastFileSize>0</LastFileSize> <IgnoreFileInformation>false</IgnoreFileInformation> <DownloadBeta>Default</DownloadBeta> <DownloadDate xsi:nil="true" /> <CheckForUpdatesOnly>false</CheckForUpdatesOnly> <VariableChangeIndicator>version</VariableChangeIndicator> <HashVariable /> <HashType>None</HashType> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <ExclusiveDownload>false</ExclusiveDownload> <HttpReferer /> <SetupInstructions /> <Variables> <item> <key> <string>schangelog</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>https://ketarin.org/forum/topic/4381-kuppet-javascript-rendered-pages-for-ketarin/</TextualContent> <Name>schangelog</Name> </UrlVariable> </value> </item> <item> <key> <string>swebsite</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>https://ketarin.org/forum/topic/4381-kuppet-javascript-rendered-pages-for-ketarin/</TextualContent> <Name>swebsite</Name> </UrlVariable> </value> </item> <item> <key> <string>version</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>.+&gt;(\d+\s\S+\s\d+)\s*Update&lt;</Regex> <Url>https://ketarin.org/forum/topic/4381-kuppet-javascript-rendered-pages-for-ketarin/</Url> <TextualContent /> <Name>version</Name> </UrlVariable> </value> </item> <item> <key> <string>sdownload</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>href="([^'"]+)"[^&gt;]*&gt;Download link</Regex> <Url>https://ketarin.org/forum/topic/4381-kuppet-javascript-rendered-pages-for-ketarin/</Url> <Name>sdownload</Name> </UrlVariable> </value> </item> <item> <key> <string>sdownloadfake</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>https://www.iana.org/_img/bookmark_icon.ico</TextualContent> <Name>sdownloadfake</Name> </UrlVariable> </value> </item> <item> <key> <string>sdownloadpath</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>{startuppath:directory}</TextualContent> <Name>sdownloadpath</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand /> <ExecutePreCommand>CALL "%localappdata%\MEGAcmd\mega-get.bat" "{sdownload}" "{sdownloadpath}" EXIT 3</ExecutePreCommand> <ExecuteCommandType>Batch</ExecuteCommandType> <ExecutePreCommandType>Batch</ExecutePreCommandType> <SourceType>FixedUrl</SourceType> <PreviousLocation /> <DeletePreviousFile>true</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <FixedDownloadUrl>{sdownloadfake}</FixedDownloadUrl> <Name>Kuppet</Name> </ApplicationJob> </Jobs>
  3. Hi, @Ambimind! Bug report: redirection-delay works perfectly for my needs, but I've noticed an issue with caching related to this feature. When I'm working on a new app and I am not yet aware of the need to use the redirection-delay feature, then the content is cached by Kuppet without it and adding the value does not force Kuppet to re-cache the value until the timeout expires or Kuppet is forcefully closed. I've been forcefully closing it, but sometimes I forget and freak out since the page isn't working. Grrr. I suggest adding a hash of the post-data collection to the key for the cache. This should allow it to break the cache when this data is changed without manually killing Kuppet. Thanks again! I've been using Kuppet for a ton of new profiles, mostly at Logitech, HP, Netgear, and so on. Loving the improved versatility.
  4. Alrighty... Here's a post-download command you can use to keep the version information up to date in the kuppet.config file. Change the $kuppetCfg path, and make sure you use a unique $browserIndex for each different browser you want to include. Also, make sure the post-download command is set to use PowerShell. This one for Firefox: # For each browser you want to include: # Use a different Index and different UA pattern # 0 = Firefox $browserIndex = 0; $browserVersion = $app.variables.version.CachedContent; $browserUA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:$browserVersion) Gecko/20100101 Firefox/$browserVersion"; $kuppetCfg = "K:\Ketarin\kuppet\kuppet.config"; # do the work $kuppetJson = Get-Content $kuppetCfg -Raw | ConvertFrom-Json; $kuppetJson.userAgentArray[$browserIndex] = $browserUA; $Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False [System.IO.File]::WriteAllLines($kuppetCfg, ($kuppetJson | ConvertTo-Json), $Utf8NoBomEncoding) This is for Chrome: # For each browser you want to include: # Use a different Index and different UA pattern # 1 = Chrome $browserIndex = 1; $browserVersion = $app.variables.version.CachedContent; $browserUA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/$browserVersion Safari/537.36"; $kuppetCfg = "K:\Ketarin\kuppet\kuppet.config"; # do the work $kuppetJson = Get-Content $kuppetCfg -Raw | ConvertFrom-Json; $kuppetJson.userAgentArray[$browserIndex] = $browserUA; $Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False [System.IO.File]::WriteAllLines($kuppetCfg, ($kuppetJson | ConvertTo-Json), $Utf8NoBomEncoding) If you're going to include additional user-agents just make sure to use a new $browserIndex value for each one, otherwise you'll overwrite the wrong one. 2022-05-02: fixed to strip the BOM (byte-order marker) from the file when it's written
  5. @Ambimindthat seems like a perfect solution. I can roll it out on only those sites where it's necessary without imposing unnecessary delays on other sites. It works perfectly. This is the first time in probably a year that I've been able to check for updates for both of those apps in the same run. Yay! Seeing the power of using the POST data feature to tune Kuppet for individual sites. Can the other options be assigned there, too? It would be nice to specify a given UA for a site that doesn't like other user-agents, for example. If you plan to implement this as a feature there's no hurry from me. I can see myself using it someday, but currently don't have a need. When I have some free time I'll be writing a script to update the kuppet.config file with the current Chrome and Firefox versions in a post-update script. I'll share that here.
  6. Hi! It's not downloading CleanTalk plugin that's the problem, but from sites that are using it. Here's an example. However, the plugin must use a connection counter or something, too, since often the first request (after several hours) works fine but any subsequent request (say, for another app from the same site) will fail. If you setup a new app and just try to capture the version number you'll see how it misbehaves by the second or third request. Here's an app for it though -- Kuppet is used in {versionstub}. setdefaultbrowser.xml
  7. Thank you, @Ambimind! That fixes the user-agent functionality I was most concerned with. Thank you for explaining the logging behavior. That makes a lot of sense and works exactly as you describe. Please do not have the kuppet.config file from the distribution named kuppet.config. It should be named "kuppet.config.sample" or similar, that way it won't overwrite any custom kuppet.config file in the destination directory. It's working perfectly with CloudFlare now. I am having problems with a site using a specific WordPress plugin, though. This is only causing problems with one site I'm downloading from, but the software there is AWESOME and the last thing I want is to lose access to it. The plugin is CleanTalk Spam Protect, and it has a built-in Anti-Crawler FireWall that's responsible for imposing a delay of between 3 and 60 seconds with a cookie and cache-buster refresh/redirect that is resulting in Kuppet not making it to the "real" content of the page. I'm sure there are going to be many similar situations but I think it can be addressed in one of two ways each time. In both cases you'll want to have an option to turn it on or off to either prevent unwanted redirects or unnecessary delays. 1) Ensure that an option exists whether to impose the redirect behavior or not (I suggest followRedirects or similar). On some sites we will not want to follow redirects. On others (like this one) we will. When this option is enabled you simply allow the JavaScript or redirect to proceed as long as it's within the page-load-timeout period. Scanning the produced content for "window.location" should be sufficient to determine whether to wait "up to page-load-timeout" seconds for the redirect to occur. I'm sure some sites will obfuscate this, but for those that don't this should be an easy check and probably the easiest solution. 2) Provide a minWait option. Like the followRedirects option above, this would provide a solution for this scenario by allowing the page to finish loading and following any redirects or timed output before considering the page fully loaded. For example, setting a minWait of 70 seconds and the page-load-timeout at 90 seconds would allow any page with a 60 second imposed delay to complete. This is probably the more universal solution since it will allow certain other scripted content sufficient time to load, too. I don't see any other way to watch for anonymous setTimeout calls other than writing a handler to capture every call to setTimeout (which is probably overkill), otherwise I would just do that so we could do something like check the execution window of each timeout and anything under page-load-timeout would be allowed to proceed until all timeouts under page-load-timeout were triggered before capturing the page contents. This would address pages where the content has an advertisement delay or other interstitial as well, though I avoid those sites like the plague.
  8. To enable visibility in kuppet.config you can use chromeVisible: "chromeVisible" : 1
  9. I've been testing it and have found a few issues: The user agent option is never properly observed. No amount of fiddling with the userAgentArray in the kuppet.config file or -u or --user-agent parameters allows this to be changed. With the kuppet.config file removed and verbose logging enabled the user-agent assignment does appear only within the header summary, but what is sent to the server remains either: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4889.0 Safari/537.36 or: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/100.0.4889.0 Safari/537.36 In either case, the deciding factor here is actually whether the --chrome-visible option is enabled. If it's visible then it uses Chrome, otherwise it uses HeadlessChrome as the user-agent. The visibility option works perfectly - in that it either shows or hides the Chrome sandbox - usually in the background of monitor 1, but not always -- for some reason this is inconsistent. I don't see that the standard logging option does anything at all. Port, timeouts, wait, website-url, verbose logging, and version, too, each work well. The only real issue is that the user-agent is never being assigned from either the kuppet.config or the parameters and the current need to enable visibility in order to avoid the HeadlessChrome user-agent is a major downer. At least it's loading in the background most of the time, though, so it's not that big of a deal. The importance of avoiding HeadlessChrome can not be understated. Almost every site now blocks it by default. The new visibility option allows this to continue to operate as expected though. Thanks, again, @Ambimind
  10. Yay! Thank you, @Ambimind. I'll report any issues here.
  11. To anyone who reads this and wants to take advantage of the feature: just select all (CTRL+A) or select the apps you want to collect data about, press CTRL+SHIFT+C and then switch to Notepad++ and paste it. Make sure you have the Elastic Tabstops plugin installed.
  12. I've been making enormous changes to how I use Ketarin. I'm relaying some updates to a distribution server I control so they're more reliable and not dependent on a nonce for download, monitoring and recording change history to a database, have vastly increased the capabilities of the external version checking routines and automatic update system I wrote, and the vast majority of it literally depends on Ketarin. Even so, one thing I still find myself having to switch to the server to extract via SQL for is the same data that is visible in the main Ketarin interface. CTRL+C always obtains the raw XML for the selected apps, and there is no equivalent (in the context menu) to "copy table" or "copy view". Even though I don't think I would use this more than once or twice per month, it would enable me to quickly identify outliers and defects without having to resort to recreating the table through SQL as I've been doing for years. I imagine it would pretty simple to implement as it would only require copying the content of the selected rows for the visible columns (including custom columns!) instead of the underlying structures for each app. I pondered the best way to implement this and thought, "Hey, CTRL+SHIFT+C would be a good shortcut. Let's see if it's currently used for anything..." Imagine my surprise when, lo and behold, IT DID EXACTLY WHAT I WANTED!!! @floele, you rock! You've made this thing so intuitive that I'm starting to think Ketarin was actually produced by distilling my own brain cells when I wasn't looking. Thanks again!
  13. Hi, @cdumdum, welcome aboard! Are you running Windows 8? What version of PowerShell do you have installed? You might be able to get around this problem by using this instead: Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process; [System.Diagnostics.FileVersionInfo]::GetVersionInfo("{file}").FileVersion Please let me know if this works for you.
  14. The many popping windows that steal focus have continued to be a problem, so here's a workaround. Change {run_kuppet} to: Add-Type -Assembly Microsoft.VisualBasic; [Microsoft.VisualBasic.Interaction]::Shell('k:\ketarin\BIN\kuppet.exe 8008 30000', 'MinimizedNoFocus') | Out-Null; The path at Shell must be the complete path to Kuppet.exe. The next parameter is the port, the next parameter is the 30s timeout. The "MinimizedNoFocus" still runs the app but loads it in the background as a minimized window so it neither steals focus nor pops as a distraction. An alternative if you don't mind the many windows popping up is to use "NormalNoFocus" which will still allow it in the foreground as a restored/normal window, but still won't steal focus. "|Out-Null" is required or else it will insert the new PID in front of the URL when created with {>}
  15. No, the registry/policy stuff should be applied separately. That stuff will improve overall PC performance and reduce information leakage to MS. Above the "start process {file}" item add a "close process msedge.exe" This will close all instances of Edge and allow the installation to complete.
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.