Jump to content
Ketarin forum

shawn

Moderators
  • Posts

    1,125
  • Joined

  • Last visited

Everything posted by shawn

  1. Alrighty... Here's a post-download command you can use to keep the version information up to date in the kuppet.config file. Change the $kuppetCfg path, and make sure you use a unique $browserIndex for each different browser you want to include. Also, make sure the post-download command is set to use PowerShell. This one for Firefox: # For each browser you want to include: # Use a different Index and different UA pattern # 0 = Firefox $browserIndex = 0; $browserVersion = $app.variables.version.CachedContent; $browserUA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:$browserVersion) Gecko/20100101 Firefox/$browserVersion"; $kuppetCfg = "K:\Ketarin\kuppet\kuppet.config"; # do the work $kuppetJson = Get-Content $kuppetCfg -Raw | ConvertFrom-Json; $kuppetJson.userAgentArray[$browserIndex] = $browserUA; $Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False [System.IO.File]::WriteAllLines($kuppetCfg, ($kuppetJson | ConvertTo-Json), $Utf8NoBomEncoding) This is for Chrome: # For each browser you want to include: # Use a different Index and different UA pattern # 1 = Chrome $browserIndex = 1; $browserVersion = $app.variables.version.CachedContent; $browserUA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/$browserVersion Safari/537.36"; $kuppetCfg = "K:\Ketarin\kuppet\kuppet.config"; # do the work $kuppetJson = Get-Content $kuppetCfg -Raw | ConvertFrom-Json; $kuppetJson.userAgentArray[$browserIndex] = $browserUA; $Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False [System.IO.File]::WriteAllLines($kuppetCfg, ($kuppetJson | ConvertTo-Json), $Utf8NoBomEncoding) If you're going to include additional user-agents just make sure to use a new $browserIndex value for each one, otherwise you'll overwrite the wrong one. 2022-05-02: fixed to strip the BOM (byte-order marker) from the file when it's written
  2. @Ambimindthat seems like a perfect solution. I can roll it out on only those sites where it's necessary without imposing unnecessary delays on other sites. It works perfectly. This is the first time in probably a year that I've been able to check for updates for both of those apps in the same run. Yay! Seeing the power of using the POST data feature to tune Kuppet for individual sites. Can the other options be assigned there, too? It would be nice to specify a given UA for a site that doesn't like other user-agents, for example. If you plan to implement this as a feature there's no hurry from me. I can see myself using it someday, but currently don't have a need. When I have some free time I'll be writing a script to update the kuppet.config file with the current Chrome and Firefox versions in a post-update script. I'll share that here.
  3. Hi! It's not downloading CleanTalk plugin that's the problem, but from sites that are using it. Here's an example. However, the plugin must use a connection counter or something, too, since often the first request (after several hours) works fine but any subsequent request (say, for another app from the same site) will fail. If you setup a new app and just try to capture the version number you'll see how it misbehaves by the second or third request. Here's an app for it though -- Kuppet is used in {versionstub}. setdefaultbrowser.xml
  4. Thank you, @Ambimind! That fixes the user-agent functionality I was most concerned with. Thank you for explaining the logging behavior. That makes a lot of sense and works exactly as you describe. Please do not have the kuppet.config file from the distribution named kuppet.config. It should be named "kuppet.config.sample" or similar, that way it won't overwrite any custom kuppet.config file in the destination directory. It's working perfectly with CloudFlare now. I am having problems with a site using a specific WordPress plugin, though. This is only causing problems with one site I'm downloading from, but the software there is AWESOME and the last thing I want is to lose access to it. The plugin is CleanTalk Spam Protect, and it has a built-in Anti-Crawler FireWall that's responsible for imposing a delay of between 3 and 60 seconds with a cookie and cache-buster refresh/redirect that is resulting in Kuppet not making it to the "real" content of the page. I'm sure there are going to be many similar situations but I think it can be addressed in one of two ways each time. In both cases you'll want to have an option to turn it on or off to either prevent unwanted redirects or unnecessary delays. 1) Ensure that an option exists whether to impose the redirect behavior or not (I suggest followRedirects or similar). On some sites we will not want to follow redirects. On others (like this one) we will. When this option is enabled you simply allow the JavaScript or redirect to proceed as long as it's within the page-load-timeout period. Scanning the produced content for "window.location" should be sufficient to determine whether to wait "up to page-load-timeout" seconds for the redirect to occur. I'm sure some sites will obfuscate this, but for those that don't this should be an easy check and probably the easiest solution. 2) Provide a minWait option. Like the followRedirects option above, this would provide a solution for this scenario by allowing the page to finish loading and following any redirects or timed output before considering the page fully loaded. For example, setting a minWait of 70 seconds and the page-load-timeout at 90 seconds would allow any page with a 60 second imposed delay to complete. This is probably the more universal solution since it will allow certain other scripted content sufficient time to load, too. I don't see any other way to watch for anonymous setTimeout calls other than writing a handler to capture every call to setTimeout (which is probably overkill), otherwise I would just do that so we could do something like check the execution window of each timeout and anything under page-load-timeout would be allowed to proceed until all timeouts under page-load-timeout were triggered before capturing the page contents. This would address pages where the content has an advertisement delay or other interstitial as well, though I avoid those sites like the plague.
  5. To enable visibility in kuppet.config you can use chromeVisible: "chromeVisible" : 1
  6. I've been testing it and have found a few issues: The user agent option is never properly observed. No amount of fiddling with the userAgentArray in the kuppet.config file or -u or --user-agent parameters allows this to be changed. With the kuppet.config file removed and verbose logging enabled the user-agent assignment does appear only within the header summary, but what is sent to the server remains either: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4889.0 Safari/537.36 or: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/100.0.4889.0 Safari/537.36 In either case, the deciding factor here is actually whether the --chrome-visible option is enabled. If it's visible then it uses Chrome, otherwise it uses HeadlessChrome as the user-agent. The visibility option works perfectly - in that it either shows or hides the Chrome sandbox - usually in the background of monitor 1, but not always -- for some reason this is inconsistent. I don't see that the standard logging option does anything at all. Port, timeouts, wait, website-url, verbose logging, and version, too, each work well. The only real issue is that the user-agent is never being assigned from either the kuppet.config or the parameters and the current need to enable visibility in order to avoid the HeadlessChrome user-agent is a major downer. At least it's loading in the background most of the time, though, so it's not that big of a deal. The importance of avoiding HeadlessChrome can not be understated. Almost every site now blocks it by default. The new visibility option allows this to continue to operate as expected though. Thanks, again, @Ambimind
  7. Yay! Thank you, @Ambimind. I'll report any issues here.
  8. To anyone who reads this and wants to take advantage of the feature: just select all (CTRL+A) or select the apps you want to collect data about, press CTRL+SHIFT+C and then switch to Notepad++ and paste it. Make sure you have the Elastic Tabstops plugin installed.
  9. I've been making enormous changes to how I use Ketarin. I'm relaying some updates to a distribution server I control so they're more reliable and not dependent on a nonce for download, monitoring and recording change history to a database, have vastly increased the capabilities of the external version checking routines and automatic update system I wrote, and the vast majority of it literally depends on Ketarin. Even so, one thing I still find myself having to switch to the server to extract via SQL for is the same data that is visible in the main Ketarin interface. CTRL+C always obtains the raw XML for the selected apps, and there is no equivalent (in the context menu) to "copy table" or "copy view". Even though I don't think I would use this more than once or twice per month, it would enable me to quickly identify outliers and defects without having to resort to recreating the table through SQL as I've been doing for years. I imagine it would pretty simple to implement as it would only require copying the content of the selected rows for the visible columns (including custom columns!) instead of the underlying structures for each app. I pondered the best way to implement this and thought, "Hey, CTRL+SHIFT+C would be a good shortcut. Let's see if it's currently used for anything..." Imagine my surprise when, lo and behold, IT DID EXACTLY WHAT I WANTED!!! @floele, you rock! You've made this thing so intuitive that I'm starting to think Ketarin was actually produced by distilling my own brain cells when I wasn't looking. Thanks again!
  10. Hi, @cdumdum, welcome aboard! Are you running Windows 8? What version of PowerShell do you have installed? You might be able to get around this problem by using this instead: Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process; [System.Diagnostics.FileVersionInfo]::GetVersionInfo("{file}").FileVersion Please let me know if this works for you.
  11. The many popping windows that steal focus have continued to be a problem, so here's a workaround. Change {run_kuppet} to: Add-Type -Assembly Microsoft.VisualBasic; [Microsoft.VisualBasic.Interaction]::Shell('k:\ketarin\BIN\kuppet.exe 8008 30000', 'MinimizedNoFocus') | Out-Null; The path at Shell must be the complete path to Kuppet.exe. The next parameter is the port, the next parameter is the 30s timeout. The "MinimizedNoFocus" still runs the app but loads it in the background as a minimized window so it neither steals focus nor pops as a distraction. An alternative if you don't mind the many windows popping up is to use "NormalNoFocus" which will still allow it in the foreground as a restored/normal window, but still won't steal focus. "|Out-Null" is required or else it will insert the new PID in front of the URL when created with {>}
  12. No, the registry/policy stuff should be applied separately. That stuff will improve overall PC performance and reduce information leakage to MS. Above the "start process {file}" item add a "close process msedge.exe" This will close all instances of Edge and allow the installation to complete.
  13. That's actually my template I suspect your problem is two-fold: 1) Your entire setup instruction is just "run {file}". 2) Edge now runs 24/7 unless you change settings or impose policy changes. There are two solutions: 1) Add an installation instruction prior to "run {file}" that closes existing apps (msedge.exe). This will close edge and allow the installation to proceed even if it was running when the process started. 2) Disable edge autorun (startup boost and background mode). This will prevent Edge from starting on its own in the background even when there's no visible windows, but it will NOT close Edge if it's already loaded. I recommend doing both. Most applications will fail to install if it's already running, so either add an instruction to close them, or expect it to fail on those where processes are already running. Edge startup boost and background mode are both resource hogs and cause more problems than they solve on any relatively recent hardware. reg add "HKLM\SOFTWARE\Microsoft\EdgeUpdate\Clients\{56EB18F8-B008-4CBD-B6D2-8C97FE7E9062}\Commands\on-logon-autolaunch" /v AutoRunOnLogon /d 0 /t reg_dword /f reg add "HKLM\SOFTWARE\WOW6432Node\Microsoft\EdgeUpdate\Clients\{56EB18F8-B008-4CBD-B6D2-8C97FE7E9062}\Commands\on-logon-autolaunch" /v AutoRunOnLogon /d 0 /t reg_dword /f reg add "HKLM\Software\Policies\Microsoft\Edge" /v StartupBoostEnabled /d 0 /t reg_dword /f reg add "HKLM\Software\Policies\Microsoft\Edge" /v BackgroundModeEnabled /d 0 /t reg_dword /f gpupdate /force There are many other policies I'd also encourage you to take advantage of (such as turning off their crapware, spying, adware, prerender, network prediction and so on).
  14. Your uploads aren't visible to me, so this is based on the content only: are you using the full installation files for Edge? These are offline installation packages: https://www.microsoft.com/en-us/edge/business/download x64: match for (http[^;]+MicrosoftEdgeEnterpriseX64.msi) x86: match for (http[^;]+MicrosoftEdgeEnterpriseX86.msi)
  15. I've been using the `LastUpdated` and a modified variation on it for `LastVersion` based on this for years. I'm going through ALL of my apps right now and it's painful to have to add the current info variables I need for my external tools, so I decided to wrap it all in a function and force it across my database. This is how I'm doing it: #### function NewAppVariable { Param( [Parameter(Mandatory)] $sVarName, $sVarValue = "", [ValidateSet("RegularExpression", "StartEnd", "Textual")] $sVarType = "Textual" ) # Derived access property, ie. app.variables.$sVarName.$varcnt $varcnt = if($sVarType -eq "Textual"){"TextualContent"}Else{"CachedContent"} # If it doesn't exist, then create a new one if(-Not $app.variables.$sVarName){ ECHO "=== $sVarName NOT FOUND, ADDING IT ===" # Add var to job/app $newvar = New-Object -TypeName Ketarin.UrlVariable -ArgumentList $sVarName,$app.variables $newvar.VariableType = $sVarType $app.variables.Add($sVarName, $newvar) $app.Save() } # If it was passed a specific value, assign it if( $sVarValue -ne "" ){ $app.variables.$sVarName.$varcnt = $sVarValue } } #### This means I can then use the following much simpler lines to add my new variables: #### ADD MISSING VARIABLES & UPDATE Last* Cache NewAppVariable "sbits" NewAppVariable "schangelog" NewAppVariable "sdownload" NewAppVariable "snotes" NewAppVariable "splatform" NewAppVariable "swebsite" NewAppVariable "svendor" NewAppVariable "version" NewAppVariable "LastUpdate" {Get-Date -UFormat %Y%m%dT%H%M} NewAppVariable "LastVersion" $sversion #### The way it's written it will never overwrite or destroy an existing variable or its contents, other than LastUpdate and LastVersion which are specifically intended for that purpose. If you need another variable it's as simple as NewAppVariable <variable name> <newvalue> <type>. Thank you again @Ambimind!
  16. It looks like there's a problem with synchronicity. The global variable for {run_kuppet} must not be getting parsed as frequently as it needs to be by Ketarin since Kuppet isn't staying alive long enough - or even launching at all if Ketarin has been open for an extended period of time. Since I always have Ketarin open this means that Kuppet often doesn't behave well. To resolve this problem I've rewritten several of my apps to avoid Cloudfront and rewrote the {run_kuppet} to simply: START "BIN\kuppet.exe" "8008 30000" This no longer checks for an existing instance of Kuppet, which means that subsequent calls will each attempt to load it, and will be ignored since an existing instance is already running. Since I don't have window:hidden flag, this results in several windows popping up briefly and disappearing almost instantaneously. This is a good thing since it indicates that the first instance is still running, and allows Kuppet to be reinstantiated on demand should it time out. Now almost every app that I'm using Kuppet for is working well, with a random stall on a couple sites that use site throttling and large files (but the file still downloads fine). I think increasing the Kuppet timeout (or eliminating it entirely) will resolve both of these problems.
  17. This is because some paths will be cached from the previous locations. For specific recommendations I would need to see the app profiles of those apps that broke for you. One of the best ways to avoid this problem is by using the built in variables for paths: {root}, {startuppath}, {file} and the related functions: {file:directory}, {file:filename} As long as your installation scripts use these path-agnostic variables and functions you should be fine. But share a couple of your defective app profiles and I'm happy to look them over.
  18. I just want to say thanks again for this, @floele. I was fiddling with my after-update scripts and this thread helped me fix another bug.
  19. There are several examples on how to use 7-zip or UniExtract to extract compressed files in the wiki. Both 7-zip and UniExtract support RAR files. The syntax for the command-line version of 7-zip is: 7z x -y -o"{root}PATH\" "{file}" If you want to include a password you would do so like this: 7z x -y -p"th3p455w@rd" -o"{root}PATH\" "{file}"
  20. This is most likely caused by the URLencoding of space and ampersand. You can either use the ":replace" or ":multireplace" functions or you can use the :urldecode/:urlencode functions - both of which can get what you're after, but depend on you knowing which ones need to be replaced for the specific website and downloaded files. "%20" is a space. "%26" is an ampersand. Many non-English characters may also be replaced - but only on some websites. Unfortunately, it's really a matter of observation and testing for the individual website. When I'm troubleshooting stuff like this I always echo the command to a text file so I can compare the values to what I expect vs what is actually being used. echo "H:\Ketarin film\wget\wget" -N --user-agent=Mozilla/5.0 --output-document="{file:directory}\{01name}.zip" "{DLpdf}" --no-check-certificate >>test.txt "H:\Ketarin film\wget\wget" -N --user-agent=Mozilla/5.0 --output-document="{file:directory}\{01name}.zip" "{DLpdf}" --no-check-certificate Then the file "test.txt" can be read to see what those variables are actually being replaced with. For example, the {01name} variable may include literal "%26" instead of "&" - which is a different file name, and that will result in the command failing in Ketarin. Changing it to this (using urldecode) might be enough: "H:\Ketarin film\wget\wget" -N --user-agent=Mozilla/5.0 --output-document="{file:directory}\{01name:urldecode}.zip" "{DLpdf:urldecode}" --no-check-certificate Good luck! Let us know how it goes. Also, when posting in the future it will help us (and you) if you better detail what the specific variables represent and what the values are that Ketarin is replacing them with. The same echo trick will work with that: echo "{file}" /// "{file:directory}" /// "{01name}" /// "{DLpdf}" >>test.txt
  21. The ability to replace the user-agent string has become essential. The Cloudfront servers that are responsible for Logitech, SnagIt and others are now completely blocking the HeadlessChrome user-agent.
  22. Or maybe a configuration file that it will parse to populate these variables.
  23. The cache appears to only survive about 10 seconds - so on a large download or on a slow site, it may not keep the cache long enough for the post-update scripts to run without re-downloading the variable webpages. I've changed the command used to extend the timeout to: if(!(PS kuppet -ea 0)) {START -WindowsStyle hidden BIN\kuppet "8008 30000"} This increases the timeout to 30 seconds and resolves my immediate problem, but it will not accept any value above 30000 (30s). I suspect there is a hard limit in place to ensure that Kuppet doesn't stay in memory forever. Unfortunately, this forces some applications to re-request data in order to perform my post-update scripts. I guarantee on some sites (just experienced it with SnagIt) this will become a problem. Re-requesting pages can get you throttled with a recaptcha or cloudflare stall. Grrrr. Please increase the hard limit to 5m or more, and/or allow us to configure caching on an execution basis. Maybe allow Kuppet to run without a hard limit (timeout=0) - and use the timeout only in relation to the cache duration. I leave Ketarin open 24/7 anyway so there's no reason I can think of to close Kuppet any more frequently than I reboot (once per month or so). Something like this: kuppet.exe [port] [cache-timeout-ms] [hard-limit-ms] [custom-user-agent] Thanks, again, @Ambimind - this has already resolved problems I've been having with about a dozen Applications.
  24. It is caching - yay! I didn't read the log very carefully. It looks like the parameters are: kuppet.exe [port] [timeout-ms] It disconnects if idle for 30s or more no matter what. It's using the following user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/93.0.4577.63 Safari/537.36 Can you make it possible to change the user-agent? Some sites block HeadlessChrome by default.
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.