Jump to content
Ketarin forum

shawn

Moderators
  • Content count

    949
  • Joined

  • Last visited

Everything posted by shawn

  1. that might be perfect. what is the "filename" param? is there a safe default I can pass?
  2. I finally got around to using Ketarin to push update information to my server. It works for hundreds of my apps, though does feel to be significantly slower than Ketarin was before I added this. I've also noticed a couple that fail. The applicable line of PS code is: 'version'=$($app.variables.ReplaceAllInString('{version}')); The error is: Drupal Exception calling "ReplaceAllInString" with "1" argument(s): "The remote server returned an error: (403) Forbidden." Exception calling "ReplaceAllInString" with "1" argument(s): "The remote server returned an error: (403) Forbidden." Exception calling "ReplaceAllInString" with "1" argument(s): "The remote server returned an error: (403) Forbidden." Exception calling "ReplaceAllInString" with "1" argument(s): "The remote server returned an error: (403) Forbidden." Since Drupal now uses IP-based brute-force prevention, I suspect that it's actually an error related to an attempt by Ketarin to re-parse {version} based on the two server requests it has to make to collect it. If that is the case, it explains why Ketarin has become so much slower since I began this integration - it would be making additional server requests with every single App and that would significantly increase timing. Is there a way in powershell at post-update to only collect the cached content of those variables without re-triggering? This is in post-update so it should have current data cached for all of them.
  3. shawn

    Protocol Violation errors: fix?

    Sure! Here's a minimal Krita x64 to demonstrate the issue: <?xml version='1.0' encoding='utf-8'?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="d145b6eb-736a-47bf-b546-66b12e20f371"> <Category>Editors</Category> <WebsiteUrl>https://krita.org/en/download/krita-desktop/</WebsiteUrl> <UserAgent>{wget}</UserAgent> <UserNotes /> <LastFileSize>101750480</LastFileSize> <LastFileDate>2018-10-11T06:20:13-07:00</LastFileDate> <IgnoreFileInformation>false</IgnoreFileInformation> <DownloadBeta>Default</DownloadBeta> <DownloadDate xsi:nil="true" /> <CheckForUpdatesOnly>false</CheckForUpdatesOnly> <VariableChangeIndicator>version</VariableChangeIndicator> <HashVariable /> <HashType>None</HashType> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <ExclusiveDownload>false</ExclusiveDownload> <HttpReferer /> <SetupInstructions /> <Variables> <item> <key> <string>version</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>/krita-x64-([\d\.]+)-setup</Regex> <Url>https://krita.org/en/download/krita-desktop/</Url> <Name>version</Name> </UrlVariable> </value> </item> <item> <key> <string>sdownload</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>['"]([^'"]+x64[^'"]+exe)['"]</Regex> <Url>https://krita.org/en/download/krita-desktop/</Url> <Name>sdownload</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand /> <ExecutePreCommand /> <ExecuteCommandType>Batch</ExecuteCommandType> <ExecutePreCommandType>Batch</ExecutePreCommandType> <SourceType>FixedUrl</SourceType> <PreviousLocation>K:\_Ketarin\Ketarin\..\Editors\Krita_(x64)-4.1.5.exe</PreviousLocation> <DeletePreviousFile>true</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <LastUpdated>2018-10-23T06:17:57.5041972-07:00</LastUpdated> <TargetPath>..\{category}\{appname:regexreplace:([\s\t\r\n\-\\&amp;\/]+):_}-{version}.{url:ext}</TargetPath> <FixedDownloadUrl>{sdownload}</FixedDownloadUrl> <Name>Krita (x64)</Name> </ApplicationJob> </Jobs> With useUnsafeHeaderParsing disabled it fails. I suspect the site is using Incapsula to prevent "bots" from scraping the site. The previous link shows that Incapsula does this intentionally to break some bots even though it really is this easy to get around it. I was wondering if you'd be interested in either globally enabling useUnsafeHeaderParsing, or exposing a per-app setting to allow this behavior that could be implemented via Reflection. This would provide safe header parsing wherever possible and only be enabled where needed. Coincidentally, implementing this change in app.config actually reduced the number of errors I was getting from ~104 to ~80 - so it has fixed several of my other apps as well. I wish I would have kept better error logs though so I could find out which ones without having to turn it back off and re-trigger the errors.
  4. shawn

    Protocol Violation errors: fix?

    I'm working to resolve my buggy apps. Some work fine some days but not on others. Some issues are surely server throttling or automation/abuse prevention efforts. A couple (Krita) stopped working recently with Protocol Violation errors. The specific error messages are: The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF or The contents of the URL can not be loaded. The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF I added the recommended app.config directive that's supposed to resolve this problem: <system.net> <settings> <httpWebRequest useUnsafeHeaderParsing="true" /> </settings> </system.net> Alone, it didn't help. Well, it actually changed the error to something else: The contents of the URL can not be loaded: The underlying connection was closed: An unexpected error occurred on a send. I changed the UA to wget and tried again. Now it works. There's probably a filter in place to allow wget to bypass whatever abuse headers are being sent. Or maybe Ketarin behaves differently when a UA is assigned. This app hasn't successfully worked since August, so this change is a "fix" of sorts, but it's a strange one, and the next release of Ketarin will probably overwrite my app.config so I will have to fix this again to allow it to get around this problem. I tried removing the new app.config stuff and just changing the UA to wget and it failed again with the protocol violation error. Added it back in and, along with the UA change it appears to consistently succeed now. I know that allowing unsafe headers is not ideal, but since we're dealing with servers that have cache control and abuse features in place, it is likely the only way to bypass these techniques. I'm open to alternative fixes if anyone has suggestions.
  5. Feature request: this is all pretty frustrating dealing with content access for different variable types and mechanisms. What would be ideal is something along the lines of ReplaceAllInString that did *not* re-parse/re-download URLs that were already parsed before or 'recently' cached. Parsing 'new' URLs/content/patterns/functions would, of course, be necessary when there was no cache hit, but ideally I should be able to use the following on Drupal without triggering two new URL requests each time: $app.variables.ReplaceAllInString("{version}") Ideally some form of temporary cache should be implemented anyway, as it could greatly improve performance and reliability. Perhaps add an option under Advanced to either ignore the cache completely or force requests again after x seconds. This would allow multiple requests to the same server to be bypassed in the case of multiple downloads (such as different architecture builds for the same app).
  6. Seeing a different issue now. Those apps that use FileHippo don't export the {version} variable correctly since I added the manual parsing method above. I suspect since it isn't a variable in the standard sense, it'll need to be addressed differently. I'm going to add a FH wrapper to ensure that the variables are properly fleshed out in the case of FH apps.
  7. Welcome, @bzowk! Many of us use Ketarin as our package downloader and push the data to our own app/device management system for installation. While you *could* put Ketarin on all your devices and keep the store local to each device, it's much easier to use Ketarin as the downloading tool and keep everything on the LAN or WAN via Ketarin, with your external package installer. You can script *from* Ketarin to SCCM with PowerShell and effectively distribute/install via SCCM the way it should be done. You have several ways to do this. You could just log things to an external file with Ketarin, then parse that log externally, or you can use the "after updating an application" global script to directly interact with the SCCM library. You could also push the data to a database or do, literally, anything else that can be done in batch, powershell, or c# to get the data from Ketarin to SCCM.
  8. Nope, using a full-page content capture and parsing it with variable functions didn't work. post-update they get triggered again. I might be able to get around it by assigning the relevant variables in powershell instead of doing so in the native Ketarin way. The download URL requires the version number, which is parsed from the release notes page, which is parsed from the download page, and the version is also used in the local filename. Thoughts/recommendations?
  9. I worked around much of this by parsing the VariableType for each variable. I don't know if it's a parsed or text variable since some apps are necessarily constructed differently. I parse the value as either TextualContent or CachedContent based on the VariableType. I then check to see if the value is equal to the variable name in braces, which would indicate that the variable didn't exist, and removes it if so. Then it checks for a curly brace in the variable contents and if present it performs a ReplaceAllInString so that stuff like file-time information is properly presented. Here's a sample code block: $sversion = switch( $App.Variables."version".VariableType ) { 'Textual' {$App.Variables."version".TextualContent;} default {$App.Variables."version".CachedContent;} } if($sversion -eq "{version}"){$sversion = "";} if($sversion -like '*{*'){$sversion = $($app.variables.ReplaceAllInString($sversion));} Unfortunately, while this resolves a couple problems and vastly improves performance since there's less re-parsing going on, it still fails when the ReplaceAllInString is called against some parsed variables. Here's the specific error case, for Drupal: "schangelogstub" is a RegEx parsed value from https://www.drupal.org/download for href="([^"]+)"[^>]*>[^'"]+notes "schangelog" is a textual variable with value https://www.drupal.org{schangelogstub} Parsing for schangelog within PowerShell forces it to re-request https://www.drupal.org/download every single time ReplaceAllInString is called, even though the value *had* to be parsed (and assumingly cached) in order for the file to have been saved. I know that variables are not parsed unless and until they're called. But the information that they depend on should be pulled from the cache when possible, right? If so, I'd like to think a potential way around this might be to use a single variable that collected the entire contents of the page "/download" instead of using it as a URL source in multiple variables, then repeatedly parsing that variable with TextualVariables. The schangelog sample above suggests that this probably won't resolve the problem. Sigh. I'll try it when I get back tonight. I guess the real question it: What's the effective "life" of cached values?
  10. shawn

    Logitech Downloads

    Hi, everyone! I spent the day yesterday trying to get around the ugly SalesForce remoting functionality on the Logitech downloads page. I ended up writing a console script that'll parse the data every hour and store it on one of my servers. If you're interested in Logitech downloads (en_us) you can parse the JSON output of the export here. Please don't abuse it, but use it to gather version numbers and direct download links for various Logitech applications.
  11. shawn

    New Version / Old Version Column

    There are several examples on the forum of downloading with post authentication or via authentication proxy. I'm not seeing the behavior you describe where, after the first cycle of collecting the old version, it would require 2 or more cycles to correctly assign values.
  12. shawn

    Running Self Extracting EXE from Ketarin

    "-spe" is a 7-zip switch that means "ignore root folder" or, more specifically "eliminate duplication of root folder for extract command." More about "-spe" here. Note the case MUST BE EQUIVALENT. "post commands" only run if the download actually happened. you may need to "force download" to cause them to run.
  13. shawn

    Could not create SSL/TLS secure channel

    you should also consider running this reg: REG ADD "HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319" /v "SchUseStrongCrypto" /t REG_DWORD /d 1 /f REG ADD "HKLM\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319" /v "SchUseStrongCrypto" /t REG_DWORD /d 1 /f
  14. version is a constructed variable for applications based on FileHippo and a manual variable for other applications. the same methods work to identify the available variables.
  15. shawn

    Remove previous folders

    Are you saving the file with the original file name? Yo could save them with just the version (since they're all win32 you don't need "-hash-windows-x86"). Saving with just the version number would allow you to skip this issue. If you'd rather do it post-extract then you can use powershell with something like this: Get-ChildItem $Path | Where{$_.Name -Match "<RegEx Pattern>"} | Remove-Item This would look something like this in practice: Get-ChildItem "k:\_ketarin\builds\pcsx2" | Where{$_.Name -Match "pcsx2-v[\d\.]+-dev-\d+-[a-z0-9]+-windows-x86"} | Remove-Item You could also use something much simpler if you want to trim it down a bit: Get-ChildItem "k:\_ketarin\builds\pcsx2" | Where{$_.Name -Like "pcsx2-v*"} | Remove-Item Or if you want to delete all of them but exclude the current build you could use: Get-ChildItem "k:\_ketarin\builds\pcsx2" | Where{$_.Name -Like "pcsx2-v*"} | Where{$_.Name -NotLike "pcsx2-v{version}*"} | Remove-Item
  16. shawn

    Running Self Extracting EXE from Ketarin

    Another option where root folders are using version numbers (as is the case with DDU) in the root folder is to rename the destination folder before unzipping and then rename it back to the generic version when finished. move "\Ketarin\Drivers\DDU" "DDU v{version}" 7z -spe ... move "\Ketarin\Drivers\DDU v{version}" "DDU"
  17. shawn

    Running Self Extracting EXE from Ketarin

    IF it was distributed with a "root folder" then you might be able to use the "-spe" feature of 7zip to ignore it. Give it a try and see if that works. Sometimes renaming the downloaded file to what would be the root folder will allow -spe to work even in situations where it otherwise fails. More about that issue here.
  18. shawn

    Zip password

    be sure to escape special characters if it doesn't work as is. this means that characters like %, &, !, and ^ should be preceeded by a ^. this is necessary for pw mgmt stuff from nirsoft, for example.
  19. shawn

    Running Self Extracting EXE from Ketarin

    have you just tried the 7z app with an output folder? 7z x -y -o"{root}PATH\" "{file}"
  20. glad you got it sorted
  21. Yes, this is all pretty much what you'd be typing at a command prompt. Maybe what you're after is robocopy or xcopy to overwrite only the older files from the archived copy to the new copy? Assuming you're running a backup first. Something like this: move "C:\Program Files\Launchbox\Emulators\Dolphin\Updates\Dolphin" Dolphin-backup "{7z}" x "{file}" -o"C:\Program Files\Launchbox\Emulators\Dolphin\Updates" -y move "C:\Program Files\Launchbox\Emulators\Dolphin\Updates\Dolphin-x64" Dolphin robocopy "C:\Program Files\Launchbox\Emulators\Dolphin\Updates\Dolphin-backup" "C:\Program Files\Launchbox\Emulators\Dolphin\Updates\Dolphin" /w:1 /r:1 /e /xo del /s /q "C:\Program Files\Launchbox\Emulators\Dolphin\Updates\Dolphin-backup\*.*" rd /s /q "C:\Program Files\Launchbox\Emulators\Dolphin\Updates\Dolphin-backup" The above *should* rename the original Dolphin folder to Dolphin-backup, extract to Dolphin-x64, rename Dolphin-x64 to Dolphin, copy your preferences and the files that are newer (such as cache files) from Dolphin-backup to Dolphin, then purge and remove Dolphin-backup. Untested, but should work.
  22. shawn

    automated script versions

    Most variables are independent to each individual app profile, so it's impossible to know them all in advance if you're using any third-party app profiles or were...uh..."less consistent" in previous apps you created for yourself. Those variables can be obtained by looking at the values in app properties, variables, or a copy & paste from the specific app profile to notepad, then looking for any items in Variables/item/key/string. Some others are listed at Special Variables on the wiki. Even more can be found with the powershell samples at the bottom of this.
  23. Sweet. Here's the official page for MEGAcmd
  24. the documentation suggests that megacmd can not access publicly shared documents - only the files within your own account.
×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.