shawn

Thanks, @floele! I might actually start reviewing the code myself, now.

7z x "{file}" -o"C:\your output folder\" -y

It's ridiculously simple to "install" a font: copy "fontfile.otf" "%windir%\fonts"

awesome! I feel like this release is all about my requests. Thanks, @floele!

Was the server/source available? Could you hit it from a browser? I suspect a firewall was blocking you for some reason. Either your IP or the specific user-agent that the app was using.

Thanks, @floele. Maybe I need to just find a different source for the Drupal version number. This is the *only* app that is misbehaving like this.

I've now tried using this and it does seem to almost always pull the data from the cache instead of re-requesting URLs. I had some problems with the date and filename information so ended up prepopulating those. Here's what I'm currently using: $sfilename = $($App.CurrentLocation); $sfiletime = [datetime]$($App.DownloadDate); $sversion = $app.variables.ReplaceAllInString("{version}", $sfiletime, $sfilename, $true); I would assume that $App.DownloadDate would have the file timestamp from the current download (ideally the date the file was originally published, but the current timestamp if that's not an option). If that's not the case, then is there another variable to pull this information directly? I switched it out with this and it seems to be working ok, any reason I should use something else? $sfilename = $($App.CurrentLocation); $sfiletime = [datetime]$( (Get-Item $sfilename).LastWriteTime ); Now... It looks like it might work, but I'm still having one problem. One of my constructed variables re-requests the URL every time regardless. Is there a reason for it? The variable is schangelog and is defined as https://www.drupal.org{schangelogstub}. schangelogstub is another variable that uses a regex capture from the site to get the actual current version number to identify the specific URL for the changelog. The "new" ReplaceAllInString is still re-requesting the URL to build schangelogstub, and returning: Drupal The remote server returned an error: (403) Forbidden. (https://www.drupal.org/8/download) The file downloads. The rest of the variables are populated. but it is still reported as an error by Ketarin since the PS chokes on re-requesting this content. Darn Drupal.

that might be perfect. what is the "filename" param? is there a safe default I can pass?

Sure! Here's a minimal Krita x64 to demonstrate the issue: <?xml version='1.0' encoding='utf-8'?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="d145b6eb-736a-47bf-b546-66b12e20f371"> <Category>Editors</Category> <WebsiteUrl>https://krita.org/en/download/krita-desktop/</WebsiteUrl> <UserAgent>{wget}</UserAgent> <UserNotes /> <LastFileSize>101750480</LastFileSize> <LastFileDate>2018-10-11T06:20:13-07:00</LastFileDate> <IgnoreFileInformation>false</IgnoreFileInformation> <DownloadBeta>Default</DownloadBeta> <DownloadDate xsi:nil="true" /> <CheckForUpdatesOnly>false</CheckForUpdatesOnly> <VariableChangeIndicator>version</VariableChangeIndicator> <HashVariable /> <HashType>None</HashType> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <ExclusiveDownload>false</ExclusiveDownload> <HttpReferer /> <SetupInstructions /> <Variables> <item> <key> <string>version</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>/krita-x64-([\d\.]+)-setup</Regex> <Url>https://krita.org/en/download/krita-desktop/</Url> <Name>version</Name> </UrlVariable> </value> </item> <item> <key> <string>sdownload</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>['"]([^'"]+x64[^'"]+exe)['"]</Regex> <Url>https://krita.org/en/download/krita-desktop/</Url> <Name>sdownload</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand /> <ExecutePreCommand /> <ExecuteCommandType>Batch</ExecuteCommandType> <ExecutePreCommandType>Batch</ExecutePreCommandType> <SourceType>FixedUrl</SourceType> <PreviousLocation>K:\_Ketarin\Ketarin\..\Editors\Krita_(x64)-4.1.5.exe</PreviousLocation> <DeletePreviousFile>true</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <LastUpdated>2018-10-23T06:17:57.5041972-07:00</LastUpdated> <TargetPath>..\{category}\{appname:regexreplace:([\s\t\r\n\-\\&amp;\/]+):_}-{version}.{url:ext}</TargetPath> <FixedDownloadUrl>{sdownload}</FixedDownloadUrl> <Name>Krita (x64)</Name> </ApplicationJob> </Jobs> With useUnsafeHeaderParsing disabled it fails. I suspect the site is using Incapsula to prevent "bots" from scraping the site. The previous link shows that Incapsula does this intentionally to break some bots even though it really is this easy to get around it. I was wondering if you'd be interested in either globally enabling useUnsafeHeaderParsing, or exposing a per-app setting to allow this behavior that could be implemented via Reflection. This would provide safe header parsing wherever possible and only be enabled where needed. Coincidentally, implementing this change in app.config actually reduced the number of errors I was getting from ~104 to ~80 - so it has fixed several of my other apps as well. I wish I would have kept better error logs though so I could find out which ones without having to turn it back off and re-trigger the errors.

Feature request: this is all pretty frustrating dealing with content access for different variable types and mechanisms. What would be ideal is something along the lines of ReplaceAllInString that did *not* re-parse/re-download URLs that were already parsed before or 'recently' cached. Parsing 'new' URLs/content/patterns/functions would, of course, be necessary when there was no cache hit, but ideally I should be able to use the following on Drupal without triggering two new URL requests each time: $app.variables.ReplaceAllInString("{version}") Ideally some form of temporary cache should be implemented anyway, as it could greatly improve performance and reliability. Perhaps add an option under Advanced to either ignore the cache completely or force requests again after x seconds. This would allow multiple requests to the same server to be bypassed in the case of multiple downloads (such as different architecture builds for the same app).

Seeing a different issue now. Those apps that use FileHippo don't export the {version} variable correctly since I added the manual parsing method above. I suspect since it isn't a variable in the standard sense, it'll need to be addressed differently. I'm going to add a FH wrapper to ensure that the variables are properly fleshed out in the case of FH apps.

I'm working to resolve my buggy apps. Some work fine some days but not on others. Some issues are surely server throttling or automation/abuse prevention efforts. A couple (Krita) stopped working recently with Protocol Violation errors. The specific error messages are: The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF or The contents of the URL can not be loaded. The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF I added the recommended app.config directive that's supposed to resolve this problem: <system.net> <settings> <httpWebRequest useUnsafeHeaderParsing="true" /> </settings> </system.net> Alone, it didn't help. Well, it actually changed the error to something else: The contents of the URL can not be loaded: The underlying connection was closed: An unexpected error occurred on a send. I changed the UA to wget and tried again. Now it works. There's probably a filter in place to allow wget to bypass whatever abuse headers are being sent. Or maybe Ketarin behaves differently when a UA is assigned. This app hasn't successfully worked since August, so this change is a "fix" of sorts, but it's a strange one, and the next release of Ketarin will probably overwrite my app.config so I will have to fix this again to allow it to get around this problem. I tried removing the new app.config stuff and just changing the UA to wget and it failed again with the protocol violation error. Added it back in and, along with the UA change it appears to consistently succeed now. I know that allowing unsafe headers is not ideal, but since we're dealing with servers that have cache control and abuse features in place, it is likely the only way to bypass these techniques. I'm open to alternative fixes if anyone has suggestions.

Welcome, @bzowk! Many of us use Ketarin as our package downloader and push the data to our own app/device management system for installation. While you *could* put Ketarin on all your devices and keep the store local to each device, it's much easier to use Ketarin as the downloading tool and keep everything on the LAN or WAN via Ketarin, with your external package installer. You can script *from* Ketarin to SCCM with PowerShell and effectively distribute/install via SCCM the way it should be done. You have several ways to do this. You could just log things to an external file with Ketarin, then parse that log externally, or you can use the "after updating an application" global script to directly interact with the SCCM library. You could also push the data to a database or do, literally, anything else that can be done in batch, powershell, or c# to get the data from Ketarin to SCCM.

Nope, using a full-page content capture and parsing it with variable functions didn't work. post-update they get triggered again. I might be able to get around it by assigning the relevant variables in powershell instead of doing so in the native Ketarin way. The download URL requires the version number, which is parsed from the release notes page, which is parsed from the download page, and the version is also used in the local filename. Thoughts/recommendations?

I worked around much of this by parsing the VariableType for each variable. I don't know if it's a parsed or text variable since some apps are necessarily constructed differently. I parse the value as either TextualContent or CachedContent based on the VariableType. I then check to see if the value is equal to the variable name in braces, which would indicate that the variable didn't exist, and removes it if so. Then it checks for a curly brace in the variable contents and if present it performs a ReplaceAllInString so that stuff like file-time information is properly presented. Here's a sample code block: $sversion = switch( $App.Variables."version".VariableType ) { 'Textual' {$App.Variables."version".TextualContent;} default {$App.Variables."version".CachedContent;} } if($sversion -eq "{version}"){$sversion = "";} if($sversion -like '*{*'){$sversion = $($app.variables.ReplaceAllInString($sversion));} Unfortunately, while this resolves a couple problems and vastly improves performance since there's less re-parsing going on, it still fails when the ReplaceAllInString is called against some parsed variables. Here's the specific error case, for Drupal: "schangelogstub" is a RegEx parsed value from https://www.drupal.org/download for href="([^"]+)"[^>]*>[^'"]+notes "schangelog" is a textual variable with value https://www.drupal.org{schangelogstub} Parsing for schangelog within PowerShell forces it to re-request https://www.drupal.org/download every single time ReplaceAllInString is called, even though the value *had* to be parsed (and assumingly cached) in order for the file to have been saved. I know that variables are not parsed unless and until they're called. But the information that they depend on should be pulled from the cache when possible, right? If so, I'd like to think a potential way around this might be to use a single variable that collected the entire contents of the page "/download" instead of using it as a URL source in multiple variables, then repeatedly parsing that variable with TextualVariables. The schangelog sample above suggests that this probably won't resolve the problem. Sigh. I'll try it when I get back tonight. I guess the real question it: What's the effective "life" of cached values?

Try this one:

I finally got around to using Ketarin to push update information to my server. It works for hundreds of my apps, though does feel to be significantly slower than Ketarin was before I added this. I've also noticed a couple that fail. The applicable line of PS code is: 'version'=$($app.variables.ReplaceAllInString('{version}')); The error is: Drupal Exception calling "ReplaceAllInString" with "1" argument(s): "The remote server returned an error: (403) Forbidden." Exception calling "ReplaceAllInString" with "1" argument(s): "The remote server returned an error: (403) Forbidden." Exception calling "ReplaceAllInString" with "1" argument(s): "The remote server returned an error: (403) Forbidden." Exception calling "ReplaceAllInString" with "1" argument(s): "The remote server returned an error: (403) Forbidden." Since Drupal now uses IP-based brute-force prevention, I suspect that it's actually an error related to an attempt by Ketarin to re-parse {version} based on the two server requests it has to make to collect it. If that is the case, it explains why Ketarin has become so much slower since I began this integration - it would be making additional server requests with every single App and that would significantly increase timing. Is there a way in powershell at post-update to only collect the cached content of those variables without re-triggering? This is in post-update so it should have current data cached for all of them.

Hi, everyone! I spent the day yesterday trying to get around the ugly SalesForce remoting functionality on the Logitech downloads page. I ended up writing a console script that'll parse the data every hour and store it on one of my servers. If you're interested in Logitech downloads (en_us) you can parse the JSON output of the export here. Please don't abuse it, but use it to gather version numbers and direct download links for various Logitech applications.

There are several examples on the forum of downloading with post authentication or via authentication proxy. I'm not seeing the behavior you describe where, after the first cycle of collecting the old version, it would require 2 or more cycles to correctly assign values.

"-spe" is a 7-zip switch that means "ignore root folder" or, more specifically "eliminate duplication of root folder for extract command." More about "-spe" here. Note the case MUST BE EQUIVALENT. "post commands" only run if the download actually happened. you may need to "force download" to cause them to run.

you should also consider running this reg: REG ADD "HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319" /v "SchUseStrongCrypto" /t REG_DWORD /d 1 /f REG ADD "HKLM\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319" /v "SchUseStrongCrypto" /t REG_DWORD /d 1 /f

version is a constructed variable for applications based on FileHippo and a manual variable for other applications. the same methods work to identify the available variables.

Are you saving the file with the original file name? Yo could save them with just the version (since they're all win32 you don't need "-hash-windows-x86"). Saving with just the version number would allow you to skip this issue. If you'd rather do it post-extract then you can use powershell with something like this: Get-ChildItem $Path | Where{$_.Name -Match "<RegEx Pattern>"} | Remove-Item This would look something like this in practice: Get-ChildItem "k:\_ketarin\builds\pcsx2" | Where{$_.Name -Match "pcsx2-v[\d\.]+-dev-\d+-[a-z0-9]+-windows-x86"} | Remove-Item You could also use something much simpler if you want to trim it down a bit: Get-ChildItem "k:\_ketarin\builds\pcsx2" | Where{$_.Name -Like "pcsx2-v*"} | Remove-Item Or if you want to delete all of them but exclude the current build you could use: Get-ChildItem "k:\_ketarin\builds\pcsx2" | Where{$_.Name -Like "pcsx2-v*"} | Where{$_.Name -NotLike "pcsx2-v{version}*"} | Remove-Item

Another option where root folders are using version numbers (as is the case with DDU) in the root folder is to rename the destination folder before unzipping and then rename it back to the generic version when finished. move "\Ketarin\Drivers\DDU" "DDU v{version}" 7z -spe ... move "\Ketarin\Drivers\DDU v{version}" "DDU"

IF it was distributed with a "root folder" then you might be able to use the "-spe" feature of 7zip to ignore it. Give it a try and see if that works. Sometimes renaming the downloaded file to what would be the root folder will allow -spe to work even in situations where it otherwise fails. More about that issue here.