Jump to content
Ketarin forum
shawn

Strange behavior with $app.variables.ReplaceAllInString

Recommended Posts

I finally got around to using Ketarin to push update information to my server. It works for hundreds of my apps, though does feel to be significantly slower than Ketarin was before I added this.

I've also noticed a couple that fail.

The applicable line of PS code is:

'version'=$($app.variables.ReplaceAllInString('{version}'));

The error is:

Drupal	Exception calling "ReplaceAllInString" with "1" argument(s): "The remote server returned an error: (403) Forbidden."
Exception calling "ReplaceAllInString" with "1" argument(s): "The remote server returned an error: (403) Forbidden."
Exception calling "ReplaceAllInString" with "1" argument(s): "The remote server returned an error: (403) Forbidden."
Exception calling "ReplaceAllInString" with "1" argument(s): "The remote server returned an error: (403) Forbidden."

Since Drupal now uses IP-based brute-force prevention, I suspect that it's actually an error related to an attempt by Ketarin to re-parse {version} based on the two server requests it has to make to collect it. If that is the case, it explains why Ketarin has become so much slower since I began this integration - it would be making additional server requests with every single App and that would significantly increase timing. 

Is there a way in powershell at post-update to only collect the cached content of those variables without re-triggering? This is in post-update so it should have current data cached for all of them.

Share this post


Link to post
Share on other sites

I worked around much of this by parsing the VariableType for each variable. I don't know if it's a parsed or text variable since some apps are necessarily constructed differently. I parse the value as either TextualContent or CachedContent based on the VariableType. I then check to see if the value is equal to the variable name in braces, which would indicate that the variable didn't exist, and removes it if so. Then it checks for a curly brace in the variable contents and if present it performs a ReplaceAllInString so that stuff like file-time information is properly presented.

Here's a sample code block:

$sversion = switch( $App.Variables."version".VariableType ) {
		'Textual' {$App.Variables."version".TextualContent;}
		default {$App.Variables."version".CachedContent;}
	}
	if($sversion -eq "{version}"){$sversion = "";}
	if($sversion -like '*{*'){$sversion = $($app.variables.ReplaceAllInString($sversion));}

Unfortunately, while this resolves a couple problems and vastly improves performance since there's less re-parsing going on, it still fails when the ReplaceAllInString is called against some parsed variables. Here's the specific error case, for Drupal:

"schangelogstub" is a RegEx parsed value from https://www.drupal.org/download for href="([^"]+)"[^>]*>[^'"]+notes

"schangelog" is a textual variable with value https://www.drupal.org{schangelogstub} 

Parsing for schangelog within PowerShell forces it to re-request https://www.drupal.org/download every single time ReplaceAllInString is called, even though the value *had* to be parsed (and assumingly cached) in order for the file to have been saved.

I know that variables are not parsed unless and until they're called. But the information that they depend on should be pulled from the cache when possible, right?

If so, I'd like to think a potential way around this might be to use a single variable that collected the entire contents of the page "/download" instead of using it as a URL source in multiple variables, then repeatedly parsing that variable with TextualVariables. The schangelog sample above suggests that this probably won't resolve the problem. Sigh. I'll try it when I get back tonight.

I guess the real question it: What's the effective "life" of cached values?

Share this post


Link to post
Share on other sites

Nope, using a full-page content capture and parsing it with variable functions didn't work. post-update they get triggered again. :(

I might be able to get around it by assigning the relevant variables in powershell instead of doing so in the native Ketarin way. The download URL requires the version number, which is parsed from the release notes page, which is parsed from the download page, and the version is also used in the local filename.

Thoughts/recommendations?

Share this post


Link to post
Share on other sites

Seeing a different issue now. Those apps that use FileHippo don't export the {version} variable correctly since I added the manual parsing method above. I suspect since it isn't a variable in the standard sense, it'll need to be addressed differently. I'm going to add a FH wrapper to ensure that the variables are properly fleshed out in the case of FH apps.

Share this post


Link to post
Share on other sites

Feature request: this is all pretty frustrating dealing with content access for different variable types and mechanisms. What would be ideal is something along the lines of ReplaceAllInString that did *not* re-parse/re-download URLs that were already parsed before or 'recently' cached. Parsing 'new' URLs/content/patterns/functions would, of course, be necessary when there was no cache hit, but ideally I should be able to use the following on Drupal without triggering two new URL requests each time:

$app.variables.ReplaceAllInString("{version}")

Ideally some form of temporary cache should be implemented anyway, as it could greatly improve performance and reliability. Perhaps add an option under Advanced to either ignore the cache completely or force requests again after x seconds. This would allow multiple requests to the same server to be bypassed in the case of multiple downloads (such as different architecture builds for the same app). 

Share this post


Link to post
Share on other sites

I'll have a look at what can be done there. Sometimes it's not easy to find a balance between performance and "correctness" unfortunately ;)

Share this post


Link to post
Share on other sites

Hm, what about using a different overload for ReplaceAllInString()?

You can use

string ReplaceAllInString(string value, DateTime fileDate, string filename, bool onlyCachedContent, bool skipGlobalVariables = false)

Parameters:
value: Same as before
fileDate: Basis for the {f:XXX} variables. Use DateTime.MinValue if you don't want to determine the value yourself.
onlyCachedContent: Use "true" in your case.
skipGlobalVariables: Optional and does what it says, use to your liking.

Share this post


Link to post
Share on other sites

I've now tried using this and it does seem to almost always pull the data from the cache instead of re-requesting URLs. I had some problems with the date and filename information so ended up prepopulating those. Here's what I'm currently using:

$sfilename = $($App.CurrentLocation);
$sfiletime = [datetime]$($App.DownloadDate);

$sversion = $app.variables.ReplaceAllInString("{version}", $sfiletime, $sfilename, $true);

I would assume that $App.DownloadDate would have the file timestamp from the current download (ideally the date the file was originally published, but the current timestamp if that's not an option). If that's not the case, then is there another variable to pull this information directly? I switched it out with this and it seems to be working ok, any reason I should use something else?

$sfilename = $($App.CurrentLocation);
$sfiletime = [datetime]$( (Get-Item $sfilename).LastWriteTime );

Now... It looks like it might work, but I'm still having one problem. One of my constructed variables re-requests the URL every time regardless. Is there a reason for it? The variable is schangelog and is defined as https://www.drupal.org{schangelogstub}. schangelogstub is another variable that uses a regex capture from the site to get the actual current version number to identify the specific URL for the changelog. The "new" ReplaceAllInString is still re-requesting the URL to build schangelogstub, and returning:

Drupal	The remote server returned an error: (403) Forbidden. (https://www.drupal.org/8/download)

The file downloads. The rest of the variables are populated. but it is still reported as an error by Ketarin since the PS chokes on re-requesting this content. Darn Drupal.

Share this post


Link to post
Share on other sites
11 hours ago, shawn said:

I would assume that $App.DownloadDate would have the file timestamp from the current download

By default, Ketarin uses the LastWriteTime from the downloaded file.

"filenme" is used  for the {url:XXX} variables. If empty, the variables will not be replaced.

11 hours ago, shawn said:

One of my constructed variables re-requests the URL every time regardless.

Hard to tell without debugging the specific case. Maybe this happens with recrusively replaced variables or something like that.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.