shawn

Are you saying that when you modify the "save as default" template, it's retroactively applying to *all* previously created apps? Are you using any other Ketarin Template functionality, or just the "default" button?

Amen to that. I hope Ketarin saves you all some time so you can spend it with family.

I've created and posted an effective template here. It was tested with a full dozen Codeplex apps, including all the ones mentioned in this thread.

Updated 2012-10-28: fixes several bugs, but still doesn't work due to a filetime parsing issue. Any ideas? Updated 2011-08-25: fixed another version parsing bug. Updated 2010-12-28: fixed version number check for a number of codeplex apps. This still has problem with a few apps (bluetooth radar, iExporter, ilyrics, iringtones, mdownloader, pst view tool, smtp4dev - possibly others). Here's a working Codeplex template with variables for the app name, category, slug, title and version preservation. Based on discussion from this thread. The template below was tested successfully with a full dozen apps on Codeplex without any problems. Enjoy! <?xml version='1.0' encoding='utf-8'?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="debb1deb-1024-1975-abcd-000000000004"> <Name><placeholder name="App name?" value="" /></Name> <Category><placeholder name="Application Category?" value="Unassigned" options="Unassigned|Browsers|Capture|Codecs|Converters|Development|Drivers|Editors|Education|Email|IM|Internet|Media|Plugins|Security|Tools|Viewers|VM|Web" /></Category> <WebsiteUrl /> <UserAgent>Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C)</UserAgent> <UserNotes /> <IgnoreFileInformation>false</IgnoreFileInformation> <DownloadBeta>Default</DownloadBeta> <DownloadDate xsi:nil="true" /> <CheckForUpdatesOnly>false</CheckForUpdatesOnly> <VariableChangeIndicator /> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <ExclusiveDownload>false</ExclusiveDownload> <HttpReferer>{sdownload}</HttpReferer> <SetupInstructions /> <Variables> <item> <key> <string>slug</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent><placeholder name="Codeplex application slug:" value="" /></TextualContent> <Name>slug</Name> </UrlVariable> </value> </item> <item> <key> <string>title</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent><placeholder name="Codeplex application title:" value="" /></TextualContent> <Name>title</Name> </UrlVariable> </value> </item> <item> <key> <string>schangelog</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>http://{slug}.codeplex.com/releases/view/{releaseId}</TextualContent> <Name>schangelog</Name> </UrlVariable> </value> </item> <item> <key> <string>swebsite</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>http://{slug}.codeplex.com/</TextualContent> <Name>swebsite</Name> </UrlVariable> </value> </item> <item> <key> <string>snotes</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent /> <Name>snotes</Name> </UrlVariable> </value> </item> <item> <key> <string>releaseId</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>href="http://{slug}.codeplex.com/releases/view/(\d+)"</Regex> <Url>http://{slug}.codeplex.com/releases/</Url> <Name>releaseId</Name> </UrlVariable> </value> </item> <item> <key> <string>fileId</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>fileId="([\d]+)" [^<>]+></Regex> <Url>http://{slug}.codeplex.com/releases/view/{releaseId}</Url> <Name>fileId</Name> </UrlVariable> </value> </item> <item> <key> <string>buildId</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>FileListItem.+?/v(\d+)/.+?".+?{fileId}</Regex> <Url>http://{slug}.codeplex.com/releases/view/{releaseId}</Url> <Name>buildId</Name> </UrlVariable> </value> </item> <item> <key> <string>filetime</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>{fileId}".+?smartDate dateOnly" title="([^"]+)"</Regex> <Url>http://{slug}.codeplex.com/releases/view/{releaseId}</Url> <Name>filetime</Name> </UrlVariable> </value> </item> <item> <key> <string>ft</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>\[(.+)\]</Regex> <Url>http://12pd.com/filetime.php?t={filetime}</Url> <Name>ft</Name> </UrlVariable> </value> </item> <item> <key> <string>version</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>(?:[\S\s][\.\sv])?([\d.]+?)(?:\s-\s[^\d]+?)?(?:\s\(.+?\))?[\r\n]*</title></Regex> <Url>http://{slug}.codeplex.com/releases/view/{releaseId}</Url> <Name>version</Name> </UrlVariable> </value> </item> <item> <key> <string>sdownload</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>http://download.codeplex.com/Download/Release?ProjectName={slug}&DownloadId={fileId}&FileTime={ft}9900000&Build={buildId}</TextualContent> <Name>sdownload</Name> </UrlVariable> </value> </item> <item> <key> <string>spost</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>http://convert-get-proxy.appspot.com/server/?method=POST&url={sdownload:multireplace:,:&,?:%26,%3F}</TextualContent> <Name>spost</Name> </UrlVariable> </value> </item> <item> <key> <string>spc</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>codeplex</TextualContent> <Name>spc</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand /> <ExecutePreCommand /> <ExecuteCommandType>Batch</ExecuteCommandType> <ExecutePreCommandType>Batch</ExecutePreCommandType> <SourceType>FixedUrl</SourceType> <DeletePreviousFile><placeholder name="Delete Previous Version?" value="true" options="true|false" /></DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <TargetPath>.\{category}\{appname:regexreplace:([\s\t\r\n\-\\&\/]+):_}-{version}.{url:ext}</TargetPath> <FixedDownloadUrl>{sdownload}</FixedDownloadUrl> </ApplicationJob> </Jobs>

You're going to be very hard pressed to find one pattern that works in every situation. Generally, you can find one that'll work across most apps on a site, but even then you'll often need to include a filepart inside (like "x86" or "x64"), so there's no silver bullet.

I think what you're after is: [='"]([^'"\s]*\.exe)[\s'"]

I'm so excited. I can't wait to start collecting stuff from Codeplex.

Well crap. I just tried again with this, and I can't get the httpx protocol handler to work. So, I looked for an online alternative. I found a webservice on AppSpot that will convert HTTP methods thru a server redirect. It works perfectly in this instance. I've copied a working app profile for WiiBaFu below. It's designed in a way that should make it really easy to create a template from it (change only "slug" and "title"). If I were actually conscious right now, I'd do that myself...but I'm really not - it just looks like it. <?xml version='1.0' encoding='utf-8'?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="ae4ad0e5-22c6-4ce7-96cb-185dcca2c255"> <SourceTemplate><![CDATA[]]></SourceTemplate> <WebsiteUrl /> <UserAgent>Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C)</UserAgent> <UserNotes /> <IgnoreFileInformation>false</IgnoreFileInformation> <DownloadBeta>Default</DownloadBeta> <DownloadDate xsi:nil="true" /> <CheckForUpdatesOnly>false</CheckForUpdatesOnly> <VariableChangeIndicator /> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <ExclusiveDownload>false</ExclusiveDownload> <HttpReferer>{sdownload}</HttpReferer> <SetupInstructions /> <Variables> <item> <key> <string>slug</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>wiibafu</TextualContent> <Name>slug</Name> </UrlVariable> </value> </item> <item> <key> <string>title</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>Wii Backup Fusion</TextualContent> <Name>title</Name> </UrlVariable> </value> </item> <item> <key> <string>schangelog</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent /> <Name>schangelog</Name> </UrlVariable> </value> </item> <item> <key> <string>swebsite</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>http://wiibafu.codeplex.com/</TextualContent> <Name>swebsite</Name> </UrlVariable> </value> </item> <item> <key> <string>snotes</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent /> <Name>snotes</Name> </UrlVariable> </value> </item> <item> <key> <string>releaseId</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>href="http://{slug}.codeplex.com/releases/view/(\d+)"</Regex> <Url>http://{slug}.codeplex.com/releases/</Url> <Name>releaseId</Name> </UrlVariable> </value> </item> <item> <key> <string>fileId</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex><a class="FileNameLink" [^<>]+ fileId="([\d]+)" [^<>]+>[^<>]+Win</Regex> <Url>http://{slug}.codeplex.com/releases/view/{releaseId}</Url> <Name>fileId</Name> </UrlVariable> </value> </item> <item> <key> <string>version</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>{slug} - Release\: {title} ([^\r\n]+)</Regex> <Url>http://{slug}.codeplex.com/releases/view/{releaseId}</Url> <Name>version</Name> </UrlVariable> </value> </item> <item> <key> <string>sdownload</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>http://{slug}.codeplex.com/releases/acceptLicense?fileId={fileId}&releaseId={releaseId}&clickOncePath=</TextualContent> <Name>sdownload</Name> </UrlVariable> </value> </item> <item> <key> <string>spost</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <TextualContent>http://convert-get-proxy.appspot.com/server/?url={sdownload:multireplace:,:&,?:%26,%3F}&method=POST</TextualContent> <Name>spost</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand /> <ExecutePreCommand /> <ExecuteCommandType>Batch</ExecuteCommandType> <ExecutePreCommandType>Batch</ExecutePreCommandType> <Category>Tools</Category> <SourceType>FixedUrl</SourceType> <DeletePreviousFile>true</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <TargetPath>.\{category}\{appname:regexreplace:([\s\t\r\n\-\\&\/]+):_}-{version}.{url:ext}</TargetPath> <FixedDownloadUrl>{spost}</FixedDownloadUrl> <Name>Wii Backup Fusion</Name> </ApplicationJob> </Jobs>

That's possible. It was only recently that URLs could contain high-bit/UTF-8 characters, and perhaps Ketarin isn't prepared for that. Alternatively, maybe it has something to do with how you're pushing the URLs out. Windows has issues with non-ascii characters in shell commands, so it's possible that simply passing the URL to the browser is causing the problem.

And Omni - I have to say this is a pretty cool implementation of page monitoring.

The Ketarin page popped on me about a dozen times during my tests. As did several of the comic pages.

This should work for you: {appname:multireplace:,\:/,?,<,>,\,/:,*,”:} However, as a rule, whitelisting characters is far safer than blacklisting. Thus, this should be more reliable: {appname:regexreplace:([^a-zA-Z0-9\_]+):-} That'll replace any non-alphanumeric or underscore sequences with a single dash. So something like "Title | my trip to mars: fun times!" will become "Title-my-trip-to-mars-fun-times-"

I assigned the "after updating all applications" to my snapshot.bat, included below. @ECHO OFF :variables REM !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! FOR /F "tokens=2-4 delims=/ " %%a in ("%date%") DO SET MM=%%a&SET DD=%%b&SET YY=%%c FOR /F "tokens=1-4 delims=:. " %%a in ("%time%") DO SET HH=%%a&SET NN=%%b&SET SS=%%c&SET LL=%%d FOR /L %%a in (0,1,9) DO @IF "%HH%"=="%%a" SET HH=0%%a SET timestamp=%YY%%MM%%DD%T%HH%%NN%%SS% REM Export current settings ketarin /export=ketarin-%timestamp%.xml Then I ran "Update All" manually over 50 times, followed by the following: FOR /L %i IN (1,1,25) DO ketarin /silent /log=ketarin-%i.log Thus, it's run about 85 times now (with "exit 2" still in place), and the jobs.db is still exactly the same size. The first and last export files are exactly the same. The first and last "log" files are different, but reasonably consistent (the significant difference appears to be that since it's doing many update checks simultaneously, the order and progress of each will be inconsistent - though the results are much the same). Bottom line: I can't replicate your problems. Sorry.

@flo - Can you elaborate on this? @omniferum - Nope. It uses a POST request type, and I was unable to get Ketarin to use a POST as a download URL. Any thoughts welcome.

Which build of Ketarin are you using?

Sadly, this is the exact type of person that www.ninite.com is designed for.

My jobs.db has been over 25mb in the past. Export & reimport shrunk it significantly - but it was working fine at that size anyway. Have you considered creating versioned XML exports between each iteration, then comparing them with something like WinMerge to see if there's a clue why it's failing? Would you mind sharing a zip of your Ketamon jobs.db so I can look into it? It's possibly related to what you're parsing, or certain characters or something, and fresh eyes can't hurt. Which build of Ketarin are you using?

DB corruption is possible under a couple scenarios: drive or ram failure are the most obvious. It's also possible that a surge or other physical defect could occur simultaneous to db writing, which could cause problems. Those are each unlikely, but not impossible. It's also possible that it's a logic failure somewhere either in Ketarin or within the mon profile. Since you're using Slashdot for testing, this could be your problem. Slashdot implements view throttles to minimize unnecessary server load. If you refresh a certain page too frequently, or if you have a feed set to recycle more often than once every couple hours, you will not receive a 200 status. Ketarin may be treating the non-200 as errant, which is why you're getting odd behavior.

Probably something like the FileZilla conflict handling: what to do if filestamp or size are different, with options to replace, replace always, ignore, ignore always for a combination of those conflicts.

I just spent a couple hours trying out sources on Codeplex. If I POST to the parsed download URL, I can access the download just fine (in fact, if I assign a variable in Ketarin with the POST data, it will actually return the binary)...but Ketarin doesn't offer a POST option for the actual download. I don't see a way around this until it does. Perhaps an option like the URL-parameterized header handling? I'm almost positive this should work, but just can't get my head in the game recently. httpx://{slug}.codeplex.com/releases/acceptLicense?header:method=POST&fileId={fileId}&releaseId={releaseId}&clickOncePath= It replaces {slug}, {fileId} and {releaseId} of course.

The only thing I see that might pose a problem (without actually trying it) is that you're including a Guid, placeholders and the SourceTemplate section. Try it again after removing the SourceTemplate section.

Sometimes the version number changes dramatically. Sometimes the result you're getting is simply not invalid. The best thing to do is to use a regex pattern match to obtain the most accurate version number, and then assign the change indicator to your version variable. Something like ">([\d\.]+)<" captures most common version numbers.

I find that Robocopy is far more reliable and stable than xcopy, and is available on all current Windows operating systems.

That's exactly my concern. Is there any way that the environment variables are set during processing where PATH might be overwritten?

"path" is a reserved variable. Try it again using a different custom variable name and you will have better luck. Since you're overwriting it, you're preventing Windows from knowing where to find xcopy and other applications that it's trying to access.