Omniferum

Each of those snippets are from different applications, the start of them. the first one starts square brackets at top with CDATA and apparently doesn't give appropriate closing brackets until the end of the job The second one lists sourcetemplate twice another example <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="190a48eb-e1dd-4112-821d-f49a8d1e4b96"> <SourceTemplate><![CDATA[<?xml version="1.0" encoding="utf-8"?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <SourceTemplate><![CDATA[]]]]></SourceTemplate> <WebsiteUrl /> There just seems to be a random number of []][]][][] for each application and duplicates of source templates for some apps. Another example, this is the top and bottom of the same app. <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="bb5072e2-c81e-4820-9f65-36a5eeb8e71c"> <SourceTemplate><![CDATA[<?xml version="1.0" encoding="utf-8"?> </ApplicationJob> </Jobs>]]></SourceTemplate> A 'healthy' looking xml for an app looks like this to me. <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="7a4a32cf-2d8e-43a4-accb-b3c834b487f1"> <SourceTemplate><![CDATA[]]></SourceTemplate>

I think i've identified what stuffed up my Ketamon .db before. After checking some of my .xml backups (I keep a regular backup of both the .db and .xml) and for no reason I can see I have inconsistent formatting. The only thing i've done is just run ketarin with the backup to xml command, haven't edited any of it. All my apps are functioning perfectly fine whilst running Ketarin as far as I know right now. I'm just posting bits, it seems to be square brackets being placed inconsistently. Whilst it may not really matter where the square brackets are if they pop up in random numbers in quasi-random locations in the xml it may cause an issue. Examples <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="f2662531-f64d-441a-921d-2c76258b0fc1"> <SourceTemplate><![CDATA[<?xml version="1.0" encoding="utf-8"?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <SourceTemplate><![CDATA[]]]]><![CDATA[></SourceTemplate> <WebsiteUrl><placeholder name="URL" value="http://www.psp-hacks.com/" /></WebsiteUrl> <LastFileDate xsi:nil="true" /> <IgnoreFileInformation>true</IgnoreFileInformation> <DownloadBeta>Default</DownloadBeta> <DownloadDate xsi:nil="true" /> <CheckForUpdatesOnly>false</CheckForUpdatesOnly> <VariableChangeIndicator>monitor</VariableChangeIndicator> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <ExclusiveDownload>false</ExclusiveDownload> <HttpReferer /> <SetupInstructions /> <Variables> <item> <key> <string>monitor</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>.*</Regex> <Url><placeholder name="URL" value="http://www.psp-hacks.com/" /></Url> <Name>monitor</Name> </UrlVariable> </value> </item> <item> <key> <string>name</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex><title>.*?(\S.*?)\s*</title></Regex> <Url><placeholder name="URL" value="http://www.psp-hacks.com/" /></Url> <Name>name</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand /> <ExecutePreCommand>start {property:WebsiteUrl} & exit 2</ExecutePreCommand> <ExecuteCommandType>Batch</ExecuteCommandType> <ExecutePreCommandType>Batch</ExecutePreCommandType> <Category>Web Page Monitoring</Category> <SourceType>FixedUrl</SourceType> <PreviousLocation /> <DeletePreviousFile>true</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <LastUpdated xsi:nil="true" /> <TargetPath>Web Page Monitoring\{appname:regexreplace:([^a-zA-Z0-9\_]+):-}</TargetPath> <FixedDownloadUrl>https://www.google.com/images/logos/mail_logo.png</FixedDownloadUrl> <Name><placeholder name="Name of application" variable="name" value="PSP Hacks · PSP Go Hacks · PSP Homebrew, PSP Downloads, PSP Custom Firmware" /></Name> </ApplicationJob> </Jobs>]]> <SourceTemplate><![CDATA[<?xml version="1.0" encoding="utf-8"?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <SourceTemplate><![CDATA[]]]]></SourceTemplate> <SourceTemplate><![CDATA[<?xml version="1.0" encoding="utf-8"?> This may be what led to my apps becoming bogged down. My question then is there something wrong with my templates that could cause this odd formatting in the exported .xml?

Ketarin uses a singeline flavor of regex, which means it treats . as matching line breaks and the such. The entire text as a single line of text rather than how it is formatted as we see it, hence right to left would serve the same purpose. At least that is how it works in theory, i've found better results with the .*() myself.

Copied and pasted this from the error screen ESET NOD32 Antivirus The specified target path 'D:\Software\Automatically Updated\{property:Category}\eav_nt64_enu.msi' is not valid.

There is a checkbox next to the regex field called RTL. Which means RightToLeft, it will achieve what you want Re: getting the newest. Alternatively put this at the beginning of your regex .*( and this at the end of your regex ) Max concurrent download from a particular server? I'm not sure what server you would need to limit, the big ones are what most people use and they allow a large enough number. Actually this would be accomplished if my feature request was implemented of being able to give your own numerical priority to apps. So if some shared the number 1, all the number 1's would be downloaded/installed first. Then after the numbers are done, whatever sorting is left would be done in that fashion.

Fair enough shawn, I remember being a roving network engineer so I can understand that necessity. I'm merely saying for general Ketarin use it does the job rather well on its own, specifying version variables for me has only been needed for the one app that keep downloading itself because Ketarin couldn't determine it correctly. I just see a lot of people new to the forums saying they've specified it, when it isn't specifically necessary.

Surely though we don't need to rely on an external website to enable this download though, can't Ketarin do this on its own anyway?

If you'd prefer this one extracts the path from the 'get the latest' and travels there and there you can further specify the file you want. I've just made it so the first exe zip rar 7z msi is found. You can edit it easily. You don't need to manually specify your mirror. As for version definitions, i'm not sure why people are so gung ho on setting this variable. I've only ever had to do it for one app because otherwise it just kept downloading. Ketarin has done a perfect job so far of determining by itself whether or not the download is new. With my template you can add the URL from any page that at least has "Looking for the latest version?" which is practically all of them and automatically names the application. Even the front page has it, just doesn't say "looking for latest version?" <?xml version='1.0' encoding='utf-8'?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <WebsiteUrl /> <UserNotes /> <IgnoreFileInformation>false</IgnoreFileInformation> <DownloadBeta>Default</DownloadBeta> <CheckForUpdatesOnly>false</CheckForUpdatesOnly> <VariableChangeIndicator /> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <ExclusiveDownload>false</ExclusiveDownload> <UserAgent>(!)</UserAgent> <HttpReferer></HttpReferer> <Variables> <item> <key> <string>download</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>([^"']+projects[^"']+files[^"']+(\.exe|\.msi|\.zip|\.rar|\.7z))</Regex> <Url>{refer}</Url> <Name>download</Name> </UrlVariable> </value> </item> <item> <key> <string>refer</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>([^"']+files[^"']+/)[^"']+\.\w\w\w</Regex> <Url><placeholder name="Application Page URL"/></Url> <Name>refer</Name> </UrlVariable> </value> </item> <item> <key> <string>name</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex><title>(.*?) </Regex> <Url><placeholder name="Application Page URL"/></Url> <Name>name</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand>/*This is a comment line, the line below is the C# script. It will run the instructions found in the Setup Tab*/ app.Install(null);</ExecuteCommand> <ExecutePreCommand /> <ExecuteCommandType>CS</ExecuteCommandType> <ExecutePreCommandType>Batch</ExecutePreCommandType> <SetupInstructions /> <Category><placeholder name="Category" /></Category> <SourceType>FixedUrl</SourceType> <DeletePreviousFile>true</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <TargetPath>{downloadpath}{category}\</TargetPath> <FixedDownloadUrl>{download}</FixedDownloadUrl> <Name><placeholder name="Name of application" variable="name" /></Name> </ApplicationJob> </Jobs>

In the end the guidelines should be the presented defaults, to help most newbies. As it stands afer making a .db of 150 apps the only real things that need to be put into place is a default variable for the download path and the installation path as they are the ones that i'm sure would be standard for everybody. All things downloaded/installed you'd want in a central location further varied by variables in that variable like {category}

Currently Ketarin does not allow variables like {category} and some others I can't recall, I have requested it a few times as have others. Just letting you know that at this moment I have to do {savepath} which is {root}Software\Automatically Updated\ and {category} so each app has a savepath of {savepath}{category}\ I'm not sure why you'd want to specify it as app version urlextension but c'est la vie.

http://ketarin.org/downloads/Ketarin/Ketarin-1.6.0.b3.zip is the latest

decided to remake my entire apps list and it works perfectly fine now for a few days where as before it would die halfway through the day. Just using the template I have. .db has not really increased in size either. So I assume there was a problem with some of the xml before, not sure what though as Ketarin happily imported it. I'll chalk this bug up to something odd I did with the old .db.

Not trying to get it to fit file extensions or specific builds, I just want it to be able to find the extension and pick the first http or " or ' it finds working its way back from \.exe. [^"'=]+[^"']+\.exe is pretty much the BEST i've come across

Hm, well that fixed a few. Some others don't actually start with = " ' and just http I keep trying to find a way to work back from the file extension. i.e. find \.exe then match the very first http as a preference, if not found then go through the " ' = group or the very last / without a space after/before it.

Hm, I just realized that perhaps I am a silly bunny. My original regex of [^"']+ worked fine, however all the problem URL's I have which have the http problem are after the = after the a href=" I think I remember omitting it because "header information may be in downloads" but forgot. So [^"'=]+64\.exe works fine, finally found my bloody answer.

I've been trying to further refine my regex [^\r]+((=|"|'|http).*\.exe) That's the furthest i've gotten, it works in the sense of 'preference' i.e. it iwll find " or ' or http first and then go all the way to the extension. The problem is though it doesn't leave out the " or ' and if I try adding a negated character class like [^"'] it stuffs the whole thing up. I want something that should work like this, but it doesn't annoyingly enough. [^\r]+(([^"=']|http).*\.exe) Any pointers? If I can get this down i'll have a regex that finds the download url for ALL my apps. Even tried working from the end, i'll admit MOST of my time finding this was not knowing that Ketarin was singleline regex and case insensitive so that sorta made a lot of time go bye bye. Then again I didn't even know that regex came in flavors so balls to me.

I do have some apps that collect 'illegal' characters from pages, asian characters. Example ビリビムI have some pages that I didn't include in the .db in mediafire. They were mainly japanese pages so perhaps this is the problem? When Ketarin detects a change and stores those characters it locks the .db?

Hm, i've been running mine as well but I have only had like 6 pages load. Some store more in the variable than others, again try 4chan or something high traffic. This is my latest template, with exit 2 in it. Assuming you want to, again i'm happy if it is just a case of maybe i'm capturing something in a variable that Ketarin doesn't like but I can't SEE anything. It would explain why YOU are ok and i'm not. Does anyone know if perhaps there is some characters that SQL databases don't like or something? Maybe i'm capturing a piece of text that is some sort of code in SQL? like class= or something along those lines, might be a long shot but i'm nothing but persistent as all buggery. <?xml version='1.0' encoding='utf-8'?> <Jobs> <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <SourceTemplate><![CDATA[]]></SourceTemplate> <LastFileDate xsi:nil="true" /> <IgnoreFileInformation>true</IgnoreFileInformation> <DownloadBeta>Default</DownloadBeta> <DownloadDate xsi:nil="true" /> <CheckForUpdatesOnly>false</CheckForUpdatesOnly> <VariableChangeIndicator>monitor</VariableChangeIndicator> <CanBeShared>true</CanBeShared> <ShareApplication>false</ShareApplication> <ExclusiveDownload>false</ExclusiveDownload> <HttpReferer /> <SetupInstructions /> <Variables> <item> <key> <string>monitor</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex>.*</Regex> <Url><placeholder name="URL" /></Url> <Name>monitor</Name> </UrlVariable> </value> </item> <item> <key> <string>name</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>RegularExpression</VariableType> <Regex><title>.*?(\S.*?)\s*</title></Regex> <Url><placeholder name="URL" /></Url> <Name>name</Name> </UrlVariable> </value> </item> <item> <key> <string>page</string> </key> <value> <UrlVariable> <RegexRightToLeft>false</RegexRightToLeft> <VariableType>Textual</VariableType> <Regex /> <Url /> <TextualContent><placeholder name="URL" /></TextualContent> <Name>page</Name> </UrlVariable> </value> </item> </Variables> <ExecuteCommand /> <ExecutePreCommand>start {page} & exit 2</ExecutePreCommand> <ExecuteCommandType>Batch</ExecuteCommandType> <ExecutePreCommandType>Batch</ExecutePreCommandType> <Category>Web Page Monitoring</Category> <SourceType>FixedUrl</SourceType> <PreviousLocation /> <DeletePreviousFile>true</DeletePreviousFile> <Enabled>true</Enabled> <FileHippoId /> <LastUpdated xsi:nil="true" /> <TargetPath>Web Page Monitoring\{appname:regexreplace:([^a-zA-Z0-9\_]+):-}</TargetPath> <FixedDownloadUrl>https://www.google.com/images/logos/mail_logo.png</FixedDownloadUrl> <Name><placeholder name="Name of application" variable="name" /></Name> </ApplicationJob> </Jobs>

Awesome, thanks shawn. Works like a charm

My .db only increases in size if there is an update to be found and the page/code is actually loaded, for my tests I pick a more high traffic site. Something with comments or a hit counter, like 4chan, slashdot comment sections or some other forums.

Got another one that's bugging me, i'm just trying to get the appname to be returned without any illegal windows characters {appname:multireplace:,:/,?,<,>,\,/:,*,”:} Just make Ketarin crash, but I assume i'm close enough to being correct. Can anyone give me that final thing i'm obviously missing?

I assume mine can get large because of my monitoring variable for changes for some sites I have to capture a fair bit of the page (should probably refine the regex) but if all my pages are 'new' it can increase by 200kb. Regardless i've gotten further now than I had before, I just assumed that Ketarin would 'flush' old data but seeing as my .db increases as such that is probably not the correct assumption.

Oh, the header thing is something floele implemented because of one of my downloads. httpx://www.pocketappreview.com/main/media/software_zip/EjectUSB.zip?header:accept=*[&x=y]+ So starting the url with httpx and the ?header:accept=(Iassumetheheaderdetailsgohere) That's what works for my ejectusb, well not anymore because the site itself has gone 'down'.

Don't suppose you ever got the codeplex to work shawn? Would just like to know how you did it in the end, for my own personal education.

Just thought i'd share this piece because I still see the odd post Re: getting sourceforge to work. All sourceforge needs is something in the User Agent field, it doesn't actually matter what it is really. I've had "Poo" and it downloads fine.