Ketarin 1.5 beta 7

[floele]

Hi again,

 

I just uploaded a new beta:

 

http://ketarin.org/downloads/Ketarin/Ketarin-1.5.0.b1.zip

 

Changes:

 

+Extended status bar information

 

+Added command line parameter /install:xmlfile for automated updating and setup

I believe this might be useful to create some kind of worry free setup mechanism. Run an XML file of your choice and let Ketarin do the work. This might need some adjustments, but I should serve a starting point for your ideas

 

+Prototype templates

Requirements: Template needs a GUID (not a {000} GUID) and it has to contain placeholders (of course). Then just import the modified template again, and it should ask if any applications that have been created from this template are to be updated.

GUID generator: http://www.guidgenerator.com/online-guid-generator.aspx

 

!Fixed: Regex match is not refreshed

 

Have fun

 

 

Beta 2

 

I only fixed the incorrect requirement of GUIDs for templates with this version.

 

http://ketarin.org/downloads/Ketarin/Ketarin-1.5.0.b2.zip

 

 

Beta 3

 

!Fixed another GUID error message

*Do not update applications on setup by default (only force update if file does not exist)

*New global variables editor

 

http://ketarin.org/downloads/Ketarin/Ketarin-1.5.0.b3.zip

 

 

Beta 4

 

!Fixed: property variable does no longer work for text fields

*Placeholder dialog is now only shown once for all placeholders

*"Update and install" option

 

http://ketarin.org/downloads/Ketarin/Ketarin-1.5.0.b4.zip

 

 

Beta 5

 

Fixed a bug with placeholders and XML import.

 

http://ketarin.org/downloads/Ketarin/Ketarin-1.5.0.b5.zip

 

 

Beta 6

 

+New attribute "variable" for placeholders

If set, Ketarin will try to determine its value based on the specified variable (name, without braces). If that fails, the user needs to enter them manually.

*"Update all and install" changed to "Update all and install updates" and adjusted the behaviour accordingly

 

http://ketarin.org/downloads/Ketarin/Ketarin-1.5.0.b6.zip

 

 

Beta 7

 

*Includes experimental sourceforge capabilities

*"Start process" instruction now more useful with environment variables and browse button (for executables)

*Store source template as CDATA

!Fixed: "Check for updates only" apps cannot be installed

 

http://ketarin.org/downloads/Ketarin/Ketarin-1.5.0.b7.zip

[shawn]

Status bar change is perfect.

 

The dropdown for "use the following variable as indicator for changes" should always include "version", even if it's not a defined variable, since this is the behavior of certain native functionality (such as filehippo and pad parsing).

 

How does the prototype template functionality work? I understand that it needs a GUID - easy enough (in DQSD it's "guidgen"). I guess what I need to know is, is the logic to handle proto-templates essentially "if the xml file has a 'placeholder' AND a GUID, it's a prototemplate - but if it only has only one or the other, it's a normal xml app profile." This is important because I currently use exported app profiles to copy and **update** the settings from one system to another. If the GUID is non-unique, or generated randomly during import instead of preserved for some reason, then it'll definitely raise issues.

[shawn]

Hm. The behavior change for templates has broken all my previous templates. Now it requires a GUID for all templates, even if the template is not intended to be a prototype (though I can respect how that really should be the default behavior). (also note: GUID must be in lowercase!)

 

I found a more reliable method of getting the current QuickTime Player (filehippo has been broken on this for several weeks), and wanted to share it --- but it's not working the way I expect it to. The online database (even after changing the GUID a half dozen times, deleting and reimportant and stuff) still tells me I'm attempting to upload "QuickTime Alternative" and asks me to rename it.

 

Also, when I do create a new app based on one of my modified templates (Guid added to sfnet template), the resulting search doesn't work, complaining of a 404 when accessing "hxxp://sourceforge.net/projects/{sftag}/files/{filepath}" (which should resolve to "hxxp://sourceforge.net/projects/dqsd/files/"). The URL works, and the SF referer is in place. Any thoughts? My guess is that there is a behavior change in the new version that no longer correctly returns an empty string within "{filepath}" since removing that portion of the URL box fixes it. Or maybe the variable replacement function for this field no longer iterates until fully resolved?

[josh]

+Prototype templates

Requirements: Template needs a GUID (not a {000} GUID) and it has to contain placeholders (of course). Then just import the modified template again, and it should ask if any applications that have been created from this template are to be updated.

GUID generator: http://www.guidgenerator.com/online-guid-generator.aspx

 

- "It needs a GUID" -> Okay

 

- "It has to contain placeholders (of course)" -> The other day i lost 36 jobs in Softpedia due to their website update, considering each job had 4 placeholders, are you telling me i needed to redefine 144 values? No way, edit XML is more simple...

 

Hm. The behavior change for templates has broken all my previous templates. Now it requires a GUID for all templates, even if the template is not intended to be a prototype (though I can respect how that really should be the default behavior).

 

I confirm this.

Edited September 27, 2010 by josh

[floele]

The dropdown for "use the following variable as indicator for changes" should always include "version", even if it's not a defined variable, since this is the behavior of certain native functionality (such as filehippo and pad parsing).

 

For the FileHippo case, a variable as indicator for changes is not necessary. So I wouldn't bother to include a "possibly existing" (and thus possibly misleading) variable if it isn't even needed.

 

How does the prototype template functionality work?

 

If it only has a GUID, it is a usual application XML file and not a template. No difference to existing files.

If it only has placeholders (and no GUID) it is a template like it has been existing earlier. It can't be used for updates, since Ketarin can't possibly recognise the template without GUID. So when updating the template, the GUID needs remain unchanged.

 

Hm. The behavior change for templates has broken all my previous templates. Now it requires a GUID for all templates, even if the template is not intended to be a prototype

 

That's an error on my side, will fix that with the next version. (See above)

 

(even after changing the GUID a half dozen times, deleting and reimportant and stuff)

 

If you do that, uploading a new version to the database is not possible. You need to keep the original GUID if you want to refer to the same application.

 

My guess is that there is a behavior change in the new version that no longer correctly returns an empty string within "{filepath}"

 

I don't know how this variable is defined. There should not be any change in the behaviour of variables though.

 

are you telling me i needed to redefine 144 values?

 

That would kinda defeat its purpose. Instead, you are supposed to only edit the template, import it and...well, that's it. The previously entered values are saved.

[floele]

Problem fixed with the new beta btw.

[josh]

Even with 1.5b2 if i import the modified template (+GUID +2 new variables) Ketarin prompts me to insert new values to replace the existing one's (hence my question above) and all it does is to create a new job. Now if i import the modified template for the 2nd time, it Ketarin recognizes that new job (1 single job) and asks me if i wanna update according to your description.

 

EDIT1: Now i remember, i once changed 1 variable in my older template... so apparently this feature is working fine, except its of no evident use to me since all i have are jobs with no template.

 

EDIT2: Something is wrong, now all i get is error message saying GUID needs to have "32 characters" (36?) with 4 dashes... i have a valid XML, got GUID directly from generator above, tried several GUIDs, create another template from scratch, "reset" database with backup file... +errors...

Edited September 28, 2010 by josh

[Omniferum]

I may just be stupid here but is there not an update and then install button/function?

 

I basically moved all my code from the "run after download" to the "setup instructions" area and I update, it downloads but doesn't actually do anything annoyingly enough.

 

I can understand a need to differentiate both places but I don't see why you need "run command after download" if it is not install instructions/options. Seeing as they essentially do the same thing and the only command you would want to run would be downloaded file specific.

 

 

Further... feature inquiry stuff. Again this shit I suggest may be too obscure or overly technical to achieve.

 

It is possible for Ketarin to parse the url found to an external application? For stuff like mediafire/megaupload links (some developers use them to save on hosting) I would like to find them. Have ketarin go "Ok, this is the link i'll now send it to Jdownloader and be done with it".

 

I'll admit my template name parsing idea is perhaps easier to achieve. I don't want Ketarin to handle the links of course, just pass them on. Even if you had the option of dumping it to a text file and from there I can just batch script the line out.

 

Hm, perhaps I could parse it from the logfile somehow? Seems a tad clunky, hmmmmmmm

 

Oh, also. Anyone care to share why this doesn't download? It used to, now it just gives a 406 error. I somewhat think it is the useragent that Ketarin uses. As EVERYTHING else is fine, the link is fine etc etc.

 

http://www.pocketappreview.com/main/media/software_zip/EjectUSB.zip

[floele]

Even with 1.5b2 if i import the modified template (+GUID +2 new variables) Ketarin prompts me to insert new values to replace the existing one's (hence my question above) and all it does is to create a new job.

 

In order to make these templates work, you need to follow these steps. I thought that it would be somewhat obvious, but it seems not so:

 

1. Create a prototype template (template with GUID and with placeholders).

2. Create X applications from the template by importing the template.

3. Change the template in any way you like, just do not add additional placeholders.

4. Import the template again. Ketarin should ask you to update X applications. Say yes, and all imported applications will be changed according to the new template.

 

EDIT2: Something is wrong, now all i get is error message saying GUID needs to have "32 characters" (36?) with 4 dashes... i have a valid XML, got GUID directly from generator above, tried several GUIDs, create another template from scratch, "reset" database with backup file... +errors...

 

What steps do you follow to get that error?

 

I may just be stupid here but is there not an update and then install button/function?

 

Nope. Wanted? Currently, installing will actually update the applications.

 

I can understand a need to differentiate both places but I don't see why you need "run command after download" if it is not install instructions/options.

 

Renaming, copying, cleanups, whatever you can imagine. Would not call that setup.

 

So maybe have an instruction "{install}" which can be used in post download commands? Or rather add an option "Update and install"?

 

It is possible for Ketarin to parse the url found to an external application?

 

Entirely depends on the external applications. If the application supports it, Ketarin can certainly pass on the downloading process, but then there won't be any status, post download commands or anything because Ketarin cannot track the completion.

 

If you do some research on apps like JDownloer and find out the command line parameters, I can certainly look into implementing it.

 

Oh, also. Anyone care to share why this doesn't download? It used to, now it just gives a 406 error.

 

No idea. Even when using this site http://web-sniffer.net/ I always get a 406 :-?

[josh]

What steps do you follow to get that error?

 

I just picked one of my Microsoft jobs (slightly modified from shawn template), export as application template, add a GUID and a couple of placeholders, import back and got that message. XML:

 

<?xml version="1.0" encoding="utf-8"?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="66ffec5e-4b83-4e3a-ae8c-48e8b23f4649">
   <WebsiteUrl />
   <UserAgent />
   <UserNotes />
   <LastFileSize>6407992</LastFileSize>
   <LastFileDate>2010-09-12T19:08:53.8377402</LastFileDate>
   <IgnoreFileInformation>false</IgnoreFileInformation>
   <DownloadBeta>Default</DownloadBeta>

   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator />
   <CanBeShared>true</CanBeShared>
   <ShareApplication>false</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <HttpReferer />
   <SetupInstructions />
   <Variables>
     <item>
       <key>
         <string>mssearch</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <TextualContent><placeholder name="What is the search term on MS/Downloads?" value="" /></TextualContent>
           <Name>mssearch</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>mskey</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>FamilyID=([a-z\d\-]+)['"&]</Regex>
           <Url>http://www.microsoft.com/downloads/en/results.aspx?freetext=%22{mssearch}%22&displaylang=en&stype=ss_rr&nr=10&sortCriteria=Date&sortOrder=Descending</Url>
           <Name>mskey</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>version</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>Version:</div>\s+<div[^>]+>([^<>]+)</div></Regex>
           <Url>http://www.microsoft.com/downloads/en/details.aspx?displayLang=en&FamilyID={mskey}</Url>
           <Name>version</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>sdownload</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>href=["'](http://[^"']+_x86\.exe)["']</Regex>
           <Url>http://www.microsoft.com/downloads/en/details.aspx?displayLang=en&FamilyID={mskey}</Url>
           <TextualContent>onclick="openDownloadWindow\('([^'"]+download[^'"]+)['"]</TextualContent>
           <Name>sdownload</Name>
         </UrlVariable>
       </value>
     </item>
   </Variables>
   <ExecuteCommand />
   <ExecutePreCommand />
   <ExecuteCommandType>Batch</ExecuteCommandType>
   <ExecutePreCommandType>Batch</ExecutePreCommandType>
   <Category>Microsoft</Category>
   <SourceType>FixedUrl</SourceType>

   <DeletePreviousFile><placeholder name="Delete Previous Version?" value="true" options="false|true" /></DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />

   <TargetPath>{root}USB-Toolkit\{category}\</TargetPath>
   <FixedDownloadUrl>{sdownload}</FixedDownloadUrl>
   <Name><placeholder name="Application Name?" value="" /></Name>
 </ApplicationJob>
</Jobs>

Edited September 28, 2010 by josh

[Omniferum]

An Update then Install button/option would make the most sense, adding extra instructions to "After download" would require extra editing when the same thing can be achieved by a single extra option.

 

As for Jdownloader if the link can be copied to the clipboard it can take it from there, the CLI arguments can be a bit iffy at times. You have to get java to call the jdownloader.jar and then parse commands etc.

 

i.e.

 

java.exe -jar jdownloader.jar -a "put link here"

[shawn]

(even after changing the GUID a half dozen times, deleting and reimportant and stuff)

If you do that, uploading a new version to the database is not possible. You need to keep the original GUID if you want to refer to the same application.

 

Understood, but if the GUID is the determining factor (and it's a DIFFERENT application - the original was Quicktime Alternative, this one is Quicktime), it shouldn't matter, right?

 

My guess is that there is a behavior change in the new version that no longer correctly returns an empty string within "{filepath}"

I don't know how this variable is defined. There should not be any change in the behaviour of variables though.

 

I can't replicate the issue right now, but {filepath} was defined as a "textual content" variable, and was empty (the default). When the URL was created for the version check, {filepath} was not replaced with "".

 

3. Change the template in any way you like, just do not add additional placeholders.

 

Can you please change this so that we *can* add additional placeholders, then prompt us what to do with them - and then a prompt to "would you like to use the following default value for all missing {whatever} placeholders, or be prompted for each individual app?" This would greatly ease my mind in using the prototemplate functions.

 

So maybe have an instruction "{install}" which can be used in post download commands?

 

Yep. Works for me.

[shawn]

are you telling me i needed to redefine 144 values? ... all i have are jobs with no template.

 

True, but if you create a new app profile using the "old" template code, then export it, you'll be able to see the source is extremely simple (simply encoded text of the template) so migrating that section to all the rest of the similar templates shouldn't be too difficult. Back it up a few times first before you do, though, since an error could be Egon Spengler "bad".

 

 

error message saying GUID needs to have "32 characters" (36?)

 

I got the same thing during most of my tests - use lowercase and the attribute name "Guid" not "guid" or "GUID". That seems to work for me.

[shawn]

I don't see why you need "run command after download" if it is not install instructions/options

...

It is possible for Ketarin to parse the url found to an external application?

 

That's a perfect example of why you need the "run command after download" functionality.

 

Parse the URL from the download source (mediafire or whatever) into a variable. Download it, or not (and download something uber-tiny like an icon file) then use the following command in the post-download command:

java.exe -jar jdownloader.jar -a "{yourUrlVaraibleName}"

Nifty, eh?

 

Oh, also. Anyone care to share why this doesn't download (pocketappreview)

 

It's a 406 error (not acceptable), so it's probably a referer/cookie/user-agent issue. Set a known good user-agent and referer in the advanced tab, and try again. Doing so doesn't work for me, so it's most likely a required cookie is missing. Checking...yep. Visiting the site assigns a cookie with a timestamp:

lastVisit=1285751139

In order to download it, we'd need to be able to assign a cookie, or at least ensure js-based cookies are scripted correctly and values assigned during subsequent page loads. Setting a cookie to the current unix timestamp would probably work (it does in my Perl tests).

[shawn]

Oh, on that note - Flo - if you decide to add cookie handling abilities, please provide a new time/date function to return the unix timestamp.

 

And speaking of date functionality....

 

 

Feature request:

 

Would it be possible to add (somehow) the ability to access the file or URL timestamp for storing within a variable ({version}). In my specific scenario, I'm trying to assign a version to a file that doesn't really have a published version number anywhere (CrucialScanner.exe). The only unique data is that of the Last-Modified header returned in the request...and while I know that the {f:...} methods can be used for file naming, I don't see any equivalent functionality for variable assignments.

[shawn]

Bug report:

 

Somewhere during the recent alphas and betas, Batch scripts that included ">" within them had the greater than character and everything to the right of it removed. This was around 9/26, if that helps at all. I didn't make incremental backups of the db, so can't tell exactly which version did it.

[josh]

are you telling me i needed to redefine 144 values? ... all i have are jobs with no template.

 

True, but if you create a new app profile using the "old" template code, then export it, you'll be able to see the source is extremely simple (simply encoded text of the template) so migrating that section to all the rest of the similar templates shouldn't be too difficult. Back it up a few times first before you do, though, since an error could be Egon Spengler "bad".

As i was saying...

Even with 1.5b2 if i import the modified template (+GUID +2 new variables) Ketarin prompts me to insert new values to replace the existing one's (hence my question above) and all it does is to create a new job.

And also...

EDIT1: Now i remember, i once changed 1 variable in my older template... so apparently this feature is working fine

 

error message saying GUID needs to have "32 characters" (36?)

 

I got the same thing during most of my tests - use lowercase and the attribute name "Guid" not "guid" or "GUID". That seems to work for me.

 

Steps to reproduce my problem @ http://ketarin.canneverbe.com/forum/viewtopic.php?pid=4156#p4156

Guid="66ffec5e-4b83-4e3a-ae8c-48e8b23f4649"

Thanks for trying to help

Edited September 29, 2010 by josh

[floele]

I just picked one of my Microsoft jobs (slightly modified from shawn template), export as application template, add a GUID and a couple of placeholders, import back and got that message. XML:

 

Ah, I see the problem now. Will be fixed in beta 3.

 

Understood, but if the GUID is the determining factor (and it's a DIFFERENT application - the original was Quicktime Alternative, this one is Quicktime), it shouldn't matter, right?

 

Yep. But also remember that the database itself also contains a GUID (a database is considered as an "author"). This is the authentication for updating the existing application. If you throw your database away, you lose the GUID and cannot update any applications that have been uploaded from that database.

 

Can you please change this so that we *can* add additional placeholders

 

Not for version 1.5 I guess, but later this might certainly be possible.

 

Oh, on that note - Flo - if you decide to add cookie handling abilities, please provide a new time/date function to return the unix timestamp.

 

Already exists, updated the docs now.

 

Cookie handling could be implemented like the post data, will look into that.

 

..and while I know that the {f:...} methods can be used for file naming, I don't see any equivalent functionality for variable assignments.

 

No? Why not create a new textual variable that consists of {f:...} variables? The file date will actually correspond to the "URL date".

 

Somewhere during the recent alphas and betas, Batch scripts that included ">" within them had the greater than character and everything to the right of it removed.

 

Once? Or at which ocassion? So far it seems to me that scripts with ">" do not cause any trouble.

[josh]

Beta 3

 

!Fixed another GUID error message

*Do not update applications on setup by default (only force update if file does not exist)

*New global variables editor

 

Thank you for this update. I'm going to need to do some further testing later, but so far i can say i really notice the difference of speed whyle installing.

[Omniferum]

I don't see why you need "run command after download" if it is not install instructions/options

...

It is possible for Ketarin to parse the url found to an external application?

 

That's a perfect example of why you need the "run command after download" functionality.

 

Parse the URL from the download source (mediafire or whatever) into a variable. Download it, or not (and download something uber-tiny like an icon file) then use the following command in the post-download command:

java.exe -jar jdownloader.jar -a "{yourUrlVaraibleName}"

Nifty, eh?

 

 

As nifty as it may be Ketarin will then be stuck 'processing' because it won't move onto the next script until the other one completes. If there is a way to stop Ketarin doing this then fair enough, I don't know of one though.

 

Also Re: cookie/post data type stuff is there a function to let Ketarin progress past the first page of a site? Specifically Blogspot, where you are required to click "I understand and wish to continue"

[shawn]

Understood, but if the GUID is the determining factor (and it's a DIFFERENT application - the original was Quicktime Alternative, this one is Quicktime), it shouldn't matter, right?

Yep. But also remember that the database itself also contains a GUID (a database is considered as an "author"). This is the authentication for updating the existing application. If you throw your database away, you lose the GUID and cannot update any applications that have been uploaded from that database.

 

I understand that...but my point is, the GUID is completely different. It's not the same "app" anymore. No more than if I were to completely replace the engine in my car -- it might have many of the same structural details, but it's not the same thing. The deciding factor - the "uniqueness" -- is determined by the GUID, right? if that really is the only determining factor, then there's no reason that the app I tried to share wouldn't be allowed: It had a different GUID.

[shawn]

As nifty as it may be Ketarin will then be stuck 'processing' because it won't move onto the next script until the other one completes. If there is a way to stop Ketarin doing this then fair enough, I don't know of one though.

 

Check out "start.exe". That provides options for shelling out of process.

 

Also Re: cookie/post data type stuff is there a function to let Ketarin progress past the first page of a site? Specifically Blogspot, where you are required to click "I understand and wish to continue"

 

You would have to consume then follow the link as a two-step process. If possible, consider using an RSS feed or tag page, when available.

[shawn]

..and while I know that the {f:...} methods can be used for file naming, I don't see any equivalent functionality for variable assignments.

No? Why not create a new textual variable that consists of {f:...} variables? The file date will actually correspond to the "URL date".

 

The problem is, that method works successfully only if the filename doesn't depend on the variable value. It can't parse the {f:...} stuff to compose the values to create the filename. Creating a test case...hm, now it's working. Aaarrrrgggghhhh. Hmm. Maybe it only raises that error if the file hasn't existed yet?

[shawn]

bug report (1.5b3):

 

Somewhere in the last few iterations the custom columns can no longer include the following as the variable source:

{property:HttpReferer}

The referer shows in the properties window, and in the exported XML, but not in the custom column as assigned.

 

Also, all my SF stuff just failed. Maybe the Referer value isn't being processed correctly right now at all?

[floele]

As nifty as it may be Ketarin will then be stuck 'processing' because it won't move onto the next script until the other one completes. If there is a way to stop Ketarin doing this then fair enough, I don't know of one though.

 

You can add a & to the end of an batch script in order to make it run in background.

 

but my point is, the GUID is completely different.

 

Ketarin won't allow you to post an app to the database which already exists with a similar name, in order to prevent "app spam". If you have created a new app, you need to find a more specific name.

 

Maybe it only raises that error if the file hasn't existed yet?

 

May be, though what error specifically? Did I miss something?

 

{property:HttpReferer}

 

Will be fixed in the next beta.