Variable as indicator for changes inconsistent

[Omniferum]

I'm getting rather strange behaviours from my Ketamon apps in regards to the variable change indicator.

 

For all my other apps this does happen every now and then, it loads when there isn't actually any changes. However slashdot is now continuously saying that the variable is changing everytime I check. Tried adding a User Agent for shits n' giggles but that didn't change much. I don't believe it would actually change page content but what the variable keeps scraping stays consistent. I can load the page in the variable screen ten times and get the same value. I just do a regular update and this keeps going "Yep, it's different". I've even changed the regex around to make it look at other things but it keeps treating it like it has changed.

 

If anyone could give me a hint as to why it keeps going "Yes! This variable value has changed so I will continue with loading everything" I would love ya'll forever. For reference sake this was working fine, with only slight intermittent problems (like with all apps) until today. The site doesn't seem to be having any difficulties.

 

<?xml version='1.0' encoding='utf-8'?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="2862ded8-7f6d-4f6c-821d-f65ba0256e8f">
   <SourceTemplate><![CDATA[<?xml version="1.0" encoding="utf-8"?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
   <SourceTemplate><![CDATA[]]]]><![CDATA[></SourceTemplate>
   <LastFileDate xsi:nil="true" />
   <IgnoreFileInformation>false</IgnoreFileInformation>
   <DownloadBeta>Default</DownloadBeta>
   <DownloadDate xsi:nil="true" />
   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator>page</VariableChangeIndicator>
   <CanBeShared>true</CanBeShared>
   <ShareApplication>false</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <HttpReferer />
   <SetupInstructions />
   <Variables>
     <item>
       <key>
         <string>monitor</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>.*</Regex>
           <Url><placeholder name="URL" value="http://slashdot.org/" /></Url>
           <Name>monitor</Name>
         </UrlVariable>
       </value>
     </item>
      <item>
       <key>
         <string>name</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>title>(.*?)</</Regex>
           <Url><placeholder name="URL" value="http://slashdot.org/" /></Url>
           <Name>name</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>page</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <Url />
           <TextualContent><placeholder name="URL" value="http://slashdot.org/" /></TextualContent>
           <Name>page</Name>
         </UrlVariable>
       </value>
     </item>

   </Variables>
   <ExecuteCommand />
   <ExecutePreCommand>start {page} & exit 2</ExecutePreCommand>
   <ExecuteCommandType>Batch</ExecuteCommandType>
   <ExecutePreCommandType>Batch</ExecutePreCommandType>
   <Category>Web Page Monitoring</Category>
   <SourceType>FixedUrl</SourceType>
   <PreviousLocation />
   <DeletePreviousFile>true</DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />
   <LastUpdated xsi:nil="true" />
   <TargetPath>:</TargetPath>
   <FixedDownloadUrl>https://www.google.com/images/logos/mail_logo.png</FixedDownloadUrl>
   <Name><placeholder name="Name of application" variable="name" value="Slashdot: News for nerds, stuff that matters" /></Name>
 </ApplicationJob>
</Jobs>]]></SourceTemplate>
   <WebsiteUrl />
   <UserAgent />
   <UserNotes />
   <LastFileSize>0</LastFileSize>
   <LastFileDate xsi:nil="true" />
   <IgnoreFileInformation>true</IgnoreFileInformation>
   <DownloadBeta>Default</DownloadBeta>
   <DownloadDate xsi:nil="true" />
   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator>monitor</VariableChangeIndicator>
   <CanBeShared>true</CanBeShared>
   <ShareApplication>false</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <HttpReferer />
   <SetupInstructions />
   <Variables>
     <item>
       <key>
         <string>monitor</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>datitle">(.*?)</</Regex>
           <Url>http://slashdot.org/</Url>
           <Name>monitor</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>name</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex><title>.*?(\S.*?)\s*</title></Regex>
           <Url>http://slashdot.org/</Url>
           <Name>name</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>page</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <Url />
           <TextualContent>http://slashdot.org/</TextualContent>
           <Name>page</Name>
         </UrlVariable>
       </value>
     </item>
   </Variables>
   <ExecuteCommand />
   <ExecutePreCommand>start {page} & exit 2</ExecutePreCommand>
   <ExecuteCommandType>Batch</ExecuteCommandType>
   <ExecutePreCommandType>Batch</ExecutePreCommandType>
   <Category>Technology</Category>
   <SourceType>FixedUrl</SourceType>
   <PreviousLocation />
   <DeletePreviousFile>false</DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />
   <LastUpdated xsi:nil="true" />
   <TargetPath>,</TargetPath>
   <FixedDownloadUrl>https://www.google.com/images/logos/mail_logo.png</FixedDownloadUrl>
   <Name>Slashdot: News for nerds, stuff that matters</Name>
 </ApplicationJob>
</Jobs>

Edited December 18, 2010 by Omniferum

[Omniferum]

To fix my problem I had to export the entire .db as XML and then delete all the apps in Ketarin and re-import the entire .xml back into Ketarin.

 

However i'm not sure if it isn't going to crap itself again and will have to do this again, does anyone else have any similar problems? I've had problems like this before that I solved just by deleting/re-importing but that seems an odd way to fix things. Could it be .db corruption of some sort?

[Tomorrow]

To fix my problem I had to export the entire .db as XML and then delete all the apps in Ketarin and re-import the entire .xml back into Ketarin.

 

However i'm not sure if it isn't going to crap itself again and will have to do this again, does anyone else have any similar problems? I've had problems like this before that I solved just by deleting/re-importing but that seems an odd way to fix things. Could it be .db corruption of some sort?

Well floele said the db corruption is pretty much impossible/out of the question so i doubt this is the case.I do a db export-import from time to time anyway.

[Omniferum]

Hm, it just happened again on me. I thought i'd try to narrow the issue down slightly and it seems that if the variable changes about 10 times or more for that app it becomes 'locked' to perpetually thinking it is always different.

 

Manually forcing downloads and using a few high traffic sites that changes every couple of minutes confirm this. Can anyone else confirm this for me? Was fixed with an export/import, still a bit annoying to have to do that.

[shawn]

DB corruption is possible under a couple scenarios: drive or ram failure are the most obvious. It's also possible that a surge or other physical defect could occur simultaneous to db writing, which could cause problems. Those are each unlikely, but not impossible. It's also possible that it's a logic failure somewhere either in Ketarin or within the mon profile.

 

Since you're using Slashdot for testing, this could be your problem. Slashdot implements view throttles to minimize unnecessary server load. If you refresh a certain page too frequently, or if you have a feed set to recycle more often than once every couple hours, you will not receive a 200 status. Ketarin may be treating the non-200 as errant, which is why you're getting odd behavior.

[Omniferum]

I did do testing with other sites, and to now yes there is difficulty with any site after ~10 variable changes until it becomes 'locked'

 

However i've used another page update checker before with slashdot and it worked perfectly fine regardless of any manual refresh. Worked in the same way, scraped the page source.

[Omniferum]

Think i've identified the problem, everytime there is a variable indicator change my jobs.db increases in size until about 1mb (I want to say 1024kb but it isn't specifically that) and then it stops being able to store anymore so I assume it is just the SQL database size limit.

 

I'm sort of curious because that would suggest that Ketarin stores EVERY variable change which leads to the .db getting full. If this is the case is there a special reason as to why it does this? I wouldn't have thought there to be much value in us retaining every variable change as we would really only need a limited number wouldn't we? between 1 and 5, is there some function in Ketarin that even allows us to access old stored variable changes?

[Tomorrow]

Well there is no global db size limit as far as i know.Poeople have seen 9-10MB databases here too.

[Omniferum]

Perhaps a per app limit then? or a limit to the variable detection specific?

[Tomorrow]

Perhaps a per app limit then? or a limit to the variable detection specific?

I doubt there is a per app limit either since how does the database distinguish between apps?.I mean it's based on sqlite(i think) so it's not some closed source thing flo wrote himself.

 

I think it's more likely that some variable catches a lot of unnessesary junk and this "junk" is what is causing problems with database.

[Omniferum]

Seems to be the most likely case, my .db just keeps growing with each new 'variable change'. I suppose nobody has found this bug before because they've never had their variable change go through so many iterations yet.

 

Nothing to do but the odd .xml export/import and wait for floele to give some insight

[floele]

1. Consider DB corruption as not happening. It is highly unlikely that you see bugs in Ketarin because of RAM failure. If at all, consider it at the very last possibility, though you should have more problems than just Ketarin.

2. There is no database size limit (at least none that would apply to Ketarin), neither is there a maximum size per app (I believe text variables can have a size of 4GB, should suffice).

 

So in conclusion you should just accept that there is a simple bug in Ketarin which I have yet to reproduce.

[Omniferum]

The only 'new' thing that I utilize that may be aiding in causing this odd behavior is the 'exit 2' command. If nobody has had 'monitor variable for change' command cause any issues before this is the only other thing i've used that hasn't been around for as long as that function. Just trying to help.

[shawn]

My jobs.db has been over 25mb in the past. Export & reimport shrunk it significantly - but it was working fine at that size anyway.

 

Have you considered creating versioned XML exports between each iteration, then comparing them with something like WinMerge to see if there's a clue why it's failing?

 

Would you mind sharing a zip of your Ketamon jobs.db so I can look into it? It's possibly related to what you're parsing, or certain characters or something, and fresh eyes can't hurt. Which build of Ketarin are you using?

[Omniferum]

http://www.mediafire.com/?0jm2h84yk8wsgrg

 

Didn't know how to attach anything to our posts (assuming we even can) so i just uploaded it to mediafire.

 

Appreciate the assistance

[shawn]

Which build of Ketarin are you using?

[Omniferum]

Latest one, 1.6.0.400.

[Omniferum]

Hm, think I have narrowed it down to the exit 2 code being the reason for the odd behaviour. My usual battery of tests has now resulted in perfect behaviour when before without fail it would 'lock', all I had to do was delete the exit 2 code.

 

If it stuffs up again i'll post, but my .db has now gone to 5mb where before it couldn't exceed the 1mb mark.

[Tomorrow]

My database is still 432 KB.Average increase ~5-10KB per week

[Omniferum]

I assume mine can get large because of my monitoring variable for changes for some sites I have to capture a fair bit of the page (should probably refine the regex) but if all my pages are 'new' it can increase by 200kb. Regardless i've gotten further now than I had before, I just assumed that Ketarin would 'flush' old data but seeing as my .db increases as such that is probably not the correct assumption.

[shawn]

I assigned the "after updating all applications" to my snapshot.bat, included below.

@ECHO OFF

:variables
REM !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
FOR /F "tokens=2-4 delims=/ " %%a in ("%date%") DO SET MM=%%a&SET DD=%%b&SET YY=%%c
FOR /F "tokens=1-4 delims=:. " %%a in ("%time%") DO SET HH=%%a&SET NN=%%b&SET SS=%%c&SET LL=%%d
FOR /L %%a in (0,1,9) DO @IF "%HH%"=="%%a" SET HH=0%%a
SET timestamp=%YY%%MM%%DD%T%HH%%NN%%SS%

REM Export current settings
ketarin /export=ketarin-%timestamp%.xml

 

Then I ran "Update All" manually over 50 times, followed by the following:

FOR /L %i IN (1,1,25) DO ketarin /silent /log=ketarin-%i.log

 

Thus, it's run about 85 times now (with "exit 2" still in place), and the jobs.db is still exactly the same size. The first and last export files are exactly the same. The first and last "log" files are different, but reasonably consistent (the significant difference appears to be that since it's doing many update checks simultaneously, the order and progress of each will be inconsistent - though the results are much the same).

 

Bottom line: I can't replicate your problems. Sorry.

[Omniferum]

My .db only increases in size if there is an update to be found and the page/code is actually loaded, for my tests I pick a more high traffic site. Something with comments or a hit counter, like 4chan, slashdot comment sections or some other forums.

[shawn]

The Ketarin page popped on me about a dozen times during my tests. As did several of the comic pages.

[shawn]

And Omni - I have to say this is a pretty cool implementation of page monitoring.

[Omniferum]

Hm, i've been running mine as well but I have only had like 6 pages load. Some store more in the variable than others, again try 4chan or something high traffic.

 

This is my latest template, with exit 2 in it. Assuming you want to, again i'm happy if it is just a case of maybe i'm capturing something in a variable that Ketarin doesn't like but I can't SEE anything. It would explain why YOU are ok and i'm not.

 

Does anyone know if perhaps there is some characters that SQL databases don't like or something? Maybe i'm capturing a piece of text that is some sort of code in SQL? like class= or something along those lines, might be a long shot but i'm nothing but persistent as all buggery.

 

<?xml version='1.0' encoding='utf-8'?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
   <SourceTemplate><![CDATA[]]></SourceTemplate>
   <LastFileDate xsi:nil="true" />
   <IgnoreFileInformation>true</IgnoreFileInformation>
   <DownloadBeta>Default</DownloadBeta>
   <DownloadDate xsi:nil="true" />
   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator>monitor</VariableChangeIndicator>
   <CanBeShared>true</CanBeShared>
   <ShareApplication>false</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <HttpReferer />
   <SetupInstructions />
   <Variables>
     <item>
       <key>
         <string>monitor</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>.*</Regex>
           <Url><placeholder name="URL" /></Url>
           <Name>monitor</Name>
         </UrlVariable>
       </value>
     </item>
      <item>
       <key>
         <string>name</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex><title>.*?(\S.*?)\s*</title></Regex>
           <Url><placeholder name="URL" /></Url>
           <Name>name</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>page</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>Textual</VariableType>
           <Regex />
           <Url />
           <TextualContent><placeholder name="URL" /></TextualContent>
           <Name>page</Name>
         </UrlVariable>
       </value>
     </item>

   </Variables>
   <ExecuteCommand />
   <ExecutePreCommand>start {page} & exit 2</ExecutePreCommand>
   <ExecuteCommandType>Batch</ExecuteCommandType>
   <ExecutePreCommandType>Batch</ExecutePreCommandType>
   <Category>Web Page Monitoring</Category>
   <SourceType>FixedUrl</SourceType>
   <PreviousLocation />
   <DeletePreviousFile>true</DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />
   <LastUpdated xsi:nil="true" />
   <TargetPath>Web Page Monitoring\{appname:regexreplace:([^a-zA-Z0-9\_]+):-}</TargetPath>
   <FixedDownloadUrl>https://www.google.com/images/logos/mail_logo.png</FixedDownloadUrl>
   <Name><placeholder name="Name of application" variable="name" /></Name>
 </ApplicationJob>
</Jobs>

Edited December 22, 2010 by Omniferum