"Independent" update verification?

[appyface]

Like many people here, I use an independent page-watching service (for me it's www.watchthatpage.com) to notify me of new programs from an author I like, updates to programs I currently use, etc.

 

Of course Ketarin is not meant to replace such a service in all things. But in going through my recent WatchThatPage notifications, I am seeing more and more where I am not getting an update through Ketarin that I intended to get.

 

So I was just thinking, maybe Ketarin can help me with this situation?

 

It's not any failure on Ketarin's part, of course. Sometimes it is just a matter of tuning a poorly written regex or variable scrape. But sometimes it just isn't possible to create an entry that is neither too generic nor too restrictive. Sooner or later that entry may simply continue to locate an old build, without error.

 

This is what I am seeing. My entry in Ketarin is working correctly, but because of the way I defined it, it just continues to locate the old build and report there is no update. The 'rules' on the webpage have changed, and I need to change Ketarin's entry to adapt to that. But how to know, if Ketarin is not having an error?

 

So... still thinking out loud... here's one idea how Ketarin could help:

 

1. Create a variable in the application the same way as we can do now -- scraping a webpage or concatenating other variables etc. But this variable will be somehow be designated as a special monitoring variable.

 

2. When asked to update an application, Ketarin compares the status of the monitor variable's contents (change, no change, error) with the status of the actual download result (change, no change. Error is separate reporting matter already). The idea being if a mismatch in the two statuses occurs, Ketarin would raise alert.

 

Of course there can be lots of reasons for the mismatch, including my poor choice of a monitoring variable, but at least Ketarin would tell me something's up based on my two 'independent' indicators.

Thoughts?

 

Thanks and regards,

--appyface

[floele]

I don't see how this would work. If you use a variable as change indicator, Ketarin will download anyways and not perform any further checks for changes.

[appyface]

Yes of course, Ketarin can only download (or not) as per the application's instructions. I must modify my entry in Ketarin to pick up the new download.

 

My point is I'd like Ketarin to be the one to warn me that I MAY need to modify my Ketarin entry so I will get the new file.

 

Otherwise I would not find out this situation very quickly, unless I use a page monitoring service, or frequently visit all my websites and manually verify whether my Ketarin entry is still retrieving the newest files.

 

My proposal was just one possible way Ketarin could be made to warn me I need to investigate. I suggested a special monitoring variable.

 

Ketarin would save the contents of this special variable from the last update request, and then compare it to the contents that are retrieved during this update request. I am only interested in the change in status, and not the actual contents: Did the variable's contents change since last time? Or does it have the same contents as last time? Did the scrape error out this time?

 

Then compare this status (changed, unchanged, error) to the same type of status of the actual downloaded file (downloaded = changed, did not download = no change).

 

If there is a mismatch in the two statuses, Ketarin should warn me so I can investigate further.

 

Of course it is not foolproof, the accuracy of the warning is only as good as the monitoring variable content I choose to scrape.

 

But it would keep me from having to rely on 3rd party page monitoring service for possible changes I need to make to Ketarin's entry.

 

I hope what I mean is clearer now? Please let me know. Thanks and regards,

--appyface

Edited January 24, 2009 by appyface

[CybTekSol]

It seems to me that this would be extremely difficult to implement and may produce as many false notifications as valid ones. If the webmaster simply changed the formatting of a page it would trigger a notification while the app version may still remain the same and the regex is still working if I'm grasping what you are proposing. Maybe a setting for "Notify if application has not updated in x number of days" that could be set at 30, 60, 90, 180 or a custom value which could possibly accomplish this? This would allow a user to investigate to determine if an update has been released and their regex needs to be adjusted.

[appyface]

It doesn't seem like it would be difficult to implement, but then again I don't really know that for sure :-)

 

In any event, you are correct, the feature is only as good as the person who is using it, makes it. I certainly wouldn't make it a required feature!

 

What I really want is to be alerted to the possibility that my Ketarin entry is not up-to-date. For me the number of days is not really useful for that. I can already see the last updated date on the display and know how long it has been since a download was made, so I can manually go inspect any of these entries. I'd like to avoid manually inspecting entries just because they might be out-of-date. But if there was some independent monitoring source that could clue me in, Ketarin could watch that too and let me know.

 

Me, I'd rather continue using WatchThatPage to let me know when something may need my attention in Ketarin, than be in the position of manually inspecting all my Ketarin entries that haven't been updated for awhile. I just thought it would be nice if Ketarin could take over that one function.

 

Thanks and regards,

--appyface

Edited January 25, 2009 by appyface

[CybTekSol]

Me, I'd rather continue using WatchThatPage to let me know when something may need my attention in Ketarin, than be in the position of manually inspecting all my Ketarin entries that haven't been updated for awhile. I just thought it would be nice if Ketarin could take over that one function.

Noted... I've never used their service, how much work would it be to setup monitoring for 500+ app pages? Could I paste them all in at once?

[appyface]

I am a supporter and send WatchThatPage a modest fee each year in exchange for making me a 'priority user' and handling about 3000 webpages for me :-) Most other services charge an arm and a leg for that many pages but these guys have never batted an eye.

 

It's free, sign up and have a look around? I have never been spammed at the email address I gave them (they are the only ones that have it) so I believe they do not sell addresses, etc. I have used them for what, must be 6 or 7 years now I think, maybe more.

 

I don't think there is any way to mass-upload, but I haven't needed to do that so I haven't looked for it. I was using another service that went defunct, that's how I switched to WatchThatPage. At the time I used a screen macro utility to 'type' my webpages into WatchThatPage, so it wasn't a hardship on me. At that time I had about 800 entries.

 

I saw mention of "Webmon" here, as another similar service, but I am not familiar with that one.

 

Best,

--appyface

Edited January 25, 2009 by appyface

[CybTekSol]

@appyface,

Thanks for the info... I'll have to give them a try.

[appyface]

The website ("GUI" if you will) can be a little buggy and some buttons don't work but look like they should. Also, some of the features are not intuitive (to me). If you have any questions please let me know, I can probably answer them.

 

I have had very few problems over the years. Unlike "speedy-Flo" the WatchThatPage guys are slow to respond to tech support and questions, but they DO eventually respond.

 

The service has been a very good value overall for me. YMMV of course!

 

Best,

--appyface

[CybTekSol]

@appyface,

It would be nice if WatchThatPage had a Firefox extension. Have you suggested that to them? I apologize Flo for getting a little off topic in the 'Feature requests' forum... this is the last question I'll ask regarding WatchThatPage.

[appyface]

I don't use Firefox, so... no I haven't.

[floele]

I guess I'll think about that after 1.0...for the time being, you could for example post a real example to help me understand the situation. While I appreciate that you bother explaining your problems in detail, you should see the question marks that pop up over my head while reading

Some examples might simplify that process...see http://i39.tinypic.com/1zxsb4i.jpg

[CybTekSol]

I LIKE that JPEG... needed a chuckle! LOL

[appyface]

LOL I have a similar design process flowchart in my office! Good one, Flo

 

Well, since I've already fixed all my examples :-) I'll have to see what I can dredge up for you, it may take me some time.

 

In the meantime, how about some more detail to explain the detail? :-p

 

Imagine this scenario:

 

1. You don't use a page watching service

 

2. You have Ketarin happily checking on, let's say 500+, applications for you so you don't have to bother to check on them manually yourself. You know that Ketarin will get you the new file whenever it shows up, right? So you can spend your time doing other things than looking for new setup packages

 

3. The authors of one or more of these programs puts out a new version of the program. However, the authors decide to make the new version have a completely new URL. Or maybe they put the new version number and information on a new webpage. Or maybe they put the URL in a different place on the same webpage where Ketarin is scraping it's URL from, or maybe the author did something else that Ketarin is not watching for....

 

Ketarin is very flexible, and can handle any of the above situations and more with ease. Unfortunately Ketarin can't know what it doesn't know!

 

Let's say that the author made a new download URL for the new version, but Ketarin entry still looks at the old version's URL. The author has not removed the old URL, so it does not give an error.

 

Ketarin happily continues checking the old URL and reports "no update". That's not what we wanted, but it's not Ketarin's fault that it doesn't know anything about the new URL.

 

That's just one example.

 

In the little scenario above, I paint the picture that just like Ketarin, WE can't know what we can't know, either. Ketarin is not telling us about a new file because it can't. We could not predict the future either, in order to code Ketarin's entry to always find the file we would want. We did the best we could when we setup the Ketarin entry, but the webpage author changed the rules on us and didn't tell us!

 

So. In the scenario above we do not manually visit all these 500+ websites ourselves, we don't use a page watching service, in short we don't use any "INDEPENDENT" process that can corroborate/deny Ketarin is still doing the job we intended. That means we don't know when Ketarin is NOT doing the job we wanted and we're missing out.

 

 

I use the WatchThatPage service for quite a few reasons, but probably 60% of the pages I watch is just for alerting me to the possibility of a new setup package to download.

 

For example, many authors put all their change information in one place on their site, regardless of how they might change their download URLs etc., Ketarin could help me corroborate/deny that its entry is doing what I intended, by also monitoring whether the contents of that one place has been updated, but a new setup package was not downloaded.

 

That alert from Ketarin MIGHT mean I need to update Ketarin's entry. Or it might be a false alarm. I'm OK with that, it must be entirely on me to choose wisely what is to be independently monitored. Same as it is entirely on me to construct Ketarin's entry so it works. Assuming no 'bug', it is not Ketarin's fault if the entry doesn't work or it downloads the 'wrong' file... the file 'wrong' only to me, Ketarin has no way of knowing that.

 

If Ketarin were to offer this independent corroboration feature, I still do not expect it to 100% replace a page watching service for this purpose. In the example above, if the author suddenly departs from putting all information on one changelog page, Ketarin is obviously not going to be independently monitoring the right data anymore.

 

So even though I'd still be using a page monitoring service to watch the home pages, news pages, or some other page on the site where I might expect to get a hint of a new program version, Ketarin could still help me out with quite a bit of it.

 

It's not a big deal for me to wait for the emails from the page watching service, and then go fix up Ketarin. That's what I've been doing. I just thought if Ketarin could do a simple independent verification for me when I run the update, the nicest thing about having that is I get the notification right then and there from Ketarin, and not from an email a few days later.

 

I think the confusion factor may very well be 'big' from such a feature, as CybTekSol has hinted. If Ketarin were to get this feature, it should be marked "advanced users only" and a warning that you need to know exactly what you're doing if you rely on this feature...

 

Flo, did I answer all the ???? you had from my earlier detailed description? Or did I make more ????

 

--appyface

[floele]

To sum it up (for when I get back to that topic at a later time):

Define a variable X. If X does change, but no update is found, show an altert.

Reason: When for example a changelog page ("X") changes, but the "URL detection" has been defined incorrectly, you'll notice.

[appyface]

Yes... but that's not all :-) Ketarin can help me identify if I'm not monitoring the right thing with variable X, too.

 

1. Ketarin does not download a new file, and variable X has not changed in content.

Statuses match. No alert.

 

2. Ketarin does not download a new file, and variable X has changed in content.

Statuses do not match. Alert.

 

3. Ketarin downloads a new file, and variable X has not changed in content.

Statuses do not match. Alert.

 

4. Ketarin downloads a new file, and variable X has changed in content.

Statuses match. No alert.

 

5. Ketarin's download attempt of a new file has error. We already get an alert for this.

 

6. Ketarin's attempt to populate variable X has an error. I'd want an alert for this.

 

Cool if this is on your list!

 

--appyface

[CybTekSol]

In the interim... until Flo decides whether this is a worthy feature to implement, for those that are interested in another alternative solution, I found a GREAT Firefox 3 extension that will monitor a web page at custom intervals with just a few mouse clicks. I am using it to monitor the pages that I am 'scraping' for the version of an app in Ketarin, therfore, if the extension has notified me of changes to to the page and Ketarin is not downloading a newer version, then I will investigate the issue. The extension is Update Scanner (add-on # 3362) located at: https://addons.mozilla.org/en-US/firefox/addon/3362 and appears to be very stable. I am using it to monitor the Ketarin forum and even highlights items that have changed in yellow. SWEET!

Edited January 26, 2009 by CybTekSol

[appyface]

Hi Flo,

 

Since I'm (still) not a Firefox user (sorry, CybTekSol!) I'd like to dredge this wish back to the top, for reconsideration?

 

Now that Ketarin's logging is so much more comprehensive than it used to be, and we have such a great variable facility now, I think this could be done easily by using variables and a bit of logging...?

 

Here's what I have in mind:

 

1. I define one or more variables (same way as for any variable) in an app. These might contain entire web page or a section of a webpage, or textual content... whatever content I'm interested in.

 

2. I have in the app definition a way to list or otherwise designate these variables as my special 'independent watch' variables. A practical limit (maybe a max of 3 such variables?) per app would be fine

 

3. For each 'independent watch' variable, during update run Ketarin compares the new scraped value with previous stored value (similar to Change Indicator Variable)

 

4. If the value of any 'watch' variable has changed, make a log entry. Something like this:

 

04/07/2009 11:42:44: Update started with 1 application(s)

04/07/2009 11:42:45: Download Test: Checking if update is required...

04/07/2009 11:42:45: Download Test: Update not required

04/07/2009 11:42:45: Download Test: Watch variable {changelog} updated

04/07/2009 11:42:45: Download Test: Watch variable {versnews} updated

04/07/2009 11:42:45: Update finished

 

Example: Say I have three 'independent watch' variables defined for this app, the above two are the ones that changed value

 

 

I would go a step further and ask if the app was NOT downloaded (for any reason, including CIV), but at least one of the watch variables was updated, that an additional entry be made in the log, perhaps something like this. I'd like a different icon on the Ketarin GUI if it's open, perhaps a red exclamation point?

 

04/07/2009 11:42:45: Download Test: Checking if update is required...

04/07/2009 11:42:45: Download Test: Update not required

04/07/2009 11:42:45: Download Test: Watch variable {changelog} updated

04/07/2009 11:42:45: Download Test: Watch variable {versnews} updated

04/07/2009 11:42:45: Download Test: Alert: No download but watch variable(s) updated

 

Thoughts on this idea? TIA as always,

--appyface

 

Edit: Actually, just thinking out loud here -- if a "watch" variable can be a concatenation of other variables, then just one special "watch" variable will do nicely...

Edited April 7, 2009 by appyface