appyface
-
Posts
459 -
Joined
-
Last visited
Content Type
Profiles
Forums
Applications
Posts posted by appyface
-
-
>>>your last idea sounds to complicated to me.
It does to me too, but just in case it did turn out to be easy :-) no harm in asking!
Thanks for providing the pieces of the determined download URL's file. I have several downloads where I'm scraping those pieces because they ARE available on a page somewhere. Now I won't have to do that for any of my downloads :-)
Thanks as always,
--appyface
-
Yes that is right, Ketarin takes the URL I have given, comes up with the correct download URL, and can download the file.
Currently I am hard-coding the base portion of the filename and the extension into my download filename, because I cannot get this info from the site. This is often the wrong filename but more importantly, is the wrong file extension, since I have no way of knowing in advance whether I am going to get a .exe or a .zip.
My personal naming convention always puts the version at the end of the basefilename portion of the downloaded file. So for me just having access to the basefilename separate from the file extension, is good enough. If Ketarin can give me {basename} and {ext}, then I can build {basename}_{vers}.{ext} just as you mentioned, because I *can* scrape the version info I want.
Most flexible would be to allow variables to access and scrape from the determined download path and filename using regex or selection start/end as the current variable function can do. But how difficult to implement?
--appyface
-
Confirmed. Thanks!
--appyface
-
From paragraph 3 of my post:
>>>Here's my issue. There is just no place on the site where I can determine what the downloaded filename is going to be
I can't scrape the info from this site. It is flat out not available. The download URL does not contain the eventual filename at all, it is a redirect which works fine in Ketarin as far as using it for a download URL.
I tried loading it into Ketarin for scraping a variable but Ketarin cannot do that, it ends up trying to download the file from within the variable definition which seems to cause all sorts of havoc.
So again my question: How I can find out the actual base filename and extension being served to Ketarin as a result of the download URL I provided?
--appyface
-
Hi Flo,
I just downloaded latest Ketarin and made sure that is what I am executing. It seems to be a prior version of Ketarin though, can you confirm? The file modification date is 11/17/08. The previous zip file I have, 0.9.9.10, has a file modification date of 12/15/08.
Thanks and regards,
--appyface
-
I think I just cannot accomplish this with Ketarin today, but if that's wrong please move topic to General Discussion... thanks!
I have a download where the name of the downloaded file is a generic one. I can scrape a version number from somewhere on the site to add to the output filename. All this is working as it should.
Here's my issue. There is just no place on the site where I can determine what the downloaded filename is going to be. Add to that, sometimes the file served is a self-extracting ZIP file, other times it is an EXE. The 'generic' name portion varies a bit from release to release too.
Right now I have Ketarin hard-coded to download to a file called myfilename_{vers}.exe
(path portion omitted in example)
This obviously does not provide the actual generic base filename portion and also does not name it with .ZIP if that is how the file was served.
Is it possible to provide built in variables for the base filename (w/o extension), and file extension portions of the download name? I can then easily construct my output filename something like {basename}_{vers}.{ext}
Alternately perhaps allow for a regex to be applied on the builtin variable {file} (which would have to be available other than for commands)?
Any thoughts on how to solve this one easily?
Thanks and regards,
--appyface
-
Great news, thanks.
Also, I just found out that once I have a cloned variable that does not populate, if I create a brand-new Ketarin entry and scrape a variable, that brand-new variable will not populate either. I quit and relaunched Ketarin to no effect. I rebooted my system and the brand-new Ketarin entry and variable is working now without modification.
Just FYI in case that affects the 'fix'.
--appyface
-
I'm sure there is a more elegant, efficient, and generic way to do this, I just threw this together and it works. As long as the webpage adheres to a few hardcoded rules, that is...
1. Add new application to Ketarin.
2. For Download Source, choose URL and enter: {url}
3. Click 'variables' button and add a variable: url
4. For Contents from URL, enter: http://www.visionapp.com/download.html
5. For Use Regular Expression, enter: (?=http://download\..{0,50}?vRD2008.*?)http.*?\.zip
6. Click OK
7. Choose a filename or folder as desired
8. For Spoof HTTP referrer, enter: http://www.visionapp.com/download.html
9. Click OK
Right-click and choose 'update' and see if you get your download.
--appyface
EDIT: I changed the regex a little bit for less backtracking. It could still be much more efficient, and generic as well.
-
Using latest (I think?) Ketarin at 0.9.9.8.
Thanks for putting in copy/paste of a download entry, that's really handy.
However, the variables in the new (cloned) entry are zero-length, they are not populated.
This seems to be true whether a regex or a start/end selection is used.
To recreate, copy and paste an existing (working) entry that scrapes a variable. Change the application name of the cloned entry so you can tell them apart. Now right-click and update the cloned entry. The variables are not populated.
Thanks and regards,
--appyface
-
That sounds like a good workaround, thanks.
If that proves to be problematic, perhaps a checkbox in individual download entry properties in Ketarin, to override and allow any file type just for that entry? Then if Ketarin doesn't download what I wanted, it's on me :-) But first let's see if globally allowing text/plain causes any problems for others...
Thanks again.
--appyface
-
Hi Flo,
Not sure if I am experiencing the same problem as #3 in discussion above (303 - redirect)?
I have this download URL:
http://midiox.com/zip/DeskInstU.msi
Ketarin will not download from this URL, I receive:
The downloaded file is not a binary file type (text/plain). Possibly there is an error page. Status code: 303 (RedirectMethod) (http://www.midiox.com/zip/DeskInstU.msi)
I can download directly from this URL by copy/paste directly into IE URL address line, also can directly copy/paste into Orbit downloader and FDM downloader. In the case of the latter two there is no referrer or session cookie, I can't find any reason for this url not to work straight away in Ketarin.
Any help is appreciated, thanks and regards,
--appyface
-
Want my db export?
-
Settings shows five. I don't recall if that is the default or if I changed it :-)
In both FDM and Orbit I have 10 threads and the page is fine...
-
Hi Flo,
First time execution using 'update now' button (we'll call this update all attempt #1), to do all entries, this Ketarin is scanning pages a LOT faster overall than any previous version.
However, it timed out on two entries from the same problematic page I gave you above. It timed out very quickly now instead of hanging for a minute or two before timing out (I have timeout set to 10 seconds, seems to honor that now).
e-sword bible gw God's Word translation The remote server returned an error: (503) Server Unavailable.
e-sword bible mkjv modern king james version The remote server returned an error: (503) Server Unavailable.
I used 'update now' button again (attempt #2). A page that was scanned successfully the first time (Serv-U) has timed out immediately, plus five entries from the problematic page timed out instead of two. Only one is in common (mkjv).
Serv-U FTP Server The operation has timed out
e-sword bible asv american standard version The remote server returned an error: (503) Server Unavailable.
e-sword bible isv international standard version The remote server returned an error: (503) Server Unavailable.
e-sword bible kjva king james version w/apocrypha The remote server returned an error: (503) Server Unavailable.
e-sword bible mkjv modern king james version The remote server returned an error: (503) Server Unavailable.
e-sword bible rv revised version The remote server returned an error: (503) Server Unavailable.
'Update now' button again (attempt #3) gives this:
e-sword bible kjv king james version The remote server returned an error: (503) Server Unavailable.
e-sword bible kjva king james version w/apocrypha The remote server returned an error: (503) Server Unavailable.
e-sword bible mkjv modern king james version The remote server returned an error: (503) Server Unavailable.
e-sword bible rv revised version The remote server returned an error: (503) Server Unavailable.
Serv-U FTP Server The operation has timed out
I next attempted to individually update one failure at a time (right click, choose 'update'), Serv-U timed out (keep in mind it scanned successfully, or so Ketarin said, the first time). The four above from e-sword all updated successfully. Then I tried Serv-U again and it updated successfully.
'Update Now' button (attempt #4) gave this:
e-sword bible asv american standard version The remote server returned an error: (503) Server Unavailable.
e-sword bible kjv king james version The remote server returned an error: (503) Server Unavailable.
e-sword bible kjva king james version w/apocrypha The remote server returned an error: (503) Server Unavailable.
e-sword bible litv literal translation of the holy bible The remote server returned an error: (503) Server Unavailable.
e-sword bible rv revised version The remote server returned an error: (503) Server Unavailable.
Serv-U FTP Server The operation has timed out
Just for grins, I ran 'old' stable Ketarin 'update now' button, no timeouts on any entries.
FYI.
--appyface
P.S. I should mention, on the problematic page from e-sword, there are 12 entries. So it isn't like Ketarin can't access that page at all....
-
It's just the one page I'm scraping on that site, another page is fine. As I mentioned, I use multiple connections on that page with Orbit and FDM so I don't think that is the issue, it's something different about the multi-thread Ketarins and going after two or more at once. The single thread Ketarin does fine on that page, as does the multi-thread version if I only update one of them at a time.
The page I scrape is: http://e-sword.net/bibles.html
Scroll down to American Standard Version, I scrape the date from the right and use it to append to the output filename.
The download URL is: http://www.e-sword.net/files/bibles/asv.exe
I do the same for several more versions on that page.
I can export my Ketarin db and send it to you if you like, if the above is not enough to get a good (e.g. failing) test. Let me know. Thanks,
--appyface
-
P.S. The timeout requires a minute or more, not the default 10 seconds in the settings. FYI.
P.S.S. Interesting. If I right-click and choose update for just ONE of those entries, 0.9.9.4 completes the job quickly. If I highlight two or more of those timeout entries, right-click and choose update, they timeout. Clue?
-
Hi Flo,
I'm using 0.9.9.4 (based on ZIP file name :-) ), in general using the two default worker threads and zooming through the checking of entries. BUT is hanging worse than ever on that site where it was before hanging on two or three entries. 0.9.9.4 timed out on ALL the entries from that webpage. Other entries from that site downloaded fine.
Previously-stable 0.9.9.2 ('regular') Ketarin does not hang on those entries.
(None of the entries require downloads, just the checking for downloads.)
Is there a logfile or other debug being produced that I can send you? I do not think multiple worker threads on that site is an issue, I have both Orbit and FDM set to use 10 and they each download from that site just fine.
Ideas? TIA,
--appyface
-
Not even in your head? Not that I can get in there... yet... working on it
LOL
-
OK I give up. Where is the next version?
-
After playing with the Custom Category a bit, it is not so useful for me as I had thought.
I need to be able to concatenate variables and some text data to display in that column. I can't always scrape what I want to see, from a single URL into a single variable.
If in the individual entry, I had a way to construct a single variable from concatenated variables and text, then I could create the Custom column's data however I would like to see it.
For example:
{mainvers} is scraped from one URL and contains the major version of the program, e.g. 3.0
{build} is scraped from another URL and contains the rest of the version info I'm intersted in, e.g. 1026
If I define my Custom column variable name as {fullvers}, then I need a way, in each of my download entries, to define
{fullvers} is made up of (whatever - already scraped variables and text data)
I can then display {fullvers} as my Custom column variable. In the above example I would define {fullvers} as being made up of
{mainvers}.{build}
In another entry {fullvers} might be directly scraped from one URL or be made up of the last modified date, etc.
Can this be done? Create new variables in a download entry, by concatenating existing populated variables and optionally some text?
--appyface
-
Different, but not better for me...
This version highlights entries reasonably quicker than previous version as one-then-two {then long pause} three-then-four {then long pause} five {long pause} {six then-seven} etc. I don't mean in any particular order, just that it seems to grab two entries more quickly but then stays on them a lot longer.
This version of Ketarin timed out five entries, three of which were the same three previous alpha version timed out on.
To double-check I ran the 'regular' Ketarin and again there are no timeouts on any of the entries. The 'regular' Ketarin performed much faster in total runtime than the alpha version.
None of my entries had new downloads today. So both the 'regular' and 'alpha' versions had similar work to do.
The winner is still the 'regular' Ketarin, it says version 0.9.3.0 in the Help -> About but I'm reasonably certain it came from a zip file something like 0.9.9. I don't still have the zip or I could tell you.
--appyface
-
Hi Flo,
Does this version of Ketarin produce a debug or trace log I should send you? Or are you collecting data from Ketarin?
I don't know if this alpha version was really faster or not. What I observed after starting Ketarin in GUI mode for full update (used Update Now button at bottom left of GUI)
1. Ketarin highlighted (light blue) one entry. This entry did not need a download.
2. While that one was highlighted, a second one highlighted (this entry also did not require a download)
3. The highlighting is slower than the 'regular' Ketarin, up to 2-3 seconds between highlights showing up, while 'regular' Ketarin proceeds within a second (or less) to the next entry. The ethernet adapter systray 'idiot lights' blink on and off, not steady, with alpha Ketarin. Using Nirsoft's "curr ports" I can see ESTABLISHED for two connections (and a few others, including your site)
4. This alternating and overlapping highlighting continued until Ketarin found an entry which DID require a download.
5. The one download commenced and no other highlighting took place (I had to scroll the GUI to examine all entries, but did not see any other highlighting)
6. Just prior to the one download finishing, Ketarin highlighted another entry. As soon as the download finished, the alternate and overlapping highlighting of two entries recommenced.
7. Ketarin did highlight two entries requiring download and proceeded to download both. The simultaneous downloading did complete faster than two back-to-back downloads. How much faster...? hmmm Noticable
8. Ketarin continued with the alternate and overlapping highlighting of entries that did not require download, then got 'stuck' on two that did not require download. No I/O idiot light activitity for several minutes. Currports showed ESTABLISHED connections to the server with the 'stuck' entries.
9. Eventually Ketarin timed out and moved on, caught one more 'stuck' entry on that same server, then continued on to the end.
10. I ran alpha Ketarin again now that everything was downloaded and current. Ketarin again got 'stuck' on first two, then later the third, entry on the one server.
11. I ran 'regular' Ketarin which proceeded to move through the entries one-by-one and did not get 'stuck' on any.
12. I ran alpha Ketarin again and still gets 'stuck' on those same three entries.
13. Still in alpha Ketarin I right-clicked one of the timed-out entries and selected Update. This did not time out though the checking was slow motion just like the alternating highlighting. I updated the other two 'stuck' entries same way and they did not time out but checking was slow.
14. Back to 'Regular' Ketarin and I right-clicked to update each of those three entries (even though there were not 'stuck' with regular Ketarin). The checking was quick, which is normal for 'regular' Ketarin.
Overall impression:
'Regular Ketarin' is faster when moving through each of the entries, calculating the variables, and discovering there are no downloads needed.
Alpha Ketarin is slow-motion in this regard, but did simultaneously complete two downloads in less time than regular Ketarin does them back-to-back.
Hope this helps,
--appyface
P.S. My DSL connection cannot be considered "on fire" :-) but is fast enough for downloading when I use FDM or Orbit. It's a 3.1 Mbps download line. Speed testing from different sites such as DSL Reports etc. consistently show 2.8Mbps or more. My DSL provider comes out once every 6-9 months and audits the line both at the point where it enters the house and from my DSL modem. The SNR is very clean on their line and over the wiring in my home. FWIW.
-
I see it and I understand it now. Thank you, I can use this!
Best,
--appyface
-
Thanks Flo {blush} but I'm not deserving. To wit see documentation thread, I'm still clueless over there! LOL
--appyface
Ketarin 0.9.9.11 old version?
in General discussion
Posted
Hi Flo,
0.9.9.11 was broken for me too. I just downloaded and get Ketarin-0.9.9.13.zip for the file, but the about shows '12' not '13', and I can't find the 'don't update urls' mentioned above?
I assume this latest feature with changing the URL is a result of my request to know the determined/derived download filename? If so, is it possible for Ketarin to keep the original request URL intact, and have a place to store the intermediate URL if it is different? Then we could keep our original URLs which might serve a different determined/derived URL on each update.
Replacing our original URL I think is counter-productive; in my case I'd get a version specific replacement URL whereas my original URL would cause the current version to be served...
If I've misunderstood all that please forgive, I can't even SEE what this is as my version of Ketarin ('12' in a '13' zip file?) doesn't have it as far as I can tell...
--appyface