appyface Posted November 23, 2008 Report Share Posted November 23, 2008 I do like to sort the Ketarin display according to last updated date/time, category, etc. I have added a LOT of entries to Ketarin and it does take a fair bit of time to go through them on an update now. Is it possible to have Ketarin respect the current sort order and start sequentially with the topmost entry? Also is it possible to limit Ketarin to one or more categories? (command line invoke or other way?) I know can highlight and right-click to update specific entry/entries. But my list is so long now.... The way I've set up my categories, my entries are grouped into frequently-changing/highly desirable category, then not so important, etc. all the way down to category I expect will hardly ever change (drivers for old hardware etc.) Would like to be able to invoke Ketarin to check my high-priorities items very often, and the lesser priority items on a more infrequent schedule. I can use task manager or other tool if I can control this with INI files or command line. Thanks as always for consideration of the feature requests for Ketarin! Best regards, --appyface Link to comment Share on other sites More sharing options...
floele Posted November 24, 2008 Report Share Posted November 24, 2008 Regarding speed, I guess it would really make a difference if Ketarin could process multiple jobs simultaneously. But using the correct order is certainly possible as well. The command line option also makes sense. Link to comment Share on other sites More sharing options...
appyface Posted November 25, 2008 Author Report Share Posted November 25, 2008 Thanks Flo! We all appreciate your consideration of all the ideas. Best regards, --appyface Link to comment Share on other sites More sharing options...
floele Posted November 25, 2008 Report Share Posted November 25, 2008 Thank you as well, it's good to have a power user observing the development Link to comment Share on other sites More sharing options...
appyface Posted November 25, 2008 Author Report Share Posted November 25, 2008 Thanks Flo {blush} but I'm not deserving. To wit see documentation thread, I'm still clueless over there! LOL --appyface Link to comment Share on other sites More sharing options...
floele Posted November 25, 2008 Report Share Posted November 25, 2008 I prepared an alpha version: http://ketarin.canneverbe.com/downloads/Ketarin/Ketarin-0.9.9.2a.zip This version downloads with 2 threads (setting still missing) simultaneously, I'd appreciate if you could check how well it actually works for you. With a decent internet connection, using 2 or more threads/workers should speed up the whole process significantly. Link to comment Share on other sites More sharing options...
appyface Posted November 26, 2008 Author Report Share Posted November 26, 2008 (edited) Hi Flo, Does this version of Ketarin produce a debug or trace log I should send you? Or are you collecting data from Ketarin? I don't know if this alpha version was really faster or not. What I observed after starting Ketarin in GUI mode for full update (used Update Now button at bottom left of GUI) 1. Ketarin highlighted (light blue) one entry. This entry did not need a download. 2. While that one was highlighted, a second one highlighted (this entry also did not require a download) 3. The highlighting is slower than the 'regular' Ketarin, up to 2-3 seconds between highlights showing up, while 'regular' Ketarin proceeds within a second (or less) to the next entry. The ethernet adapter systray 'idiot lights' blink on and off, not steady, with alpha Ketarin. Using Nirsoft's "curr ports" I can see ESTABLISHED for two connections (and a few others, including your site) 4. This alternating and overlapping highlighting continued until Ketarin found an entry which DID require a download. 5. The one download commenced and no other highlighting took place (I had to scroll the GUI to examine all entries, but did not see any other highlighting) 6. Just prior to the one download finishing, Ketarin highlighted another entry. As soon as the download finished, the alternate and overlapping highlighting of two entries recommenced. 7. Ketarin did highlight two entries requiring download and proceeded to download both. The simultaneous downloading did complete faster than two back-to-back downloads. How much faster...? hmmm Noticable 8. Ketarin continued with the alternate and overlapping highlighting of entries that did not require download, then got 'stuck' on two that did not require download. No I/O idiot light activitity for several minutes. Currports showed ESTABLISHED connections to the server with the 'stuck' entries. 9. Eventually Ketarin timed out and moved on, caught one more 'stuck' entry on that same server, then continued on to the end. 10. I ran alpha Ketarin again now that everything was downloaded and current. Ketarin again got 'stuck' on first two, then later the third, entry on the one server. 11. I ran 'regular' Ketarin which proceeded to move through the entries one-by-one and did not get 'stuck' on any. 12. I ran alpha Ketarin again and still gets 'stuck' on those same three entries. 13. Still in alpha Ketarin I right-clicked one of the timed-out entries and selected Update. This did not time out though the checking was slow motion just like the alternating highlighting. I updated the other two 'stuck' entries same way and they did not time out but checking was slow. 14. Back to 'Regular' Ketarin and I right-clicked to update each of those three entries (even though there were not 'stuck' with regular Ketarin). The checking was quick, which is normal for 'regular' Ketarin. Overall impression: 'Regular Ketarin' is faster when moving through each of the entries, calculating the variables, and discovering there are no downloads needed. Alpha Ketarin is slow-motion in this regard, but did simultaneously complete two downloads in less time than regular Ketarin does them back-to-back. Hope this helps, --appyface P.S. My DSL connection cannot be considered "on fire" :-) but is fast enough for downloading when I use FDM or Orbit. It's a 3.1 Mbps download line. Speed testing from different sites such as DSL Reports etc. consistently show 2.8Mbps or more. My DSL provider comes out once every 6-9 months and audits the line both at the point where it enters the house and from my DSL modem. The SNR is very clean on their line and over the wiring in my home. FWIW. Edited November 26, 2008 by appyface Link to comment Share on other sites More sharing options...
floele Posted November 26, 2008 Report Share Posted November 26, 2008 Hm, does this version behave better? http://ketarin.canneverbe.com/downloads/Ketarin/Ketarin-0.9.9.2a2.zip Link to comment Share on other sites More sharing options...
appyface Posted November 27, 2008 Author Report Share Posted November 27, 2008 Different, but not better for me... This version highlights entries reasonably quicker than previous version as one-then-two {then long pause} three-then-four {then long pause} five {long pause} {six then-seven} etc. I don't mean in any particular order, just that it seems to grab two entries more quickly but then stays on them a lot longer. This version of Ketarin timed out five entries, three of which were the same three previous alpha version timed out on. To double-check I ran the 'regular' Ketarin and again there are no timeouts on any of the entries. The 'regular' Ketarin performed much faster in total runtime than the alpha version. None of my entries had new downloads today. So both the 'regular' and 'alpha' versions had similar work to do. The winner is still the 'regular' Ketarin, it says version 0.9.3.0 in the Help -> About but I'm reasonably certain it came from a zip file something like 0.9.9. I don't still have the zip or I could tell you. --appyface Link to comment Share on other sites More sharing options...
floele Posted November 27, 2008 Report Share Posted November 27, 2008 Hm, that's a little strange. For me, there is actually a notable difference. 1 thread (previous version) takes 53s, the current version with 2 threads is 41s. I'll look into a bit... Link to comment Share on other sites More sharing options...
floele Posted November 27, 2008 Report Share Posted November 27, 2008 Ah, I found another optimisation. Make sure to try the next version as well Link to comment Share on other sites More sharing options...
appyface Posted November 27, 2008 Author Report Share Posted November 27, 2008 OK I give up. Where is the next version? Link to comment Share on other sites More sharing options...
floele Posted November 28, 2008 Report Share Posted November 28, 2008 It's not yet anywhere Link to comment Share on other sites More sharing options...
appyface Posted November 28, 2008 Author Report Share Posted November 28, 2008 (edited) Not even in your head? Not that I can get in there... yet... working on it LOL Edited November 28, 2008 by appyface Link to comment Share on other sites More sharing options...
floele Posted November 28, 2008 Report Share Posted November 28, 2008 I released 0.9.9.3 now (still too lazy to update the version number in the about screen), which should work fine. Link to comment Share on other sites More sharing options...
appyface Posted December 1, 2008 Author Report Share Posted December 1, 2008 (edited) Hi Flo, I'm using 0.9.9.4 (based on ZIP file name :-) ), in general using the two default worker threads and zooming through the checking of entries. BUT is hanging worse than ever on that site where it was before hanging on two or three entries. 0.9.9.4 timed out on ALL the entries from that webpage. Other entries from that site downloaded fine. Previously-stable 0.9.9.2 ('regular') Ketarin does not hang on those entries. (None of the entries require downloads, just the checking for downloads.) Is there a logfile or other debug being produced that I can send you? I do not think multiple worker threads on that site is an issue, I have both Orbit and FDM set to use 10 and they each download from that site just fine. Ideas? TIA, --appyface Edited December 1, 2008 by appyface Link to comment Share on other sites More sharing options...
appyface Posted December 1, 2008 Author Report Share Posted December 1, 2008 (edited) P.S. The timeout requires a minute or more, not the default 10 seconds in the settings. FYI. P.S.S. Interesting. If I right-click and choose update for just ONE of those entries, 0.9.9.4 completes the job quickly. If I highlight two or more of those timeout entries, right-click and choose update, they timeout. Clue? Edited December 1, 2008 by appyface Link to comment Share on other sites More sharing options...
floele Posted December 1, 2008 Report Share Posted December 1, 2008 What site is it? Maybe it doesn't like multiple connections? Link to comment Share on other sites More sharing options...
appyface Posted December 1, 2008 Author Report Share Posted December 1, 2008 It's just the one page I'm scraping on that site, another page is fine. As I mentioned, I use multiple connections on that page with Orbit and FDM so I don't think that is the issue, it's something different about the multi-thread Ketarins and going after two or more at once. The single thread Ketarin does fine on that page, as does the multi-thread version if I only update one of them at a time. The page I scrape is: http://e-sword.net/bibles.html Scroll down to American Standard Version, I scrape the date from the right and use it to append to the output filename. The download URL is: http://www.e-sword.net/files/bibles/asv.exe I do the same for several more versions on that page. I can export my Ketarin db and send it to you if you like, if the above is not enough to get a good (e.g. failing) test. Let me know. Thanks, --appyface Link to comment Share on other sites More sharing options...
floele Posted December 4, 2008 Report Share Posted December 4, 2008 OK, I think I have found the problem. Try http://ketarin.canneverbe.com/downloads/Ketarin/Ketarin-0.9.9.5.zip Link to comment Share on other sites More sharing options...
appyface Posted December 5, 2008 Author Report Share Posted December 5, 2008 (edited) Hi Flo, First time execution using 'update now' button (we'll call this update all attempt #1), to do all entries, this Ketarin is scanning pages a LOT faster overall than any previous version. However, it timed out on two entries from the same problematic page I gave you above. It timed out very quickly now instead of hanging for a minute or two before timing out (I have timeout set to 10 seconds, seems to honor that now). e-sword bible gw God's Word translation The remote server returned an error: (503) Server Unavailable. e-sword bible mkjv modern king james version The remote server returned an error: (503) Server Unavailable. I used 'update now' button again (attempt #2). A page that was scanned successfully the first time (Serv-U) has timed out immediately, plus five entries from the problematic page timed out instead of two. Only one is in common (mkjv). Serv-U FTP Server The operation has timed out e-sword bible asv american standard version The remote server returned an error: (503) Server Unavailable. e-sword bible isv international standard version The remote server returned an error: (503) Server Unavailable. e-sword bible kjva king james version w/apocrypha The remote server returned an error: (503) Server Unavailable. e-sword bible mkjv modern king james version The remote server returned an error: (503) Server Unavailable. e-sword bible rv revised version The remote server returned an error: (503) Server Unavailable. 'Update now' button again (attempt #3) gives this: e-sword bible kjv king james version The remote server returned an error: (503) Server Unavailable. e-sword bible kjva king james version w/apocrypha The remote server returned an error: (503) Server Unavailable. e-sword bible mkjv modern king james version The remote server returned an error: (503) Server Unavailable. e-sword bible rv revised version The remote server returned an error: (503) Server Unavailable. Serv-U FTP Server The operation has timed out I next attempted to individually update one failure at a time (right click, choose 'update'), Serv-U timed out (keep in mind it scanned successfully, or so Ketarin said, the first time). The four above from e-sword all updated successfully. Then I tried Serv-U again and it updated successfully. 'Update Now' button (attempt #4) gave this: e-sword bible asv american standard version The remote server returned an error: (503) Server Unavailable. e-sword bible kjv king james version The remote server returned an error: (503) Server Unavailable. e-sword bible kjva king james version w/apocrypha The remote server returned an error: (503) Server Unavailable. e-sword bible litv literal translation of the holy bible The remote server returned an error: (503) Server Unavailable. e-sword bible rv revised version The remote server returned an error: (503) Server Unavailable. Serv-U FTP Server The operation has timed out Just for grins, I ran 'old' stable Ketarin 'update now' button, no timeouts on any entries. FYI. --appyface P.S. I should mention, on the problematic page from e-sword, there are 12 entries. So it isn't like Ketarin can't access that page at all.... Edited December 5, 2008 by appyface Link to comment Share on other sites More sharing options...
floele Posted December 5, 2008 Report Share Posted December 5, 2008 How many threads to you use? Link to comment Share on other sites More sharing options...
appyface Posted December 6, 2008 Author Report Share Posted December 6, 2008 Settings shows five. I don't recall if that is the default or if I changed it :-) In both FDM and Orbit I have 10 threads and the page is fine... Link to comment Share on other sites More sharing options...
floele Posted December 6, 2008 Report Share Posted December 6, 2008 503 is not really a timeout. It rather sounds like the server denies the connection, but I could't reproduce it yet. Link to comment Share on other sites More sharing options...
appyface Posted December 6, 2008 Author Report Share Posted December 6, 2008 Want my db export? Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now