Jump to content
Ketarin forum

shawn

Moderators
  • Posts

    1,179
  • Joined

  • Last visited

Posts posted by shawn

  1. Don't take this the wrong way, but addressing the issues with assigning header values this way is very funky, and I'm concerned about proper encoding.

     

    For example, cookies have multiple values separated by ampersands (potentially multiple on the same line) and carriage returns within the HTTP protocol. How do we accomplish this with the new httpx protocol? Will the ampersands be URL-encoded automatically, or do we need to do that ourselves? Do we need to include multiple "cookie:name=value" parameters?

     

    I really feel that this implementation is going to be a burden to maintain, and that your best shot is to use a datagrid, as you've done with global variable assignments and post data. Preferably with the ability to sort values in order of assignment, so the headers can be composed to comply with certain user-agent profiles (a common anti-spidering/anti-caching method is to enforce header sequence and conformity for popular UA's, and forbid other UA's). A datagrid that enabled us to assign both the sequence, form and structure of these headers would allow us to correctly obtain content from these strict sources.

  2. Tomorrow, here's a working uTorrent profile:

    <?xml version='1.0' encoding='utf-8'?>
    <Jobs>
     <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="98e52edb-3990-43de-b09d-06c5515c9d84">
       <SourceTemplate><![CDATA[]]></SourceTemplate>
       <WebsiteUrl>http://www.utorrent.com/</WebsiteUrl>
       <UserAgent>{ie32}</UserAgent>
       <UserNotes />
       <LastFileSize>328568</LastFileSize>
       <LastFileDate>2010-10-16T03:55:24.9037236</LastFileDate>
       <IgnoreFileInformation>false</IgnoreFileInformation>
       <DownloadBeta>Default</DownloadBeta>
       <DownloadDate>2009-05-20T22:20:53</DownloadDate>
       <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
       <VariableChangeIndicator>updated</VariableChangeIndicator>
       <CanBeShared>false</CanBeShared>
       <ShareApplication>false</ShareApplication>
       <ExclusiveDownload>false</ExclusiveDownload>
       <HttpReferer />
       <SetupInstructions />
       <Variables>
         <item>
           <key>
             <string>swebsite</string>
           </key>
           <value>
             <UrlVariable>
               <RegexRightToLeft>false</RegexRightToLeft>
               <VariableType>Textual</VariableType>
               <Regex />
               <TextualContent>http://www.utorrent.com/downloads</TextualContent>
               <Name>swebsite</Name>
             </UrlVariable>
           </value>
         </item>
         <item>
           <key>
             <string>schangelog</string>
           </key>
           <value>
             <UrlVariable>
               <RegexRightToLeft>false</RegexRightToLeft>
               <VariableType>RegularExpression</VariableType>
               <Regex>"([^"]+)">Changelog</Regex>
               <Url>http://www.utorrent.com/downloads/?</Url>
               <EndText>">Changelog</EndText>
               <Name>schangelog</Name>
             </UrlVariable>
           </value>
         </item>
         <item>
           <key>
             <string>snotes</string>
           </key>
           <value>
             <UrlVariable>
               <RegexRightToLeft>false</RegexRightToLeft>
               <VariableType>Textual</VariableType>
               <Regex />
               <TextualContent />
               <Name>snotes</Name>
             </UrlVariable>
           </value>
         </item>
         <item>
           <key>
             <string>updated</string>
           </key>
           <value>
             <UrlVariable>
               <RegexRightToLeft>false</RegexRightToLeft>
               <VariableType>RegularExpression</VariableType>
               <Regex>\(US\)\s*-\s*(\S+\s*\d+,\s*\d+)<</Regex>
               <Url>http://www.utorrent.com/downloads/?</Url>
               <Name>updated</Name>
             </UrlVariable>
           </value>
         </item>
         <item>
           <key>
             <string>aversion</string>
           </key>
           <value>
             <UrlVariable>
               <RegexRightToLeft>false</RegexRightToLeft>
               <VariableType>RegularExpression</VariableType>
               <Regex>/([\d\.]+)/[^"]+exe</Regex>
               <Url>http://www.utorrent.com/downloads/complete?os=win</Url>
               <Name>aversion</Name>
             </UrlVariable>
           </value>
         </item>
         <item>
           <key>
             <string>bversion</string>
           </key>
           <value>
             <UrlVariable>
               <RegexRightToLeft>false</RegexRightToLeft>
               <VariableType>RegularExpression</VariableType>
               <Regex>Version ([\d\.]+\s\(build\s*[\d]+)\)</Regex>
               <Url>{schangelog}&</Url>
               <Name>bversion</Name>
             </UrlVariable>
           </value>
         </item>
         <item>
           <key>
             <string>url</string>
           </key>
           <value>
             <UrlVariable>
               <RegexRightToLeft>false</RegexRightToLeft>
               <VariableType>RegularExpression</VariableType>
               <Regex>http://download.utorrent.com/[\d\.]{3,}/utorrent.exe</Regex>
               <Url>http://utorrent.com/downloads/complete?os=win</Url>
               <Name>url</Name>
             </UrlVariable>
           </value>
         </item>
         <item>
           <key>
             <string>version</string>
           </key>
           <value>
             <UrlVariable>
               <RegexRightToLeft>false</RegexRightToLeft>
               <VariableType>Textual</VariableType>
               <Regex />
               <TextualContent>{bversion:regexreplace:[^\d\.]+:.}</TextualContent>
               <Name>version</Name>
             </UrlVariable>
           </value>
         </item>
       </Variables>
       <ExecuteCommand />
       <ExecutePreCommand />
       <ExecuteCommandType>Batch</ExecuteCommandType>
       <ExecutePreCommandType>Batch</ExecutePreCommandType>
       <Category>Internet</Category>
       <SourceType>FixedUrl</SourceType>
       <DeletePreviousFile>true</DeletePreviousFile>
       <Enabled>true</Enabled>
       <FileHippoId />
       <LastUpdated>2010-10-16T03:55:24.9037236</LastUpdated>
       <TargetPath>.\{category}\{appname:regexreplace:([\s\t\r\n\-\\&\/]+):_}-{version:replace: build :.}.exe</TargetPath>
       <FixedDownloadUrl>http://download.utorrent.com/{aversion}/utorrent.exe</FixedDownloadUrl>
       <Name>uTorrent</Name>
     </ApplicationJob>
    </Jobs>

  3. I think this would be great, especially now that Google Reader has abandoned webpage monitoring. I think Omni is more after forking Ketarin to provide this functionality as an alternative tool, not doing this (webpage monitoring) in the core Ketarin. I know I would keep them separate in every way, even with the portability - they just don't belong in the same app.

     

    I've not used WebMon, but I'll check it out.

  4. Dunno how else to justify except to confirm changes without exiting window.

    In which situation would you do that?

     

    It's not necessarily a situation, but a user behavior. Many people *expect* to see it, so they get nervous when it isn't there.

     

    Personally, I could care less if an "apply" button exists, but it would be nice if there were a "dirty" indicator in the title bar (just an asterisk or something to indicate the changes haven't yet been saved).

  5. If you make the online database also hold extra data like "How is this program installed? Via unzipping or universal extractor" and it could then propagate their install instructions from whatever was defined in the wizard.

     

    Yep. This, along with other meta-data, such as a "dependencies" field which would include such things as global variable assignments, sourceforge account, 7z or ahk. each of these should be listed as a dependency of the app profile. Dependencies could be detected automatically from the XML.

     

    Also, I think you really need to settle on naming conventions. I get all hot and bothered every time I try to describe a single app profile. What's the correct terminology? The profile itself isn't actually the "app" so calling it an "app" really bothers me. And what do you call the XML exports for one or many profiles?

     

     

    I would still love if global variables would allow stuff like {category} if that would ever be possible

     

    Me, too! It would seriously minimize the amount of crap I have to prefix within many profile fields. Maybe global vars (or even, maybe, ALL variables) should be recursively processed in order to process all variables, even those that are built dynamically during the variable assignment process.

  6. the problem is when you need to assign a useragent to a bunch of records. currently, the only way is to manually assign it to each one as a static string. when i need to go change that (which I am sure I inevitably will), I have to find a way to first locate, then update all those records. it would be far easier to use global variables to assign the UA string you want, then just insert something like {curl}, {ie64} or {chrome} in the UA box. if I want to update the UA I have been using for chrome, I just have to edit the value in one place - global variables (where this kind of global option belongs).

  7. yep. that would be cool. i'd LOVE to be able to put links on my sites to enable users to monitor my apps (or other apps) with Ketarin. nifty.

     

    bug report (IMHO):

    the "useragent" field doesn't parse variables (contrast with the "referer" which does appear to parse vars).

  8. this is possible in the current beta, available here:

    http://ketarin.canneverbe.com/forum/viewtopic.php?id=557

     

    in a specific app profile, click the "advanced settings" tab. Here's a few UA strings to start you out:

    ie32    Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C)
    ie64    Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Win64; x64; Trident/4.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Media Center PC 5.0; SLCC1; Tablet PC 2.0; .NET4.0C)
    firefox    Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10
    opera    Opera/9.80 (Windows NT 6.1; U; en) Presto/2.6.30 Version/10.62
    chrome    Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3
    wget    wget/1.9+cvs-stable+(red+hat+modified)
    curl    pycurl/7.18.2
    

  9. No, i don't think an installer is a good idea UNLESS you're going to map a filetype/extension to the program. As it is, I *love* the simplicity and portability of the app, which would only be reduced by the inclusion of an installer.

     

    I do understand that it still wouldn't REQUIRE an installer, and that those of us that really prefer the portability of Ketarin could continue to use the simple "unzip" format -- but including an installer would likely conduse some people about the applications function and portability.

     

    However, I do like the idea of mapping a certain filetype to Ketarin. For example, when I open a URL with a content-type of "text/ketarin" (which is simply an XML formatted file like an app dump file) which opens directly as an import in Ketarin, possibly with a potential new element to provide the ability to immediately download+install+save the app profile, download only + save the app profile, or either of those two but operating as a temp, without storing the app profile permanently within Ketarin. On that, it'd be kinda nifty to be able to build the contents of the xml file in such a way as to refer to external files to import, locally or by URL (similar to the format of sitemaps)

  10. What you're actually seeing is a special behavior applied to SF for the curl and wget engines. Because SF is used as a distribution network for Linux libraries and other packages, allowing curl & wget a direct "pass" to the real files, effectively bypassing the whole framed and redirection distribution network.

     

    I really should have realized that before when I was creating the template, 'cause it could save a lot of time just by adding a wget or curl UA header to the SF apps. Sigh.

  11. Begin/End only works on certain HTML, too, though - so that's not really a good justification to exclude XSL. Even Regex is only useful if you can find a *unique* pattern that works for the specific data you're trying to capture. XSL is really a useful tool when it *can* work, and if it becomes an option within Ketarin, it could be very useful to those geeks like me that will find ways to make it work. :)

     

    Also, if it's implemented as a function, instead of only a capture type, we would be able to utilize it with well-formed data once we've mostly sanitized it using other methods.

     

    That said, I would really like the simplicity of using the IE "slice" method of simply tagging the returned region with an anchor reference in the URL.

  12. Maybe I was a little too excited. All SF apps need updated again, and again, and again...

    10/12/2010 2:30:43 AM: 7-Zip cmd: Update required, file modified dates do not match
    ...
    10/12/2010 2:30:45 AM: 7-Zip Portable: Checking if update is required...
    10/12/2010 2:30:45 AM: 7-Zip Portable: Update required, file modified dates do not match

     

    The SF behavior is broken in this sense. The anticipated behavior is that it *should* be updating the "file modified date" on local files to ensure that the timestamps are accurate with the remote (and to better enable sorting and stuff).

     

    Meanwhile, I can use a manually-assigned "version" variable check for changes, but this would have to be updated on all my SF apps, which would be a PITA to add and remove. :(

  13. Yay!

     

    I added a global variable "sfprefix" with my SF login credentials in this format:

    sf://username:password@

    ...then added that variable to prefix the "download" field for all my SF downloads, as so.

    {sfprefix}{dl}

    {sfprefix}http://downloads.sourceforge.net/project/xbmc/XBMC%20for%20Windows/{tag}%20-%20{version}/xbmc-{version}.exe

    {sfprefix}http://downloads.sourceforge.net/project/dqsd/dqsd/{filename}

     

    They ALL work!

     

    Some modification to my apps were required for a couple apps that returned the "Source file on server is empty (ContentLength = 0)" error. This indicates that the URL is *wrong* for these files, so correcting the regex capture to eliminate extraneous characters resolved it in every case.

  14. Thanks, Gozi, but I'm actually more interested in obtaining a random source link from a regex-matching pattern from an HTML result. Such as the download source lists on MySQL, IrfanView, and similar sites where the list uses something much different than an appended number to identify the different sources.

  15. when attempting to post a reply a moment ago I received a MySQL error.

     

    An error was encountered
    --------------------------------------------------------------------------------
    Unable to connect to MySQL and select database. MySQL reported: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
    
    The error occurred on line 44 in [path removed]/mysqli.php

     

    This is common when mysql exceeds the number of allowed connections, the hdd is failing, during updates, or when the amount of memory (or other resources) allocated to MySQL is insufficient.

     

    I don't know what level of control you have over this, but if you are interested in free hosting (on resource-safe servers), I will provide.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.