Jump to content
Ketarin forum

No caching 3


Recommended Posts

Guest guest

Often I need to extract several pieces of information from one and the same site for one application. I always assumed Ketarin would do clever caching so to load the site only once even if I access it through several variables. However now I checked and it doesn't. (I guess that's because all variables try to access the same site almost simultaneously before the site has fully loaded.)

What's the best practice here?

Save the complete site in a variable and then use the regexp function several times to extract the needed data?

Link to post
Share on other sites
  • 3 weeks later...
  • 1 month later...
Guest guest

Saving to a variable doesn't work either: everytime I access the variable it resolves its content again, i.e. it accesses the website again.

Some websites I use have a quota and accessing the same content several times is counter-productive. Or maybe I'm just to stupid to do it right.

For example, how would I make Ketarin read out the date and users of all posts of this conversation while accessing this site only once?

Link to post
Share on other sites
  • 3 weeks later...

In order to provide feedback for a specific problem it's always best to share an app XML profile so we can see what you're doing. Select it within the Ketarin list and press CTRL+C to copy it and paste it in a `code` block here.

The problem you describe is one I've personally experienced, though. The fix is obvious once you realize it: store the contents of the entire page in a single variable and use variable parsing functions to extract the data you need from the new variable. You can use a regex type variable with ".+" to collect the entire page contents. Use variables of type textual to parse subsequent variables.

Link to post
Share on other sites
Guest guest

Here's an example where Ketarin downloads the same padfile over and over again just because I extract different pieces of information from it using the regex function.

How can I prevent this?

(Note: This is just for demonstrating the issue. So please no answers about how to use a padfile properly.)

<?xml version='1.0' encoding='utf-8'?>
<Jobs>
  <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="7d17b202-0aa8-49aa-b33f-e5f21941eb07">
    <WebsiteUrl />
    <UserAgent />
    <UserNotes />
    <LastFileSize>0</LastFileSize>
    <LastFileDate xsi:nil="true" />
    <IgnoreFileInformation>false</IgnoreFileInformation>
    <DownloadBeta>Default</DownloadBeta>
    <DownloadDate xsi:nil="true" />
    <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
    <VariableChangeIndicator>result</VariableChangeIndicator>
    <HashVariable />
    <HashType>None</HashType>
    <CanBeShared>true</CanBeShared>
    <ShareApplication>false</ShareApplication>
    <ExclusiveDownload>false</ExclusiveDownload>
    <HttpReferer />
    <SetupInstructions />
    <Variables>
      <item>
        <key>
          <string>cost</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Cost_Dollars&gt;(.+?)&lt;/Program_Cost_Dollars&gt;:1}</TextualContent>
            <Name>cost</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>padfile</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>RegularExpression</VariableType>
            <Regex>.+</Regex>
            <Url>https://ketarin.org/padfile</Url>
            <Name>padfile</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>result</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{version}|{month}|{day}|{year}|{cost}|{type}|{status}|{support}|{os}|{lang}</TextualContent>
            <Name>result</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>version</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Version&gt;(.+?)&lt;/Program_Version&gt;:1}</TextualContent>
            <Name>version</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>month</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Release_Month&gt;(.+?)&lt;/Program_Release_Month&gt;:1}</TextualContent>
            <Name>month</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>day</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Release_Day&gt;(.+?)&lt;/Program_Release_Day&gt;:1}</TextualContent>
            <Name>day</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>year</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Release_Year&gt;(.+?)&lt;/Program_Release_Year&gt;:1}</TextualContent>
            <Name>year</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>type</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Type&gt;(.+?)&lt;/Program_Type&gt;:1}</TextualContent>
            <Name>type</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>status</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Release_Status&gt;(.+?)&lt;/Program_Releas_Status&gt;:1}</TextualContent>
            <Name>status</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>support</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Install_Support&gt;(.+?)&lt;/Program_Install_Support&gt;:1}</TextualContent>
            <Name>support</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>os</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_OS_Support&gt;(.+?)&lt;/Program_OS_Support&gt;:1}</TextualContent>
            <Name>os</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>lang</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Language&gt;(.+?)&lt;/Program_Language&gt;:1}</TextualContent>
            <Name>lang</Name>
          </UrlVariable>
        </value>
      </item>
    </Variables>
    <ExecuteCommand />
    <ExecutePreCommand />
    <ExecuteCommandType>Batch</ExecuteCommandType>
    <ExecutePreCommandType>Batch</ExecutePreCommandType>
    <Category />
    <SourceType>FixedUrl</SourceType>
    <PreviousLocation />
    <DeletePreviousFile>true</DeletePreviousFile>
    <Enabled>true</Enabled>
    <FileHippoId />
    <LastUpdated xsi:nil="true" />
    <TargetPath />
    <FixedDownloadUrl />
    <Name>No caching example</Name>
  </ApplicationJob>
</Jobs>

 

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.