Jump to content
Ketarin forum

No caching 3


Guest guest
 Share

Recommended Posts

Guest guest

Often I need to extract several pieces of information from one and the same site for one application. I always assumed Ketarin would do clever caching so to load the site only once even if I access it through several variables. However now I checked and it doesn't. (I guess that's because all variables try to access the same site almost simultaneously before the site has fully loaded.)

What's the best practice here?

Save the complete site in a variable and then use the regexp function several times to extract the needed data?

Link to comment
Share on other sites

  • 3 weeks later...
  • 1 month later...

Saving to a variable doesn't work either: everytime I access the variable it resolves its content again, i.e. it accesses the website again.

Some websites I use have a quota and accessing the same content several times is counter-productive. Or maybe I'm just to stupid to do it right.

For example, how would I make Ketarin read out the date and users of all posts of this conversation while accessing this site only once?

Link to comment
Share on other sites

  • 3 weeks later...

In order to provide feedback for a specific problem it's always best to share an app XML profile so we can see what you're doing. Select it within the Ketarin list and press CTRL+C to copy it and paste it in a `code` block here.

The problem you describe is one I've personally experienced, though. The fix is obvious once you realize it: store the contents of the entire page in a single variable and use variable parsing functions to extract the data you need from the new variable. You can use a regex type variable with ".+" to collect the entire page contents. Use variables of type textual to parse subsequent variables.

Link to comment
Share on other sites

Here's an example where Ketarin downloads the same padfile over and over again just because I extract different pieces of information from it using the regex function.

How can I prevent this?

(Note: This is just for demonstrating the issue. So please no answers about how to use a padfile properly.)

<?xml version='1.0' encoding='utf-8'?>
<Jobs>
  <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="7d17b202-0aa8-49aa-b33f-e5f21941eb07">
    <WebsiteUrl />
    <UserAgent />
    <UserNotes />
    <LastFileSize>0</LastFileSize>
    <LastFileDate xsi:nil="true" />
    <IgnoreFileInformation>false</IgnoreFileInformation>
    <DownloadBeta>Default</DownloadBeta>
    <DownloadDate xsi:nil="true" />
    <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
    <VariableChangeIndicator>result</VariableChangeIndicator>
    <HashVariable />
    <HashType>None</HashType>
    <CanBeShared>true</CanBeShared>
    <ShareApplication>false</ShareApplication>
    <ExclusiveDownload>false</ExclusiveDownload>
    <HttpReferer />
    <SetupInstructions />
    <Variables>
      <item>
        <key>
          <string>cost</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Cost_Dollars&gt;(.+?)&lt;/Program_Cost_Dollars&gt;:1}</TextualContent>
            <Name>cost</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>padfile</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>RegularExpression</VariableType>
            <Regex>.+</Regex>
            <Url>https://ketarin.org/padfile</Url>
            <Name>padfile</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>result</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{version}|{month}|{day}|{year}|{cost}|{type}|{status}|{support}|{os}|{lang}</TextualContent>
            <Name>result</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>version</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Version&gt;(.+?)&lt;/Program_Version&gt;:1}</TextualContent>
            <Name>version</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>month</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Release_Month&gt;(.+?)&lt;/Program_Release_Month&gt;:1}</TextualContent>
            <Name>month</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>day</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Release_Day&gt;(.+?)&lt;/Program_Release_Day&gt;:1}</TextualContent>
            <Name>day</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>year</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Release_Year&gt;(.+?)&lt;/Program_Release_Year&gt;:1}</TextualContent>
            <Name>year</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>type</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Type&gt;(.+?)&lt;/Program_Type&gt;:1}</TextualContent>
            <Name>type</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>status</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Release_Status&gt;(.+?)&lt;/Program_Releas_Status&gt;:1}</TextualContent>
            <Name>status</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>support</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Install_Support&gt;(.+?)&lt;/Program_Install_Support&gt;:1}</TextualContent>
            <Name>support</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>os</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_OS_Support&gt;(.+?)&lt;/Program_OS_Support&gt;:1}</TextualContent>
            <Name>os</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>lang</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>{padfile:regex:&lt;Program_Language&gt;(.+?)&lt;/Program_Language&gt;:1}</TextualContent>
            <Name>lang</Name>
          </UrlVariable>
        </value>
      </item>
    </Variables>
    <ExecuteCommand />
    <ExecutePreCommand />
    <ExecuteCommandType>Batch</ExecuteCommandType>
    <ExecutePreCommandType>Batch</ExecutePreCommandType>
    <Category />
    <SourceType>FixedUrl</SourceType>
    <PreviousLocation />
    <DeletePreviousFile>true</DeletePreviousFile>
    <Enabled>true</Enabled>
    <FileHippoId />
    <LastUpdated xsi:nil="true" />
    <TargetPath />
    <FixedDownloadUrl />
    <Name>No caching example</Name>
  </ApplicationJob>
</Jobs>

 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.