addons.mozilla.org high CPU usage due to regex [SOLVED]

[Tomorrow]

I had this problem once before.So i re-did all the regex rules and that fixed it.Until now.

Im assuming they changed the website again and when i click update all apps it literally drags my system to a crawl.

 

Has anyone run into this too?

I have 2 variables for each app (examples):

 

changelog @ https://addons.mozilla.org/en-US/firefox/addon/1865/versions/'>https://addons.mozilla.org/en-US/firefox/addon/1865/versions/ with:

(?<=<div class="prose userinput".*?>).+?(?=\<\/div\>)

 

prettyversion @ https://addons.mozilla.org/en-US/firefox/addon/1865 with:

(?-s:(\d+(?:\.\d+)+.*?|\d b[a-z]* ?\d+|\d+ sp\w+|r\d+)(?= ?([<\[\]]))|(?-s:20\d\d-\d\d-\d\d))

 

Template:

<?xml version='1.0' encoding='utf-8'?>
<Jobs>
 <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Guid="ba766f51-4941-47ae-bbe7-dd37509fc161">
   <WebsiteUrl>https://addons.mozilla.org/en-US/firefox/addon/1865/</WebsiteUrl>
   <UserAgent />
   <UserNotes>Changelog:
14.07.2011 - Moved to regex based version capturing.
18.08.2011 - Changed version capture regex as the old one stopped working and was causing high CPU usage.</UserNotes>
   <LastFileSize>627675</LastFileSize>
   <LastFileDate>2011-09-30T10:11:09.6786866</LastFileDate>
   <IgnoreFileInformation>true</IgnoreFileInformation>
   <DownloadBeta>AlwaysDownload</DownloadBeta>
   <DownloadDate xsi:nil="true" />
   <CheckForUpdatesOnly>false</CheckForUpdatesOnly>
   <VariableChangeIndicator />
   <CanBeShared>true</CanBeShared>
   <ShareApplication>true</ShareApplication>
   <ExclusiveDownload>false</ExclusiveDownload>
   <HttpReferer />
   <SetupInstructions>
     <SetupInstruction xsi:type="StartProcessInstruction">
       <EnvironmentVariables />
       <FileName>"{file}"</FileName>
       <Parameters />
       <WaitForExit>true</WaitForExit>
     </SetupInstruction>
   </SetupInstructions>
   <Variables>
     <item>
       <key>
         <string>changelog</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>(?<=<div class="prose userinput".*?>).+?(?=\<\/div\>)</Regex>
           <Url>https://addons.mozilla.org/en-US/firefox/addon/1865/versions/</Url>
           <StartText><blockquote>
   <div class="prose userinput" >
     </StartText>
           <EndText><br><br>Detailed changelog: <a href="http://outgoing.mozilla.org/v1/d2308aa5f75f84c19709c30d5ac9c6c84c106865/http%3A//adblockplus.org/en/changelog-</EndText>
           <Name>changelog</Name>
         </UrlVariable>
       </value>
     </item>
     <item>
       <key>
         <string>prettyversion</string>
       </key>
       <value>
         <UrlVariable>
           <RegexRightToLeft>false</RegexRightToLeft>
           <VariableType>RegularExpression</VariableType>
           <Regex>(?-s:(\d+(?:\.\d+)+.*?|\d b[a-z]* ?\d+|\d+ sp\w+|r\d+)(?= ?([<\[\]]))|(?-s:20\d\d-\d\d-\d\d))</Regex>
           <Url>https://addons.mozilla.org/en-US/firefox/addon/1865</Url>
           <Name>prettyversion</Name>
         </UrlVariable>
       </value>
     </item>
   </Variables>
   <ExecuteCommand>echo {changelog:regexreplace:[\n|\r]:}>>"{Apps x86p}\Ketarin\CHANGELOG.txt"
echo ---------------------------------------------------->>"{Apps x86p}\Ketarin\CHANGELOG.txt"</ExecuteCommand>
   <ExecutePreCommand />
   <ExecuteCommandType>Batch</ExecuteCommandType>
   <ExecutePreCommandType>Batch</ExecutePreCommandType>
   <Category>Firefox Addon</Category>
   <SourceType>FixedUrl</SourceType>
   <PreviousLocation>D:\Downloads\Firefox Backups\Addons\[Adblock Plus] v.1.3.10.xpi</PreviousLocation>
   <DeletePreviousFile>true</DeletePreviousFile>
   <Enabled>true</Enabled>
   <FileHippoId />
   <LastUpdated>2011-09-30T10:11:09.6786866</LastUpdated>
   <TargetPath>{Firefox Backups}\Addons\{appname} v.{prettyversion}.{url:ext}</TargetPath>
   <FixedDownloadUrl>https://addons.mozilla.org/en-US/firefox/downloads/latest/1865</FixedDownloadUrl>
   <Name>[Adblock Plus]</Name>
 </ApplicationJob>
</Jobs>

 

The problem may not be so apparent with a single app but try 32 while ketarin checks 20 at a time...

 

Im considering moving to a simpler regex on these apps:

(?<=)[0-9a-zA-Z. ]{1,}(?=)

Edited October 5, 2011 by Tomorrow

[shawn]

This is wrong:

(?<=<div class="prose userinput".*?>).+?(?=\<\/div\>)

The classes have changed to "desc prose", and I would also personally recommend changing it to this to be simpler and also trim unnecessary whitespace:

<div class="desc prose".*?>\s*(.+?)\s*</div>

 

And if you don't mind me saying so, this is another perfect example of where using a template would be ideal. The template would make updates to all 32 profiles to replace this simple regex, while also making it extremely easy to add another addon to your monitoring list.

 

As far as the delays you'rs experiencing, I'm just not seeing it.

 

HOWEVER, I did notice that this profile is using SSL, and I've seen about a dozen reports in forums, security blogs and from clients with SSL-related slowdowns/crashes over the last 3 days, and I suspect this is the case with you, as well. I personally haven't had any problems, but I use OpenDNS on all my hardware (love it!), and that's protected me from a number of different MITM attacks such as these DNS poisoning situations. Give it a shot and see if that resolves your problems. You MAY have to reboot for Ketarin to fully purge the DNS resolver cache (IE9+ does).

[Tomorrow]

Thanks.I'll test that regex on monday or tuesday as i really don't want to do this on a weekend.

 

EDIT 05.10.2011:

 

Thanks shawn.The nonworking changelog regex was at fault.By using your fixed one i got rid of the high CPU usage.

Edited October 5, 2011 by Tomorrow