4

I have created a PHP script which scrapes some particular data from a website, looks for the relevant information and store this in a database. The script works fine on the local machine but when i run it on a live server using cron job, it stops after doing 10 database inserts saying

"Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 35 bytes) in /home/content/36/78632936/html/scripts/simple_html_dom.php on line 809
"

I checked the same script on my local using get_memory_peak_usage() and the maximum usage comes around 8 MB .

I am puzzled as to why the memory usage is spectacularly high on the live machine.. Any help will be appreciated..

Glimpse of my code:

DB_table1: contains list of 60000 rows of data

Main code starts with calling the DB1 and get the data and then use each row of data to form an URL. Each of these URLs will be scraped using simple_html_dom() function and the script will look for some particular information on each page and store this data in another table DB_table2.

Please let me know if you need to know anything else . Thanks :)

Improve this question
5
  • 1
    What OS, php version and mysql version are the live and the local server using?
    – gnur
    Aug 25, 2011 at 14:07
  • are you running the latest version of php? :)
    – dan
    Aug 25, 2011 at 14:07
  • 2
    This is happy guessing as long as we do not see what happens in simple_html_dom.php at line 809 (and the lines above and below that, ideally the whole block...)
    – ty812
    Aug 25, 2011 at 14:08
  • Do you scrape the whole website and then start doing inserts? or you insert-as-you-go?
    – Quamis
    Aug 25, 2011 at 14:08
  • I just noticed.. My production server uses PHP version 5.2 while my localhost is on 5.3.1 .. Can this be the sole reason of this difference?
    – ankit
    Aug 25, 2011 at 15:21

2 Answers 2

Reset to default
0

Don't forget to use unset() after while/foreach to cleanup no longer required (DB) results.

Improve this answer
0

Update:
Looking at the php change log there are a lot of memory issues that are address. If you are using part of PHP that has a leak in it in 5.2 it is possible that could be a cause. If you want to know for sure try wrapping memory intensive part of your code with differences in memory_get_usage and see if there are any discrepancies between the memory usage.

Question/Answer:

afuzzyllama: Is your production environment the same as your local environment? It is possible your production db has a lot more data than your sandbox.

user743914: Unfortunately I didn't notice the difference previously, My production server is running php version 5.2 while my localhost is on 5.3.1 ... Production DB has same amount of data as that of localhost...

Simple solution:
What is the maximum memory limit your server allows?

Search for 'memory_limit' on this page [php.net].

As you see in the docs, depending on your version of PHP your limit could be as low as 8mb. I will guess that your limit is set to ~64M right now since your script is failing at ~67mb.

If you need to raise it temporarily you can use ini_set() or you can set it in your php.ini file and restart your server.

Improve this answer
1
  • Thanks a lot for your input...Unfortunately I didn't notice the difference previously, My production server is running php version 5.2 while my localhost is on 5.3.1 ... Production DB has same amount of data as that of localhost... Can the version difference be the sole reason of the memory usage difference? I am using unset at each step wherever possible..
    – ankit
    Aug 25, 2011 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.