Magento Import Catalog Products A Horror Story

It's been so many days i have been spending my time trying to figure out how to import products using CSV in Magento.

I recommended client to go for better server and as a result we moved to VPS. But that didn't help. Hosting company was so generous to work along with me to figure out what was actually wrong. They also spent many hours and haven't charged for VPS yet and I am free to use the server as I need. But I am not that good in server settings, but learnt a lot along the way :)

ini_set('memory_limit', '128M');

I increased memory limit in so many files that import uses but it was always failing halfway. Also, i increased memory through php.ini and .htaccess file.

I was so frustrated. And found another option to increase memory, CPU and Proc using rlimit as:

RLimitMEM 1280000000
RLimitCPU 150 200
RLimitNPROC 25 30

That helped a lot in memory but CPU and PROC were throwing errors "Must be uid 0 to raise maximum RLimitNPROC" and "Must be uid 0 to raise maximum RLimitCPU".

The hosting company asked to add those on apache config file. So, when i was going to add that I remembered there was option to set number of records to update per update while importing.

I couldn't remember where i could set "number_of_records". I guess i set it in file somewhere, but can't find anywhere.

Then i remembered when i was looking into profile information of Import All Products, I saw 10000 on "Number of records" field. And, that time it was failing to import and we had 16000 records so i had increased to 30000 on that field as i didn't realise or didn't spend time on finding it's use.

So, today went back to that field and updated it to 2 records to see if that helps, wohoo! finally import is running like a leopard :)

Finally i can move to another project that i love to work on and my days of horror are over.

FYI: you can increase number of records to max your server can handle like 500 or 1000 etc.

Location:

Magento Admin >> System >> Import/Export >> Dataflow - Profiles >> Import All Products >> Profile Wizard >> Profile Information >> Number of records.

This is what my "Profile Actions XML" tab looks after that update.

<action type="dataflow/convert_parser_csv" method="parse">
   <var name="delimiter"><![CDATA[,]]></var>
      <var name="enclose"><![CDATA["]]></var>
      <var name="fieldnames">true</var>
      <var name="store"><![CDATA[0]]></var>
      <var name="number_of_records">2</var>
      <var name="decimal_separator"><![CDATA[.]]></var>
      <var name="adapter">catalog/convert_adapter_product</var>
      <var name="method">parse</var>
</action>