Author - Vinai Kopp

I am a passionate Web Developer and open source enthusiast since 1998. Today my main focus in on Training Developers how to customize Magento.
Since March 2008 I have specialized on the Magento platform, which, at the time, still was in the later Beta stages.
Working as a freelancer for my own clients and in large teams I was able to gather valuable experience in projects ranging from very small to quite large.
This enabled me to move into training Developers how to effectively customize Magento.
Since then I have spent thousands of hours giving trainings, both in-house and open to everybody, in all aspects of the Magento platform.

Since March 2010 I also am a Instructor for the official Magento U developer trainings for Magento.
I spent two years as an employed trainer for Magento, but decided I prefer the freedom to focus on projects I choses myself, and now work as a freelancer again since
January 2014.
I am a member of the Magento Certification Advisory Board and was one of the first to pass the MCD+ exam.
Also, I am an co-author of the German Magento Developer Handbook published by O'Reilly in 2010.

More Info » Follow me on Twitter »

Reader Comments (137)

  1. Fabrizio Branca
    August 2, 2011 at 7:16 pm /

    Great post, Vinai! Nice to see that you’ve found a nice solution to the shortcomings of the file cache backend. I will definetly try your extension.

    Bye, Fabrizio 🙂

    Reply
  2. Marc Jakubowski
    August 2, 2011 at 8:12 pm /

    Nice idea Vinai!
    Will try it out today and provide some feedback.

    Reply
  3. oliver.brildor
    August 2, 2011 at 8:40 pm /

    Netzarbeiter_Cache.0.0.2.tgz from magento connect not contain “code / community / Netzarbeiter” files!! downloads from github for good choise.

    Reply
    1. oliver.brildor
      August 2, 2011 at 9:13 pm /

      Created 897 symlinks create own magento store dev site, 40% performance improve! 🙂

      I found an error when trying compiler via shell “php compiler.php enable”

      PHP Fatal error: Uncaught exception ‘Zend_Cache_Exception’ with message ‘cache_dir must be a directory’

      more info: https://github.com/Vinai/Symlink-Cache/issues/1

      Great module, thx 🙂

      Reply
  4. Marc Jakubowski
    August 3, 2011 at 12:38 am /

    I created a little test script that creates 50.000 cache entries with 2 to 5 random tags from a 10 tag list.
    The symlink approach was rather slower than the pure file backend.

    File Backend load IDs by tag:
    IDs by tag “TAG_0”: 3.674868
    IDs by tag “TAG_1”: 3.958485
    IDs by tag “TAG_2”: 4.082589
    IDs by tag “TAG_3”: 4.274734
    IDs by tag “TAG_4”: 4.153434
    IDs by tag “TAG_5”: 4.499756
    IDs by tag “TAG_6”: 4.194537
    IDs by tag “TAG_7”: 4.182449
    IDs by tag “TAG_8”: 4.194028
    IDs by tag “TAG_9”: 4.153169

    Symlink Backend load IDs by tag:
    IDs by tag “TAG_0”: 4.073611
    IDs by tag “TAG_1”: 11.370826
    IDs by tag “TAG_2”: 11.975922
    IDs by tag “TAG_3”: 7.671180
    IDs by tag “TAG_4”: 6.697377
    IDs by tag “TAG_5”: 12.255260
    IDs by tag “TAG_6”: 7.763530
    IDs by tag “TAG_7”: 8.118395
    IDs by tag “TAG_8”: 6.722158
    IDs by tag “TAG_9”: 5.552058

    The symlink approach got faster / more stable times when repeated a few times. I guess this is due to the filesystem cache when listing files. Maybe you even get other results when using other filesystems.

    Reply
  5. Tim Reynolds
    August 3, 2011 at 3:13 am /

    Vinai,

    Very interesting write-up. I had no problem using the regular backend, but when I set it as the slow backend with apc as my frontend I get the following error in my apache2 error.log:

    PHP Fatal error:  Class 'Netzarbeiter_Cache_Model_Symlink' not found in /mnt/hgfs/vaporpro/html/upgrade/lib/Zend/Cache.php on line 153, referer: http://www.thevaporpro.com/upgrade/index.php/admin/cache/

    Note, I am using roughly the latest version of Pro, not Community.
    My config in local.xml looks like:

    <cache>
    	<backend>apc</backend>
    	<slow_backend>Netzarbeiter_Cache_Model_Symlink</slow_backend>
    </cache>
    

    Thanks
    Tim

    Reply
  6. Fabrizio Branca
    August 3, 2011 at 5:34 am /

    Hi Vinai,

    I tried your extension and it works fine. Up to now my solution to this problem was to not delete the cache at all, but to save the delete command in a queue and process this queue asynchroniously by a cron task. See (http://www.fabrizio-branca.de/magento-asynchronous-cache.html). But obviously this approach has some drawbacks if you need to rely on cache entries to be deleted suddently.

    Your solution looks like a perfect solution to this. Your approach is to create an index of the tags by creating directories for each tag and linking the original files to this directory. With 40k of raw cache file this might add up to more than 100k files (incl. symlinks), doesn’t it?

    Why not use other storages to store the relation of the tags and cache entries? Each key-value-store would be fine, wouldn’t it? So why didn’t you create a normal cache entry to store comma-separated list of filenames of the related files for each tag?

    Reply
    1. Marc Jakubowski
      August 3, 2011 at 9:39 am /

      The problem with tag->IDs lists is that they are hard to maintain.
      While it’s pretty easy to append new entries to a tag file, you would have a lot duplicates in it and you would have to keep track of IDs that change their tags on every save and crop those IDs out of the list.
      One extreme example would be the FullPageCache where you have up to 100 tags per page that change regularly, because every single model object that is loaded on one page would result in an additional tag.

      Reply
  7. Marc Jakubowski
    August 3, 2011 at 9:52 am /

    Today I managed to implement Tagging with memcached by using a fairly clever approach that results in invalidating tags _instantly_.
    I hope I will be allowed to share the code with the community. At least I will be able to describe the concept, if people are interested in it.

    Reply
  8. Improving the File Cache Backend – Extensions – Magebase | Magento Training Course

    […] developers, and to share one method of overcoming some limitations of the file cache storage class. Link – Trackbacks source: Topsy – magento tutorial – Improving the File Cache Backend […]

  9. Roland
    August 4, 2011 at 12:27 am /

    When I go to System -> Tools -> Symlink Cache, I see an empty Magento page. So there is no error, magento’s header and footer are loaded, but no content on the page. Running on Enterprise 1.8.0.0.

    I turned on developer mode and display errors, but still nothing.

    Also checked that APC is active.

    Reply
  10. Links 31/2011: Magento, e-Commerce und PHP | Matthias Zeis

    […] Kopp hat sich in einem Gast-Posting bei Magebase mit dem Caching beschäftigt und beschreibt, wie man das File-Caching-Backend verbessern kann.Damian und Tobias haben eine weitere Ausgabe des sympathischen Magento-Podcasts produziert. Auf […]

  11. Karol
    Karol
    August 11, 2011 at 6:47 pm /

    hello, is it working with 1.4.1 ?

    Reply
  12. Paul Bates
    August 12, 2011 at 8:32 am /

    Very interesting post, is the extension limited to Magento 1.5 use? I thought it wise to ask before I give it a try.

    Reply
    1. Karol
      Karol
      August 12, 2011 at 8:04 pm /

      i installed it on 1.4.1. It seems to work ? Any errors. Extension made a lot of directories in var/tabs/ … but they are empty. And I dont see any speed imporovement, and files in cache.

      Reply
      1. Paul Bates
        August 12, 2011 at 9:47 pm /

        Karol – Did you go to the Symlink Cache page and hit the “Initialize Tag Symlinks” button?

        Reply
  13. Paul Bates
    August 12, 2011 at 9:44 pm /

    Yes it works on 1.4.1.1.

    I did seem to have instant performance gains, my test server has always been historically slow and it was very refreshing to see pages load within seconds. (No APC, No image CDN etc.)

    My live server operates in the RackSpace Cloud environment with APC, Fooman and all the other performance mods which provides anything from page loads from 0.8 – 2 seconds. This has been helped along by One PICA’s image CDN which is also being served by RackSpace’s Cloud Files – So I am looking to test this out in this environment to see what happens. I will post my results.

    That’s the good news now for some not so good news. As with all things they need to be broken to see where the limitations lie and it was relatively easy to break my website – Just by running Compiler, not that I use Compiler! My test site is now in a very sorry state with total lockdown and no access to admin, which means a few hours to bring it back to life and start testing again. So a word of caution backup.

    Reply
    1. Karol
      Karol
      August 12, 2011 at 9:54 pm /

      yes, i hit this button. do you have any files in /var/tags/ directories ? i have not. All new directories are empty.
      Do you change your local.xml ?
      i have something like this

      apc
      MAGENTOO_
      Netzarbeiter_Cache_Model_Symlink

      Reply
      1. Paul Bates
        August 12, 2011 at 10:17 pm /

        Hello Karol, Yes you have to insert the code in local.xml. There are two options of code types, if you are using APC it’s the 2nd code and none APC the first. Once inserted click the Initialize Tag Symlinks. You can keep pressing the Initialize Tag Symlinks as pages are cached and it will keep building symlinks.

        Reply
      2. Jerome
        Jerome
        February 2, 2012 at 9:59 pm /

        Hello

        Yes the new directories “tags” are empty via ftp like fizilla but the directories are not empty.. you can see the files in directory with WinSCP by example

        Reply
  14. Karol
    Karol
    August 12, 2011 at 10:30 pm /

    Hej, Paul look for my screen please

    it seems that config and creation are correct.

    after this i have still empty directories in var/cache/tags/….

    do you have any files in /var/tags/ directories ?

    i dont know how to test that extension works….

    Reply
    1. Paul Bates
      August 12, 2011 at 10:39 pm /

      Looks to be functioning correctly from the screen grab.

      I do have files in the var/cache/tabs folder.

      For testing I simply compared page load times from the unmodified site to the modified version in different web browsers (just in case Chrome was giving faster page loads as standard).

      Are you using APC? Can you post how your code index.xml file (without the sensitive data).

      Reply
      1. Karol
        Karol
        August 12, 2011 at 10:46 pm /
        {
        
         		<cache>
        			<backend>apc</backend>
        			<prefix>MAGENTOO_</prefix>
        			<slow_backend>Netzarbeiter_Cache_Model_Symlink</slow_backend>			
        		</cache>
        
        
        }
        Reply
        1. Paul Bates
          August 12, 2011 at 11:01 pm /

          Looks correct to me Karol. I did notice that Tim used…

          <cache>
              <backend>apc</backend>
              <slow_backend>Netzarbeiter_Cache_Model_Symlink</slow_backend>
          </cache>

          without the

          <prefix>

          I trust you are on a test server not a live environment, perhaps it’s worth dropping Mr Kopp an email to see if he can shed some light on the issue. Would be good to know the result. WHATEVER you do, do not run Compiler!!!

          Reply
          1. Vinai Kopp
            October 12, 2011 at 2:27 am /

            Hi Karol, hi Paul,

            thanks for your comments!
            The is only needed if several Magento instances share one memory based cache backend, i.e. APC or Memcached.
            I will not support the Compiler because I prefere using APC with apc.stat = 0, which does the same thing as the compiler only better 🙂
            However, if someone wants to look into the compiler issue and submit a patch I’ll be happy to integrate it.

            Reply
  15. Srdjan
    August 19, 2011 at 4:19 pm /

    Thanks for sharing this. Will try it out

    Reply
  16. Colin Mollenhour
    August 26, 2011 at 12:32 pm /

    Nice work Vinai! I’ve experienced this slow-down lately myself. However, I have another possible solution: Redis.

    Redis 2.4 (currently rc6) supports operating on multiple keys for many operations now so I figured it should make a great cache since it can support tags without the TwoLevels backend. I would love to see how it performs against your Symlink-Cache. Would you care to give it a shot?

    https://github.com/colinmollenhour/Zend_Cache_Backend_Redis

    Also, please post your benchmark code if you don’t mind. Thanks!

    Reply
    1. Matthias Zeis
      August 30, 2011 at 5:59 am /

      Hi Colin and Vinai,
      thanks for the implementation and the benchmark. I’m eager to give it a try, it’s a pity I will have to wait until after my holiday (well – actually it’s not that much of a pity, I’m really looking forward to my holiday ;)).

      Reply
      1. Vinai Kopp
        August 30, 2011 at 6:04 am /

        Hey Matthias,
        please report your results when you get a chance. Enjoy the holiday!

        Reply
  17. Andy
    Andy
    August 30, 2011 at 6:28 am /

    >So, lets use RAM as our cache storage, right?

    >Sounds good, except for one problem: APC and memcached only support storing simple key-value pairs, so the cache lifetime and cache tags are lost!

    Wont dumping var/cache/ into tmpfs get around this limitation altogether?

    Reply
    1. Vinai Kopp
      August 30, 2011 at 7:14 am /

      Partly. On a call to getIdsMatchingAnyTags() PHP still needs to read every cache metadata file, unserialize the contents and loop through the associated tags.
      This is a very inefficient, and the File IO is the least part of it. Most of the resources used are CPU cycles. So putting the cache on a tmpfs mount will speed up cache reads, but most of the TAG related operations won’t benefit a lot.
      Because of this, using the Symlink backend on tmpfs would be a good possible choice for a single server.
      On the other hand, if I would be the one with root access to the server to set up a ramdisk, I would investigate further into Radis instead, since it’s support for hash-lists gives much better improvement on all fronts.

      Reply
    2. Colin Mollenhour
      August 30, 2011 at 7:27 am /

      Andy, putting var/cache onto tmpfs has arguable merit. Linux already does filesystem caching so if you have enough free memory you are probably already making use of memory for your cache for most read operations. Also, the issue with the filesystem backend is that the tag implementation scans all keys for tags when doing getIdsMatchingTags. This is going to be slow no matter what for several reasons:

      1. listing contents of directories with lots of files
      2. reading and unserializing O(N) files from disk (whether tmpfs or disk or buffered disk)

      In both cases, the tag-related operations will get slower and slower as the number of keys increases. Vinai’s Symlink cache alleviates 1 by listing directories that only have as many files as keys with the given tag, and 2 by not having to read the metadata data files from disk or unserialize the metadata files to get their tag lists. The directory listing and symlink reading will still take time whether or not the var/cache directory is mounted on disk or tmpfs and I’d guess the difference is negligible to very little depending on your amount of free memory and disk speed.

      Reply
  18. tenderodigital
    September 1, 2011 at 5:31 am /

    I’ve installed your tool cache. “Magento” now works much faster than Prev.

    But then, by accessing the “Magento Connect,” an error “Must be a directory cache_dir.” If you remove your tool is working again. Anyone know what can be

    Reply
    1. Vinai Kopp
      September 1, 2011 at 5:39 am /

      Hi,
      I can’t reproduce the problem. Please open an issue on https://github.com/Vinai/Symlink-Cache since the forum on Magento Connect currently seems to be offline. If possible include a full stack trace of the error.

      Reply
  19. Andy
    Andy
    September 1, 2011 at 8:55 am /

    How many products do you think before any noticeable difference is seen? I have tested this on a demo store with sample product list and cant measure any measurable difference.

    I have installed redis.. does any one know where I can find info on integrating with magento? i could have swarn i say a tutorial somewhere but have lost track of it

    thanks

    Reply
  20. Colin Mollenhour
    September 1, 2011 at 10:01 am /

    The redis mention is a few comments up: https://github.com/colinmollenhour/Zend_Cache_Backend_Redis
    The readme has an installation section.

    Reply
    1. Andy
      October 26, 2011 at 3:11 am /

      Hi Colin,
      regarding redis extension.. the cache does not seem to be wiping so to make changes live it is necessary to take redis server down then up.. you come across this?
      cheers
      andy

      Reply
      1. Colin Mollenhour
        October 26, 2011 at 4:44 am /

        I think maybe your phpredis library or your redis version is out of date.. Try using the <use_redisent>1</use_redisent> option. Also I’ve made several changes since release so make sure you are using the latest version of the backend. You can run the unit tests to confirm it is working. The cache tag clearing is definitely working for me: http://screencast.com/t/qJ5cQBzRQsh1 (each big dip is clearing the cache)

        Reply
  21. Mike
    Mike
    September 9, 2011 at 10:27 am /

    Would you mind adding the database backend to your benchmarks? Preferably as both the backend and as the low_backend in the two level cache. Or could you share your testing methodology and I could do it…

    I too noticed this issue when users were adding to cart. We had about 70k cached files. Currently we are using two memcached nodes with the slow_backend as database.

    Thank you for the great article as well as for updating it with the Redis implementation.

    Reply
  22. Colin Mollenhour
    September 9, 2011 at 6:09 pm /

    Mike, see Vinai’s previous comment to me with the same question regarding branchmark code: http://magebase.com/magento-tutorials/improving-the-file-cache-backend/comment-page-1/#comment-1578

    Keep in mind these benchmarks only test getting ids by tag, not general performance. I thought about writing a general performance benchmark, but then I just decided it wasn’t worth it since I’m pretty sure no backend can outperform my Redis backend. I could be biased, but you’re welcome to disprove me. 🙂

    Reply
  23. Jake
    Jake
    September 11, 2011 at 4:38 am /

    I have Symlink Cache enabled for both Fast and Slow and I “had” the compiler set up before I installed this. It’s working great and I have noticed a good increase in performance on 26K products.

    My only problem is that I am continuously getting an error from cron.php

    Fatal error: Uncaught exception 'Zend_Cache_Exception' with message 'cache_dir must be a directory' in /home/jbuhrman/public_html/lib/Zend/Cache.php:209
    Stack trace:
    #0 /home/jbuhrman/public_html/lib/Zend/Cache/Backend/File.php(154): Zend_Cache::throwException('cache_dir must ...')
    #1 /home/jbuhrman/public_html/lib/Zend/Cache/Backend/File.php(121): Zend_Cache_Backend_File->setCacheDir('var/cache')
    #2 /home/jbuhrman/public_html/app/code/community/Netzarbeiter/Cache/Model/Symlink.php(37): Zend_Cache_Backend_File->__construct(Array)
    #3 /home/jbuhrman/public_html/lib/Zend/Cache.php(153): Netzarbeiter_Cache_Model_Symlink->__construct(Array)
    #4 /home/jbuhrman/public_html/lib/Zend/Cache.php(94): Zend_Cache::_makeBackend('Netzarbeiter_Ca...', Array, true, true)
    #5 /home/jbuhrman/public_html/app/code/core/Mage/Core/Model/Cache.php(117): Zend_Cache::factory('Varien_Cache_Co...', 'Netzarbeiter_Ca...', Array, Array, true, true, true)
    #6 /home/jbuhrman/public_html/app/code/core/Mage/Core/Model/Config.php(1254): Mage_Core_Model_Cache-> in /home/jbuhrman/public_html/lib/Zend/Cache.php on line 209
    Reply
    1. Vinai Kopp
      September 12, 2011 at 7:14 pm /

      Hi Jake,

      EDIT: Thanks to a patch from Collin this issue should be resolved now in version 0.0.5.

      Reply
      1. Jake
        Jake
        October 11, 2011 at 10:34 am /

        Vinia,

        Negative, it’s still killing my cron with the same error, In addition I can not access my connect manager. It just loads a page with “cache_dir must be a directory” at the top.

        Reply
        1. Vinai Kopp
          October 12, 2011 at 2:31 am /

          Hi Jake,
          You can also configure the cache dir in the local.xml settings.
          For a quick fix, try setting your cache dir in the constructor of the symlink cache backend class.
          Vinai

          Reply
          1. Jake
            Jake
            October 12, 2011 at 12:31 pm /

            Removing the backend options in the local.xml file fixed this issue. Back to fast page loads!

            Reply
  24. Tomas
    September 22, 2011 at 6:52 pm /

    Thanks Vinai,

    This is very informative and well written post. I find it very usefull!

    Reply
  25. Andy
    Andy
    September 26, 2011 at 3:15 am /

    anyone tried this on EE 1.10.1.1 ?

    Reply
  26. Nick
    Nick
    September 28, 2011 at 3:07 am /

    Thanks for this post, it was extremely helpful. I decided to try Colin’s redis code and it is definitely much faster than the stock Magento cache, and not very difficult to install. I’m curious how many others are using it?

    Reply
  27. Andy
    Andy
    October 18, 2011 at 9:16 pm /

    can any of you chaps think of a reason this would not work on enterprise edition?

    Reply
    1. Vinai Kopp
      October 19, 2011 at 2:22 am /

      The Symlink cache module is in production use on several Enterprise instances.
      I don’t know about Redis.

      Reply
  28. Andy
    October 20, 2011 at 7:06 am /

    OK I installed the redis using https://github.com/colinmollenhour/Zend_Cache_Backend_Redis there are a lot of hoops to jump through but it was a relatively easy install.

    Using modman installed this on EE 1.10.1.1 with 12000 products..

    You can see the results here.. immediate decrease by half the load time of the site

    http://screencast.com/t/adMgCUAd

    next step is an CE with 50 000 products… will feed back

    nice work chaps.

    Reply
    1. Paul
      October 20, 2011 at 9:46 am /

      Andy, Do you think you could document your CE Redis installation, this would be very helpful having a walkthrough available to us non-programers. Thanks Paul.

      Reply
    2. Robert Popovic
      October 20, 2011 at 12:12 pm /

      @Andy, to add a suggestion to Paul’s comment, if you feel like it, you could do a guest post on Magebase. Let me know -> info [at] this domain.

      Reply
      1. Andy
        October 26, 2011 at 3:10 am /

        Sure, will pull some notes together and give you a shout

        Reply
  29. Walter Steinbach
    Walter Steinbach
    October 25, 2011 at 10:03 pm /

    Hi,
    great article. After reading

    “If you use a cluster of webservers, you should use memcached for the fast backend and the database for the slow backend. The latter has more overhead then the APC/files combo, but having all servers share one cache pool makes up for that.”

    we made a test with our System (Magento 1.4.1.1, 4 Frontendserver, 1 DB server, 1 NFS server, 1 Loadbalancer).

    We called 3 sites for each configuration.

    Fast: Memcache/ Slow: Memcache
    —————————————————-
    Avg. Click Time [ms]
    Site 1: 3.155
    Site 2: 1.024
    Site 3: 2.044

    Fast: Memcache/ Slow: File
    —————————————–
    Avg. Click Time [ms]
    Site 1: 3.144
    Site 2: 1.147
    Site 3: 2.351

    Fast: Memcache/ Slow: Database
    —————–
    Avg. Click Time [ms]
    Site 1: 4.412
    Site 2: 2.316
    Site 3: 3.199

    Using Memcache as fast backend and database for slow backend was the slowest. Additional the load of servers was significant higher.

    Maybe our config is wrong?

    Greetings from Leipzig. walter.

    Reply
  30. Vinai Kopp
    October 25, 2011 at 11:31 pm /

    Hi Walter,

    thanks for your post. Here a few points you might want to consider:
    Using memcached for the slow backend doesn’t work right because cache tags are not supported. memcached/memcached will not work (at least not with a clever custom adaption of the memcached backend) as intended because clear actions will clear the whole cache pool. .

    Regarding option two, memcached/file, on multiple web-nodes. Using a shared fast cache backend and a slow backend that isn’t shared between the cluster nodes makes no real sense, unless the file backend was e.g. on a NFS mount (which is problematic in it’s own way). So the measurements for option 2 are flawed, regardless which pages where benchmarked.

    Option three (memcached/DB) is the only fully functional variation in your setup. And it’s no surprise it’s slower then the other two, but at least it should work over all conditions (but consider the issues Fabrizio raises in his post http://www.fabrizio-branca.de/magentos-database-cache-backend-mess.html )

    The other question is which pages you benchmarked. Did the action include a getidByTag() call or did they remove records by tag? If not, they aren’t really effective measurements, since simple read operations always are quite fast, the problems mostly is the cache tag handling.

    Reply
    1. Walter Steinbach
      Walter Steinbach
      October 26, 2011 at 1:24 am /

      Hi Vinai,
      yes, I know using memcache for fast and slow backend is not very goog.

      For the option memcache/file NFS is bad – like you said. But I think we found a solution to have it fully functional for us: Slow backend (file) is usind tmps on each frontend server. I know that means 4 x same content cached. But it seems to be the fasted way and we have enough RAM.

      Reply
      1. Vinai Kopp
        October 26, 2011 at 2:30 am /

        Since the cache meta data in the tmpfs cache directory doesn’t match the contents of the memcached pool, deleting a segment of the cache would only remove those entries from the memcached that happen to be referenced by the cache meta data on that host where the clearCache() call is executed (i.e. that where written by that hosts Magento instance). On the other hosts cache records might be read that should have been deleted.

        On the other hand, you will have left over meta records in the cache directories of the other hosts that reference now non-existant records in the memcached instance.

        So the result is an inconsistant cache.
        I suggest using the redis backend of Collin, it is faster then a two-level backend and avoids all those problems.

        Reply
  31. Matthew
    Matthew
    November 12, 2011 at 1:01 am /

    Hi! Thank you for the article! After reading, a question arises in my mind:

    >> every time a cache entry is written,
    >> a directory with the name of each tag is created and
    >> a symlink to the metadata file is placed into it

    Couldn’t you instead dynamically use the “cache_dir” option of the Zend_Cache backend to make several caching directories (based on the name of the tags) and put the cache files in there?

    Reply
  32. Vinai Kopp
    November 12, 2011 at 1:15 am /

    Hi Matthew,

    neat idea, try implementing it. What would the benefits be?

    Reply
    1. Matthew
      Matthew
      November 12, 2011 at 1:26 am /

      Vinai, sorry that would imply a lot of duplicated files in case of multiple tags:) oops!! I’m not using Magento and was thinking too much about my application which may show performance problems when generating huge number of cache files.
      However it may still be interesting to create mutliple caching directories in some cases, e.g. I have about 100 users using a private web app. For each of these users there is a huge amount of cache files caching results of complex SQL queries. Sometimes I need to clean all the cache files for only one specific user, thus instead of tagging cache files with username, I could create for each user a cache directory and clean all the cache files of that directory. That will result in huge optimization my side (because currently I am using tags, and as you very well explained in your article, all cache files (of all users) will be opened, read, … to get the user tag).

      Reply
      1. Vinai Kopp
        November 12, 2011 at 1:34 am /

        Thank you for the clarification. I like the thought – still, using the cache-tag-directory-with-a-symlink approach would probably solve that, too, as long as there aren’t too many other tags involved. Thank you for the interesting input!

        Reply
  33. Andy
    Andy
    November 15, 2011 at 2:35 am /

    @vinai,

    genious, made the changes and cli tools are back up again

    many thanks.
    Andy

    Reply
  34. Tino
    Tino
    November 25, 2011 at 7:59 am /

    Thanks a bunch for this. We got it to work nicely, but now our /downloader page is giving the following error:
    “cache_dir must be a directory”

    This occurs only when trying to access Magento connect manager = /downloader page.
    tmp folder has been set to root with 755 (tried 777 as well) access. no help.

    What do you think is causing this?

    We are using Litespeed Enterprise instead of Apache.

    Reply
    1. Andy
      November 25, 2011 at 9:29 am /

      @Tino Rename the line (in File.php in directory /lib/Zend/Cache/backend) ‘cache_dir’ => null, to ‘cache_dir’ => ‘var/tmp/’ and make sure var/tmp exists and is writeable by webserver

      Reply
  35. Tino
    Tino
    November 25, 2011 at 10:59 am /

    @Andy Thanks for your reply. I did the change, flushed cache/relogin but now the /downloader page is completely empty (white, source code is empty). Made the folder /var/tmp, chmod 777 and chown to correct user just in case. No help.

    Reply
    1. Vinai Kopp
      December 31, 2011 at 5:01 am /

      Tino, to you the same applies as to Jérôme’s comment further below http://magebase.com/magento-tutorials/improving-the-file-cache-backend/comment-page-1/#comment-1799

      Reply
  36. Jérôme
    Jérôme
    November 27, 2011 at 5:40 am /

    Hi,

    The button “Initializ Tag Symlink” is use one time (after installation) or must i use it from time to time or tags are create automatically afterward?

    I use (magento 1.5.1, debian 5 and PHP 5.2.17) is it compatible with Symlink ?

    Can i use “Symlink” if i use a tmpfs for : /var/cache/ on RAM ?

    If i use symlink extension i must desactived aoe_cache cleaner extension ?

    Best regards

    Reply
    1. Vinai Kopp
      November 27, 2011 at 6:56 am /

      Jérômoe,
      you only need to use the Initialize Symlinks button once after installation. After that, the tag directories are created automatically.
      Looks like your installation should work fine with the symlink cache extension, as far as I can tell.
      Symlink on tmpfs is a good idea.
      I don’t know about compatibility with the Aoe_Cachecleaner extension, but my guess is that they should play together fine.

      Reply
  37. Diego Padilha
    December 1, 2011 at 8:53 am /

    Vinai, first of all, I’d like to thank you for this excellent tutorial.
    Is this tutorial compatible with the community edition?
    Do you know if anyone has tested this tutorial in these conditions?

    Thanks.

    Reply
    1. Vinai Kopp
      December 1, 2011 at 9:22 am /

      Hi Diego,

      the contents of this article apply to the community edition and enterprise edition alike.

      Reply
  38. Jerome
    Jerome
    December 11, 2011 at 3:59 am /

    Hi Vinai,

    Symlink cache works with zend server ?

    Thanks.

    Reply
    1. Vinai Kopp
      December 11, 2011 at 7:21 am /

      I haven’t tried Zend Server, but I believe it should work.

      Reply
  39. Jérôme
    Jérôme
    December 29, 2011 at 6:35 am /

    Hi Vinai

    When i use the following configuration in my local.xml : (i use apc)

    apc
    MYPREFIX_
    Netzarbeiter_Cache_Model_Symlink

    I’ve got a problem with cron : all my cron dont work any more, not more scheduled and runs any more

    My aplogies for my bad english

    Reply
  40. Jérôme
    Jérôme
    December 29, 2011 at 6:38 am /
    <cache>
                <backend>apc</backend>
                <prefix>myprefix_</prefix>
                <slow_backend>Netzarbeiter_Cache_Model_Symlink</slow_backend>
            </cache>
    Reply
    1. Vinai Kopp
      December 29, 2011 at 8:50 pm /

      Jérôme,

      please check the __construct() method of the class Netzarbeiter_Cache_Model_Symlink matches the one on github: https://github.com/Vinai/Symlink-Cache/blob/master/app/code/community/Netzarbeiter/Cache/Model/Symlink.php
      Is that the case?
      If not – how did you install the Symlink cache module?

      Reply
      1. Jerome
        Jerome
        January 7, 2012 at 11:02 pm /

        Hello,

        I ulpload the last version by ftp
        I cha

        Reply
  41. Sanjay
    Sanjay
    December 31, 2011 at 1:50 am /

    Is it possible to make both fast and slow backend point to Redis?

    Reply
    1. Vinai Kopp
      December 31, 2011 at 1:53 am /

      Because Redis supports cache segmentation there is no need for a two level cache backend.
      With Redis you only have a single cache backend, there is no separation into slow and fast backend.

      Reply
  42. Sanjay
    Sanjay
    December 31, 2011 at 2:40 am /

    Another question – what about a setup where Admin and Frontend are on different servers?

    Reply
  43. Sanjay
    Sanjay
    December 31, 2011 at 3:19 am /

    Thanks Vinai for the answer, my 2nd question was from the context of using symlink and memcache. Will it differ if we have admin and frontend on two different servers?

    Reply
    1. Vinai Kopp
      December 31, 2011 at 3:44 am /

      If you are using two servers, I don’t recommend using a file based slow backend (i.e. separated between the servers) and a shared fast cache backend (i.e. shared between the servers).
      In that case you will have inconsistent cache and meta data records on both servers.
      Use memcached/database in that case or another solution that enables all servers to use the same cache pool (fast and slow).

      Reply
  44. Sanjay
    Sanjay
    December 31, 2011 at 4:13 am /

    thanks a ton, Vinai.

    Reply
  45. Sanjay
    Sanjay
    January 1, 2012 at 6:19 pm /

    Hi Vinai,
    Just to give you an update: we deployed your code and the site became really fast, the checkout times which used to be in 25 secs+ came down to 10 secs.
    As I had previous experience working with Redis and knew what it was capable of, we got a bit greedy and went for the Collin’s Zend_Cache_Backend_Redis, the site became blazing fast. There is a driver issue and corrupt cache records (when we connect the Admin to the Redis service) and we have raised an issue, hopefully will get it solved soon.
    Wish you and your blog readers a Very Happy New Year!
    Cheers
    Sanjay

    Reply
    1. Vinai Kopp
      January 2, 2012 at 12:08 am /

      Thanks for the feedback, Sanjay – happy new year to you, too!

      Reply
  46. Jerome
    Jerome
    January 7, 2012 at 11:06 pm /

    Hello,

    I ulpload the last version by ftp – I changed my local.xml – i intilialize symnlink button – but my crons are stoped

    Reply
    1. Paul Bates
      Paul Bates
      January 8, 2012 at 12:07 am /

      I can’t get my crons to work, interesting to read you comment.

      Reply
    2. Jerome
      Jerome
      January 8, 2012 at 12:39 am /

      Hi,

      When I remove the following code my crons work again

      <cache>
                  <backend>apc</backend>
                  <prefix>myprefix_</prefix>
                  <slow_backend>Netzarbeiter_Cache_Model_Symlink</slow_backend>
              </cache>

      my crons work again

      Reply

Add a Comment & Join the Discussion

Insert small snippets of code by using [code]{your_code_here}[/code]
For larger code blocks please use http://pastebin.com and paste your link.

You may also use the following HTML in your comment: <a href="" title=""> <abbr title="">
<acronym title=""> <blockquote cite=""> <cite> <em> <strike> <strong>