Last night we performed some upgrades and restarted a server with nearly a year of uptime. This morning I was checking on the status of the server, and saw this graph. It is a beautiful example of the kinds of things you can see with Munin.
You are looking at a graph from munin, showing the usage of memcache of a large Ruby on Rails application.
After the server restarted, the cache was empty. As the server traffic picked up this morning, more and more database calls and page fragments were generated by users hitting the site. They were tucked away in memcache so they could be served up faster to the next visitor. As we get more traffic, this cache fills up, and the website actually gets faster.
The performance gains aren't the subject of this post though - I thought this was a beautiful representation of a day in the life of a server process. This represents one day period, with time going from left to right, and the three lines representing the amount of data in bytes (green), the number of objects (whether that be objects from the database or pre-rendered pieces of html - red), and the number of connections our rails processes are making to the memcached server (blue).
What does this show me? I can see that memcache is doing its job - that the rails app is putting data in and getting it out. I can also see that the cache is filling up... based on the 'hockey stick' curve, I can see we have allocated enough memory to hold the data we are putting in there. If we hadn't allocated enough memory, I would visually be able to see it 'hit the ceiling'.
I can also tell things about the system overall by comparing to other graphs from munin and other tools. Comparing this to graphs of mysql queries, I can see the impact of using memcached on mysql (by the number of reduced database queries) By comparing it to processor utilization, I can see the amount that pre-rendering reduces processor load (since we don't spend as much time rendering).
