My Hidden Room

So one of the issues that I deal with a lot is tuning web applications to not suck.  This is done by a few things; by monitoring, by profiling, caching, caching (CACHING!), and by tuning.  The process for making a web application more awesome basically boils down to this list of steps:

1. Monitor your application performance (http threads, cpu, memory, thread response time, etc)
2. Profile your code
3. Fix slow requests/implement caching
4. Tune your web-server
5. Goto 2.

Monitoring

Monitoring the response time of your application is useful and awesome for making positive changes to your environment .  This means paying attention to your application response time, cpu, memory, network traffic, disk IO, disk capacity, etc.  All those metrics that say whether your application is healthy or not.  There are a few different tools available for this that all work pretty well, here’s an incomplete list:

• Cacti – http://www.cacti.net
• Munin – http://www.munin-monitoring.org
• Cricket – http://cricket.sourceforge.net

They all work well and solve slightly different problems.  Pick the one you like most.  I’m a fan of Cacti.

Profiling

Profiling means being able to see how long each call that an application makes takes to execute.  It’s invaluable for getting a feel for what parts, what components, of your application perform badly or perform well.

Caching

Whenever an application fetches data from a resource, that’s an opportunity to improve performance.  Every time something is fetched, there’s the ability to take that result set, and keep it.  Caching the results of database calls, of external API lookups, of intermediate steps, all these things leave lots of room for improving performance. Memcached is the de facto standard for a caching engine.

Cache early, cache often!

(Apache) Tuning

A well configured web-server is crucial to a happy environment.  This means not running into swap, not wasting time with connections that are dead, and other such things that waste time.  In short, don’t look up what you don’t need, don’t save what you don’t need, and be efficient.  Here are some basic things that apply to Apache:

   KeepAlive          Off (Or On, see below, it depends on workload)
   Timeout            5
   HostnameLookups    Off
   MaxClients         (RAMinMB * 0.8) / (AverageThreadSizeInMB)
   MinSpareServers    (0.25 * MaxClients)
   MaxSpareServers    (0.75 * MaxClients)

About these parameters:

• KeepAlive – this controls whether when one request from a client to the web-server is completed whether that thread will remain connected to the clients for subsequent requests.  In high-scale applications, this can lead to contention for available resources.  Some workloads, however, benefit from keeping this on.  If you are serving lots of different content types on a page to a client, leaving this on can be a good thing.  Test it out, YMMV.
• Timeout — how long before we assume that the client has gone away and won’t be requesting further data.  The default is 300.  It is in seconds.  This value is aggressive.
• HostnameLookups — this is for logging, and if it is on, each client will cause a DNS request to be made.  This slows down the request.
• MaxClients — the total number of threads that the server will allow to run.  Each thread consumes memory.  This model assumes that 20% overhead for other system tasks is appropriate and will keep us out of swap.  On machines above 16GB of ram, use 0.9 instead of 0.8.
• MinSpareServers — the fewest threads that Apache will leave running.  Setting this too low will result in load spikes when traffic increases.
• MaxSpareServers — the most spare, unutilized threads that Apache will leave running.  Setting this too low will result in lots of process thrashing and threads are used and then terminated.  The tradeoff is utilized Ram.

There are a lot of other things that can be done as well, so don’t take this as a complete set…

These are my handout style tips on performance tuning.  There are whole volumes of books dedicated to this topic.  Some great resources include:

• http://www.mysqlperformanceblog.com/
• The Art of Capacity Planning (Allspaw)
• http://www.standalone-sysadmin.com/blog/

-Gabriel

So at work, I had a blog.  Not my blog, of course.  One I support.  So this blog was thrown together quickly to facilitate business goals.  Like you do.  And that’s great.  We met the deadline.  We got the product functional.  But performance kinda sucked.  A little backstory.  Here’s how it got set up to start.

We love virtualization here.  Everyone should.  It’s a fantastic way to take adventage of hardware that you’ve got laying around that’s being idle.  Idle hardware is lame. So we run KVM.  This means we get to manage machines more effectively, and can provision things faster.  That’s awesome.

So this blog.  It’s a cluster of boxes, made up of a pair of webservers, and a pair of DB backends.  The DB backends are a master, and a backup.  This is a small project, so these got set up with just 4GB of ram.  So that’s fine.  The web servers are each a VM with 1 vcpu, 4GB of ram, and some disk space.

So we set that all up, and it did fine.  But not great.  here’s what it did.

Incidentally, if you’ve got data you’ve collected for performance with Pylot, and need to CSV the averages to make a pretty graph, here’s my way of extracting the averages from the results HTML to feed into gnuplot.

for file in load-dir/*/results.html ; do cat $file  | egrep 'agents|avg' | head -3 | perl -e 'while(<>){s/<\/?t[dr]>//g; s/(\d+\.?\d*)/\t$1/g; chomp; @p=split /\t/,$_; push @r,$p[1]; }; printf "%s\n", join ",",@r; '; done


So what we can see then from these graphs is that a pair of single core boxes, tuned with the default cherokee+php installation don’t do all that great.  They can handle 1-2 simultaneous requests, but past that, the response time gets pretty bad. That’s where the project got left for a while, until the other day I got the request to make it handle 500 requests per second.  “Wow, shit.” I thought.  So I dove in to see what I could do to improve performance on the blog, and it turned out that there was a lot I could do.

1. Single-core boxes don’t have a lot of performance available to them.
2. Cherokee uses PHP in FastCGI mode, and does a terrible job of tuning it out of the box.  It defaults to a single PHP thread.
3. WordPress is very hungry for CPU time.  It chews up a lot of resources doing its DB queries and wrangling them.

To address these points, I did the following.

The VMs themselves were restarted with more CPU cores — four cores per box.   This allowed me to dive into discovering that Cherokee wasn’t tuned well.  Under a single core, I saw 80% cpu utilization on PHP, with high system wait time.  That sucked.   But after I bumped up the core count, it still looked bad.  Still only one CPU @ 80%. WTF.  So then I turned to Cherokee, and I tuned PHP so that it would invoke 16 children, enough to handle the request volume.  This helped a lot, but there was still room to do better.  So I added APC, the Alternative PHP Cache, to the configuration.  This helped out a lot.  I then looked at wordpress specific caching engines, and settled on using W3-Total-Cache to help out.  This brought performance into the range of fulfilling the customer’s request.  I felt great about it.

I used pylot to graph performance at various points through this project so I could figure out how I was doing a better or worse job of tuning the boxes.

Here’s the performance data showing the improvements that caching at various layers added to the party:

So it turns out that these are some great ways to make sure your blog performs:

• Make sure that there’s enough hardware supporting it
• Cache in depth and breadth, early and often
• Tune your PHP/WebServer to handle the project
• Employ performance testing to measure your real improvements

With some hard work and performance tuning, you can turn an abysmal 7 requests/sec peak performing blog into one that can sustain 350 requests per second, and do it in a tiny fraction of the time.

-Gabriel

Texting me early in the morning without identifying yourself is a fantastic way to piss me off. If it was your goal to antagonize me against your position and company, you did a great job. Also: When I text you back, “do not txt me,” it does NOT mean that you should then call me five minutes later!

Advice to recruiters who haven’t yet pissed me off: Email. It’s great. If I’m interested in the position you have, I’ll let you know! Don’t cold-call me. Don’t cold txt me.

So it’s time for the annual pilgrimage to Burning Man.  Like so many other folks, I’m going.  Unlike a lot of people, I haven’t had the ticket scramble that the selling out process led to — I bought mine back in January.  Anyway, this post isn’t so much about tickets, as it is about the experience, and how I feel about the upcoming trip.

I’m super excited about it.

This year, I’m bringing my own big art project to the Burn.  The Air Kraken has been a ton of work this year, and I’m so happy that I’ve gotten it done, and that it’ll be on playa.  The project has gone really well due to help from my friends by way of Kickstarter contributions, INW’s Art Grant, and construction assistance.   I’m super appreciative to all their wonderful help!

Check out more about the Air Kraken on my project page.

So what else is going on for the Burn?  I’m planning on taking a ton of photographs this year, and that means bringing a lot of storage… I’m bringing 28GB of CF cards, and 6GB of SD cards for two cameras.  I’m bringing a cool selection of lenses, too:  Nikkor 50mm f/1.8 D, Sigma 21-35mm Alpha f/3.5-5.6, Nikkor 70-210mm f/4-5.6 D, and on my D80 the 18-200mm alphabet soup lens as backup.  There should be lots of cool stuff to see.

If you see me on playa, I demand you ask for a portrait.    See you in the desert…

So this past Friday, I went up with some friends to Rattlesnake Lake off exit 32 on I-90.  It was a lot of fun, and we laughed and talked and drank good wine.  It was a lovely time.  I shot a lot of photos, but didn’t catch any meteoroids.  The moon was too full and bright, and there just weren’t that many.  I saw about 10 trails in the sky, which was quite pleasant.   Here are some photos I did shoot.  Funny thing about the full moon… It’s a lot like the full sun.

Specifically, 30 sec @ f/5.6, ISO 200 looks like a daytime exposure.  It’s pretty awesome to have images that look like daytime shots with star trails in the backgrounds!

One thing that I realized about this trip was that I was lacking a good release for my D700.  Fortunately, a quick trip over to Amazon and my Prime membership means that next time, that won’t be a problem for me.

Until next time…

So on Friday, I attended the launch of the Flash Bus talk series by Joe McNally and David Hobby.  It was awesome.  Both of these presenters do a fantastic job of getting their points across in a clear and concise way.  Even as an experienced shooter, I got a lot out of the experience.  Watching them work with subjects and coax out awesome photographers was great.

So afterwards, I was inspired to go out and do some environmental portraits.  Near my studio is a lot of excellent brickwork and old, cool buildings.  These buildings make great locations for doing a shoot.  The shot to the left was done with two speedlights: a SB800 through an umbrella for fill, and a SB900 camera left for key.  This was a fun set to do.

The other shot I did was with my friend H at Golden Gardens park.  This is one of my favorite places to go, and to shoot at.  There’s lots of cool variety to see; train tracks, trails, beach with sand, with cobbles.  Lots of fantastic imagery to pick from.  We went there Monday afternoon, around 5PM.  It was a windy and rainy day, so the shooting was challenging.  The wind LOVES to grab hold of a shoot-through umbrella and carry it away. 

This shot (at right) was done with a single bare SB900 on a lightstand held down into the cobbles with rocks.  It was done in aperture priority mode with -1 EV exposure compensation to get nice colors out of the fading sunset through the dark and stormy clouds.  The wind was fun, but did present some difficulties, including the heart stopper of having your flash fall over onto the rocks.

Fun stuff!

So coming up in two months, and announced yesterday is a photography work shop by David “Strobist” Hobby and Joe McNally.   It will be fantastic.  They’re starting their trip in Seattle, so I’ll get to experience this great event first hand.

Check it out, if lighting is in your bag: http://www.theflashbus.com/.

I can’t wait!

So today, like a few tens of thousands of other burners, I was after a ticket to go burn in the desert. I got in line promptly at 10am, and got the initial queue position of 2008. SWEET, I thought.

But no, there were server problems on their ticketing vendor’s side of things, and that resulted in the queuing system misbehaving and generally punting people way back in line.

So what I did, after waiting for a number of hours was I figured out how to retrieve my old queue position in line, and get my ticket.

Here’s what I did, and what you can, too. This only works on Firefox.

1) Pull up a blank page, and go to the URL about:cache. That’ll pull up the cache entries page where you can look through your browser cache for old URLs you’ve visited. Search the memory cache and then the disk cache for the string “key_queue=” — this will be found in URLs for the Burning Man ticket page links. You’ll want to find all of them, and collect them.

2) Then take the URLs you’ve collected, and load them into your browser. I was able to get immediately to the ticket purchase page doing this, and bought my second-tier ticket.

This worked for me. It might work for you, it might not. But I thought I’d share.

This guide leans heavily on the work of http://simplygenius.com/2010/08/jconsole-via-socks-ssh-tunnel.html — what I’ve done is collect his work into something a little more manageable for my environment.  In here, I will go over the shell commands that can be used to make doing this easy.

Read more…

So it’s coming up on Halloween. One of my favorite holidays, and that of many other people, too. This year, I’m helping out my wife with her costume by making her some goggles that employ SMD LEDs to create a number of neat effects. To do this, I had to make some boards to fit inside the goggles, etch them, and solder up them. The embedded flickr set shows the process I engaged in in making these.

Each of the eyes of the goggles will have seven lights in it, and will support a variety of light patterns (to be determined), executed with an Arduino and two shift registers, one for each eye.

More design notes, and build updates to follow.  Enjoy the show.