My Hidden Room

CC licensed image via alancleaver on flickr.

Today I went to breakfast at Voula’s Offshore Cafe in the University District.  I had the steak and eggs with rye toast for breakfast.  It was quite tasty.  But this isn’t about my breakfast so much as it’s about the interesting example of information leakage.

I use foursquare because it’s fun, and to find where my friends are going for food, activities, and other things.  I don’t link my foursquare to any other accounts, but other people do.   This morning at Voula’s, I checked in, and saw one other person there.  I’m curious, and so I look at the foursquare profile. Turns out there’s a facebook link on the foursquare login, so I follow it.  Turns out she was sitting at the table next to mine.   I said hello and made a comment about foursquare.  Then I returned to my meal.

Her facebook profile said a lot of things: that we have quite a few friends in common, who she’s seeking, and her full name.   Now, I didn’t say anything to her about any of this, nor attempt to engage in a conversation at this time.  It would have been weird and creepy.  I didn’t want to be That Guy, or a weirdo.  But it was very much an interesting lesson in how we chose to interact online can affect us in the offline world.

When we all got out to the car, I had a conversation with the teenagers about information that you share online, and used this very real example from just now to illustraite that what you do online can have real results in the rest of your life.   Pay attention to what you post, to what you link, to all those things.  It’s very easy to draw connections between people via the social graphs.

There are many wonderful things that I get out of connecting my social graphs.   I get to point people whom I want to connect with me to where they can find me, and how they can do so.  But it bears consideration before you blithely do so.  Everything connected means there aren’t any secrets.

Ramblings

Today we had to do a very large alter table on some production data.  The table in question is big. 47 million rows big.  We have to make some changes to drop a column and change some indexing to support our application.  Fine, ALTER TABLE blah blah blah.  But how do we keep an eye on the progress of the alter?

There’s nothing in the documentation for MySQL that speaks to a method, nor does any google searching pull up anything (except for the one InnoDB file per table patch.)

So what’s a clever systems administrator to do?

I looked at the output of lots of status commands from MySQL, and finally I came across something useful in the InnoDB engine status output.  I noticed that there was an entry in the TRANSACTION section that speaks to the undo log entries.  I noticed that after several hours of our ALTER TABLE running that the undo log entries value was up around 33 million.  And then it occurred to me that since InnoDB does row-level operations, that this value might correspond to the number of rows that have been operated on throughout the process of the ALTER:

---TRANSACTION 0 2769988559, ACTIVE 53210 sec, process no 4115, OS thread id 1157654880 inserting, thread declared inside InnoDB 91
mysql tables in use 2, locked 2
111667 lock struct(s), heap size 13580272, undo log entries 33707735
MySQL thread id 948913367, query id 1662722204 localhost root copy to tmp table


So I wrote some bash.  And some awk.  And some perl, and I made a hacky little one-liner script to dump out the rate of change, the average rate of change, the % complete, and an estimate (based on average rate of change) as to when the query might be complete.

That script is below, and you are welcome to it.  I have yet to wrap it in more clever perl to dump out an actual status bar, but I’m happy with it, and hopeful that this is a meaningful way of keeping an eye on the rate of change, and the progress of ALTER statements.

The script:

#!/bin/bash
(while(true); do  \
(mysql -e 'show innodb status \G' | grep undo\ log\ entries ; sleep 1 ;  \
mysql -e 'show innodb status \G' | grep undo\ log\ entries ) |    \
egrep '[0-9][0-9][0-9][0-9]' |awk '{print $10;}' ; done ) | \
perl -e '$t = ROWS_IN_TABLE; while(1) { \
$n ++; $nn; $a = <>; $b = <>; $nn += ($b-$a); \
printf "Rate: %d, avg: %d, %0.3f%% complete, done in %d sec\n", \
$b-$a, $nn/$n, ($b/$t)*100, ($t-$b)/($nn/$n); }';


I hope you find this tool useful.

It took a LONG time for the query we performed to run, and it was very helpful to have this to gauge the time it’d take.   The moving average is not the best for long-term accuracy; restart the script periodically for better views (and wait for ~100+ trips through the loop.)

It was within three minutes of accurate.  It look a little longer than the script indicated (log undo entries was greater than the # of rows, but not by more than a small percentage (less than 0.01%.)  The total query time was 22 hours, 9 minutes, and 33 seconds.  So it was off by only 1 hundredth of a percent.

This is only valid for ALTER TABLE, but appears VERY useful.

WOOHOO!

MySQL, Ramblings alter table, database administration, hacks, MySQL, spiffy new discoveries

So the opening for my show is this evening.  Wow, I’m excited.

Ramblings

So in June, I’m having my first art opening.  That’s exciting!   Over the last half-year I’ve been talking to the folks at Utilikilts about whether they’d be interested in seeing my works in their store for Art Walk.  They expressed interest that yes, indeed, they would like that.  So we talked, and figured things out, and now I’ve got an opening coming up.

This collection of works ranges over a few areas of interest for me: Burner Culture, People, and some other pieces.   Please come see my work next month, on June 4th for Art Walk at the Pioneer Square Utilikilts store.  There will be a live DJ and drinks to be had.

It’s very exciting, and I’m looking forward to it.  I hope to see everyone there.

Ramblings

I attended MySQL Conf, and it’s been an neat conference.  The most surprising thing was the news was the purchase of Sun by Oracle.  This sounds to me like a play for a solid vertically integrated market for Oracle.  I can appreciate that — they get Sun’s hardware and OS skills, they bring their own DB services to the table, and they get a lot of community with MySQL.

Lots of people have spoken about their fears that Oracle will suffocate MySQL, but I don’t think they’ll do that.  I think that Oracle realizes how centrally important to the web community that MySQL is.  I do think they’ll try to sell Oracle to high-volume MySQL sites, and that makes sense.  And I think they’ll continue MySQL to bring DB share under the Oracle roof.

MySQL is hugely popular, it wouldn’t make any sense to throw away that good will.

Other things from the conference have been that I’ve run into people I haven’t seen in a while, and I’m learning things.

One of the most interesting talks I went to was Rockyou’s talk about how they built their federated platform for supporting their Facebook applications.    Very neat to see their approach, especially since we’re buidling our own sort of Facebook applications, specifically Bejeweled Blitz.   Good times.

I met the photographer called Julian Cash, a very nice guy located in San Francisco.  He did a light painting portrait of me, which was fun.   We talked about various things for a while, and he talked to me about his project that he’s working on.  More to come later.

Today is the last day.  Things are winding down, and I’m preparing to go home.

MySQL, Photography, Ramblings, adventure MySQL

This past weekend was Norwescon, a sci-fi/fantasy convention.  It was awesome.   I put on a shoot for free out of my room, and I got a good turnout, and I had a lot of fun.  It was a good experience, and good practice.  Next year, I know some things I’ll do better.

I got a lot of good feedback, and pleased participants.  Woohoo.

I think it’s important to do things like this to give back to the community — to let people come if they want to have a photo made, and share in a creative process.  The best photos were those when the subject played along, and had ideas about what to do.  I like it when it’s a collaborative project.

Anyway, enjoy.  Click the image for the flickr.

-Gabriel

Photography, Ramblings

It’s been a long time since I’ve updated this thing. Lots of changes, some photos. Good times.

Christmas has come and gone, thank goodness. The holidays aren’t for me. Winter is over, and I’m glad. But it’s been too long without updating. Without creating. Without feeling good about things. It’s time for that to stop, and for things to get back on track.

So, I’ve got some things cooking that’ll be good. I have a strong lead on a show in the summer (more to follow later), some leads on good ideas/concepts to push myself, and I’m biking to work now. 11 miles in the morning is good fun.

But where to go from here, that’s the thing. I have ideas, but to make them real…

Photography, Ramblings

I’ve made a few changes to the blog: I’ve switched themes to something that I think fits better with what I want htis to be, and I’ve updated my photography website to have a better portfolio set of images.  Go take a look.

Ramblings

This weekend I went to the Hive-Mind Halloween party.  It’s a pretty cool scene, and the money raised from it goes to support local arts organizations and other charities.  It’s money for a good cause, and it’s a great time.   I had a good time catching up with people I hadn’t seen in a while, and taking lots of pictures of people.

I played around more with a hand-held shoot-thru umbrella setup there.  It worked well, and it went very well with my steampunkish costume.   The results, and full photo set can be see on my flickr page.

It was a fun time.

-Gabriel

Ramblings

So today I decided to do another pendant.  And it went well.  I’m happy with my progress, like I’m gaining skill (like I want.)

Metal flows through the mold
Flowing swiftly ’til it turns cold
Shapes cast from a fleeting master
I see you freed in your sprue tree
Polished to a shine
Cast metal, frozen solid.

I’m thinking that I’m going to try sandcasting as an approach for doing lots of these.  I takes a lot of time to prepare these as one-offs of wax, to sprue them, invest them, burn them out, and pour them.  2-3 hours each.  I think by sand-casting, I can get that down to 30 minutes for batches of ten.  That’d be awesome.

Hopefully in a couple weeks, I’ll have the materials on hand to do that.  See you soon…

Burning Man, Photography, Ramblings, Things I made, casting