Hueekend at Hueco

IMG_7277

BJ near the top of Cakewalk – his first multi-pitch route.

My friend BJ and I made a quick weekend trip to Hueco Tanks last weekend.  Here’s the tick list:

Saturday:

  • Cakewalk
  • Uriah’s Heap
  • Sea of Holes
  • Window Pain
  • All The Nasties

Sunday:

  • Divine Wind
  • Malice in Bucketland
  • Rainbow Bridge (first pitch)
  • Lunch Rock  Direct
  • Fox Trot

IMG_7301 (1)

Selfie at the top of the first pitch of Rainbow Bridge – my favorite route at Hueco Tanks

My New Favorite Crag

photo

Pitch 1 of the Uber-Classic Corrugation Corner

Blake and I just returned from a 4-day weekend of climbing at Lover’s Leap.  Lover’s Leap is 20 minutes from Lake Tahoe, at about 6500 feet of elevation, making it one of the top summer climbing destinations in the US.  This was our first visit to the Leap and it didn’t disappoint.  In fact, the combination of stellar climbing, easy access/approach, great weather, and idyllic setting may make Lover’s Leap my new favorite destination for summer cragging.

Because this was our first visit, we decided to focus on the classics — and we ticked a bunch of them (29 total pitches of climbing in 3.5 days).  We had been warned that Lover’s Leap is crowded in summer and it’s not unusual to wait in line to climb a classic route.  However, we had no problem with crowds and had the entire crag to ourselves much of the time.  We also lucked into a great little cabin right next to a river, which was perfect for cooling off after a long day of climbing.

Here’s our tick list:

Thursday:

  • Haystack, 5.8 (3p)
  • Bear’s Reach, 5.7 (3p) – Maybe the best 5.7 I’ve done anywhere!
  • The Line 5.9 (3p) – Stellar! We linked P2 and P3.

Friday:

  • Surrealistic Pillar, 5.7 (3p) – I didn’t think it was as great as advertised.
  • Hospital Corner, 5.10a (2p) – The second pitch is probably the best single pitch we did at Lover’s Leap.
  • Tombstone Terror, 5.10c (1p) – Hard and sustained!  If you told me it was 5.11 I wouldn’t disagree.
  • Boothill, 5.11 sport – Powerful AND technical.  Maybe the best sport climb I’ve ever climbed!
  • The Groove, 5.8 – Meh.

Saturday:

  • Corrugation Corner, 5.7 (3p) – Lives up to the hype.
  • Scimitar, 5.9 (3p) – Fantastic, varied climbing.  One of my favorites of the trip.
  • Surrealistic Pillar Direct, 5.10b (1p) – Great line.  We also did the left variation.

Sunday:

  • Traveler Buttress, 5.9 (4p) – This route is one of the “50 Classic Climbs of North America” but I don’t think it’s that great by today’s standards.  The offwidth section on P2 is heinous.

 

48 Hours in Moab

ImageLowering down the splitter thin hand crack on Pente, Indian Creek

One of the (few) benefits of traveling frequently for business is that I accumulate lots of airline miles.  And the great thing about airline miles is that you can use them for last minute flights that would otherwise be very expensive.  Last week our realtor suggested that we vacate our house (which recently went on the market) over the weekend for an open house and showings.  Megan had a tennis tournament and the kids went to grandma’s so you can probably guess what came to my mind.  I checked the forecast at a few climbing hotspots and decided on Moab.

I flew in to Salt Lake City Friday evening and drove 4 endless hours across barren desert to Moab.  This was a solo trip, so I had arranged for a climbing guide for the weekend.  When you have limited time and aren’t familiar with an area it’s nice to hire a local.  It’s also an opportunity to climb harder routes than I normally would.  And I had a very specific route in mind:  Fine Jade (5.11a, 5 pitches).  Fine Jade follows a gorgeous crack system on the southern prow of The Rectory, facing Castleton Tower.  The route is considered by many to be one of the very best climbs in the desert.

Image

Approaching The Rectory, Castle Valley, Utah

Saturday
The climbing season in the desert southwest is rapidly coming to an end and the forecast for the weekend called for highs in the upper 80’s.  So we got an early start on Saturday and were able to do the short but steep approach hike – and the majority of the route – in the shade.  I can’t say enough about the quality of Fine Jade.  Every pitch is great.  I don’t get to do much pure crack climbing so it felt difficult – especially since the strenuous crux comes low on the very first pitch.  If you want the blow-by-blow, here’s my GoPro video of the first pitch that includes a fall (at the 2:50 mark) after botching the crux sequence of moves.  Fine Jade definitely goes on my top 5 list of all-time favorites.  I want to go back and lead it soon.

Image

Summit of The Rectory via Fine Jade

Sunday
To become an expert trad climber you have to master crack climbing.  Cracks come in a wide range of sizes – tips, fingers, off-fingers, thin hands, hands, off-hands, fist, offwidth, squeeze chimney, and chimney – and each requires different technique.  If you want to hone your technique, the very best place in the world to go is Indian Creek, about an hour south of Moab.  It’s a crack climbing mecca with literally thousands of clean-cut cracks of all sizes.

I had never been to Indian Creek, so I was excited to check the area out.  Living in Fort Worth, most of my climbing happens in a gym where it’s nearly impossible to practice crack climbing.  So I was curious to see how I would fare at Indian Creek.

We visited Reservoir Wall, one of the better shaded crags at Indian Creek and did a handful of routes.   The highlight was Pente (5.11-), widely considered one of the area’s best routes.  Because I was with a guide I followed Pente but it felt pretty casual so I’m confident that I could lead it.  In fact, I felt pretty solid on all of the routes we did (mostly 5.10s).  I’ll definitely be going back to Indian Creek when the temperatures cool this fall.  And I might even build a crack machine for my man cave to train at home…

Inti Watana

The spring climbing season in the desert southwest is coming to an end, so Blake and I flew to Vegas last week for a few more days on the rock.  These days I consider Red Rocks my “home crag” since it’s almost as easy to fly there (usually with frequent flyer miles) as it is to drive to the (far inferior) Wichita Mountains in Oklahoma.

Our primary objective for this trip was a route called Inti Watana (5.10+, 12 pitches), a 1500 foot route on the steep north face of Mount Wilson (which, at 7070′, is the tallest peak at Red Rocks).  Here’s how it played out:

Wednesday
We flew to Vegas early and headed straight to the crags to climb Adventure Punks (5.10d, 5 pitches), one of Red Rocks’ “rediscovered classics” .  We had visited this route last year, but I backed off the first pitch due to a dangerous runout.  This year I returned with specialized gear (a #1 ball nut) to protect the runout start.  Even with this gear, the consequences of a fall in the wrong place would be very severe.

Image
A #1 (blue) ball nut deep in a shallow hueco 10 feet off the deck is all the protection you get for the first 25+ feet of 5.10 climbing on Adventure Punks.  You can see the flake high above that offers the first real protection.

It was a spooky lead and it set the tone for the pitches to follow.  We got a late start and only did the first 3 pitches but they all had a bold, adventurous feel to them.

Thursday
Inti Watana has been on our tick list for a long time.  It’s a big route with a long, steep approach so we got up at 5am and were hiking less than an hour later.  I had stopped in Vegas the previous week on my way back from California to preview the approach with my friend Stan so I knew exactly where to go.  However, in my pre-dawn mental haze I parked at the wrong pullout and didn’t realize it until we had hiked the wrong trail for 15 minutes.  It didn’t turn out to be a big deal – since I had scoped the area a few days earlier I was able to correct the mistake, adding only about 10 minutes to our hike.  The approach involves a lot of 4th class scrambling and took us exactly 90 minutes – not bad considering that we were carrying full packs and did some “bonus” hiking.

The 5.10c crux of the route is on the second pitch and involves awkward semi-liebacking with poor feet.  Much of the route is protected by bolts but you need a light rack (single cams to 2 inches) to protect non-bolted sections.

Image
Pitch 7, the “S crack”, about halfway up Inti Watana

The weather was perfect and the pitches fell quickly so we finished the route in 5 hours, linking pitches 9/10 and 11/12.  Many multi-pitch routes feel like they’re pushing you down, guarding the summit, but Inti Watana seems to pull you up higher and higher.  The only downside was the lack of comfy belay ledges.  There are only two good ledges on the whole route so our feet were pretty beat up by the time we reached the top.  The route tops out on a pinnacle a few pitches below the true summit of Mount Wilson.  You can either continue up Resolution Arete and hike down the back of Mount Wilson (adding maybe 5 hours to an already long day) or rap down Inti Watana, which we did.  We rapped down in about an hour with two 60m ropes, linking 12/11, 10/9, 6/5, 4/3, and 2/1.  We ate a leisurely lunch at the bottom, hiked down, and were back at the car at 3:30pm for a car-to-car time of just under 10 hours (9.5 if you don’t count the parking error).  Inti Watana was a really fun route.  I wouldn’t say that the climbing itself is 5 star but the overall experience certainly is.  Highly recommended!

Friday
We expected to be pretty tired from the previous day’s effort, so we didn’t have any specific plans for Friday.  However, we woke up early and felt OK, so we headed to Black Velvet Canyon intending to do Triassic Sands, one of my all-time favorite routes.  However, one of the 4 climbing teams in the canyon was already headed there so we decided to do Wholesome Fullback (5.10a, 2 pitches) instead.  I had done this route before in 2008 and remembered that it was hard for the grade.  The overwhelming consensus on Mountain Project is that the crux move is significantly harder than 10a and I agree.  It felt more like 10c to me.  After finishing the route we top-roped the stellar third pitch of Our Father, a route that I want to come back and lead next year.

Image
Finishing the opening thin fingers crux on Wholesome Fullback

Temperatures at Red Rocks are beginning to top 90 degrees – a little warm for multi-pitch.  That means it’s time to move on to higher latitudes and higher altitudes.  Next stop – Eldorado Canyon…

Red Rocks Delivers

HoldemBlake on the first pitch of Texas Hold’em, Black Velvet Wall

Just returned from a long weekend of climbing at Red Rocks.  As my climbing partner Blake says, “Red Rocks always delivers”, and this time was no exception.  We’ve done many of the 5 star routes at Red Rocks so we’re starting to climb off the beaten path a bit.

Here’s our tick list:

Thursday
We flew to Vegas early, grabbed lunch at our favorite cafe, and headed to the cliffs for a little afternoon cragging…

solarPsyched x 3.  Me, Blake, and Andy at the top of Solar Flare.

Friday
We hooked up with our friend Andy and hiked into Oak Creek Canyon.  This was my first time to climb as a party of 3 where the leader brings both followers up at the same time using a Petzl Reverso.  I’m normally not a fan of 3-man rope teams but this system worked really well and was only about 20% slower than climbing as a party of 2.  You have to bring two ropes anyway for many routes at Red Rocks (for rappels), so why not use them for a third climber?

  • Solar Flare (5.10, 5p) – An OK route – wouldn’t do it again.
  • Beulah’s Book (5.9, 3p) – We did the first two pitches.  The second pitch is one of the best of its grade at Red Rocks!
  • Red Zinger  (5.10+, 2p) – Pure “Indian Creek style” crack climbing – it felt hard and sustained!  We only did the first pitch because we were getting tired and the overhanging thin hands second pitch looked a little intimidating.

photo 2 (2)The stellar second pitch of Beulah’s Book

photo 1 (1)Leading Red Zinger

Saturday
Each time we visit Red Rocks we like to spend at least one day in Black Velvet Canyon.  The canyon’s namesake wall is the crown jewel of the area.  At over 2000 feet tall, Black Velvet Wall has perhaps the highest concentration of high quality (and mostly moderate) multipitch trad routes in the world.  Andy had recommended a route called Texas Hold’Em, which checks in at 5.11c over 9 pitches.  However, the first 6 pitches can be climbed at a more manageable (but still challenging) 5.10d rating.  Black Velvet Wall is huge and intimidating, and this route had a pretty serious feel to it.  Every pitch – even the ones with easier ratings – has something to keep your attention.  No gimmes!

The crux pitch was long and sustained, beginning with an overhanging 10d fist crack followed by 150 feet of  sustained 5.10 face and thin crack.  I was psyched to get the onsight – and also exhausted both mentally and physically.  After rapping down we realized that we were the only people in Black Velvet Canyon – very unusual for a Saturday in April.  We enjoyed the quiet hour-long hike back to the car (and also the mountain of sushi we ate later).

photo 3 (1)Looking back at Black Velvet Wall on the hike out.  Beautiful and quiet.

Sunday
With a flight to catch in the afternoon, we needed an objective with a relatively short approach and a quick descent.  We have had our eye on the “new Red Rocks classic” route La Cierta Edad (5.10d, 5p) for a while so we decided to give it a go.  It was fantastic – the pitches were varied (runout face, thin crack, wide crack, chimney, etc.) and each was challenging in its own way.  The 5.9+ chimney on the second pitch gave me fits (chimneys are my weakness – Blake cruised it) but the 10d offwidth felt surprisingly easy.  Our favorite pitch was the 3rd, with a sustained 10a off-fist crack and lots of stemming.  We moved quickly and were back at the car before 1pm.

photo 5 (1)La Cierta Edad, P3

photo 4 (1)Hanging belay selfie on La Cierta Edad

Another great trip – Red Rocks always delivers.  Time to heal up and plan the next one…

 

The Weather Window

RR3

Sending Potato Chips, Kraft Boulders

For most of the country this has been a harsh winter.  Even in Texas we’ve had unusually cold temperatures and lots of ice.  When skies are grey I (somewhat obsessively) monitor the weather at my favorite climbing destinations, looking opportunities to escape to warmer temps.   Last weekend I saw a window of good weather at Red Rocks, so I gave Blake a ring and we decided to go for it.  Through the miracle of frequent flyer miles we were able to book flights and hotels at the last minute at zero cost.  And it paid off – we climbed in t-shirts and shorts while Fort Worth suffered through yet another ice storm.

Here’s our tick list:

Saturday
We flew in Saturday morning and went directly to Brass Wall to do some short warmup climbs:

Sunday
We’ve done most of the classic moderate routes at Red Rocks so we were looking for something a bit off the  beaten path.  We decided to try Orange Clonus, a 5-6 pitch 5.10d on Straight Shooter wall.  It ended up being more than we bargained for!  Both the first pitch and the crux 5th pitch felt quite a bit harder than 5.10d (and lots of people on Mountain Project agree).  After a few attempts at the crux thin hands corner (which for me was off-fingers with slick feet) I got frustrated and we bailed near the top of the route.  The descent was pretty involved due to the wandering nature of the route.  We ended up rapping a pitch, down-climbing two easy pitches, then rapping into a gully.  If you climb enough it’s inevitable that you’ll have “adventures” like that so you just roll with it.  To regain our confidence we climbed the classic finger crack Straight Shooter before hiking out of Pine Creek Canyon.

RR1

Blake styling Panty Raid, a steep, crimpy 5.10 line on Panty Wall

Monday
We had a flight to catch in the afternoon, so we went to Calico Basin and climbed Classic Crack of Calico, a wonderful 3 pitch 5.9+ on Kraft Mountain.  I can’t say enough good things about this route.  The third pitch may be the best 5.8/5.9 pitch I’ve done at Red Rocks.  After summiting and hiking down, I did a quick lap on the classic V2 Potato Chips at the Kraft Boulders just so I could say I did a boulder problem on this trip.

When we got back to the car, we had messages from American Airlines – our flights to Dallas later that afternoon had been cancelled due to icy conditions.  It seemed crazy since we were getting sunburned in Vegas.  Needless to say, we were NOT upset.  We went back to town to grab lunch and then headed to Panty Wall in the Calico Hills to bang out a few more routes:

With the forecast calling for continued sun and warm temps we were sort of hoping for another cancelled flight on Tuesday.  But all good things must come to an end…

Chilly Hueco

Bundled up on the Hueco mega-classic Nobody Here Gets Out Alive

Blake and I made a quick weekend trip to Hueco Tanks right after Christmas.  Located in the Texas desert, Hueco is one of the best winter climbing destinations in the U.S.  However, it’s still a dice roll – I’ve climbed in a t-shirt in January and I’ve also climbed with temps in the teens.  This trip was on the chilly side, with morning temps in the 30’s and afternoon highs in the high 40’s or low 50’s.  35 degrees in the shade on cold rock can be pretty tough – especially if the wind is blowing.  However, we managed to get plenty of climbing done.  Here’s our tick list:

Friday:

  • Uriah’s Heap (5.8, 2p)
  • Hueco Syndrome (5.10, 2p)
  • Malice in Bucketland (5.9)
  • All the Nasties (5.10)
  • Window Pain (5.10)

Saturday:

  • Head Fox (5.9+)
  • Fox Trot (5.10+)
  • Indecent Exposure (5.9+, 2p)
  • Alice in Banana Land (5.10-)

Sunday:

The weather was overcast, colder, and windy so we decided to skip roped climbing and boulder/explore on North Mountain.  I did a couple of laps on Nobody Here Gets Out Alive, which is considered one of the best V2 problems in the world.  I also worked Lobster Claw, a V5 problem that has eluded me for a while.  I came very close to sending but was tired from the previous two days.

All in all, a fun trip but I’m looking forward to spring weather!

Open Source Bouldering

UPDATE:  Someone on Reddit informed me that the idea I describe below has already been done in the UK and it’s called the Moonboard.  It’s almost identical to the system I envisioned.  Seems like a great idea but apparently it hasn’t proven to be very popular – probably because the number of climbers that are motivated enough to build something like this (and have the space to do it) is relatively small.  However, this is sort of a network effect idea – the more people that have it, the more appealing it becomes…

I’m been thinking about an interesting idea for rock climbing (specifically home and gym bouldering).  The idea was inspired by two things:

1) IFSC speed climbing walls, which use a standardized route pattern, holds, and wall angle so that climbers around the world can practice and compete on exactly the same route.

2) Crossfit, which uses standardized workouts called Workouts of the Day (“WODs”).  This allows people around the world to share the same workout experience, motivate (or compete with) each other (in person and online), and track their performance.  This has been a big part of Crossfit’s incredible success.

What if the ideas of standardization and sharing were applied to bouldering?  Imagine hundreds or even thousands of climbers across the world with the same simple, low cost bouldering wall at home or in their local gym.  Using the simple system described below they could set up identical boulder problems and connect with people across the country (and world) that are working the same problems.  It wouldn’t take long for the community to amass a large collection of “open source” boulder problems that could be shared, discussed, and rated by users for difficulty and quality.

Here’s how it could work:

1) The system could be based on a simple, inexpensive “woody” bouldering wall, similar to the walls many of us have in our garages and basements.  The wall size and overhang would be standardized – perhaps three 4 x 8 foot panels stacked to form an 8 x 12 foot wall.  With a 1 foot kicker panel and an overhang of about 35% this wall would fit in a room with an 8 foot ceiling.

fig1

Figure 1.  Side view of a basic wall design. Standardized, simple, and affordable.

2) Panels would be manufactured to ensure precise bolt hole locations  and, more importantly, they would have “alignment holes” around each threaded bolt hole to make sure holds are oriented properly (more on hold orientation below).  Rows and columns of bolt holes would be labeled so that the coordinates of each location could be easily referenced (e.g. “J16″ as shown in Figure 2 below).

fig2

Figure 2.  The wall panels would have a pre-drilled grid of bolt holes,
with column and row labels so that boulder problem “blueprints”
could reference hold locations.

3) One of the big challenges with this idea is hold orientation.  Any climber knows that turning a hold a few degrees one way or the other can dramatically change the difficulty.  This problem is easily solved with a peg & hole alignment system.  Holds would be molded with short alignment pegs on the back that fit into small holes around each bolt hole in order to precisely and securely orient holds in fixed positions.  This feature would be crucial to make sure that boulder problems could be accurately replicated on any Open Source wall.

fig4

Figure 3.  Each bolt hole would be surrounded by alignment holes
that enable precise orientation of holds that are molded with
matching pegs on the back.  The design shown here would support
four orientations of a hold (0, 90, 180, & 270 degrees).

4) Boulder problem “blueprints” would be a simple list of each hold to be used (holds designed for this system could each have a name), its location on the wall (column and row), and its orientation.  With this system, problems could be set from blueprints in a matter of minutes with zero routesetting skill – a big plus for people who would rather climb than set problems.  Lots of boulder problem blueprints would be available online so that climbers could choose problems by difficulty, style, quality rating, and other factors (and of course submit new problems).  The website would allow users to post beta and comments and upload photos and videos, much like Mountain Project (in fact, the community could use Mountain Project pretty much as is).  It would be easy to connect with climbers of similar ability and climbing style.  You can imagine that talented routesetters would develop a following and perhaps pro climbers would contribute.  Wouldn’t it be fun to try a problem designed by Chris Sharma from the comfort of your garage?

5) With a relatively small set of holds (maybe 50?) a wide range of problems would be possible.  Ideally, the standards for this system would be shared by all climbing hold manufacturers and they could sell hold sets designed for specific problems (sell holds by making them part of a great boulder problem!).

I like this idea for a few reasons:

  • It would make training more social and fun.
  • It would allow climbers to quickly set high quality routes on their home wall.
  • It would allow climbers to track their ticks online and compare with others.
  • Problems could be documented forever and favorites could be revisited (re-set) from time to time.
  • It would allow for benchmarking of ratings by many climbers, similar to what happens on sites like Mountain Project.

Does this idea have legs?

Going Bandit at Red Rocks

triassicThe top of P2 of Triassic Sands – one of my all time favorites.

Last month my friend Stan and I went to Red Rocks for a long weekend of climbing.  Due to the government shutdown the park was technically closed but the rangers were allowing climbers to hike in as long as they didn’t use park roads.  That meant longer hikes to some areas but wasn’t a big limitation.  Stan is new to climbing (he’s a cyclist) and this was his first real outdoor trip so we mostly stayed on easy stuff.  Here’s our tick list:

Friday:  Oak Creek Canyon
I’ve climbed at Red Rocks at least a dozen times but had never visited the Solar Slab area in Oak Creek Canyon.  Our objective was the classic moderate Johnny Vegas on Lower Solar Slab and maybe something on Upper Solar Slab if we moved quickly.  Unfortunately, several other parties had the same idea – including some guys from an AMGA course that was underway.  So we didn’t get started on Johnny Vegas until almost noon.

  • Johnny Vegas (5.7, 3p)
  • Arch Enemy (5.9, first pitch only)

Saturday: Black Velvet Canyon

  • Schaeffer’s Delight (5.7, 3p)
  • Triassic Sands (5.10b, pitches 1 & 2 only)
  • The Misunderstanding (5.9, fist pitch only)
  • Miss Conception (5.10a, 2p)

Sunday: Lots of hiking + Calico Basin
Our plan was to do the moderate mega-classic Olive Oil.  We hiked in from the highway (not the normal approach) and after an hour and a half of looking for the route I realized we were in the wrong canyon (Juniper Canyon).  Dumb mistake.  By that point we weren’t too motivated to hike further in (since we had parked on the highway) and do a long-ish route so we hiked back out, grabbed lunch, and went to Calico Basin.

  • Over the Hill to Grandmother’s House (5.9+, 1p)
  • Baseboy Direct (5.11a, 1p, sport)
  • Caustic (5.11b, 1p, sport)

Monday: Calico Basin
We had to catch a flight in the afternoon, so we headed back to Calico Basin and did a few routes at Gnat Man Crag.

  • Bottoms Up (5.7, 1p)
  • P-coat Sleeve (5.10a, 1p)
  • Knock the Bottom Out Of It (5.10a, 1p)

Big Data Democratized

How I mashed-up Google and NOAA’s massive computing infrastructure to answer a Big Data question in 3 hours with no tech skills.  (It cost me $1.39.)

top25

Democratization of Big Data
2.5 exabytes – the equivalent of one trillion digital photos – is captured each day by computer systems across the globe.  This number is growing exponentially with proliferation of mobile devices, cameras, and other information sensors at the edge of the network.  This presents both an enormous challenge and a game-changing opportunity.  The challenge is how to deal with data at a scale that is almost inconceivable.  The opportunity is this:  If we can tame the data, it will usher in the next generation of technological breakthroughs.   It’s already happened in fields as far-flung as genomics, banking, and national security and it’s still early days.

Big Data refers to datasets so large that they exceed the capabilities of traditional database systems.  By today’s standards that means a few dozen terabytes.  Datasets this large require big tech infrastructure – massively parallel software running on hundreds, thousands, or in some cases millions of servers.  Well-funded, tech savvy companies are investing heavily in this technology, but what about startups and other small businesses that can’t afford the computing infrastructure required to deal with their own ever-expanding data assets?  Small businesses – or even individuals – might want to use big data to out-fox larger competitors or simply make better informed decisions.

It’s no secret that cloud computing has removed barriers to access and driven the cost of computing power down dramatically.  But just how far?  With this question in mind, I set out to conduct a small experiment to determine how quickly an individual with limited tech skills could spin up a big (or at least big-ish) data infrastructure and use it to solve a simple but data-intensive problem.

The Experiment

Which US cities have the most days of perfect weather?

It’s a question I ask myself each September, after enduring five months of relentless heat and humidity in Texas.  As an outdoor enthusiast, weather plays a major role in my psyche.  Too often I feel like I’m limited to just a few short, precious windows of great weather.  Maybe it’s time to relocate – but where?

There’s no shortage of weather information on the web, but most of it is based on averages – and averages often don’t tell the whole story.  I was more interested in a new metric:  Perfect Weather Days.  Put simply, how many days each year is the weather ideal for outdoor activities?  The definition of a Perfect Weather Day is, of course, highly subjective.  My personal definition of Perfect Weather had four simple criteria:

  1. No precipitation (it doesn’t necessarily have to be sunny as long as there’s no rain or snow)

  2. A low temperature no lower than 50 degrees (as they say in Texas, I’m “warm blooded”)

  3. A high temperature of at least 65 degrees (after all, it’s not a perfect day if you can’t wear a t-shirt and shorts)

  4. High temperature no higher than 82 degrees

I considered other criteria such as humidity and wind speed but in the end decided to keep it simple.  If there’s no rain and good temps then I’m not going to complain.

My first task was to find a source of daily weather data for US cities.  It turns out that NOAA makes its enormous database of global weather data available for free.  Using an online tool on the NOAA site I submitted a request for a dataset containing the following daily measures for thousands of weather stations across the US for the 8 year period between 2005 and 2012 (I learned on the NOAA site that the quality of data prior to 2005 is lower):

  • station ID

  • latitude / longitude

  • date

  • precipitation

  • low temperature

  • high temperature

It took about a day for NOAA to deliver the dataset, which totaled 46.9 million records.   Granted, this may not technically qualify as big data.  However,  it turns out that I could have just as easily completed the project with a dataset 10 or even 100 times larger.

The Technology
Just a few years ago the idea of spinning up IT infrastructure to answer one trivial question would be preposterous.  But with a few minutes of digging, I discovered BigQuery, a new product from Google that seemed to be purpose-built for my little experiment.  In Google’s words:

Querying massive datasets can be time consuming and expensive without the right hardware and infrastructure. Google BigQuery solves this problem by enabling super-fast, SQL-like queries against append-only tables, using the processing power of Google’s infrastructure. Simply move your data into BigQuery and let us handle the hard work.

Score!  In less than an hour I was able to upload my NOAA dataset to Google and start using a web-based SQL query tool to play with the data.  Keep in mind that I have almost no programming skills – I remembered a bit of SQL from college and spent a few minutes reading the BigQuery documentation.  With a little trial and error I came up with a query to rank order weather stations by the annual percentage of Perfect Days.  I was blown away by how fast and easy BigQuery is.  My queries usually took just 3 or 4 seconds to run and I could move large datasets around just as quickly. I didn’t have to think about database schema, indexing, or optimization at all.  It all just worked.

Here’s the query I used (it’s not nearly as complex as it looks).

SELECT
weatherdata.stations_US_clean.station as station,
weatherdata.stations_US_clean.state as USstate,
weatherdata.stations_US_clean.name as stationname,
weatherdata.stations_US_clean.latitude as lat,
weatherdata.stations_US_clean.longitude as lon,
count(weatherdata.2005_2012.date)/1460 as percentperfectdays,
count(weatherdata.2005_2012.date)/4 as perfectdays,
count(weatherdata.2005_2012.date) as totalperfectdays
FROM [weatherdata.2005_2012]
JOIN [weatherdata.stations_US_clean] ON weatherdata.stations_US_clean.station = weatherdata.2005_2012.station
WHERE weatherdata.2005_2012.prcp=0
AND (weatherdata.2005_2012.tmin*.18 + 32) >= 50
AND (weatherdata.2005_2012.tmax*.18 + 32) >= 65
AND (weatherdata.2005_2012.tmax*.18 + 32) <= 82
group by station, stationname, USstate, lat, lon
order by percentperfectdays DESC

The Results
With BigQuery I was able to crunch 47 million records down to a summary of Perfect Weather Days for 7,102 weather stations across the country.  This dataset was small enough to move into Excel for more analysis.  It’s worth noting that the weather data is for weather stations, which don’t necessarily correlate to cities (many weather stations are located on mountain tops, in national or state parks, or other remote areas).  So I had to manually map top-ranking weather stations to cities using latitude/longitude coordinates and Google Maps.

As I sifted through the results I noticed a couple of things.  First, it was clear that California dominated the top of the list.  No surprise there – California weather is legendary.  Second, I noticed that many cities that ranked high in number of Perfect Days are cities that I know to have extreme weather.  In other words, they may have a large number of Perfect Days each year, but they also have a lot of bad weather, making it impractical (or at least uncomfortable) to be outside.  For example, my home town, Fort Worth, might have 40-50 Perfect Days a year (not too bad) but we also get months on end of daily highs near 100 degrees.  Conversely, other cities might have mild summers but brutally cold winters.  So I concluded that I needed to also consider the number of Bad Weather Days for each city.   I defined a Bad Weather Day as a day that meets any of these criteria:

  1. Half an inch or more of rain

  2. Low temperature above 85 degrees

  3. Low temperature below 15 degrees

  4. High temperature above 95 degrees

  5. High temperature below 32 degrees

(Again, this is highly subjective.  If you’re into snow skiing, a day with clear skies and a high temp below 32 might be perfect.)

I then combined Perfect Days and Bad Weather Days into a score using this simple formula:

Weather Score = Perfect Days – ½ * Bad Days

My logic was simple (and yet again, subjective):  It takes two Bad Weather Days to erase the joy of a Perfect Day.  Now I could rank order each weather station based on my Weather Score.  Back to the results…

At the top of the list is Kula on the Hawaiian island of Maui, a true paradise where 74% of all days are Perfect.  While Hawaii claims the top spot, California dominates the top of the list with San Diego and LA running neck and neck for the highest scores in the continental U.S.  Those cities average over 200 Perfect Days each year with less than 10 Bad Weather Days.  In contrast, Fort Worth gets 47 Perfect Days but 86 Bad Days (about 2 Bad Days for every Perfect Day!).

Here’s a list of the top 25 cities by Weather Score:

top25

And on a map…

map

Every city in the top 25 is coastal.  I probably should have included humidity or heat index in my definition of perfect days since I know first hand that 80 degrees in Key West can be far more uncomfortable than 85 in Phoenix or Boulder.

I also ranked the 50 largest US cities by weather score.  It’s interesting how sharply the number of Perfect Days drops off outside of California.

top50

The bottom of the list was dominated by Alaska.  No surprise given that my definition of a Perfect Day involves fairly warm temps and no precipitation.  In dead last is Deadhorse, Alaska with only 2 Perfect Days and 235 Bad Weather Days each year (I’m guessing that the town’s name was inspired by its brutal winters).  In the continental U.S., the lowest score goes to Lajitas, TX in Big Bend country, with just 5 Perfect Days and 154 Bad Days.  (Interestingly, just 30 miles away in Big Bend National Park is the Chisos Basin which ranks near the top of the list.)

For those of us that prefer mountain air, I took a quick look at locations above 4,000 feet of elevation.  Many of these locations (which were mostly in California) didn’t correspond to cities, so there was a lot of noise in the data.  For example, the highest ranking location above 4,000 feet is the aforementioned Chisos Basin in Big Bend National Park with 83 perfect days and 16 bad.  Yosemite Valley (my favorite place on Earth) also ranked high.  The highest ranking cities at altitude include:  Carson City (NV), Alamogordo (NM), Cortez (NM), Sierra Vista (AZ), Silver City (NM), and Provo (UT).  I was surprised that one of my favorite mountain towns, Boulder, CO, didn’t rank very high with only 14 perfect and 48 bad days.  It’s clear that mountain towns get penalized by my definitions of Perfect and Bad, which favor warmer temperatures.

The Bill
When all was said and done, this experiment confirms the obvious  – for great weather California wins by a landslide.  But it also provides a quantitative measure to compare other cities.  My home town of Fort Worth scored a 4 compared to a 203 in LA.  Ouch.

But the most interesting part of this project was the project itself.  With about 3 total hours of late night work I was able to sift through an impressive amount of data to answer my question.  And because BigQuery runs on top of Google’s massively parallel infrastructure, I could have completed the project in about the same amount of time if my dataset had been billions of records instead of tens of millions of records.

What about the cost?  Here’s my statement from Google totaling $1.39.

bill

That’s $0.19 for storage, $1.13 for BigQuery compute cycles, and $0.07 in tax.

Clearly democratization of Big Data is happening, and that means small businesses and even individuals like me can take advantage of massive computing power for almost no cost and with minimal technical skills.  Imagine millions of businesses and individuals with the power to hack huge amounts of data covering just about every field of human interest.  It’s like the Internet all over again – the Internet of data.