How a few dedicated Amazonians saved Christmas
And other stories from a collection of old t-shirts
A few days ago I finally got the courage to clean up my closet. I’m terrible at this and I’ve literally never thrown away an old t-shirt in my life regardless of stains, holes and overall fit. Some are most definitely never to be worn in public again. Some probably shouldn’t be worn in private either.
I’ve always suspected I had built a vast collection of t-shirts, sweaters, hoodies and coats from my days at Amazon, Microsoft and Google, but it wasn’t until I started to sort them out I realized just how many (about 30!).
The other thing I realized is that despite my best intentions to get rid of some of my stuff, I simply could not get rid of any of those shirts. Each one had sentimental value, because it held a story and it brought back a feeling. Someday I’ll pass away and whoever has the job of cleaning up my closet will look at them, shrug and put them either in the donation pile or in the garbage, but until then, they’re safe in a section of my closet.
How a few dedicated Amazonians saved Christmas
The amazon.com site has thousands of critical moving parts. Most of them you take for granted, some you don’t even know exist, but if they were to break, you would notice pretty quickly.
As we were nearing Christmas of 2014, the Wishlist feature was in trouble. Wishlist is a neat little feature of the retail site where you can create a list of things that you would love your friends and family to purchase for you, perhaps for your birthday, or Christmas. People who aren’t great at naturally picking appropriate presents without a little hinting (such as myself) very much appreciate this. After 20 years of marriage, my wife has given up on dropping subtle hints and she uses Wishlist extensively, so I personally love this feature. And, apparently, so do millions of other Amazon customers worldwide.
Wishlist was a fairly old crufty service, with a few architectural bottlenecks. An engineer in the team cranked up some numbers and forecasted that given the traffic trends, it was very likely to die a few weeks before Christmas. We could keep going, and take the risk of a very high profile operational issue during the season. We could disable the feature entirely until we could figure things out, but that might be after Christmas, and millions of people had added millions of things to their wishlists already.
Amazon does Customer Obsession like nobody else. Neither option was acceptable to either the engineers or the leaders. So the decision was simple. We’re going to rewrite, test and deploy, an entirely new Wishlist service, able to handle Amazon production scale, in four days. That was an ambitious goal.
I was not in the team, and I wasn’t even under the same SVP. But for years I had been involved in load and performance testing efforts, tools and guidelines across the entire company, so my name came up as a Tribute. Brad, one of the Distinguished Engineers from the org, asked me if I could make some time to help out. I innocently said “Sure! When?” There was a pause. “Hum, right now?” I almost choked on the coffee I was sipping. I had underestimated the urgency. I cleared it with my manager right away, and then I headed over to the War Room in the other side of Amazon’s campus.
War Rooms in the tech world are physical or virtual groups of people who get quickly pulled in to handle an emerging situation. More often than not, it’s a virtual group responding to a high profile operational issue. In this case, it was a physical group of people gathered in a big room. When I walked in, there were boxes and arrows drawn in every possible square inch of the whiteboards, and an impressive gathering of Principal Engineers. We were definitely not in Kansas anymore. I was in awe of being in the same room as some of these people.
My specific job was to learn about the product, come up with a test strategy, and implement it, so that we could validate that the new version of Wishlist was going to improve things. And to do that in the next 24-hours. Being in a physical War Room was a really neat experience, because you could feel the energy, urgency and excitement in the room, and you could exchange ideas and iterate very quickly. It was Thursday. We stayed late into the night, drinking large quantities of coffee and soda. At 7pm, the VP ordered a bunch of pizzas. At 9pm, I wasn’t feeling so sharp anymore so I went home, but came back the next morning around 7am.
I had the easy job. I finished the test strategy, and had a working suite of tests, ready to go by Friday afternoon. The engineers who were actually rewriting the system stayed through the weekend. That was a much harder job! I did remain on-call just in case they had problems with the tests and needed my help. By Monday morning, there was a brand new Wishlist ready to be deployed. I ran all the tests, looked at a plethora of dashboards to assess how the system was scaling, and gave the final signoff for deployment to prod.
By Tuesday, Amazon.com was running an entirely different Wishlist service, happily serving millions of people their precious data. It handled the load flawlessly during that season. Likely none of our customers ever knew just how hard a dedicated group of Amazonians had worked over a very intense 4–5 days, to save Christmas.
And, we all got shirts to commemorate the experience!
As I was cleaning my closet, I couldn’t help but smile when I looked at my Wishlist shirt. No, I absolutely could not get rid of that shirt. It captured a precious and intense moment in time.
Up the River and into the mountains
Somewhere in 2014, I saw one of the Directors I worked with wearing a cool hoodie. It had the logo “Up The River”. I wanted one! So I asked him how he had gotten it. He smiled and said, “You’ve got to earn it!”
I’m a sucker for a challenge so that definitely piqued my interest. Once a year, Amazon rented Suncadia, a vast, plush resort tucked away in the Cascade mountains, about two hours from our headquarters in Seattle. Every Director got to nominate a couple of their engineers, submitting a written justification on why they deserved to go. Attendees were generally expected to be Top Tier (the highest performing engineers in the company) and hand-picked. Level didn’t matter as much, as one of the goals was to expose hungry and promising junior engineers to more senior role models to accelerate their career. The idea was: let’s get together Amazon’s most driven engineers, have them eat, drink, play and talk for 2–3 days in the middle of nowhere, and let’s see what kind of cool ideas get cross-pollinated.
Up The River had just happened, so I had an entire year to work towards being nominated for the following Up The River. I wanted that hoodie!
The following 12 months were some of the most intense months of my life, and I loved every second of it. I had just been promoted to Senior Engineer, and I wanted more. I had a great project, a great manager and a great Director, and I was not coy or shy about my desire to be in his list for Up The River.
My heart stopped a beat the day an email with the title “Congratulations! You’ve been selected for Up The River 2015!” showed up on my inbox. I also realized that this was a responsibility and a privilege. My Director had used up one of his very few spots to hand-pick me, and I didn’t want to let him down.
Suncadia was a gorgeous setting, with mountains and forests all around. There were maybe 200 Amazonians, of all levels, roles, tenures, and teams. A lot of it was Open Space format, so anybody could get up on stage, pitch a topic, pick a time and a place, and host a session. If you were interested in that topic, you’d show up for the session and have a spirited debate with whoever else happened to show up. We shared meals together, exchanged ideas, made friends, came up with plans, drank large quantities of coffee during the day and even bigger quantities of whiskey after dark, and played a massive and vicious nerf-gun battle well into the night, to the dismay of the resort staff.
Up The River no longer exists, sadly. I think the company got too big. It’s hard to quantify the impact of a conference (and education and networking in general), but I can tell you this. Up The River 2015 was where I developed some critical relationships that have lasted my entire career, and it was a definite inflection point in my transition from a Senior Engineer to a Principal Engineer. And, of course, I did get a fine hoodie out of it.
Do Amazonian Vampires Exist?
A hoodie with a vampire-inspired Amazon logo is awesome in general (particularly at Halloween), but the fact that it has its own story makes it even more important to me.
Romania holds a special place in my heart. In 2010, I joined a team that had half its staff in Seattle and the other half in Iași, Romania. And so a lifetime of business travel started for me. I would end up traveling to Iași six times in four years to visit my Romanian counterparts, and becoming a manager of ten engineers there for a while. My wife tagged along on the first trip. We rented a beat up Dacia and drove all over the Carpathians, exploring impossibly picturesque villages and castles that looked like Dracula movie sets. The country is stunning. Another time, my connection to Bucharest was in Istanbul, so I spent a week backpacking through Turkey. Another time, I found out just as I was landing in Bucharest that the Iași airport had unexpectedly closed, so I ended up improvising and after much arguing in broken English with a stern official, taking an unforgettably rickety overnight train to Western Moldavian. I loved the people, I loved the country, and I loved heading over there every six months.
I had not really done a lot of business travel prior to joining Amazon. Amazon opened my eyes to the world, as I would end up traveling to all continents except Antartica… 44 business trips (mostly international) in 11 years. And the very first trip, the one that started it all, was that trip to Iași.
So when my Romanian coworkers decided to make an Amazon Vampire shirt as a joke given all the times people asked them about Dracula, I was ecstatic to wear it. And I’m still today.
Pay It Forward
This is not one but two long-sleeve shirts, a few years apart, with a Pay-It-Forward story.
In 2012, I was looking for ways to increase my scope and responsibilities, and pushing myself out of my comfort zone in as many ways as I could. Amazon had a company-wide weekly technical talk series called the Principals of Amazon (“POA”) Talks. It had a large audience of maybe ~1000 people (in-room and connecting remotely). Anybody who wanted to speak at a POA was assigned a couple of opinionated Principal Engineers as coaches who gave endless feedback over endless dry-runs. These Principals had to eventually approve it as worthy of the POA brand before the speaker went up on stage. Giving one of these talks was a badge of honor, and I was very uncomfortable with public speaking, so I signed up to do one.
Once you successfully gave your POA talk, you got a shirt that you could proudly wear around the Amazon campus to show off your achievement.
Fast forward to 2014, and I was promoted to Principal Engineer. My experience as a POA Speaker had been very positive, and that was in huge part thanks to my POA Coaches. These Principals had patiently listened to many versions of my talk, and insightfully suggested many improvements. I had grown significantly from the experience. It was time for me to pay it forward and be on the other side. So I volunteered to be a POA Coach.
I shepherded a few fantastic talks through the POA process, and enjoyed seeing the people I mentored becoming Principal Engineers themselves within a few years. And, guess what? I got a second shirt, this time, the “Coach” shirt!
I like wearing these shirts because both are badges of honor. But I love wearing them because they remind me, every time, that as I grow in seniority, it is my duty to mentor the next generation of engineers that will follow me.
