2020 a Year in Review

Even writing this feels weird, but so many of the year’s rituals have been up-ended or discarded, that it seemed necessary to continue with one that I can easily do on my own, so here we go. 2020 presented a lot of challenges, however I was incredibly fortunate and privileged to have been able to work from home this whole year, stay in touch with family and friends via Zoom, Teams, and FaceTime, and escape into the outdoors often to recharge.

Work

The little Microsoft Research (MSR) team I joined in the summer of 2017 continues to grow and hit major milestones.  On February 24th 2020 Azure Sphere became a Generally Available Product. 

I also appeared on the Azure Friday show talking about End-to-end Iot device security with Azure Sphere and the Internet of Things Show to give an Introduction to the Azure Sphere Security Service.  Both of these were recorded pre-covid hence the lack of social distancing and masks.

My team, the Azure Sphere Security Services Team, continued to grow hiring 8 new developers and 2 new PMs.  I couldn’t be more proud of this team for its resilience, collaboration, and willingness to tackle challenges while caring for each other. 

Writing

I wrote  two blog posts in 2020.

I also wrote two essays for the book 97 Things Every SRE Should Know

  • On-Call Health: The Metric You Could Be Measuring
  • Helping Leaders Prioritize On-Call Health

Wanderlust

In February I spent a long weekend on Oahu, HI playing ultimate frisbee with my women’s master team Mint at the Kaimana Klassik.  The sun, sand and surf were a warm welcome from gray rainy Seattle.  We played on fields lined with palm trees, and flanked by mountains in the distance.  There was a considerable amount of wind which made moving the disc on offense a struggle at times, but overall spirits were high.  So high that we won the tournament’s team Spirit Award, a highly coveted prize!

I fondly remember one of the evenings sitting around playing cards, and laughing so hard my stomach hurt and cheeks ached from smiling.  The sport of ultimate frisbee  and its community continues to bring me an immense amount of joy. Even during Covid, when we could not play on the fields together, we found ways to support each other remotely, and took up a new form of virtual competition via weekly Zoom Trivia. Team ConfineMINT has had several strong showings this year.

Covid kept me grounded and confined to Washington state for the vast majority of the year, as all travel got indefinitely postponed.  Thankfully Washington state has so much beauty and variety to offer, that once I got over my initial angst at having to cancel highly anticipated trips, I felt an immense amount of gratitude to live in such a stunning locale.

I relied heavily on escapes into nature this summer to reset and recharge from the constantly online life I now was living.  From July through November I was perpetually escaping into the backcountry, hiking over a 150 miles, and many of those carrying a 30+ pound pack.  We hiked to alpine lakes, watched sunsets at fire lookouts and camped on top of ridgelines.  The days and nights I spent in the back country this year were some of the best parts of 2020. 

I started a public Instagram account IntrepidTechie to practice and share my photography and post pictures from my explorations, feel free to follow along. 

Books

I read 36 books this year.  As I often did as a teenager I escaped into books as a way to explore, while stuck in a particular spot.  Some favorites by genre include

  • Memoir – Thirst: 2600 Miles to Home: Tells the story of Heather Anderson’s Fastest Known Time (FKT) record attempt on the Pacific Crest Trail.  Anderson is a masterful story teller, and this was a gripping account of her attempt, I did not know the out come when I picked up the book, and found myself rooting and cheering for her as the book progressed, and feeling waves of elation when she reached the northern terminus. 
  • Fiction – Disappearing Earth: Julia Phillips weaves a tail of two girls in north eastern Russia who go missing.  The story is told through a series of chapters each, each told from a different perspective and exploring a different set of characters on the Siberian peninsula.  Each chapter is a masterful short story in and of itself with remote and rugged vistas and nuanced characters.  Phillips deftly links all the stories together in the final chapter in a surprising satisfying conclusion. 
  • Non Fiction – The End of Everything (Astrophysically Speaking) – Katie Mack is an astro physicist and describes the leading theories on how the universe will end.  This book is exceptionally well written, and Mack clearly and passionately describes the physics, in simplified terms, to take the reader on a journey from big bang to universe destruction.  I found myself giddy reading this book, remembering my summer working at Stanford’s Linear Accelerator Center as an intern.  There I heard about quantum physics and dark matter for the first time, and how it broke and stretched my brain simultaneous.  Mack’s book brought back all the excitement, wonder and awe I felt that summer, along with new knowledge along with feelings of amazement at the workings and mystery of the universe around us

What’s Next

As 2020 comes to a close and 2021 begins, a lot will still not be back to normal, so its difficult to say what’s next. I am eagerly anticipating the return of ultimate frisbee hopefully at some point in 2021.  I am also looking forward to next summer and backpacking as my list of hikes and wilderness explorations continues to grow.

Upon reflection, the opportunity of 2020 was a lesson in resilience.  I learned to continue, to change, to move forward. To practice not forming attachments, and living in the present, even as we desire so feverishly to get back to that fictional normal state.  It gave me the opportunity to practice gratitude even when it was challenging.  There were plenty of opportunities to feel the angst, the despair, the discomfort acknowledge it, feel it fully, and then figure out how to move past it.  With all the challenges it presented it gave me a chance to take stock, to think, and reflect on what’s important. 

So with all that said I know that 2021 will come with its own set of challenges and opportunities to explore and experience.

2019 A Year In Review

My last blog post talked about 2017 being a year of change.  2018 and 2019 have been ones of intense growth and discovery.  It’s felt like a whirlwind, so let’s catch up on the basics. 

Home

I’m still living in and loving Seattle.  Moving back here proves over and over again to be the right decision for me.  I was recently asked if I could live anywhere in the world where would you live?  As participants went around the circle answering the ice breaker with exotic locales like, New Zealand, Italy, France, I pondered the question.  I’ve lived in 5 cities in the U.S. and traveled to dozens more domestically and internationally, but for me the answer is Seattle.  

Plus that house I bought, is amazing.  I’ve settled in, made it my own and created a mini indoor jungle in the living room full of tropical plants including my giant Calathea which just bloomed for the first time in two years.  The roof deck continues to be my favorite feature of the house. I’ve watched dozens of sunsets up there.  I often would rush home from frisbee practice, or being out with friends, climb the three stories to the roof just in time to catch the sky burst into pinks and oranges, staying up there til grays, blues and purples took over and I was ready for bed.  The roof deck has also been host to some incredibly fun parties this year, and I’m looking forward to hosting even more now that I’ve finally purchased a grill. 

Work

I’m also still working for that little team I joined in Microsoft Research (MSR) back in June of 2017, however we are no longer in MSR and we are no longer so little.  In April of 2018 we announced the Azure Sphere project to the world at RSA, a major security industry conference.  Azure Sphere is an end to end solution for securing Internet of Things (IoT) devices.  It’s a vertically integrated solution including custom silicon, operating systems, and cloud services.  It is by far the most full stack project I’ve ever worked on. 

Since joining the team I’ve not only architected and implemented the cloud services, but stepped into a role leading the Azure Sphere Security Services team as well.  In the past 2.5 years I went from Individual Contributor (IC) to Dev Lead, to my current role as Dev Manager of multiple teams.  The team’s grown to include multiple disciplines including Developers, Project Managers, and Site Reliability Engineers.  I’ve had the great privilege of hiring everyone on the current Azure Sphere Security Services team with the exception of one developer who hired me :).  The services team is a kind, caring, collaborative, fearless, and inclusive bunch of people, and I love working with them. 

It has certainly been a wild ride so far on such a fast growing project and team.  We are currently in Public Preview, and have customers like Starbucks.  Azure Sphere MCUs are currently available from one silicon manufacturer MediaTek, and in 2019 we announced that Qualcomm and NXP will be making Azure Sphere certified chips as well.  We ship our services daily, and new features in our operating system quarterly.  We are moving fast, and currently working towards our General Availability date in February 2020!

Ultimate Frisbee

In 2018 I began playing with a Masters (30+) Women’s Ultimate Frisbee team in Seattle called Mint.  In 2019 I began my second season with the Minties.  The camaraderie and friendships I’ve experience on this team have been truly special.  We work hard, have fun, and push each other to be better.  We party like we are back in college together, chow down together, and run wind sprints together.  Growing up and pursuing Computer Science, I have frequently been surrounded by groups of men, often I’m the only woman in the room.  Playing with Mint has been the antidote for the gender imbalance in my life.  

Post Practice Socializing

Perhaps the most special moment occurred at the end of Nationals this year, played on an incredibly hot weekend at the end of July in Aurora Colorado.  We had just won our game to claim 9th place and hold seed.  It was a battle from behind the entire game.  At one point we were down 9-12.  But this team is gritty.  We went on a tear scoring a streak of 4 goals to win the game.  After the cheers, spirit circle, and dancing to Lizzo’s song Phone, we squeezed together sitting cross-legged under a shade tent for a round of popcorn.  Popcorn is an activity where each person picks a teammate and gives them a compliment.  This continues until everyone on the team has been praised.  Typically this takes 15 minutes, but this session went on for over an hour, as love and admiration for our teammates poured out.  A single compliment from a single person wasn’t sufficient.  Teammates plus one’d, snapped, and piled on espousing everything from killer lay out Ds and Hucks at the tournament, to loving someone’s laugh and kind spirit, to their mad dance skills.  By the end we were all nursing aching stomachs from laughing too hard and wiping tears away from our eyes.  My heart felt so full, I could not love this group of women more. 

Wanderlust

Even though I love Seattle, I’ve continued to fuel my wanderlust resuming my international and domestic travels in 2019.

Nor Cal

I traveled back to San Francisco and Northern California twice this year.  In February I spent a long weekend practicing yoga, lounging in a cozy cabin, eating vegetarian food, and disconnecting from the internet at a Tibetan Budhist Retreat Center, Ratna Ling Lodge.  I’ve been practicing yoga on and off for a decade now, and I always find the time flowing through vinyasas incredibly centering, no matter how long I’ve been away from my mat. 

Scribe Winery

In October I spent another long weekend celebrating my 33rd birthday with friends from San Francisco and Seattle in Sonoma.  We spent the weekend wine tasting and lounging at the Carneros Resort and Spa.  It was fun to get the two groups together, and to catch up with my San Francisco people. 

Galapagos & Peru

In April I set off on a bigger adventure heading back to South America on an REI Adventures Trip.  I spent the first week island hoping in the Galapagos hiking, snorkeling, kayaking, and even driving a boat.  The Galapagos have been a dream destination of mine for a long time.  Biology and evolution have always fascinated me.  I even took Evolutionary Biology in college incurring the ire of several pre-med students as I messed up the curve on a course that was required for their major.  Traveling to the islands which inspired Darwin’s Theory of Evolution blew away all of my expectations.  Being in the Galapagos is like living inside a Nature Documentary.  Within an hour of arriving I saw a Darwin Finch perched atop a prickly pear cactus, and then in another moment, a sea lion, a sally crab, and a blue footed booby all perched on the same rock outcrop. 

A Darwin Finch perched on a Prickly Pair Cactus

I snorkeled with sea turtles, and sea lions.  The sea lions there are just playful water puppies with no fear of humans, and so we dove and swam circles around each other.  One even played chicken with me as we swam towards each other blowing bubbles than rolled away to swim to the side or under me at the last moment.

Isabella Island

On the second leg of my South America trip I headed to Cusco Peru, to explore the Sacred Valley and visit Machu Picchu.  In the scared valley Incan ruins, and alpacas were a common feature of the landscape.  The Andes surrounded us soaring towards the sky.  The equator allows for giant green covered mountains which tower at 13,000 – 14,000 ft above sea levels.  Snow and glaciers were reserved for only the tallest peaks pushing 19,000 ft of elevation or more. 

One night before heading to Machu Picchu we stayed at a resort where I had a private casita nestled at the base of a mountain in the Sacred Valley.  The rooms were incredibly comfortable but I loved the quiet and the calm even more.  The darkness settled in blanketing us.  revealing the southern constellations, including the Southern Cross.  I’ve been to the southern hemisphere before, but on this trip seeing the Southern Cross made me realize and consider how limited our perspectives are.  Something we have always known to be true, what stars are in the sky when we look up, is not quite so universal, and all it requires is a change of location. 

Incan Ruins and Alpacas

Visiting Machu Picchu was everything I imagined and more.  We took the train in and spent two days in the ruins.  I had seen all the picture, and wasn’t quite sure what it would be like to finally experience this much photographed place in person.  It was stunning.  Photos cannot truly capture the etherealness of the place.  A giant Incan city perched atop a mountain top, with a river wrapping around its base.  It is hard to fathom that is was possible to construct, even though the proof of its existence is directly in front of you. 

On our first afternoon, we spent our time exploring the city, passing by iconic view points, and learning about the mountain gods that surround the city and the totems the Incans built to honor them.  As we wound through the various sections a light rain broke out, and then another Incan god, the Rainbow graced us with her presence.  Standing in that city, it was easy to see how the Incans could worship the surrounding mountains, rainbows, and thunder as deities. 

Rainbows at Machu Picchu

On our second afternoon, I headed up Montana Machu Picchu.  This is the taller of the two mountains you can climb, and it overlooks the city.  The climb is strenuous.  You start at 8,000ft of elevation and climb 2,000ft straight up, since the Incans were so kind as to build stone staircases up the mountain.  During the trek I could definitely feel the elevation, my breath would become short, and my legs would start to feel wobbly, like Jello.  When this happened a quick 5 minute break to catch my breath would make me feel as good as new!  It was all worth it though.  The views from the top were outstanding.  It was a clear blue sky day with almost no clouds, so we had 360 degree visibility of the Andes, and Machu Picchu looking like a doll house tucked next to Huayna Picchu below.  We sat and took in the view for 30 minutes or so before heading back, but I could have stayed for hours. 

Machu Picchu viewed from Montana Machu Picchu

Todos Santos Mexico

In November as the days grew shorter and darker in Seattle, my boyfriend, Will, and I migrated south to spend a week on the Baja Peninsula.  We spent the week lounging pool side at Hotel San Cristobal.  We had no agenda for this trip besides to relax, and eat all of the tacos of course.  We consumed 52 tacos between the two of us from 7 different places and I kept a running log on Instagram of our Tacumentary. 

Balandra Beach

In between consuming tacos we managed to explore a lot of the wonderful things in Baja.  We took a day trip to La Paz to snorkel with Whale Sharks and swim in the calm waters of Balandra Beach.  We rented ATVs and went cruising inland towards the Sierra de la Laguna mountains using arroyos, dried riverbeds, as our roads.  We explored a hidden beach which required a journey down an unmarked dirt road and a short trek through a Palm Tree forest.  We watched many spectacular sunsets.  One from a restaurant aptly named the Mirador, which sat on a high point looking out over the water.  We watched another while sitting in the Hotels hot tub sipping a frozen watermelon margarita and watching a local band play, and yet another from the comfort of our rooms balcony.  All of them were stunning. 

2017 a Year in Review

2017 was a year of change, personal and professional.  I started the year in San Francisco, working at Twitter as an Individual Contributor, and in a long term relationship.  I ended the year in Seattle, working at Microsoft Research as a Lead, sans that long term relationship, and a brand new home owner.  

Change can be terrifying, especially when you are comfortable, when you are content.  Nothing was terribly wrong, but I got the nagging feeling that perhaps nothing was going terribly right either.  I was no longer content with being content.  So in 2017 I began to change some things up to make space for new opportunities.  

I made a conscious effort in 2017 to be less busy, to travel and speak a bit less.  2016 was a year of constant travel visiting 19 cities, 7 countries, and 3 continents.  I visited Twitter offices, spoke 15 times at conferences and meetups, and managed to squeeze in trips to see family and friends.  It was an amazing experience, but not a sustainable one for me.

So I made a conscious effort to slow down and was incredibly selective about the talks and travel I took on.  I declined several opportunities to speak and travel to great conferences and locations this year.   I wanted to take a moment to thank all the conference organizers who reached out, I greatly appreciate all of the invitations and fantastic opportunities and unfortunately did not have the bandwidth to do more this past year.  

Talks

I gave versions of my The Verification of Distributed Systems talk to larger audiences at Devoxx San Jose in March, and Velocity San Jose in June.  While I’ve given this talk numerous times, I think it’s perennially important, and people consistently tell me how much they learn from it.  

I wrote a brand new talk Distributed Sagas: A Protocol for Coordinating Microservices which I gave at J on the Beach in May and at Dot Net Fringe in June.  This was a passion project for me, as I’d been exploring the ideas for multiple years, and wanted to share the progress I had made.  

I also wrote another new talk for the inaugural Deconstruct Conf in Seattle, The Path Towards Simplifying Consistency in Distributed Systems.  This conference was my favorite of the year.  A single track filled with excellent speakers that focused not only on technology, but the culture and community in tech.  The cherry on top was its location The Egyptian theater in Seattle’s Capitol Hill neighborhood, my old stomping grounds.  

I also spoke at two chapters of Papers We Love, San Francisco and Seattle.  I presented Barbra Liskov’s paper Distributed Programming in Argus.  This brings my total times speaking at Papers We Love chapters to 7, which I think once again makes me the record holder :).  All joking aside Papers We Love is one of my favorite organizations and I love attending and speaking at the meetups because of the community it fosters bringing together academia and industry and the culture of curiosity it inspires.  

Writing

I wrote a single blog post in 2017.  Resources for Getting Started with Distributed Systems which is a collection of materials that have greatly influenced me, and attempts to answer the perennial question I get asked “How do I get started with Distributed Systems.”

Work

Earlier this year an old colleague recommended I take a phone call with a group at Microsoft Research.  After a couple phone calls, and an onsite interview, I was convinced that this was a rare opportunity with an amazing team and an industry defining project.  So in June, after 2.5 years of working at Twitter, I decided to leave the flock.  

Working at Twitter was a truly great experience.  It was an incredible ride where I got to learn and work on so many amazing projects including being the Tech Lead of the Observability team, speaking at Twitter Flight, digging into Distributed Build, shipping Abuse Report Notifications, and facilitating TWIG (Twitter’s Engineering Leadership Program).  I also feel very fortunate to have worked with and met so many incredible people.

In July I started as a Principal Software Engineer in Microsoft Research, and have loved every minute of it.  I’m getting to stretch, learn, and grow every day on a project that I truly believe will change the world.  I also adore my teammates, this is by far the smartest and nicest team I have ever worked on.  We consistently talk and live our cultural values of trust, kindness, and fearlessness.  I couldn’t ask for a better team.  And just incase that wasn’t enough change for one year in November I stepped into the Lead role, a hybrid Tech Lead and People Manager, for the Service’s Team, which is another new exciting challenge and opportunity that I’m loving.  

Personal

Leaving San Francisco felt inevitable.  I moved to San Francisco to experience the tech scene, to live the cultural phenomenon.  But after 2.5 years I was ready to move on.  San Francisco was not my forever home, our words just did not match.  

Moving back to Seattle was an easy decision.  I first fell in love with Seattle when I moved here after college, and still love it.  Even after all my nomadic wanderings and travel when I visited Seattle in April for Deconstruct Conf I instantly felt like I was home.  I also realized I was quite nostalgic for Seattle earlier in the year when I began marathoning episodes of Grey’s Anatomy again.  

And if all the warm and fuzzy feelings about Seattle weren’t enough, the stars magically aligned and within a week of moving back I made an offer on a house, and it was accepted!  New job, new/old city, and a new homeowner too!

I jokingly tell friends that I blew up my whole life earlier this year, which isn’t entirely untrue.  The top three stressors in life are commonly reported as job change, relationship change, and moving.  I did all three within the span of about two months.  I’d like to take quick moment to thank my community of family, friends, and colleagues who helped and supported me through this whirlwind transition.  I could not have done it without your support.  

Even with all the stressors I honestly could not be happier (with my personal and professional life, the political nightmare of 2017 still fills me with dread, despair, and anger).  I no longer feel comfortable or content.  In fact I often feel decidedly uncomfortable, but in the way that signals learning and growth.  And instead of contentment I often feel a wild unbridled joy and excitement.  I’m energized to go to work every day.  I’ve sang and danced and laughed until my stomach hurts more times than I can count since blowing up my life.  So I guess the lesson once again is, “You are braver than you believe, stronger than you seem, and smarter than you think.”  Oh and always take the phone call :).  

 

2016: A Year in Review

2016 was a year of constant movement, I visited 19 cities, in 7 countries, on 3 continents.  In the middle of the year I switched teams inside of Twitter and began working on a new challenge, Distributed Build.  I also spoke 15 times at conferences and meetups.  Its been a really long, wonderful and exhausting year, filled with great people, travel to many exciting places, and new challenges.

Below is a summary of the articles, interviews, talks, and programming committees I participated in in 2016.

Articles & Books

My article “The Verification of a Distributed System” was published in  the February issue of Communications of the ACM.  I also began writing Go this year and wrote a Blog Post “A Quick Guide to Testing in Golang” describing the testing methodologies and project setup I use with my team.

I also was a Technical Editor for James Turnbull’s The Art of Monitoring.

Interview & Podcasts

At QConNY I recorded a podcast about Engineering Effectiveness at Twitter and Verifying Distributed Systems.  I was also honored to be included in Helena Dagmar’s Techies Project, which captured the images and stories of silicon valley tech employees who are usually underrepresented.

13522976_10103121808447645_6746242372821133955_o
Myself & Ines Sombra giving a Keynote at Velocity Santa Clara

Programming Committees

I participated in several programming committees this year both academic and industry focused.  I was an industry member of the Principles and Practices of Computing for Distributed Data Workshop co-located with EuroSys.  I also hosted a track on Data & Distributed Systems at GOTO Chicago 2016, and finally I was on the inaugural programming committee for Systems We Love, a one day conference inspired by Papers We Love, but focused on Computer Systems.

Talks

After a lot of positive feedback and based on numerous requests I gave versions of my talk Scaling Stateful Services at CraftConf, CurryOn, and at Nike Tech Talks.

I turned my article The Verification of Distributed Systems into a talk, now with more rantifestos, and gave it at GOTO Chicago, QCon New York, and YOW Melbourne, Brisbane, and Sydney.

At the end of June I was honored to give a keynote at Velocity Santa Clara alongside Ines Sombra called So We Hear You Like Papers 2 in which we discuss the academic paper Unreliable Failure Detectors for Reliable Distributed Systems and how it applies to our jobs in industry.

Also at the end of June I spoke at Monitorama about Tackling Alert Fatigue, which summarized the strategies my team used over the past half year to reduce the number of critical alerts the Twitter Observability Services fired by 50%.

13305164_10103059412165295_4668730123625827164_o
Members of my Distributed Systems Track at GOTO Chicago: Christopher Meiklejohn, Myself, Boaz Avital, Peter Bailis, and Aysylu Greenberg

Christopher Meiklejohn and I gave an academic and industry perspective on the Remote Procedure Call in a talk at CodeMesh called A Brief History of Distributed Programming: RPC, and a second time at YOW Brisbane.

I also spoke at three different chapters of Papers We Love this year, bringing my total PWL talks to 5.  In April I filled in last minute for a cancelled speaker at the San Francisco chapter and spoke about Sagas.  While in New York for QCon I also spoke at a special edition of Papers We Love New York, along with Eric Brewer, Evelina Gabasova, and Ines Sombra, on my favorite paper of 2014 Simple Testing Can Prevent Most Critical Failures.  Finally, while in Portland for Monitorama I spoke at the newly formed PDX chapter about Detection of Mutual Inconsistency in Distributed Systems.

I also combined all my Github talk repos into one repo CaitieM20/Talks, which contains references and links to slides and videos for each talk.

2015: A Year in Review

2015 has been a whirlwind of a year, which started off in a new city, with a new job as the Tech Lead of  Observability at Twitter.  The year was full of travel spanning 10 states, 3 different countries, and 2 continents.  This year I also had numerous opportunities to share my experiences with programming and distributed systems via, talks, blog posts, podcasts, and articles.  Below is the recap.

Articles

Interviews & Podcasts

Programming Committees

Talks

Orleans: A Framework for Cloud Computing

Presented at Papers We Love SF: Video & Slides [February 19th 2015]

Abstract

Caitie McCaffrey stops by and talks about the Orleans: Distributed Virtual Actors for Programmability and Scalability paper by Bernstein, Bykov, Geller, Kliot, and Thelin.

Orleans is a runtime and programming model for building scalable distributed systems, based on the actor model.  The Orleans programming model introduces the abstraction of Virtual Actors.  Orleans allows applications to obtain high performance, reliability, and scalability.  This technology was developed by the eXtreme Computing Group at Microsoft Research and was a core component of the Azure Services that supported that powered Halo 4, the award winning video game.

gotochicago
Distributed Systems Track at Goto Chicago: Neha Narula, Caitie McCaffrey, Chris Meiklejohn, Kyle Kingsbury

Building the Halo 4 Services with Orleans

Presented at Qcon London: Video & Slides [March 5th 2015]

Abstract

Halo 4 is a first-person shooter on the Xbox 360, with fast-paced, competitive gameplay. To complement the code on disc, a set of services were developed to store player statistics, display player presence information, deliver daily challenges, modify playlists, catch cheaters and more. As of June 2013 Halo 4 had 11.6 million players, who played 1.5 billion games, logging 270 million hours of gameplay.

Orleans, Distributed Virtual Actors for Programmability & Scalability, is an actor framework & runtime for building high scale distributed systems. It came from the eXtreme computing group in Microsoft Research, and is now Open Source on Github.

For Halo 4, 343 Industries built and deployed a new set of services built from the ground up to support high demand, low latency, and high availability using using Orleans and running in Window Azure. This talk will do an overview of Orleans, the challenges faced when building the Halo 4 services, and why the Actor Model and Orleans in particular were utilized to solve these problems.

Architecting & Launching the Halo 4 Services

Presented as the Closing Keynote of SRECon15: VideoSlides [March 17th 2015]

Abstract

The Halo 4 services were built from the ground up to support high demand, low latency, and high availability.  In addition, video games have unique load patterns where the majority of the traffic and sales occurs within the first few weeks after launch, making this a critical time period for the game and supporting services. Halo 4 went from 0 to 1 million users on day 1, and 4 million users within the first week.

This talk will discuss the architectural challenges faced when building these services and how they were solved using Windows Azure and Project Orleans. In addition, we’ll discuss the path to production, some of the difficulties faced, and the tooling and practices that made the launch successful.

12029606_10102616259451885_1497979433309626479_o
On stage during Strange Loop 2015 at the Peabody Opera House

The Saga Pattern

Presented at Craft Conf 2015 & Goto: Chicago 2015 Video & Slides [April 23rd 2015 & May 12th 2015]

Abstract

As we build larger more complex applications and solutions that need to do collaborative processing the traditional ACID transaction model using coordinated 2-phase commit is often no longer suitable. More frequently we have long lived transactions or must act upon resources distributed across various locations and trust boundaries. The Saga Pattern is a useful model for long lived activities and distributed transactions without coordination.

Sagas split work into a set of transactions whose effects can be reversed even after the work has been performed or committed. If a failure occurs compensating transactions are performed to rollback the work. So at its core the Saga is a failure Management Pattern, making it particularly applicable to distributed systems.

In this talk, I’ll discuss the fundamentals of the Saga Pattern, and how it can be applied to your systems. In addition we’ll discuss how the Halo 4 Services successfully made use of the Saga Pattern when processing game statistics, and how we implemented it in production.

Scaling Stateful Services

Presented at StrangeLoop 2015 Video & Slides [September 25th 2015]

This talk was incredibly well received, and I was flattered to see write-ups of it featured in High Scalability and InfoQ

Abstract

The Stateless Service design principle has become ubiquitous in the tech industry for creating horizontally scalable services. However our applications do have state, we just have moved all of it to caches and databases. Today as applications are becoming more data intensive and request latencies are expected to be incredibly low, we’d like the benefits of stateful services, like data locality and sticky consistency. In this talk I will address the benefits of stateful services, how to build them so that they scale, and discuss projects from Halo and Twitter of highly distributed and scalable services that implement these techniques successfully.

12237996_10102700397259045_5884837663522330773_o
Ines Sombra & Caitie McCaffrey’s Evening Keynote  at QconSF

On the Order of Billions

Presented at Twitter Flight: Video & Slides [October 21st 2015]

Abstract

Every minute Twitter’s Observability stack processes 2+ billion metrics in order to provide Visibility into Twitter’s distributed microservices architecture. This talk will focus on some of the challenges associated with building and running this large scale distributed system. We will also focus on lessons learned and how to build services that scale that are applicable for services of any size.

So We Hear You Like Papers

Presented as the Evening Keynote at QconSF with Ines Sombra: Video, Slides, Resources, & Moment [November 16th 2015]

Abstract

Surprisingly enough academic papers can be interesting and very relevant to the work we do as computer science practitioners. Papers come in many kinds/ areas of focus and sometimes finding the right one can be difficult. But when you do, it can radically change your perspective and introduce you to new ideas.

Distributed Systems has been an active area of research since the 1960s, and many of the problems we face today in our industry have already had solutions proposed, and have inspired new research. Join us for a guided tour of papers from past and present research that have reshaped the way we think about building large scale distributed systems.