Light Hoodie

A few weeks ago Hoop Somuah and I decided that since the Seahawks were going to the Super Bowl we needed proper attire, so we decided to show our 12th man pride by creating custom Light Hoodies.

We ordered NeoPixel Strands, a Trinket Arduino board, a capacitor, and a battery pack from Adafruit, and set out to do some soldering, sewing and programming.

NeoPixels are full LEDs and each pixel in the strand is individually programmable.  The Trinket is not 100% Arduino compatible but it can be programmed using the Arduino SDKs, and at around $8 its a great little micro controller running at 8MHz with 8K of flash memory and 512 bytes of SRAM.

_MG_4727

The one caveat with using a Trinket to power the Neopixel strand is that each pixel takes 3 bytes of RAM to run.  The Neopixel strips we cut for our hoodies had ~100 pixels so we used 60% of our memory budget just to support the pixels.  When we added a button to control the light patterns we blew our memory budget and had to do some refactoring to reduce our memory footprint.  For larger projects a larger board like the Flora would be necessary.

The code to power our hoodies is up on Github.  So far we have written three programs.  The original Hoodie contains the Seahawk code patterns.  usaHoodie is USA themed for Sochi Olympics, and portraitHoodie is a set of solid lights I have been using in my photography class.

This was a super fun project, and a great introduction to electronics for me.  The Adafruit tutorials were great, and the products were easy to use.  I’d highly recommend Adafruit for any DIY electronic projects.  Currently Hoop and I are working on designs for Hoodie Mark 3.  It may involve sound!  Stay Tuned!

You should follow me on Twitter here

Node Summit 2013 Retrospective

This year I attended Node Summit 2013 from December 3rd-4th in San Francsico.  I went to support Michael Shim, an HBO coworker who was speaking on a panel about Node.js in the Digital Media Universe, and to get a crash course into the culture, tech, and community surrounding this framework.  Being brand new to Node.js Node Summit was a great opportunity to get a feel for the community and how other companies were using it.

Below are some of the thoughts and overall themes that I took away from the conference.

Node Can Perform at Enterprise Scale

There were dozens of examples of companies using and deploying Node.js at enterprise scale.  Walmart had great success this past Black Friday.  Groupon, LinkedIn, Ebay, & Paypal have all recently made the switch over to running Node in their services’ stack, and had great success.  Media companies like NPR, CondeNast, and Direct TV are also starting to use Node with success.

The big takeaway was this framework is being deployed by several companies, at large scale, and it is performing.  The fledgling tech does have some challenges to work through, especially surrounding security and patching, which were addressed in a panel on day 2, but overall Node.js is a very viable option to build large scale services on.

Node & Functional Programming

On Day One of the summit, many presentations and panels talked a lot about the process of incorporating Node.js into their tech stacks, which was not surprising.  However, I was surprised to hear that as part of the process many companies, like LinkedIn and PayPal, also switched from Object Oriented Programming models to Functional Programming models.

Thinking back on this it shouldn’t be surprising since the asynchronous and event-driven nature of Node.js applications fits extremely well with Functional Programming models.  Now if only message passing between node processes was easier…

Fast Mean Time to Recovery 

An overall theme throughout several talks was how easy it is to bring up and tear down Node.js processes.  This is not only great for developer efficiency, but it is also incredibly important in production.  The ability to kill a misbehaving process and quickly replace it with a new instance means the Mean Time To Recovery (MTTR) is low.  Having a low MTTR means you can fail fast and often.

It was also cited that Mean Time to Failure (MTTF) is no longer as important of a metric.  This is a bit of a paradigm shift, when thinking about more traditional services.

No More Monoliths

Node.js is really useful when you structure your services into small modular systems.  Node services should be broken down into lots of little processes that communicate via messages passed through queues or streams.

Small processes are more easy to reason about, make your application more testable, and the code more manageable.  The Node Package Module (NPM) makes breaking your code into modules and managing dependencies extremely easy.  NPM was touted throughout the conference as Node’s killer feature.

The Node Community really loves Node 

This may sound like a no brainer, but the community was incredibly passionate.  Its no wonder since the use of Node.js has grown tremendously over the past year, and the number of modules in NPM has also been growing exponentially.  It was awesome to be around such a group of motivated passionate developers.

In addition companies are finding it easy to hire developers, because developers really love Node.js.  Even if they don’t have prior experience developers want to learn and work with Node.js.  I should know I’m one of them :)

and not a lot else…

The down side of this passion came with a lot of other tech bashing, that felt more religious than factual.  Java was lambasted in every talk and panel.  To be fair asynchronous programming in Java is quite cumbersome, whereas the asynchronous model is built into the Node.js framework.  But at its core Java is a language and Node.js is a framework, so apples to apples comparisons seemed odd without the mention of Java frameworks, like Typesafe, that also try and solve this problem.

.Net tech was barely mentioned beyond snide remarks.  I found these remarks odd since I have worked rather extensively with the Async/Await programming model introduced in .Net 4.5, and in my experience the async/await model is easy to use and resulted in clean concise code.

Ruby on Rails also took a lot of heat especially in the Why Large Scale Mobile & E-Commerce Apps Use Node.js panel.  Groupon and LinkedIn both started with a Ruby on Rails stacks.  Both companies switched to a Node stack citing that the Ruby on Rails services could not scale and that the code base was unmanageable.  However, in a later talk Sean McCullogh, a Groupon Engineer, did mention that Groupon’s original architecture using Ruby on Rails was broken and that there were changing business needs which led them to switch to Node.js.  I greatly appreciated this honesty instead of just blaming the Ruby stack for all their problems.

As a developer I think Node.js is fantastic.  The framework embraces a lot of the core principles that make developing and deploying large scale web services easy.  And to be fair, most tech communities, and tech specific conferences tend to run into this problem.  However, there are tradeoffs in any technology and lessons to be learned from other solutions.  While this was a Node conference I wish there would have been a more honest discussion about these tradeoffs, and an openness to other technology.

Favorite Talks

Reflections on Three Years of Node.js in Production – Bryan Cantril

This was one of the more technical talks,  and Bryan was immensely entertaining.  Dropping quotes like,  “You gotta love open source sometimes a magical pony comes and poops out a rainbow on your code.”

The majority of his talk focused on logging and debugging Node.js services.  Javascript core dumps are really hard to trace and debug, but Joyent has done a lot of work to demystify this process by creating tools that work on SmartOS.  However general Linux users hadn’t been able to take advantage of this functionality…Until Now!  Bryan did a live demo of taking a core dump from an Ubuntu Linux box and getting the same deep analysis using ephemeral compute within the Joyent Manta Storage Service.  More details are available on the Joyent Blog.  Pretty cool!

Node From the Battlefield – Eran Hammer 

Eran, a Senior Architect at Walmart, gave a highly entertaining talk on how Walmart moved a large portion of their service’s stack over to Node.js and the lead up to their biggest test, Black Friday.

While preparing for Black Friday the Walmart team encountered a suspected memory leak.  To help diagnose the problem the team increased analytics and monitoring which signaled that the problem was actually in the core of Node.js.  TJ Fontaine at Joyent tracked down the memory leak fixed it and released a new version of Node which the Walmart team picked up just in time for holiday shopping.

On Black Friday Eran live tweeted Walmart Devops under the hashtag #nodebf, and it was surprisingly boring.  With the memory leak fixed, their services performed well consuming low amounts of memory and CPU despite high amounts of traffic.

At the end of his talk Eran read a “bed time story” to TJ on stage called the Leek Seed.  It was quite comical and had the audience in stitches.

This talk demonstrated the ability for Node to scale and perform in large production environments.  In addition this talk reinforced my opinion that rigorous logging, monitoring, and testing at load is the only way to discover some issues of the most nefarious issues in distributed systems.

Conclusion

I had an excellent time at NodeSummit, I learned a lot.  Personally I like more technical talks and code at conferences, and less panels, but there are other Node.js related conferences that are tailored to that.  Overall it was a great crash course into the Node world and community, and I look forward to applying what I learned, getting more involved in the community, and developing services on Node in the future.

You should follow me on Twitter here

A WebSocket Primer

Over the past year, prior to leaving 343, I spent a large amount of time working with the WebSockets protocol and upgrading the Halo Services to support it.  In order to solidify my knowledge and provide a handy refresher for when this information invariably gets context switched out of my brain in the future, I decided to write a primer on WebSockets.  Hopefully other people will find the introduction to this new protocol useful as well.

Overview

In December 2011 the IETF standardized the WebSocket protocol.  Unlike the typical Request/Response messaging patterns provided by HTTP, this network protocol provides a full-duplex communication channel between a host and a client over TCP.  This enables server sent events, reactive user experiences, and real time components.

The WebSocket protocol provides some advantages over the traditional HTTP protocol.  Once the connection has been established, there is a point to point system of communication where both devices can communicate with one another simultaneously.  This enables server sent events without using a work around like Comet or Long Polling.  While these technologies work well, they carry the overhead of HTTP, whereas WebSocket frames have a wire-level overhead of as little as two bytes per frame.  The full-duplex communication and low packet overhead make it an ideal protocol for real-time low latency experiences.

An important note:  The WebSocket protocol is not layered on top of HTTP, nor is it an extension of the HTTP protocol.  The WebSocket protocol is a light weight protocol layered onto of TCP.  The only part HTTP plays is in establishing a WebSocket connection via the HTTP Upgrade request.  Also the HTTP Upgrade request is not specific to WebSockets but can be used to support other hand-shakes or upgrade mechanisms which will use the underlying TCP connection.

Open a WebSocket Connection

A client can establish a WebSocket connection by initiating a client handshake request.  As mentioned above the HTTP Upgrade request is used to initiate a WebSocket connection.

GET /chat HTTP/1.1
HOST: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Origin: http://example.com
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13

If all goes well on the server and the request can be accepted then the server handshake will be returned.

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

If an error occurs and the server cannot accept the request, than a HTTP 500 should be returned to indicate that the request has failed and that the protocol is still HTTP.

Once the client server handshake is completed the TCP connection used to make the initial HTTP request has now been upgraded to a WebSocket connection.  Messages can now be sent from either the client to the server or the server to the client.

Code

As a developer most of the nuances of the WebSocket handshake are hidden away by the platform specific APIs and SDKs.  In the .NET world Windows 8 and Windows Server 2012 introduced native support for the WebSocket protocol.  In addition Internet Explorer 10 introduced native support for the WebSocket protocol as well. Also a variety of other platforms support WebSockets.

Client

Using the .NET 4.5 Framework the client code to establish a WebSocket connection in C# would look like this.

ClientWebSocket webSocket = null;
webSocket = new ClientWebSocket();
await webSocket.ConnectAsync(new Uri("ws://localhost/Echo"), CancellationToken.None);

Once the connection succeeds on the client the ClientWebSocket object can be used to receive and send messages.

Server

Using the .Net 4.5 Framework on a simple server using HttpListener, the C# code to accept a WebSocket request and complete the hand-shake would look like this.

HttpListenerContext listenerContext = await httpListener.GetContextAsync();
if (listenerContext.Request.IsWebSocketRequest)
{
    WebSocketContext webSocketContext = await listenerContext.AcceptWebSocketAsync();
    WebSocket webSocket = webSocketContext.WebSocket
}
else
{
    //Return a 426 - Upgrade Required Status Code
    listenerContext.Response.StatusCode = 426;
    listenerContext.Response.Close();
}

The call to AcceptWebSocket request returns after the server handshake has been returned to the client.  At this point the WebSocket object can be used to send and receive messages.

WebSocket Messages

WebSocket messages are transmitted in “frames.”  Each WebSocket frame has an opcode, a payload length, and the payload data.  Each frame has a header.  The size of the header is between 2-14 bytes.  As you can see the header overhead is much smaller than the text based HTTP headers.

 Headers

0 1 2 3 4 5 6 7 8 9 A B C D E F
Final Reserved Bits OpCode Mask Payload Indicator
Extended payload length ( present if payload is longer than 125 bytes )
Extended payload length  ( present if payload length is >= 2^16 )
Extended payload length  ( present if payload length is >= 2^16 )
Extended payload length  ( present if payload length is >= 2^16 )
MaskingKey ( present if masking bit is set )
MaskingKey ( present if masking bit is set )

The first 9 bits sent in every WebSocket frame are defined as follow

  • Final Bit (1 bit) – Indicates whether the frame is the final fragment of a message, as a large message can be broken up and sent over multiple frames.  A message that is one frame long would also set this bit to 1.
  • Reserved (3 bits) – These must be 0, and are currently reserved for extensions.
  • OpCodes (4 bits) – Opcodes define how the payload data should be interpreted
  • Masking (1 bit) – Indicates if the payload data is masked.  The WebSocket protocol specifies that all messages sent from a client to a server must be XOR masked.

The variable length of a WebSocket header is based on the size of the payload and the masking-key

  • Payload Length (7 bits, 7 + 16 bits, 7 + 64 bits) – Bits 10-16 of the header are the payload indicator bits. The number of bits used to encode the payload length varies based on the size of the payload data.
    • 0-125 bytes: payload length encoded in the payload indicator bits
    • 126 – 65,535 bytes: The payload indicator bits are set to 126, and the next two bytes are used to encode the payload length.
    • >65,535 bytes: 127 is encoded in the payload indicator bits, and the next 8 bytes are used to specify the payload length.
  • Masking-key (0 or 16 bits) – If the masking bit is set, then the 32 bit integer used to Mask the payload is specified in this field.  If the masking bit is not set than this is omitted.

 OpCodes

The following table below defines WebSocket frame OpCodes.  Applications should only set the Text or Binary OpCodes to specify how the payload data in the frame is interpreted.

Code Meaning Description
0x0 Continuation Frame The payload in this frame is a continuation of the message sent in a previous frame that did not have its final bit set
0x1 Text Frame Application Specific – The payload is encoded in UTF-8
0x2 Binary Frame Application Specific – The payload is a binary blob
0x8 Close Connection Frame Specifies that the WebSocket connection should be closed
0x9 Ping Frame Protocol Specific – sent to check that the client is still available
0xA Pong Frame Protocol Specific – response sent after receiving a ping frame.  Unsolicited pong messages can also be sent.

Code

Sending and receiving WebSocket messages is easy using the .NET Framework APIs.

Receiving a Message

byte[] receiveBuffer = new byte[receiveBufferLength];
while (webSocket.State == WebSocketState.Open)
{
     WebSocketReceiveResult receiveResult = await webSocket.ReceiveAsync(new ArraySegment<byte>(receiveBuffer), CancellationToken.None);
}

The WebSocketReceiveResult object contains the information sent in one WebSocket frame including the OpCode, Final Bit Setting,  Payload Length, and CloseStatus & Reason if its a Close Connection Frame.  The receiveBuffer will be populated with the data sent in the payload.

Sending a Message

Sending a message is also simple and an Async method is provided in the .NET 4.5 Framework.  The code below echos the message received back over the channel.  The data, Message Type, and Final Bit are specified in the parameter list.

await webSocket.SendAsync(new ArraySegment<byte>(receiveBuffer, 0, receiveResult.Count), WebSocketMessageType.Binary, receiveResult.EndOfMessage)

Close a WebSocket Connection

Either endpoint can close the WebSocket connection.  In order to do this the endpoint starts the WebSocket Closing Handshake.  The initiating end point sends a WebSocket message with a closing status code, and an optional close reason (text), and sets the Opcode in the message to the Close Connection Frame (0x8).  Once the message is sent the endpoint will close the WebSocket connection by closing the underlying TCP connection.

As an application developer it is important to note that either endpoint, server or client, can initiate the closing handshake.  Practically this means both endpoints need to handle receiving the close frame.  It also means that some messages may not be delivered, if the connection is closed while the messages are in transit.

Connection Close Code

Connection Close frames should include a status code, which indicates the reason the WebSocket connection was closed.  These are somewhat analogous to HTTP Status Codes.

Code Definition Description
1000 Normal Closure The purpose for which the connection was established has been fulfilled
1001 Endpoint Unavailable A server is going down, or a browser has navigated away from a page
1002 Protocol Error The endpoint received a frame that violated the WebSocket protocol
1003 Invalid Message Type The endpoint has received data that it does not understand.  Endpoints which only understand text may send this if they receive a binary message and vice versa
1004 -1006 Reserved Reserved for future use
1007 Invalid Payload Data The payload contained data that was not consistent with the type of message
1008 Policy Violation Endpoint received a message that violates its policy
1009 Message Too Big Endpoint received a message that is too big for it to process.
1010 Mandatory Extension An endpoint is terminating the connection because it expected to negotiate one or more extensions
1011 Internal Error The server is terminating the connection because it encountered and unexpected error
1015 TLS Handshake Used to designate that the connection closed because the TLS handshake failed.

Connection Close Code Ranges

Code Definition
0-999 Not Used
1000-2999 Reserved for use by Protocol Definition
3000-3999 Reserved for use by libraries, frameworks & applications.  These should be registered with IANA
4000-4999 Reserved for private use and can’t be registered.

 Code

Once again most of the details are dealt with by WebSocket libraries in your framework of choice.  Application developers must decide when the connection should be closed, should set the appropriate connection close code and may also set a connection close reason.

The .Net Framework makes this very easy, by providing an asynchronous method, which takes in the connection close code, and close reason as parameters.

await webSocket.CloseAsync(WebSocketCloseStatus.NormalClosure, "Normal Closure", CancellationToken.None);

Microsoft WebSocket Implementations

As mentioned before Windows 8 and Windows Server 2012 introduced native support for the WebSocket protocol.  Also because the Xbox One is running a variant of the Windows 8 operating system it also has built in support for WebSockets.

.Net 4.5

Version 4.5 of the .NET framework introduced support for WebSockets through the System.Net.WebSockets namespace.  The underlying connection is passing through HTTP.sys in the kernel so timeout settings in the HTTP.sys layer might still apply.

WinRT

WinRT only exposes APIs for creating a WebSocket client connection.  There are two classes to do this in the Windows.Networking.Sockets namespace, MessageWebSocket & StreamWebSocket.

 Win32 (WinHTTP)

The WinRT API is also available to C++ developers.  For developers that want more control WinHTTP provides a set of APIs for sending  WebSocket upgrade request, and sending and receiving data on WebSocket connections.

 JavaScript

All the latest versions of common browsers, with the exception of Android, support the WebSocket protocol and API as defined by the W3C.

SignalR

The ASP.NET team has built a high-level bi-directional communication API called SignalR.  Under the hood SignalR picks the best protocol to use based on the capabilities of the clients.  If WebSockets are available it prefers to use that protocol, otherwise it falls back to other HTTP techniques like Comet and Long Polling.  SignalR has support for multiple languages including .NET, Javascript, and iOS and Android via Xamarin.  It is an open source project on GitHub.

Conclusion

WebSockets are a great new protocol to power real time applications and reactive user experiences due to its lightweight headers, and bi-directional communication.  It is also a great protocol for implementing Pub/Sub messaging patterns between servers and clients.  However WebSockets are not a silver bullet for networked communications.  WebSockets are incredibly powerful but do also have their drawbacks.  For instance because WebSockets require a persistent connection, they are consuming resources on the server and require the server to manage state.  HTTP and RESTful APIs are still incredibly useful and valid in many scenarios and developers should consider the uses of their APIs and applications when choosing which protocol to use.

You should follow me on Twitter here

A New Adventure

343e3_featureimage

My last day at Microsoft and 343 Industries was two weeks ago.  I left to take a new job at the recently opened HBO Codelabs to work on HBO Go.  After 5 awesome years at Microsoft it was not an easy decision to leave, but personally I was ready for a change, and was incredibly excited by the people, problems, and possibility offered by the HBO team.

Over the past 5 years, I’ve worked on some amazing projects, with some truly outstanding people, so I thought I would take a moment to reflect.

Gears of War 2

In July of 2008 I joined Microsoft to work on Gears of War 2.  I was beyond ecstatic to have the opportunity to work in the games industry, and on one of my favorite franchises.  Shipping Gears of War 2 was a blur, I was quickly thrown into the deep end.  I owned the testing of Co-op and helped out with matchmaking.  There were a lot of long nights, but I was having a blast and learning an incredible amount.  Getting Gears 2 to go Gold required several long days, weekends, and an all-nighter which involved me taking a quick 20 minute power nap on the floor of my office, but it was all worth it come launch night!  Plus Lancers!

645_600267432095_6954_n

Halo Waypoint V1

In the fall of 2009 I was working on Gears of War 2 sustain, and dedicated servers for Gears of War 3, when the fledgling 343 Industries asked our team for resources to help with the first Halo Waypoint Title Application.  This was one of the first Xbox 360 title applications, and so I relished in learning the new tech and quickly bonded with the 343i team when I was loaned out on the project.

343 Industries

In early 2010 I was offered a position to join 343 Industries as web service developer #2 working on the Halo Game Services.  It was thrilling to join the studio when we were only 60 people and watch it grow and band together to ship Halo 4.  I worked on the Halo Presence Service, Playlist Service, Statistics Processing Service, and the automated Cheating & Banning Service, along with owning the Network Data Protocol.

One of the things I am most proud of is the transition we made from physical machines (IIS & SQL) to an Azure Cloud powered solution.  It was a huge undertaking, requiring a re-architecture of the entire service, but we pulled it off and now the Halo Services are in a great position to grow and scale in the future.

343_featureimage

Not only did I get to dive head first into cloud computing with Windows Azure, but I also got to work with some amazing Microsoft Research Teams.  Post Halo 4 I was lucky enough to get to share my experiences and learnings from building and shipping these services via podcasts, web videos, and conference talks.

There are too many excellent memories to enumerate over the past 4 years I spent at 343.  But some highlights include, wrangling Spartans at Halo Reach launch events, showing off Halo 4 at E3, laughing so hard I cried when taking a break to watch Corgi videos late at night, and watching Australia and New Zealand come online and hit our new cloud services on launch night.

Over the years I’ve worked with too many awesome people to list them all.  Of course no discussion of my career at 343i would be complete without a huge shout out to Hoop Somuah.  Over the years he taught me so much, helped me grow immensely as a developer, and became a very good friend.  WAH!

Leaving 343 Industries was bittersweet.  I have so much love for Halo, the studio, and the project.  But I am so immensely excited for the new adventures ahead!  “Let the Wild Rumpus Begin!”

You should follow me on Twitter here

Enforcing Idempotency at the Data Layer

Idempotency

In Computer Science idempotent operations are defined as operations that produce the same result if they are executed once or multiple times.

Practically in an application or service this means that idempotent operations can be retried or replayed without the fear of processing the data multiple times or causing unwanted side effects.  As a Web Service Developer having Idempotent operations allows us to have simpler logic for handling failures.  If a request fails we can simply retry the request by replaying it.  In services and messaging systems having idempotent operations is the easiest way to handle “at least once messaging”  Jimmy Bogard has written a great post on this topic: Los Techies (Un) Reliability in Messaging Idempotency and De-Duplication

In services and messaging systems having idempotent operations is the easiest way to handle “at least once messaging”

Most operations are not mathematically idempotent, so developers must write application level logic to enforce Idempotency of requests.  However, If we can enforce Idempotency in our operations at the Data Storage Layer than the need for special case logic is minimized.

Using Azure Table to Enforce Idempotency

Azure Table Storage provides Transaction support inside of a single Table Partition.  We can take advantage of this to enforce Idempotency at the Data Storage layer of applications by using the Aggregate-Event Data Model described in a pervious post.

In order for this to work the data needs to be structured in the following way.

  1. The Aggregate Entity and Event Entities must be stored in the same Partition.
  2. Updating the Aggregate Entity and Adding the Event Entity to storage must occur in the same Batch Operation.

By updating the Aggregate Entity and adding the Event Entity in the same TableBatchOperation either both writes will succeed or both writes will fail, leaving your data in a consistent state whether you have received the event once, or many times.

If the Batch Operation fails, you can determine that it was because the data was already processed, and not caused by some other failure, by checking the Storage Exceptions HTTP Status Code.  If the Status Code equals 419 – Conflict, then one of Entities marked for Add in the Batch Operation already exists in the table.

Simple Stats Example

To see this in practice, we’ll go through a simple example for storing game statistics for a game like Galaga, using Azure Table Storage.  Each player has a unique PlayerId and at the end of each game players will upload data to store their statistics for that game.

public class PlayerGameData
{
   public Guid GameId { get; set; }
   public Int32 GameDurationSeconds { get; set; }
   public bool Win { get; set; }
   public Int32 Points { get; set; }
   public Int32 Kills { get; set; }
   public Int32 Deaths { get; set; }
}

In the previous post on Immutable Data Stores I also share code for the TableEntities used in the Simple Stats Example, which will not be repeated here.

Below is an example of how to Process a Simple Stats Game and store it such that Idempotency is enforced at the Data Layer.

public static void ProcessGame(Int64 playerId, PlayerGameData gameData)
{
   // Create the batch operation.
   TableBatchOperation batchOperation = new TableBatchOperation();

   PlayerEntity player = PlayerEntity.GetPlayerEntity(playerId);
   UpdatePlayerEntity(player, gameData)
   batchOperation.InsertOrReplace(player);

   //Create PlayerGame Row
   PlayerGameEntity playerGame = new PlayerGameEntity(playerId, gameData);
   batchOperation.Insert(playerGame);

   try
   {
      StorageManager.Instance.PlayersTable.ExecuteBatch(batchOperation);
   }
   catch (Exception ex)
   {
       //Check if the error occurred because we already processed the data
       if (ex is Microsoft.WindowsAzure.Storage.StorageException)
       {
          var storageEx =  (Microsoft.WindowsAzure.Storage.StorageException)ex;
          if(storageEx.RequestInformation.HttpStatusCode == (int)HttpStatusCode.Conflict)
             return;
       }
       throw ex;
   }
}

There are two options when a game is processed:

1. The system has not processed this Game: In this case the above code will create a new PlayerGame Entity and update the in memory copy of the Player Entity.  The Batch Operation will succeed, and the PlayerGame along with the updated Player Entity will be stored in the table.

2. The system has processed this Game: In this case the above code will create a new PlayerGame Entity and update the in memory copy of Player Entity.  However when the Batch Operation executes it will fail since the Event Entity already exists in this partition.  Because of the per partition transaction support provided by Azure Storage, the updates to the Player Entity will also not be stored.  The data in storage will be the same as if the Game had only been processed once.

By using Azure Table Store to enforce Idempotency of event processing you no longer have to write application level logic to handle at least once messaging.  In addition by using this pattern you get all the benefits of Immutable Data Stores as well.

You should follow me on Twitter here

Creating Immutable Data Stores

Immutability

Immutability is the property of data to not change once it is created.  Most storage systems provide basic C.R.U.D. (Create, Read, Update, Delete) functionality.  However, to have truly Immutable data we should only use the Create and Read functionality provided by storage systems.

Nathan Marz makes an excellent case for why we need Immutable data in his talk at Strata Conf 2013, Human Fault – Tolerance.  The basic premise is people make mistakes.  We are fallible beings: we write code with bugs, we accidentally delete data from production, etc… He goes on to posit that the worst kinds of errors are those which cause Data Loss or Data Corruption, because it is impossible to recover from them.  By having Immutable Data stores we minimize the impact bugs or mistakes can have on our systems.

By having Immutable Data stores we minimize the impact bugs or mistakes can have on our systems.

Aggregate-Event Data Model

Building truly Immutable data stores can be inefficient.  For instance in a game statistics service calculating the total number of games a player has ever played would require reading all of the data from storage about a player’s past games and computing the number on every request.  For an active player who has played hundreds or thousands of games this can be time consuming and expensive.  Caching can help solve this problem, but keeping in memory aggregates for every user can also be costly depending on the size of your audience.

What we really want is the benefits of Immutable data stores, but with the efficiency of precomputed aggregates.  We can achieve this using a hybrid model, made up of Event Entities and Aggregate Entities

Event Entities store information about the systems events.  These Entities are Immutable, and should only support Create and Read operations.  The Event Entities are the source of truth for your application.

Aggregate Entities store the precomputed aggregates.  These Entities are Mutable, and will support all C.R.U.D. operations.  These can be used in your application to respond to requests and should be updated when new events are received.

agg-event model

In this model the Aggregate Entity is really just a function over a set of the corresponding Event Entities.  The Aggregate Entity can be recomputed at any time from the corresponding Event Entities if the Aggregate Entity is deleted, a bug is found in the aggregate computation code, or a new aggregate is desired to support a new feature.

Simple Stats Example

To see this in practice, we’ll go through a simple example for storing game statistics for a game like Galaga, using Azure Table Storage.  Each player has a unique PlayerId and at the end of each game players will upload data to store their statistics for that game.

public class PlayerGameData
{
  public Guid GameId { get; set; }
  public Int32 GameDurationSeconds { get; set; }
  public bool Win { get; set; }
  public Int32 Points { get; set; }
  public Int32 Kills { get; set; }
  public Int32 Deaths { get; set; }
}

The system needs to store the player statistics for each game.  In addition players will want to view their aggregate statistics across all of their games.

In this example the Events in the system are the game statistic uploads.  The Aggregates in this system will be aggregate statistics players will see across all their games, like Total Points or Total Kills.

The Event Entities in the system will store the information for each game a player plays. The PartitionKey will be based on the PlayerId and the RowKey will be based off of the GameId. These will be Created at the end of each game. These Entities are Immutable, they are the source of truth for the system.

public class PlayerGameEntity : TableEntity
{
   private static string RowKeyString = "game_{0}";

   public PlayerGameEntity(Int64 playerId, Guid gameId)
   {
      PartitionKey = String.Format("{0:x16}", playerId);
      RowKey = String.Format(RowKeyString, gameId);
   }
   public PlayerGameEntity(){}

   public Int64 Points { get; set; }
   public bool Win { get; set; }
   public Int32 Kills { get; set; }
   public Int32 Deaths { get; set; }
   public Int64 GameDuration { get; set; }
}

The Aggregate Entities will store the aggregate player statistics. The PartitionKey will be based off of the PlayerId and the RowKey will be a static string to identify the entity type since there will only be one of these entities per player. These entities are Mutable. They should be updated at the end of each game.

public class PlayerEntity : TableEntity
{
   private static string RowKeyString = "Players";

   public PlayerEntity(Int64 playerId)
   {
      PartitionKey = String.Format("{0:x16}", playerId);
      RowKey = RowKeyString;
    }
    public PlayerEntity() { }

    public Int64 TotalPoints { get; set; }
    public Int32 TotalGames { get; set; }
    public Int32 TotalWins { get; set; }
    public Int32 TotalKills { get; set; }
    public Int32 TotalDeaths { get; set; }
    public Int64 TotalSecondsPlayed { get; set; }
}

This will create Storage Partitions that look like the example below, captured using Cerebrata Tools. Each PlayerEntity (the Aggregate Entity) will have a set of corresponding PlayerGameEntities (Event Entities).  As you can see in the example below the PlayerEntity Values can all be recalculated from the PlayerGameEntities.

agg-event model storage exp

This model has another benefit besides avoiding Data Loss & Corruption; the Data Layer is now a record of all events that have gone through the system.  This is extremely useful in testing and debugging in development and production.

You should follow me on Twitter here

Halo Pride

Halo Pride _feature image

Pride Weekend in Seattle is always a good time, the city especially Capitol Hill is decked out in rainbows, and the Pride parade is always a spectacle to be seen.

This year the Warthog made a special appearance in the Pride Parade.  A bunch of 343ers, including myself, marched along to show support, along with our General Manager Bonnie Ross, Executive Producer Kiki Wolfkill, and Pricipal Producer CJ Saretto rode in the back of the Warthog tossing out t-shirts.

I might not be the same, but that’s not important
No freedom till we’re equal, damn right I support it

-Same Love, Macklemore

The crowds went nuts for the Warthog.  It was awesome to see people’s faces as they recognized the vehicle as it rolled down the street.  In addition to the Warthog, we had a DJ booth on a flatbed truck following behind us providing music to march to, including Halo 4 Soundtrack Remixes.

At one point along the route the Warthog was boarded by a Drag Queen who was hosting the parade.  As we drove up she proclaimed that we STOP!  At which point she climbed on board, with sequins and all!  She was quite impressed with our ride!

WP_20130630_02120130630143251

Working at 343 has offered me some awesome opportunities, like attending E3 & PAX.  Plus as a studio we try to give back to the community by supporting charities like Make a Wish and Child’s Play, which always leaves a smile on my face.  Having Halo show up at Pride in a big way to show our support was yet another awesome experience to add to the list!  I loved being able to march and show Allied support for Pride.  Plus we got to rock some really awesome Pride themed Halo shirts!  Next year I think we need a bubble turret!

You should follow me on Twitter here

Distributing Text Templates via Nuget

A few months ago I began to look into how to distribute T4 Text Templates via Nuget to do Code Generation.  Creating a Nuget package including T4 files is easy to do, just include them as content files.  Content files in Nuget Packages are copied into the root of your application when the package is installed.

Content Files in Nuget packages are added to the project with the default settings for the file type.  T4 files are added to a project along with a dependent file, which will add the following items to your project file.

<ItemGroup>
  <None Include="CodeGeneration.tt">
    <Generator>TextTemplatingFileGenerator</Generator>
    <LastGenOutput>CodeGeneration.txt</LastGenOutput>
  </None>
</ItemGroup>

For most use cases this may be adequate, however in my case I was doing complex code generation where multiple T4 files contributed to the generation of a single output file.  In order to suppor this scenario I needed the T4 files to be added without the Generator and LastGenOutput Tag.  I took to Twitter to see if there was an easy way to specify the default add behavior in Nuget.

nuget

Since there was no out of the box solution to my problem I decided to use the Nuget 2.0+ functionality of running custom Powershell scripts on install.  Some poking around on StackOverflow turned up a couple solutions which made use of XML parsing to modify the project file.  These were verbose and hard to read so I came up with a more compact, reusable, easier to understand solution by utilizing the passed in Project parameter and the DTE.

param($installPath, $toolsPath, $package, $project)

  #Get all of the T4 files in the package extension .tt
  $files = $package.GetFiles() | Where-Object {$_.EffectivePath -match "tt$"}

  foreach ($file in $files)
  {
     $fileName = $file.EffectivePath
     $ProjectItem = $project.ProjectItems | Where-Object {$_.Name -eq $fileName}

     #Set CustomTool to empty not TextTemplatingFileGenerator
     $CustomTool = $ProjectItem.Properties | where-object {$_.Name -eq "CustomTool"}
     $CustomTool.Value = ""
  }

Install.ps1 is also available on gist

This short script can be added to any Nuget Package as the Install.ps1 script to discover all T4 Text Template files included in the package and remove the extra tags and dependent files from the solution.

In addition when needing to modify the Project File on Nuget Package install I would highly recommend programming against the DTE Project Interface instead of using XML, like I did above, as it results in solutions with less lines of code, that is easier to understand.

You should follow me on Twitter here

Patterns & Best Practices For Moving From RDBMS to Azure Storage

dotNetConf_featureImage

I recently presented at DotNetConf 2013 on how to make the paradigm shift from the relational data world like SQL to the non-relational world of Azure Storage.  I had a lot of fun giving the talk.  This was my first public technical speaking event, so any constructive feedback is very welcome!

In the talk I discuss:

  • A quick overview of Azure Storage with a focus on Table Store
  • The trade off between scale and consistency in Azure Table
  • The importance of Table Partitions in Azure Table
  • A Code example of a Simple Stats Storage Layer in SQL and Azure.
  • The benefits of Immutable data, and how you can realistically implement this
  • Using the Saga Pattern for distributed or long running transactions

The Slides for the Presentation are posted on SpeakerDeck
All of the code I use in my slides is available on GitHub.

Resources & References

If you want to dig in deeper.  Below is a great list of resources.  Also I include Links to the quotes and papers I referenced in my talk.

Relational vs Non-Relational

Getting Started with Azure Table

Saga Pattern

Immutable Data:

Google Spanner

 

You should follow me on Twitter here

Origin Story: Becoming a Game Developer

becoming a gamer_featureimage

Over the past few weeks I have been asked over a dozen times how I got into the Games Industry, so I thought I would write it down.

TLDR; My first Console was a SNES.  I learned to program in High School. I attended Cornell University and got a B.S. in Computer Science.  My first job out of college was as a network tester on Gears of War 2 & 3.  I joined 343 industries as a Web Services Developer in January of 2010, and recently shipped Halo 4 on November 6th 2012.

In the Beginning

My story starts out in the typical fashion I fell in love with Video Games after my parents got me an SNES as a kid.  However, here is where my story diverges, my career in the games industry was not decided at 7.

In fact I had already chosen my career a few years earlier.  When I was 5, I announced to my mother that I did not need to learn math because I was going to be a writer when I grew up.  I had an active imagination, and loved exercising it by writing stories of my own.  My first major work was a story about ponies entitled “Hores.”  Luckily my parents would not let me give up on math, and helped me with my spelling.

It turned out that I actually did enjoy math, I just was ahead of my classmates in comprehension which is why I found it boring in grade school.  In Middle School I was placed into the Advanced Math program along with about 25 other students selected to take accelerated courses.  I enjoyed the problem sets and challenges, and more importantly I excelled at them.  This put me on Mrs. Petite’s short list of students to recruit.

The Way of the Code

Mrs. Petite taught Computer Science at my High School, and she notoriously recruited any advanced math or science student to take her class.  She was stubborn and didn’t take no for an answer so Sophomore year instead of having an extra period of study hall, like I originally intended, I was in her Intro to programming class, writing a “Hello World” application in Visual Basic.

Mrs. Petite quickly  became my favorite teacher and I took AP level Computer Science classes Junior and Senior year learning C++ and Java, respectively.  We learned programming basics, object oriented programming, and simple data structures with fun assignments like writing AI for a Tic-Tac-Toe competition, programming the game logic in Minesweeper, and creating a level in Frogger.

During High School I began to realize that I wasn’t just good at programming, but I truly enjoyed it.  Computer Science wasn’t just a science, it was a means of creation.  Like writing, programming gave me the power to start with a blank canvas and bring to life anything I could imagine.

“Programming gave me the power to start with a blank canvas and bring to life anything I could imagine.”

Throughout Middle School and High School I played my fair share of video games.  Most notably I acquired a PlayStation and raided dozens of tombs with Lara Croft, and played Duke Nukem 3D my first First Person Shooter, but games were still not my main focus.  I ended up spending more of my time programming, playing lacrosse, singing in choir, participating in student council, and spending time with my friends.  Video Games were great, but I still had not decided to pursue a career in the Games Industry.

I graduated from High School not only having learned to program in Visual Basic, C++, and Java, but with a passion for programming.  In the Fall of 2004 I decided to continue on my coding adventure by enrolling in the Engineering School at Cornell University focusing on Computer Science.

College

I entered Cornell University expecting to major in Computer Science, but to be sure I dabbled in other subjects Philosophy, Evolutionary Biology, and Civil Engineering before declaring my major.  To this day I still have a diverse set of interests and I enjoyed all of these subjects immensely, but none of them lived up to the joys of coding.

We Made It!
Computer Science Best Friends at Graduation

College was this beautiful, wonderful, stressful blur.  I ran on massive amounts of caffeine and memories of crazy weekends spent with friends.  We worked really hard, but played really hard too.  Even with all the pressure, stress, and deadlines I was having the time of my life.  The classes were fast paced, I was being challenged, and I was learning an immense amount from Data Structures to Functional Programming to Graphics to Security.

Sophomore year I declared myself for CS, and also became a Teaching Assistant for CS 211 (Object Oriented Data Structures and Programming).  In addition another immensely important event happened in the fall of my Sophomore year: I bought an Xbox 360, and Gears of War.  I loved the game, and spent many nights during winter break staying up till 2am chainsawing locusts.  I also spent a significant amount of time playing Viva Piñata that break, like I said diverse set of interests.  This new console, some fantastic games, and the Xbox Live enabled social experiences reignited my passion for gaming.  Now I began to consider Game Development as a career.

Internships

After Sophomore year I took a somewhat unconventional but completely awesome internship at Stanford’s Linear Accelerator Center (SLAC).  I lived in a house with 20 brilliant physics majors, learned about black holes, dark matter, and quantum computing while helping to manage the Batch farm which provided all the computing power for the physicists working at the center.  It was an absolutely amazing experience.

After Junior year I once again went West for the summer.  This time to Redmond Washington as a Microsoft intern working on Windows Live Experiences (WEX).  During that summer I got to exercise my coding chops and most importantly fully solidified the opinion that I wanted to be a developer.  I left the Pacific North West at the end of summer with two job offers in WEX, but by then I knew I really wanted to work on games.  So after some negotiation and another round of interviews I managed to secure a 3rd offer in Microsoft Game Studios as a Software Engineer in Test working on the Networking and Co-op of Gears of War 2.  I was beyond thrilled.

I graduated from Cornell in 2008 with a Bachelors of Science in Computer Science from the Engineering School.  It was a bittersweet moment, I had loved my time at Cornell and most of my friends were staying on the East Coast, but I knew exciting things were waiting for me in Seattle.

The Real World (Seattle)

In July of 2008 I moved out to Seattle, and joined the Microsoft Game Studios team working on Gears of War 2.  I quickly was thrown into the fire as I was assigned ownership of testing the co-op experience.  It was terrifying and exciting to be given so much responsibility right away.  I eagerly jumped into the project and joined the team in crunching immediately after starting.

The first few months in Seattle were a whirlwind as we pushed to get the game through to launch.  The hours were long but I was passionate about the project and I was learning a lot.  It was an amazingly gratifying experience the day Gears of War 2 went Gold.  When the game launched I had another immensely satisfying moment; my computer science best friend from college and I played through the game in co-op and at the end we saw my name in the credits. Life Achievement Unlocked!

Midnight Launch Halo 4
Midnight Launch Halo 4

I love social game experiences, both collaborative and competitive; So post launch I focused a lot of my energy on improving my skills in the areas of networking and services.  So as we moved into sustain on Gears of War 2 I began focusing on the matchmaking and networking experience.  I spent my free time diving through the Xbox XDK, learning about the networking stack, and playing around with Xbox Live Services.  As work began on Gears of War 3 I took ownership of testing the matchmaking code and became very involved in dedicated servers for multiplayer.

In the Fall of 2009 I was asked to temporarily help the fledging 343 Industries studio ship one of the first Xbox Title Applications, Halo Waypoint.  I knew it would mean extra hours and a lot of work, but the opportunity to work on new technology, and make connections in other parts of Microsoft Game Studios was too good to pass up.  I dove headfirst into the transport layer of the Waypoint Console app, and helped get them through launch in November 2009.

The next few months I began to evaluate what I wanted to do next in my career.  Working on Gears of War 3 was a great opportunity, but I really wanted to do be a developer.  The parts of my testing job that I found most satisfying were designing systems, coding internal tools, and researching new technology.  So when the opportunity to join 343 Industries as a developer appeared in January 2010 I jumped at it.  It was a perfect fit.  After reaching out to my contacts in 343 and then participating in a full round of interviews I was offered a position on the team as a web services developer to write code that would power the Halo Universe and enable social experiences; I excitedly accepted!

One of my first tasks at the studio was working on the Spartan Ops prototype.  I was elated that I got to utilize both my technical and creative skills to help create a brand new experience; my Spartan adventures were off to an amazing start!  The rest is history and a few years later we shipped Halo 4.  After launch I once again had a intense moment of elation after playing through Halo 4 on co-op with my college bff and seeing my name in the credits.  It never gets old.

Final Thoughts

Some thoughts, all my own and anecdotal. To be successful as a Game Developer first and foremost you have to be passionate about what you do, whether it is programming, art, design, writing, or something else.  You need to be passionate about games and your chosen field.  In addition I believe my love of learning has been a huge asset in my career development and growth.  I am not afraid to dive into new technologies, or get my hands dirty in a code base I do not understand.  I believe doing this helped me get into the industry, and continuing to do so makes me valuable.  Lastly do not be afraid to ask for what you want, no one is going to just hand you your dream job.  Of course there is a bit of luck and timing involved in breaking into the Industry, but working incredibly hard is the best way I know to help create those opportunities.

You should follow me on Twitter here