Enforcing Idempotency at the Data Layer


In Computer Science idempotent operations are defined as operations that produce the same result if they are executed once or multiple times.

Practically in an application or service this means that idempotent operations can be retried or replayed without the fear of processing the data multiple times or causing unwanted side effects.  As a Web Service Developer having Idempotent operations allows us to have simpler logic for handling failures.  If a request fails we can simply retry the request by replaying it.  In services and messaging systems having idempotent operations is the easiest way to handle “at least once messaging”  Jimmy Bogard has written a great post on this topic: Los Techies (Un) Reliability in Messaging Idempotency and De-Duplication

In services and messaging systems having idempotent operations is the easiest way to handle “at least once messaging”

Most operations are not mathematically idempotent, so developers must write application level logic to enforce Idempotency of requests.  However, If we can enforce Idempotency in our operations at the Data Storage Layer than the need for special case logic is minimized.

Using Azure Table to Enforce Idempotency

Azure Table Storage provides Transaction support inside of a single Table Partition.  We can take advantage of this to enforce Idempotency at the Data Storage layer of applications by using the Aggregate-Event Data Model described in a pervious post.

In order for this to work the data needs to be structured in the following way.

  1. The Aggregate Entity and Event Entities must be stored in the same Partition.
  2. Updating the Aggregate Entity and Adding the Event Entity to storage must occur in the same Batch Operation.

By updating the Aggregate Entity and adding the Event Entity in the same TableBatchOperation either both writes will succeed or both writes will fail, leaving your data in a consistent state whether you have received the event once, or many times.

If the Batch Operation fails, you can determine that it was because the data was already processed, and not caused by some other failure, by checking the Storage Exceptions HTTP Status Code.  If the Status Code equals 419 – Conflict, then one of Entities marked for Add in the Batch Operation already exists in the table.

Simple Stats Example

To see this in practice, we’ll go through a simple example for storing game statistics for a game like Galaga, using Azure Table Storage.  Each player has a unique PlayerId and at the end of each game players will upload data to store their statistics for that game.

public class PlayerGameData
   public Guid GameId { get; set; }
   public Int32 GameDurationSeconds { get; set; }
   public bool Win { get; set; }
   public Int32 Points { get; set; }
   public Int32 Kills { get; set; }
   public Int32 Deaths { get; set; }

In the previous post on Immutable Data Stores I also share code for the TableEntities used in the Simple Stats Example, which will not be repeated here.

Below is an example of how to Process a Simple Stats Game and store it such that Idempotency is enforced at the Data Layer.

public static void ProcessGame(Int64 playerId, PlayerGameData gameData)
   // Create the batch operation.
   TableBatchOperation batchOperation = new TableBatchOperation();

   PlayerEntity player = PlayerEntity.GetPlayerEntity(playerId);
   UpdatePlayerEntity(player, gameData)

   //Create PlayerGame Row
   PlayerGameEntity playerGame = new PlayerGameEntity(playerId, gameData);

   catch (Exception ex)
       //Check if the error occurred because we already processed the data
       if (ex is Microsoft.WindowsAzure.Storage.StorageException)
          var storageEx =  (Microsoft.WindowsAzure.Storage.StorageException)ex;
          if(storageEx.RequestInformation.HttpStatusCode == (int)HttpStatusCode.Conflict)
       throw ex;

There are two options when a game is processed:

1. The system has not processed this Game: In this case the above code will create a new PlayerGame Entity and update the in memory copy of the Player Entity.  The Batch Operation will succeed, and the PlayerGame along with the updated Player Entity will be stored in the table.

2. The system has processed this Game: In this case the above code will create a new PlayerGame Entity and update the in memory copy of Player Entity.  However when the Batch Operation executes it will fail since the Event Entity already exists in this partition.  Because of the per partition transaction support provided by Azure Storage, the updates to the Player Entity will also not be stored.  The data in storage will be the same as if the Game had only been processed once.

By using Azure Table Store to enforce Idempotency of event processing you no longer have to write application level logic to handle at least once messaging.  In addition by using this pattern you get all the benefits of Immutable Data Stores as well.

You should follow me on Twitter here