Sunday, August 26, 2018

DynamoDB table getting throttled

Usually when a table is throttled while the consumed capacity is well below the provisioned capacity, it is an indicator of hot partition or a “burst” of read or write activity. Hot partition means one single partition is accessed more than other partitions and hence more capacity is consumed by that partition. Hot partition can be caused by unevenly distributed data across partitions due to hot partition key. A “burst” of read or write activity occurs when workloads rely on short periods of time with high usage, for example batch data upload.

I recommend the following approaches to help mitigate this situation:

1. Buffer write requests in your application and then send them out sequentially during the time when the capacity is underutilized (blue color- cold spots) so all the write capacity can be fully utilized. This helps reduce bursts in write operations.

2. Distribute your upload work across your partition key values. For example, you can upload one or a few items from each partition key value (resource_type_id) first. Then you repeat the pattern for the next set of sort key values (TimeStamp) for all the items until you upload all the data. This way the workload is distributed across your partition key values, keeping more DynamoDB servers busy simultaneously and improving your throughput performance. It helps reduce hot partitions.

3. Temporarily increase write capacity for the table to handle short term bursts in write operations. After the load has decreased, you can reduce it again.

4. Revisit your table schema design if needed (best long term solution): DynamoDB performs the best when data are evenly distributed across partitions and read/write workloads are distributed across partition key values. For long term solution, you can check your data pattern against partition keys, sort keys and GSI in the table to see if there are improvements can be made. 


Post a Comment