About me
DynamoDB
DynamoDB Backups
Still using AWS DynamoDB Console? Try Dynobase to accelerate DynamoDB workflows with code generation, data exploration, bookmarks and more. Check it out .
I’ve seen many situations where a team has realised the initial Key Schema they’ve selected for their tables isn’t a good fit, for performance, scale reasons, or simply they recognised a need for a Local Secondary Index.
Global Secondary Indexes allow you to create indexes with alternative Partition Key / Sort Key but this isn’t ideal, as it has additional storage + capacity costs. You also lose the ability to use consistent reads on this index.
In this situation, try leveraging DynamoDB Streams and AWS Lambda to remodel data as needed.
A nice way to restructure your table definition is to leverage DynamoDB triggers, following these steps:
Create a new table (let us call this NewTable), with the desired key structure, LSIs, GSIs.
Enable DynamoDB Streams on the original table
Associate a Lambda to the Stream, which pushes the record into NewTable. (This Lambda should trim off the migration flag in Step 5)
[*Optional*] Create a GSI on the original table to speed up scanning items. Ensure this GSI only has attributes: Primary Key, and Migrated (See Step 5).
Scan the GSI created in the previous step (or entire table) and use the following Filter: FilterExpression = "attribute_not_exists(Migrated)" Update each item in the table with a migrate flag (ie: “Migrated”: { “S”: “0” }, which sends it to the DynamoDB Streams (using UpdateItem API, to ensure no data loss occurs). **NOTE** You may want to increase write capacity units on the table during the updates.
Update each item in the table with a migrate flag (ie: “Migrated”: { “S”: “0” }, which sends it to the DynamoDB Streams (using UpdateItem API, to ensure no data loss occurs).
**NOTE** You may want to increase write capacity units on the table during the updates.
The Lambda will pick up all items, trim off the Migrated flag and push it into NewTable.
Once all items have been migrated, repoint the code to the new table
Remove original table, and Lambda function once happy all is good.
The great part about DynamoDB Streams it respects ordering, which means there will be no data loss.
I’ve used this a few times to great effect.
Code used in this post is available here:
https://github.com/abhayachauhan/change-table-schemas
Hit me up if you would like more details. Good luck!
Click to share on Twitter (Opens in new window)
Click to share on LinkedIn (Opens in new window)
Click to share on Reddit (Opens in new window)
Click to share on WhatsApp (Opens in new window)
Click to share on Skype (Opens in new window)
Click to share on Facebook (Opens in new window)
Click to email this to a friend (Opens in new window)
mnwk · Reply June 28, 2018 at 7:17 AM So why not just make your code in step 5 do the work of step 3 instead of just flagging? Abhaya Chauhan · Reply Author June 28, 2018 at 9:51 AM Hey mnwk, Thanks for the question. You certainly could do the work for Step 3 in Step 5, but then you could have a chance of losing data (assuming your system is still live in production). If it is not, then your suggestion will work well. The lambda ensures that all data changed is constantly pushed to the new table, until the application stops using the original table and data stops changing in it. Alternatively, to speed up Step 5, you could just table create an “on demand backup” and restore it. This is a new feature which could speed up this process, but essentially you still need the lambda to ensure no data is lost in the migration. HTH Abhaya
So why not just make your code in step 5 do the work of step 3 instead of just flagging?
Abhaya Chauhan · Reply Author June 28, 2018 at 9:51 AM Hey mnwk, Thanks for the question. You certainly could do the work for Step 3 in Step 5, but then you could have a chance of losing data (assuming your system is still live in production). If it is not, then your suggestion will work well. The lambda ensures that all data changed is constantly pushed to the new table, until the application stops using the original table and data stops changing in it. Alternatively, to speed up Step 5, you could just table create an “on demand backup” and restore it. This is a new feature which could speed up this process, but essentially you still need the lambda to ensure no data is lost in the migration. HTH Abhaya
Hey mnwk, Thanks for the question.
You certainly could do the work for Step 3 in Step 5, but then you could have a chance of losing data (assuming your system is still live in production). If it is not, then your suggestion will work well.
The lambda ensures that all data changed is constantly pushed to the new table, until the application stops using the original table and data stops changing in it.
Alternatively, to speed up Step 5, you could just table create an “on demand backup” and restore it. This is a new feature which could speed up this process, but essentially you still need the lambda to ensure no data is lost in the migration.
HTH
Abhaya
wlemcke · Reply July 24, 2018 at 1:11 AM Did you also tried this using cloudformation? abhayac · Reply Author September 26, 2018 at 10:17 PM DynamoDB APIs and CloudFormation do not allow you to modify Key Schemas. You can add GSIs but the base table cannot be modified.
Did you also tried this using cloudformation?
abhayac · Reply Author September 26, 2018 at 10:17 PM DynamoDB APIs and CloudFormation do not allow you to modify Key Schemas. You can add GSIs but the base table cannot be modified.
DynamoDB APIs and CloudFormation do not allow you to modify Key Schemas. You can add GSIs but the base table cannot be modified.
Anonymous · Reply December 6, 2018 at 1:17 PM This is perfect. I tried to update my schema to add a LSI this morning; quickly realizing that to have a new schema, I must have a new table. Your migration strategy is exactly what I needed, so thank you!
This is perfect. I tried to update my schema to add a LSI this morning; quickly realizing that to have a new schema, I must have a new table. Your migration strategy is exactly what I needed, so thank you!
Ahri · Reply March 7, 2019 at 4:18 AM A couple questions – 1) How many records did you have in your table when you ran this? I am curious, since we are running ours at a pretty big scale (~100 million records) and I am wondering if this sort of solution can handle it. 2) Where did you run your step 5 from?
A couple questions – 1) How many records did you have in your table when you ran this? I am curious, since we are running ours at a pretty big scale (~100 million records) and I am wondering if this sort of solution can handle it.
2) Where did you run your step 5 from?
AWS Specialist, passionate about DynamoDB and the Serverless movement.
Currently focusing on helping SaaS products leverage technology to innovate, scale and be market leaders.
Introduction to CloudFormation
Getting Started with DynamoDB
DynamoDB in Production
Intro to CloudFormation Course
Using DynamoDB in Production – New Course
DynamoDB: Monitoring Capacity and Throttling
Pluralsight Course: Getting Started with DynamoDB
Partition Throttling: How to detect hot Partitions / Keys
AWS
Lambda
Pluralsight
Presentations
Rants
Uncategorized
Type and press Enter to search