About me
DynamoDB
DynamoDB Backups
Still using AWS DynamoDB Console? Try Dynobase to accelerate DynamoDB workflows with code generation, data exploration, bookmarks and more. Check it out .
The recent AWS Reinvent 2017 event saw some major announcements, including a couple of exciting ones for DynamoDB users. Global tables and On-Demand backups. In this post, I’ll be taking a quick look at on-demand backups, how they work and how we can schedule them to regularly take full backups.
On-Demand backups are a feature built into the DynamoDB service (Accessible via the API, AWS Management Console and CLI as usual), which allows you to take a full backup of a table at a point in time.
This task has no impact on performance or availability to your tables. All backups are automatically encrypted, cataloged, easily discoverable, and retained until you explicitly delete them.
Additionally, you can restore these backups to a new table at any point.
Along with data, the following is included in the backups:
Global secondary indexes (GSIs)
Local secondary indexes (LSIs)
Streams
Provisioned read and write capacity
The following is NOT included in the backups:
Auto scaling policies
AWS Identity and Access Management (IAM) policies
Amazon CloudWatch metrics and alarms
Tags
Stream settings
Time To Live (TTL) settings
Charges vary between regions, but you are charged for storage of the backups only, which means the size of the table you are backing up. The price is approx 40% of standard DynamoDB storage costs. (Approx $0.10-0.11 per GB-month).
According to the AWS Docs:
When you create an on-demand backup, a time marker of the request is cataloged. The backup is created asynchronously by applying all changes until the time of the request to the last full table snapshot. Backup requests are processed instantaneously and become available for restore within minutes.
The above explains how the backups are taken without impacting performance.
Navigate to the DynamoDB Console
Click on the new Backups option under Tables
Click on Create Backup
Select the table you wish to backup, and type in a name for the Backup.
Click Create Backup
Click okay on the success message
You should now see your backup
In the Backups section in DynamoDB Console
Select the backup you wish to restore, and click Restore Backup
Enter the table you wish to restore the backup to. This needs to be a new table.
Click Restore
You should see the new table being restore back in DynamoDB
Unfortunately, there is no native functionality that allows you to schedule these backups regularly, but they do recommend you to use CloudWatch Events and AWS Lambda together to trigger these backups. Lets see how we can set this up:
Create a new Lambda Function using NodeJS 6.10 using this code on github . (As at Dec 3rd 2017) Keep in mind, the native Lambda AWS SDK has not been updated to include the backup and restore APIs. Include node_modules/aws_sdk as part of the Lambda function code. This seems to be resolved now
(As at Dec 3rd 2017) Keep in mind, the native Lambda AWS SDK has not been updated to include the backup and restore APIs. Include node_modules/aws_sdk as part of the Lambda function code. This seems to be resolved now
Use the following IAM Policy for the Lambda function Role
Create a CloudWatch Event Rule Set a schedule for the backups to run (ie: Every 60 minutes) Set the target as the Lambda function Configure the input to be a constant (JSON text). This allows you to configure backups for tables at independent intervals. ie: Seperate rules for each table / interval. Pass in constant JSON like this (seperated by comma): Done! The Lambda should run as often as configured, and take backups of the configured input. You should see backups using the name format: [ TABLE_NAME ][ YYYYMMDDTHHMMSS ]
Set a schedule for the backups to run (ie: Every 60 minutes)
Set the target as the Lambda function
Configure the input to be a constant (JSON text). This allows you to configure backups for tables at independent intervals. ie: Seperate rules for each table / interval. Pass in constant JSON like this (seperated by comma):
This allows you to configure backups for tables at independent intervals. ie: Seperate rules for each table / interval.
Pass in constant JSON like this (seperated by comma):
Done! The Lambda should run as often as configured, and take backups of the configured input. You should see backups using the name format: [ TABLE_NAME ][ YYYYMMDDTHHMMSS ]
Although a valid backup option for DynamoDB, it is worth validating whether this solution meets your needs.
No impact on performance
Backups taken in very little time
No limit on backups
Restores are very straight forward
Scheduling for backups is possible (including alerts when scheduling fails)
Storage of the backups can get expensive (compared to S3)
Backups are full snapshots only (no incremental options)
Not available in all regions (yet)
You can only restore to a new DynamoDB table
No ability to backup to a separate account (Essential for Disaster Recovery in case of security breach)
There is news that a new Point in Time backup / restore option will be available in DynamoDB in early 2018, looking forward to seeing that as a more complete solution. My big hope is that it will provide the ability to move the backups to an isolated account, for DR scenarios. Fingers crossed!
HTH
Click to share on Twitter (Opens in new window)
Click to share on LinkedIn (Opens in new window)
Click to share on Reddit (Opens in new window)
Click to share on WhatsApp (Opens in new window)
Click to share on Skype (Opens in new window)
Click to share on Facebook (Opens in new window)
Click to email this to a friend (Opens in new window)
Tony · Reply January 4, 2018 at 8:48 AM Could you please enlarge the image after this: “Configure the input to be a constant (JSON text).”? It is hard to see? Abhaya Chauhan · Reply Author January 4, 2018 at 11:31 AM Hey Tony, Thanks for the request. I’ve made the screenshot full size. Let me know if it is still too small. Abs
Could you please enlarge the image after this: “Configure the input to be a constant (JSON text).”? It is hard to see?
Abhaya Chauhan · Reply Author January 4, 2018 at 11:31 AM Hey Tony, Thanks for the request. I’ve made the screenshot full size. Let me know if it is still too small. Abs
Hey Tony, Thanks for the request. I’ve made the screenshot full size. Let me know if it is still too small. Abs
bisonfute (@bisonfute) · Reply May 9, 2018 at 11:33 PM I think, the aws-sdk is updated now. If you use node8.10, it’s creating backups without installing the aws-sdk package in node_modules abhayac · Reply Author May 10, 2018 at 11:23 AM Thanks for the message, I’ll update the post.
I think, the aws-sdk is updated now. If you use node8.10, it’s creating backups without installing the aws-sdk package in node_modules
abhayac · Reply Author May 10, 2018 at 11:23 AM Thanks for the message, I’ll update the post.
Thanks for the message, I’ll update the post.
Anonymous · Reply September 20, 2018 at 4:01 AM Thanks Abhaya. Do you have a way to age out backups? You have this run everyday, but you only want the last 7. As soon as the 8th day hits, the oldest backup will be removed.
Thanks Abhaya. Do you have a way to age out backups? You have this run everyday, but you only want the last 7. As soon as the 8th day hits, the oldest backup will be removed.
AWS Specialist, passionate about DynamoDB and the Serverless movement.
Currently focusing on helping SaaS products leverage technology to innovate, scale and be market leaders.
Introduction to CloudFormation
Getting Started with DynamoDB
DynamoDB in Production
Intro to CloudFormation Course
Using DynamoDB in Production – New Course
DynamoDB: Monitoring Capacity and Throttling
Pluralsight Course: Getting Started with DynamoDB
Partition Throttling: How to detect hot Partitions / Keys
AWS
Lambda
Pluralsight
Presentations
Rants
Uncategorized
Type and press Enter to search