DMS instead of Datapipeline

In a previous post I detailed my trials and fails with using Datapipeline to archive Aurora tables to Redshift. This led to a comment from rjhintz about using AWS DMS instead. I initially went with Datapipeline because we would eventually truncate the source table and would not want that replicated to Redshift, deleting the archive data. But I would still take some timeout to checkout the DMS service.

AWS DMS initial use case is to help people migrate to AWS from an on-premise DB installations. As it says in the name 😉 My use case would be for archiving. We initially used Datapipeline to achieve this. The setup was pretty tedious! To say the least. Once it is up and running, we still have to check that the jobs have completed correctly and that nothing has gone wrong.

This weekly checking had become a chore. This is where DMS comes in. We only have one table that needs truncating, whereas all it’s related tables simply need to be in-sync. It took a day to get up to speed with DMS, after that it we migrated all our Datapipelines except for one to DMS.

Create an instance that has access to your source and target databases. This was needed in Datapipeline as well. It’s just much much easier in DMS. No AMIs needed with larger instance stores.

Screen Shot 2016-08-13 at 10.41.56 PM.png

Create your database endpoints.

Create a task that will migrate and then keep in sync your desired tables.

Screen Shot 2016-08-13 at 10.44.15 PM.png

That’s it. Done.

Advertisements

Tags: , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s