Posts Tagged ‘DMS’

DMS instead of Datapipeline

April 27, 2017

In a previous post I detailed my trials and fails with using Datapipeline to archive Aurora tables to Redshift. This led to a comment from rjhintz about using AWS DMS instead. I initially went with Datapipeline because we would eventually truncate the source table and would not want that replicated to Redshift, deleting the archive data. But I would still take some timeout to checkout the DMS service.

AWS DMS initial use case is to help people migrate to AWS from an on-premise DB installations. As it says in the name 😉 My use case would be for archiving. We initially used Datapipeline to achieve this. The setup was pretty tedious! To say the least. Once it is up and running, we still have to check that the jobs have completed correctly and that nothing has gone wrong.

This weekly checking had become a chore. This is where DMS comes in. We only have one table that needs truncating, whereas all it’s related tables simply need to be in-sync. It took a day to get up to speed with DMS, after that it we migrated all our Datapipelines except for one to DMS.

Create an instance that has access to your source and target databases. This was needed in Datapipeline as well. It’s just much much easier in DMS. No AMIs needed with larger instance stores.

Screen Shot 2016-08-13 at 10.41.56 PM.png

Create your database endpoints.

Create a task that will migrate and then keep in sync your desired tables.

Screen Shot 2016-08-13 at 10.44.15 PM.png

That’s it. Done.