AWS Storage Gateway deployment on VMware

The AWS Storage Gateway facilitates the management of the storage in the AWS Cloud. It’s an appliance that runs in On-Premises and Online environments within the Amazon Web Services ECS instances. It offers access to the File storage (Object storage) based on Amazon “S3” protocol, Volume Storage on EBS instances (Block level storage) and Tape Gateway supporting VTL on S3 storage and Glacier for longer retention periods. Moving data to the Cloud has moved from being simply desirable to a necessity for a number of reasons. If only a few years ago the trend was to “avoid” the Cloud now the trend has completely changed also for companies that manage sensitive data. More importantly companies and users in the first place require data to always be available irrespective of the platform used.

There are really main benefits when using the Cloud. And in particular the AWS Storage Gateway provides that “air-gap” feature that is typical of tape backups. It is fully compatible and tested with major backup applications and eliminates the need of “physically” manage the tapes as the VTLs will be stored and archived online. This also reduces the costs of exporting the data with tapes into a separate “tape-center” miles away from the Production environment. Now all this is available “online” mimicking the standard procedures where the VTL tapes can be easily archived even in different regions. Subject to current Data Sovereignty rules of course.

Data is growing and managing longer retention with On-Premises storage is becoming cost prohibitive in some cases. AWS offers the advantage to store data for even longer periods online and the flexibility to integrate with the current backup infrastructure. Good news is AWS also offers a free tier trial for 12 months. Suitable for testing in production environments and for homelabs too!

The purpose of this article series is to explore the configuration steps and benefits of storing data to the Cloud. In particular using the AWS Storage Gateway to store Veeam Backup data to VTLs located in the Amazon Cloud. All Veeam backups from the production environment including VMware vSphere, Microsoft Hyper-V and Nutanix AHV can be saved into the Cloud fully adhering to the 3-2-1 industry best practice which suggests 3 copies of the data, stored on two different media being one offline/offsite.

The article series will cover different steps:

• AWS Storage Gateway deployment in VMware
• AWS virtual appliance configuration
• Configuring Veeam Backup with AWS VTL tapes
• Running Veeam backup jobs to AWS tapes

Deploy AWS Storage Gateway on VMware

In order to use the AWS free tier trial it is required to sign and log-in. The process is really quick. Once the AWS user is created we can access the main services page and click on AWS Storage Gateway from the Storage section.

A new web page will start the wizard from where we can download and configure the virtual appliance.

So let’s start by selecting the gateway type. In reality it is possible to install and configure one AWS Storage Gateway per AWS account. The wizard just helps on the first installation type. After initial deployment, same AWS Storage Gateway can be used for remaining purposes. In this example the AWS gateway will provide tape storage to support Veeam Backup to tape (VTL) jobs.

In this step we can decide on image type. For this example we’ll use the VMware based image template to deploy on VMware vSphere. Other offers include a template image for Microsoft Hyper-V and the ability to directly run the appliance from an Amazon EC2 instance. There are details on the AWS documentation as per the recommended requirements. Generally speaking the AWS Storage Gateway appliance should have 4 CPUs, 16 GB of RAM Memory and initial 80 GB of disk space for installation.

After downloading the VMware OVA image we can use Veeam or other solutions to copy or move the OVA to a convenient location to perform the initial deployment.

When deploying an OVA file we can point to the file directly or provide the HTTP link to the file directly from the vSphere Datastore. when logging to the VCSA or vSphere address and adding /folder like (https://vcsaIPaddress/folder) will show the datastores we can browse and copy the HTTP link to deploy the OVA file. Either method is fine.

At this point let’s choose a name and select the folder location in the Data Center infrastructure.

Next is to associate the vSphere Host to run the AWS Storage Gateway appliance. Recommended requirements are 4 CPUs, 16 GB or Memory and 80 GB of installation disk plus cache and upload buffer as we’ll see later.

Let’s review the main settings. I would recommend to run thick provisioning in the next screens for stable performances over time.

In this step we can define the datastore containing the AWS appliance virtual disk files. Considering this appliance will serve VTL tapes as backup copies we can also choose a less performing storage with more space available. In general secondary copies to tapes have less restrictive time windows to complete. Of course it is important to evaluate different factors that might affect the job completion before the end of the chosen time window. More information in this in a separate article on the series.

By default the AWS appliance ships with one network card. It should be on the same Port Group as Production or Backup Network. In the next steps once the appliance is deployed we’ll configure the networking parameters.

Time to review the main settings and kick off the AWS Gateway deployment.

Before powering on the AWS Storage Gateway appliance there are some settings that we need to change to accommodate the Tape Gateway configuration. In particular, we need to add 2 virtual disks of at least 150 GB each. One for cache and one for the upload buffer. The AWS documentation also provides a formula to calculate the storage requirements based on data “ingest” and finally to the VTL Cloud. I’m trying my luck and reduced the Memory from 16 GB to 8 GB and see how things go performance wise. First jobs will not be massive anyways so eager to understand more about resources utilization.

Last bit before powering on the appliance is to enable the synchronization with the vSphere Host. At the same time let’s make sure both vSphere Hosts and vCenter use the same NTP servers.

Once ready we can hit OK and start the AWS Storage Gateway appliance. The deployment process is very quick and upon first boot to make configuration setting effective is using a small fraction of the assigned memory: about 80 MB. Whilst the appliance is working will consume more of course.

This concludes the first part about deploying AWS Storage Gateway in VMware vSphere. Next step is the appliance configuration and registration against the AWS account.