AWS News Blog – Welcome to AWS Storage Day 2020

Our first-ever Storage Day in November 2019 (Welcome to AWS Storage Day) was a big success. We were able to take a multitude of significant announcements related to AWS Storage services and summarize them in a single post, with longer and more detailed posts as needed.

Today, we are doing it again, so welcome to AWS Storage Day 2020! If you use Amazon Simple Storage Service (S3), Amazon FSx for Lustre, Amazon FSx for Windows File Server, Amazon Elastic Block Store (EBS), Amazon Elastic File System (EFS), AWS Transfer Family (AWS Transfer for SFTP, FTPS, or FTP), AWS DataSync, AWS Storage Gateway, AWS Backup, or a device in the AWS Snow Family, I am sure that you will find something relevant and useful. Unless otherwise noted, all of these features are available now and you can start using them today!

Like I said in 2019, let’s get into it…

Amazon Simple Storage Service (S3)
S3’s Intelligent-Tiering storage class now supports automatic data archiving to new Archive Access and Deep Archive Access storage tiers. The Archive Access tier supports expedited (1-5 minutes), standard (3-5 hours, and bulk (5-12 hours) retrievals, with an extra charge for the expedited option. The Deep Archive Access tier supports standard (within 12 hours) and bulk (within 48 hours) retrievals. You simply select this storage class for the desired objects, opt in to one or both Archive Access tiers, and S3 will take care of the rest, moving objects between the four tiers of storage based on access patterns. S3 Intelligent-Tiering delivers millisecond latency and high throughput for data that is accessed frequently, and now the lowest storage cost in the cloud for data that is not accessed for long periods of time. To learn more, read S3 Intelligent-Tiering Adds Archive Access Tiers. For more information on Optimizing costs with S3 Intelligent-Tiering Archive Access Tiers you can view this video.

You can now opt in to Delete Marker Replication when you use S3’s Replication Time Control feature. These features were previously mutually exclusive, and you can now use them together to have confidence that deletions in a source bucket will be reflected in the target bucket, while also taking advantage of S3’s Replication SLA. To learn more, read Amazon S3 Replication Adds Support for Replicating Delete Markers.

You can now monitor replication progress on a rule-by-rule basis. You can examine Bytes Pending Replication, Operations Pending Replication, and Replication Latency and use S3 Event Notifications to receive replication failure events. To learn more, read Amazon S3 Replication Adds Support for Metrics and Notifications.

S3 Object Ownership is now available in all AWS regions and is now supported by AWS CloudFormation. You can use this feature to automatically assume ownership of objects that are uploaded to a bucket by another AWS account. To learn more read Amazon S3 Ownership Now Generally Available with CloudFormation Support and Amazon S3 Update – Three New Security & Access Control Features.

Amazon FSx for Lustre
You (or your IT administrator) can now set, monitor, and enforce storage quotas on your Amazon FSx for Lustre file systems using native Lustre commands (lfs setquota and lfs quota). This allows multiple users and groups to share a single high-performance file system while keeping storage usage within defined limits. The limits can be hard (strictly enforced) or soft (subject to a grace period that can also be configured). To learn more, read FSx for Lustre Storage Quotas.

Amazon FSx for Windows File Server
Amazon FSx for Windows File Server file shares can now be accessed using user-friendly DNS aliases, allowing you to move applications and data to the cloud without making code or configuration changes. You can set up one or more aliases when you create each share, and you can add, change, or remove them later. Then you create a CNAME record in your DNS server, and connect to the share (we recommend Kerberos authentication & encryption in transit). To learn more, read Managing DNS Aliases.

Container-based Windows workloads running on Amazon ECS can now access Amazon FSx for Windows File Server file shares. This allows ECS to run stateful workloads, content management systems, and other applications that require persistent shared storage. You simply specify the file system id and the mount point in your task definition, and ECS will take care of the rest. The latest versions of the ECS-Optimized Amazon Windows AMIs include the necessary modules; to learn more read Using Amazon FSx for Windows File Server as Persistent Storage on Windows Containers.

Amazon Elastic Block Store (EBS)
The EBS Cold HDD (sc1) volume type gives you low-cost magnetic storage for your large, sequential, cold-data workloads: Hadoop Clusters, log processing, and big data. Effective today (November 9, 2020) we are reducing the price for sc1 storage by 40% in all regions where EBS is available. To learn more, read Price Reduction for Amazon EBS SC1 Volumes.

You can now use Amazon Data Lifecycle Manager (DLM) to manage the lifecycle of your Amazon Machine Images (AMIs) and the EBS snapshots that are associated with them. The AMI Lifecycle Policies that you create can select target instances for AMI creation, set schedules for creation and retention, copy newly created AMIs to other regions, share them with other AWS accounts, and even add tags to the AMI and the snapshots. On the deletion side, you can choose to simply deregister the AMI or to delete the associated snapshots as well. To learn more, read about AMI Lifecycle Management Available for Data Lifecycle Manager.

Amazon Elastic File System (EFS)
You can now create an Amazon Elastic File System when you use the EC2 Launch Instance Wizard to launch an instance. You can then add the new file system to the instance and it will be mounted automatically when the instance is launched. To learn more, read Easy-to-Use Amazon EFS Integrations with AWS Compute Services.

AWS Transfer Family
If you operate in a shared services VPC environment, you can now use AWS Transfer Family to help you meet your compliance requirements when you segment your AWS environment using tools such as AWS Landing Zone for security, cost monitoring, and scalability. With this launch, when you create a server, you can host your server’s endpoint in a VPC shared by the VPC owner’s account and associate security groups during server creation. To learn more, read Using VPC Hosted Endpoints in Shared VPCs with AWS Transfer Family.

AWS DataSync
You can now use AWS DataSync to perform fully-automated data transfers between datasets stored in Amazon S3, Amazon EFS, or Amazon FSx for Windows File Server, without deploying DataSync agents. You can create and schedule the data transfer tasks with a couple of clicks, and then benefit from automatic encryption of data in-transit, data integrity verification both in-transit and at-rest, and highly granular visibility into the transfer process, all for datasets that might span millions of files or objects and multiple AWS Regions. To learn more, read AWS DataSync Announces Fully Automated Transfers Between AWS Storage Services.

In addition, you can now exercise more granular control over the network bandwidth used by DataSync tasks, by adjusting the per-task limit up or down while the task is running. This can come in handy when you are moving data from your own data center to AWS for an extensive period of time, over a network connection that is shared with other workloads. To learn more, read You Can Now Adjust Network Bandwith Used by Running AWS DataSync Tasks and watch the video.

AWS Snow Family
You can now import virtual machine images in raw (disk image) format as AMIs into Snowball Edge Storage Optimized and Snowball Edge Compute Optimized devices while they are onsite and deployed. This allows you to make changes in minutes without having to order and await delivery of a new device. This feature is available on devices that are ordered on or after November 9, 2020; to learn more, read AWS Snowball Edge Supports Importing Virtual Machine Images to Deployed Snow Devices.

You can now import Windows 2012 and Windows 2016 virtual machine images and use them to launch instances on Snowball devices. You can use the onsite AMI feature that I described in the previous paragraph, or you can use VM Import/Export to import your image into AWS and then have it loaded onto your Snowball device when you order it. You are responsible for bringing your own Windows license in either case. To learn more, read AWS Snowball Edge Supports Windows Operating Systems.

AWS Storage Gateway
We have added four new features that increase scalability, security, and ease of use for File, Tape, and Volume Gateways.

You can now create Tape Gateway and Volume Gateway local storage caches as large as 64 terabytes, four times as large as before, for gateways that are running on a virtual machine. This will allow you to keep more of your data closer to your on-premises applications and users. You can create larger caches today on newly created gateways, and after the next scheduled software update for existing ones. To learn more, read AWS Storage Gateway Increases Cache 4x and Enhances Bandwidth Throttling.

You can now create a schedule to control the maximum network bandwidth consumed by your Tape and Volume Gateways. You can, for example, regulate use of your on-premises bandwidth during peak time and use all available bandwidth at other times. To learn more, read AWS Storage Gateway Increases Cache 4x and Enhances Bandwidth Throttling.

Uploads to File Gateway can now trigger Amazon CloudWatch or Amazon EventBridge notifications that you can use to initiate automated workflows. This allows you to easily use AWS to process on-premises file data using AWS Lambda functions and other event targets. To learn more, read Processing File Upload Notifications from AWS Storage Gateway on Amazon S3.

File Gateway now supports access-based enumeration. You can enable this feature to ensure that your users see only the SMB file shares, folders, and files that they have permission to open. In addition to helping protect sensitive information, this feature provides users with a simplified folder view by showing them only what they can access. To learn more, read Enhance Privacy Controls with Access-Based Enumeration for File Gateway.

AWS Backup
You can now use AWS Backup to create scheduled, policy-driven backup plans for your Amazon FSx file systems. The backups take advantage of Amazon FSx’s existing backup functionality, and can be used for Amazon FSx for Windows File Server file systems, and for persistent Amazon FSx for Lustre file systems that are not linked to S3 or another data repository. To learn more, read AWS Backup Provides Centralized Data Protection Across Your AWS Resources.

Join us Online
If you are ready to learn even more about any or all of these launches, please register for and attend our AWS Storage Day 2020 Online Event tomorrow (November 10, 2020). We’ve got nine sessions on the agenda, with four hours chock-full of content. I’ll be the concluding speaker, and I’m looking forward to the event.

That’s All, Folks
As you can see, this year’s Storage Day launches focused on four pillars: cost, performance, management, and integration. We chose these pillars (and the features that they represent) after speaking with and listening to thousands of customers for these services. Take a close look at what we launched, build something amazing, and let me know about it!

— Jeff;