The “Rsync” program is a UNIX system-based tool that allows you to copy content from a local machine to a remote machine—your system to a NetStorage storage group. This tool is particularly useful because it quickly synchronizes content by transferring only the changed files. It does not overwrite standing files you’ve already transferred. This makes it easier to simply transfer what has changed from a local copy, at the expense of extra remote read operations.

NetStorage offers support for two “versions” of Rsync:

• Secure Rsync. With this version, you connect by resolving a Secure Shell (SSH) key pair. You then issue Rsync commands via this connection.
• Non-secure Rsync. With this version, you create a unique password and include it for authentication when issuing Rsync commands.

Rsync-specific guidelines and requirements

Use the same Rsync version as NetStorage

For best results, we recommend that the Rsync client installed the system that will perform the transfers (the “client”) be as close in version as possible to the Rsync server used by NetStorage. This ensures compatibility with supported rsync command options. Older versions can be problematic, and newer ones may offer functionality that is not supported.

NetStorage currently supports version: 3.1.2

Rsync file total limitations

There's no specific limit to the number of Rsync operations that target a NetStorage storage group. However, the storage group itself is subject to technical limits.

Recommendations when targeting a directory

The nature of Rsync is to scan directories and only update content it finds to be new or updated. This is the benefit of Rsync—you can be lenient on tracking your file content, because there is reduced fear of complete overwrite. We recommend that you be specific in targeting subdirectory trees you know to contain new or changed content. Rsync under-performs if you target a large directory structure, because it continually scans until it finds this content.

📘

The more directories and files Rsync has to scan, the longer an operation will take.

Know your source environment

Be mindful of your local source environment and protect your directory tree configuration. This helps to avoid sudden large target changes as a result of accidental source changes. For example, there could be an accidental configuration change on your end that selects a new, empty source directory. Rsync could inadvertently use it to synchronize content in NetStorage. Since the new source is empty, Rsync will faithfully attempt to delete everything in the storage group to make it conform with this empty directory.

Rsync and smaller quantity transfers

Due to the nature of Rsync and its typical configuration, we recommended you use Rsync to transfer larger quantities of files. Since Rsync must scan the entirety of a directory tree to determine what to transfer, it may not be suited for
the immediate transfer of files. For example, if you are looking to quickly transfer five or six files, you would be better off uploading them via the CMShell system (if SSH security is desired), or an FTP connection.

Using Rsync on the Sun Microsystems Solaris operating system

The Solaris operating system truncates any Rsync password greater than eight (8) characters (i.e., its getpass() function call in Rsync only takes the first eight characters). If your password is longer than this, you can work around the problem by using the --password-file option, or by setting the environment variable, “RSYNC_PASSWORD”.

The “--delete” command and the Force Case feature

🚧

Use this combination of features with caution

Misconfiguration of your system may have undesired results that leads to additional processing of files and additional latency due to geographic replication.

NetStorage’s Force Case functionality only affects individual file lookup and creation, not file enumeration (that is, any call that results in a list.) An end user’s file requests are converted to the desired case, as are uploaded file names.

Force Case will likely behave unexpectedly if used with Rsync's --delete option. This option removes files from the destination directories being synchronized if they are no longer present on the source.

Example unsupported scenario

This example demonstrates an incorrect use of Force Case with Rsync and the --delete option. The symptoms will appear as slow processing of files, excessive processing, origin churn, and geographic replication latency.

1. Force Case: Set to force file names to lowercase.
2. Initial upload of a local file called TextFile.txt.
3. Force Case processes the file and stores it on NetStorage as textfile.txt.
4. When you attempt a new Rsync operation using the --delete option:
   1. Your client notes that it does not have a local file named textfile.txt.
   2. Your client directs NetStorage to delete textfile.txt.
   3. Your client then re-uploads the local file TextFile.txt.
   4. Force Case subsequently renames to textfile.txt.

Two work-around options are currently available:

• Don't use the --delete option. Instead, use Rsync to update any changed files first. Then perform a local scan for deleted files and use CMShell or another mechanism to delete those files from NetStorage.
• Rename all local files to match your Force Case settings. In this case, --delete will work as expected.

Secure Rsync

Use SSH with rsync for secure transfers.

With this method, you define an SSH key pair—a “private” and “public” key set. You apply the public instance of the key to an upload account that has been configured to access the desired storage group. You then tell Rsync which Secure Shell (SSH) client to use so content is encrypted as it traverses the public Internet. The private key is incorporated within commands to resolve with the public instance to grant access to the target storage group. Other requirements apply, and are discussed here.

Before you begin using secure Rsync

You must generate and apply an SSH key for use.

• If you have already applied an SSH key to an applicable upload account, this key can also be used. However, for added security, it is recommended that a unique key be used.
• You must know the upload domain name associated with the target storage group.
• You must have an Rsync client installed on your local system.
• The Rsync tab in the Access Methods section of the NetStorage Groups UI is not used if you wish to use secure Rsync. That functionality only applies to the use of non-secure Rsync.
• If you intend to use an existing SSH key for connection, you will need to know the “passphrase” associated with it, if one has been applied to it. If you don’t know the passphrase, you’ll need to create a new SSH key pair.

Additional requirements for secure Rsync

If you are using secure Rsync, you should also meet the following requirements:

• OpenSSH version 6.7 or later. For multi-threading and higher window sizes. See additional information on the OpenSSH.
• High Performance Network Patches are recommended. These patches are applied to remove a networking bottleneck that exists in the base OpenSSH code. Additional details are available via an FAQ.
• If your client CPU maxing out. This is unlikely if you are using a wide-area network, but if it does happen, you can use the multi-threaded AES-CTR cipher if that meets your security needs.

How secure Rsync commands are issued

Rsync commands are issued via a Terminal Session (Linux/UNIX-based OS) or a terminal session emulator such as Cygwin (Windows OS).
Regardless of the type of command used, various values must be set in the syntax when using secure Rsync:

• Call the Rsync SSH Command Option: The Rsync command option -e ssh must be included in the syntax.
• Call Out the “Private Key” - You will need to include the -i option along with the Rsync SSH command option (above), and include the path to the private key on your local system. Additionally, the entire “ssh” segment must be
  enclosed in quotes: -e "ssh -i /<path>/<Private Key File>".
• Username - When providing the username, the value sshacs must be used. This is the case for all Secure Access Methods in NetStorage.

Secure Rsync usage examples

Here are examples to securely view directory lists, uploading, and downloading files.

Upload a single file with secure Rsync

In this example, a single file is to be uploaded. Here, the <src> variable is included first, to target the file for upload, and the <dest> variable exists as the path in the target storage group.

rsync <option> -e "ssh -i <private key>" <src> sshacs@<Upload Domain Name>:/<CPCode>/<dest>/

rsync -av -e "ssh -i usr/local/scripts/keys/private_1.ppk" /uploads/footage_03-13.mp4 sshacs@baseball.rsync.upload.akamai.com:/123456/baseball/

📘

Ensure that the path included for the destination ends in a trailing slash (“/”). Otherwise the operation will rename the file. In the example above, if the trailing slash was left out, “footage_ 03-13.mp4” would be renamed to “baseball.”

Download a single file with Secure Rsync

In this example, a single file is targeted for download. Here, the <src> variable represents the path to the target file in the storage group, and the <dest> is included last to specify where on the local system the file is to be downloaded.

rsync <option> -e "ssh -i <private key>" sshacs@<Upload Domain Name>:/<CPCode>/<src> <dest>

rsync -av -e "ssh -i usr/local/scripts/keys/private_1.ppk" sshacs@baseball.rsync.upload.akamai.com:/123456/baseball/footage_03-13.mp4 /downloads/baseball/

View a directory contents with Secure Rsync

In this example, a single directory is called out to view its contents. Here, a <src> variable is not included because a specific file isn't being moved. The <dest> variable serves to call out the specific directory to be viewed.

rsync <option> -e "ssh -i /<private key>" sshacs@<Upload Domain Name>:/<CPCode>/<dest>

rsync -av -e "ssh -i usr/local/scripts/keys/ssh/private.ppk" sshacs@baseball.rsync.upload.akamai.com:/123456/baseball/

Secure Rsync connection syntax

Syntax variable definitions and their use in example commands.

Non-secure Rsync

With "non-secure Rsync," you develop a unique password and associate it with an upload account you use to access a specific storage group.

When you issue commands via your Rsync client to this storage group, you include this password value for use in authentication.

In difference to FTP, the password is protected by a challenge-response mechanism. However, content is still sent “in the clear” via the public Internet. It is not encrypted, as is the case with Rsync via SSH.

Before you begin using non-secure Rsync

You should use secure connections when possible, but if there's a need for non-secure connections, here's what you need to know.

• You must generate a unique Rsync in an upload account that has been configured to access the desired storage group.
• You must also know the username ("Id") of the upload account in which you set up the password.
• You must know the Upload domain associated with the target storage group.
• You must have an Rsync client installed on your local system.

Non-secure Rsync (password) and Solaris

The Solaris operating system truncates any Rsync password greater than eight (8) characters. (Its getpass() function call in Rsync only takes the first eight characters). If your password is longer than this, you can work around the problem by using the --password-file option, or by setting the environment variable, “RSYNC_PASSWORD”.

Non-secure Rsync usage examples

Here are examples to non-securely view directory lists, uploading, and downloading files.

Each NetStorage access method has an optimized upload domain. Use <Domain name prefix>.rsync.upload.akamai.com for rsync uploads.

Upload a single file with non-secure Rsync

In this example, a single file is to be uploaded. Here, the <src> variable is included first, to target the file for upload, and the <dest> variable exists as the path in the target storage group.

rsync <option>  <src> <user@upload_domain_name>::<Username>/<CPCode>/<dest>/

rsync -av /uploads/footage_08-17.mp4 editor@sports.rsync.upload.akamai.com::editor/123456/baseball2017/September/

📘

Ensure that the path included for the destination ends in a trailing slash (“/”). Otherwise the operation will rename the file. In the example above, if the trailing slash was left out, “footage_ 08-17.mp4” would be renamed to “September.”

Download a file with non-secure Rsync

In this example, a single file is targeted for download. Here, the <src> variable represents the path to the target file in the storage group, and the <dest> is included last to specify where on the local system the file is to be downloaded.

rsync <option>  <src> <user@upload_domain_name>::<Username>/<CPCode>/<src> <dest>

rsync -av editor@sports.rsync.upload.akamai.com::editor/123456/baseball2017/September/footage_08-17.mp4 C:/Downloads/Baseball/

View a directory listing with non-secure Rsync

In this example, a single directory is called out to view its contents. Here, a <src> variable is not included because a specific file isn't being moved. The <dest> variable serves to call out the specific directory to be viewed.

rsync <option> <user@upload_domain_name>::<Username>/<CPCode>/<dest>

rsync -v editor@sports.rsync.upload.akamai.com::editor/123456/baseball2017/

Non-secure Rsync connection syntax

Syntax variable definitions and their use in example commands.

Supported Rsync command options

This table shows rsync commands supported with NetStorage. Rsync may offer additional options, but they're not supported.

Rsync accepts both long (double-dash + word “--option”) and short (single-dash + letter “-o”) options, but some options only have a long variant, not a short. If an option can be specified in more than one way, the choices are comma-separated. If the option takes a parameter, the parameter is only listed after the long variant, even though it must also be specified for the short. When specifying a parameter, you can either use the form --option=<param> or replace the = with a whitespace.

Support key

Yes. The option is supported.
No. The option is not supported.