Use Rsync

The “Rsync” program is a UNIX system-based tool that allows you to copy content from a local machine to a remote machine—your system to a NetStorage storage group. This tool is particularly useful because it quickly synchronizes content by transferring only the changed files. It does not overwrite standing files you’ve already transferred. This makes it easier to simply transfer what has changed from a local copy, at the expense of extra remote read operations.

NetStorage offers support for two “versions” of Rsync:

  • Secure Rsync. With this version, you connect by resolving a Secure Shell (SSH) key pair. You then issue Rsync commands via this connection.
  • Non-secure Rsync. With this version, you create a unique password and include it for authentication when issuing Rsync commands.

Rsync-specific guidelines and requirements

Use the same Rsync version as NetStorage

For best results, we recommend that the Rsync client installed the system that will perform the transfers (the “client”) be as close in version as possible to the Rsync server used by NetStorage. This ensures compatibility with supported rsync command options. Older versions can be problematic, and newer ones may offer functionality that is not supported.

NetStorage currently supports version: 3.1.2

Rsync file total limitations

There's no specific limit to the number of Rsync operations that target a NetStorage storage group. However, the storage group itself is subject to technical limits.

Recommendations when targeting a directory

The nature of Rsync is to scan directories and only update content it finds to be new or updated. This is the benefit of Rsync—you can be lenient on tracking your file content, because there is reduced fear of complete overwrite. We recommend that you be specific in targeting subdirectory trees you know to contain new or changed content. Rsync under-performs if you target a large directory structure, because it continually scans until it finds this content.

📘

The more directories and files Rsync has to scan, the longer an operation will take.

Know your source environment

Be mindful of your local source environment and protect your directory tree configuration. This helps to avoid sudden large target changes as a result of accidental source changes. For example, there could be an accidental configuration change on your end that selects a new, empty source directory. Rsync could inadvertently use it to synchronize content in NetStorage. Since the new source is empty, Rsync will faithfully attempt to delete everything in the storage group to make it conform with this empty directory.

Rsync and smaller quantity transfers

Due to the nature of Rsync and its typical configuration, we recommended you use Rsync to transfer larger quantities of files. Since Rsync must scan the entirety of a directory tree to determine what to transfer, it may not be suited for
the immediate transfer of files. For example, if you are looking to quickly transfer five or six files, you would be better off uploading them via the CMShell system (if SSH security is desired), or an FTP connection.

Using Rsync on the Sun Microsystems Solaris operating system

The Solaris operating system truncates any Rsync password greater than eight (8) characters (i.e., its getpass() function call in Rsync only takes the first eight characters). If your password is longer than this, you can work around the problem by using the --password-file option, or by setting the environment variable, “RSYNC_PASSWORD”.

The “--delete” command and the force case feature

Use this combination of features with caution. Misconfiguration of your system may have undesired results.

NetStorage’s Force Case functionality only affects individual file lookup and creation, not file enumeration (that is, any call that results in a list.) An end user’s file requests are converted to the desired case, as are uploaded file names. However, Force Case will likely behave unexpectedly if used with Rsync's --delete option. This option removes files from the destination directories being synchronized if they are no longer present on the source.

For example, if you enable Force Case to force file names to lowercase, your initial upload of a local file called TextFile.txt creates a file called textfile.txt at the destination. When you attempt a new Rsync operation using the --delete option, your client notes that it does not have a local file named textfile.txt and directs the server to delete it at the destination. Your client then uploads the local file TextFile.txt, which the server subsequently renames to textfile.txt.

Two work-around options are currently available:

  • Don't use the --delete option. Instead, use Rsync to update any changed files first. Then perform a local scan for deleted files and use CMShell or another mechanism to delete those files from NetStorage.
  • Rename all local files to match your Force Case settings. In this case, --delete will work as expected.

Secure Rsync

Use SSH with rsync for secure transfers.

With this method, you define an SSH key pair—a “private” and “public” key set. You apply the public instance of the key to an upload account that has been configured to access the desired storage group. You then tell Rsync which Secure Shell (SSH) client to use so content is encrypted as it traverses the public Internet. The private key is incorporated within commands to resolve with the public instance to grant access to the target storage group. Other requirements apply, and are discussed here.

Before you begin using secure Rsync

You must generate and apply an SSH key for use.

  • If you have already applied an SSH key to an applicable upload account, this key can also be used. However, for added security, it is recommended that a unique key be used.
  • You must know the upload domain name associated with the target storage group.
  • You must have an Rsync client installed on your local system.
  • The Rsync tab in the Access Methods section of the NetStorage Groups UI is not used if you wish to use secure Rsync. That functionality only applies to the use of non-secure Rsync.
  • If you intend to use an existing SSH key for connection, you will need to know the “passphrase” associated with it, if one has been applied to it. If you don’t know the passphrase, you’ll need to create a new SSH key pair.

Additional requirements for secure Rsync

If you are using secure Rsync, you should also meet the following requirements:

  • OpenSSH version 6.7 or later. For multi-threading and higher window sizes. See additional information on the OpenSSH.
  • High Performance Network Patches are recommended. These patches are applied to remove a networking bottleneck that exists in the base OpenSSH code. Additional details are available via an FAQ.
  • If your client CPU maxing out. This is unlikely if you are using a wide-area network, but if it does happen, you can use the multi-threaded AES-CTR cipher if that meets your security needs.

How secure Rsync commands are issued

Rsync commands are issued via a Terminal Session (Linux/UNIX-based OS) or a terminal session emulator such as Cygwin (Windows OS).
Regardless of the type of command used, various values must be set in the syntax when using secure Rsync:

  • Call the Rsync SSH Command Option: The Rsync command option -e ssh must be included in the syntax.
  • Call Out the “Private Key” - You will need to include the -i option along with the Rsync SSH command option (above), and include the path to the private key on your local system. Additionally, the entire “ssh” segment must be
    enclosed in quotes: -e "ssh -i /<path>/<Private Key File>".
  • Username - When providing the username, the value sshacs must be used. This is the case for all Secure Access Methods in NetStorage.

Secure Rsync usage examples

Here are examples to securely view directory lists, uploading, and downloading files.

Upload a single file with secure Rsync

In this example, a single file is to be uploaded. Here, the <src> variable is included first, to target the file for upload, and the <dest> variable exists as the path in the target storage group.

rsync <option> -e "ssh -i <private key>" <src> sshacs@<Upload Domain Name>:/<CPCode>/<dest>/
rsync -av -e "ssh -i usr/local/scripts/keys/private_1.ppk" /uploads/footage_03-13.mp4 sshacs@baseball.rsync.upload.akamai.com:/123456/baseball/

📘

Ensure that the path included for the destination ends in a trailing slash (“/”). Otherwise the operation will rename the file. In the example above, if the trailing slash was left out, “footage_ 03-13.mp4” would be renamed to “baseball.”

Download a single file with Secure Rsync

In this example, a single file is targeted for download. Here, the <src> variable represents the path to the target file in the storage group, and the <dest> is included last to specify where on the local system the file is to be downloaded.

rsync <option> -e "ssh -i <private key>" sshacs@<Upload Domain Name>:/<CPCode>/<src> <dest>
rsync -av -e "ssh -i usr/local/scripts/keys/private_1.ppk" sshacs@baseball.rsync.upload.akamai.com:/123456/baseball/footage_03-13.mp4 /downloads/baseball/

View a directory contents with Secure Rsync

In this example, a single directory is called out to view its contents. Here, a <src> variable is not included because a specific file isn't being moved. The <dest> variable serves to call out the specific directory to be viewed.

rsync <option> -e "ssh -i /<private key>" sshacs@<Upload Domain Name>:/<CPCode>/<dest>
rsync -av -e "ssh -i usr/local/scripts/keys/ssh/private.ppk" sshacs@baseball.rsync.upload.akamai.com:/123456/baseball/

Secure Rsync connection syntax

Syntax variable definitions and their use in example commands.

VariableDescription
...This indicates that multiple instances of the variable can be used. For example, <option>... indicates that multiple options can be applied).
<option>This indicates that you can specify at least one command option.
<private key> (optional)This represents the complete path to the private instance of the SSH key on your local system (including its extension, if applicable). After execution of the command, this is resolved with the public instance of the SSH key you have applied to the applicable upload account for access.
sshacs@<Upload Domain Name>The required secure access username is sshacs, followed by your domain name prefix and upload domain.

For example, if your domain name prefix is baseball, you would look up the correct upload domain name for your storage group, this value would be: .rsync.upload.akamai.com. Combine all of the values to create the full connection string: sshacs@baseball.rsync.upload.akamai.com

<src>This is the complete path to one or more source files to be targeted. This value can apply to content on your local system (to be uploaded), or content in your NetStorage storage group. (For example, to target it for download or view stats).
<CP Code>This is a Content Provider (CP) code that has been provisioned for use with your instance of NetStorage, and has been configured as the root directory in the storage group.
<dest>This is the complete path to the desired destination for the operation. This value can apply to a directory in your NetStorage storage group (a destination for an upload), or to one on your local system (for a download).

Non-secure Rsync

With "non-secure Rsync," you develop a unique password and associate it with an upload account you use to access a specific storage group.

When you issue commands via your Rsync client to this storage group, you include this password value for use in authentication.

In difference to FTP, the password is protected by a challenge-response mechanism. However, content is still sent “in the clear” via the public Internet. It is not encrypted, as is the case with Rsync via SSH.

Before you begin using non-secure Rsync

You should use secure connections when possible, but if there's a need for non-secure connections, here's what you need to know.

  • You must generate a unique Rsync in an upload account that has been configured to access the desired storage group.
  • You must also know the username ("Id") of the upload account in which you set up the password.
  • You must know the Upload domain associated with the target storage group.
  • You must have an Rsync client installed on your local system.

Non-secure Rsync (password) and Solaris

The Solaris operating system truncates any Rsync password greater than eight (8) characters. (Its getpass() function call in Rsync only takes the first eight characters). If your password is longer than this, you can work around the problem by using the --password-file option, or by setting the environment variable, “RSYNC_PASSWORD”.


Non-secure Rsync usage examples

Here are examples to non-securely view directory lists, uploading, and downloading files.

Each NetStorage access method has an optimized upload domain. Use <Domain name prefix>.rsync.upload.akamai.com for rsync uploads.

Upload a single file with non-secure Rsync

In this example, a single file is to be uploaded. Here, the <src> variable is included first, to target the file for upload, and the <dest> variable exists as the path in the target storage group.

rsync <option>  <src> <user@upload_domain_name>::<Username>/<CPCode>/<dest>/
rsync -av /uploads/footage_08-17.mp4 editor@sports.rsync.upload.akamai.com::editor/123456/baseball2017/September/

📘

Ensure that the path included for the destination ends in a trailing slash (“/”). Otherwise the operation will rename the file. In the example above, if the trailing slash was left out, “footage_ 08-17.mp4” would be renamed to “September.”

Download a file with non-secure Rsync

In this example, a single file is targeted for download. Here, the <src> variable represents the path to the target file in the storage group, and the <dest> is included last to specify where on the local system the file is to be downloaded.

rsync <option>  <src> <user@upload_domain_name>::<Username>/<CPCode>/<src> <dest>
rsync -av editor@sports.rsync.upload.akamai.com::editor/123456/baseball2017/September/footage_08-17.mp4 C:/Downloads/Baseball/

View a directory listing with Secure Rsync

In this example, a single directory is called out to view its contents. Here, a <src> variable is not included because a specific file isn't being moved. The <dest> variable serves to call out the specific directory to be viewed.

rsync <option> <user@upload_domain_name>::<Username>/<CPCode>/<dest>
rsync -v editor@sports.rsync.upload.akamai.com::editor/123456/baseball2017/

Non-secure Rsync connection syntax

Syntax variable definitions and their use in example commands.

VariableDescription
...This indicates that multiple instances of the variable can be used. For example, <option>... indicates that multiple options can be applied).
<option>This indicates that you can specify at least one command option.
<user@upload_domain_name>This is the Username ("Id") of the upload account that has access to the desired storage group (and in which you have set up an Rsync password), followed by the Upload Domain Name for that storage group. For example, if your upload account Id was “editor” and your Domain Name prefix was “sports,” you would look up the correct Upload Domain Name for your storage group, this value would be:editor@sports.rsync.upload.akamai.com

::<Username>This is a duplicate instance of the upload account Id. Using the example above, you would input editor for this value. This is always preceded by two colons (“::”) as its separator.
<CP Code>This is a Content provider (CP) code that has been provisioned for use with your instance of NetStorage, and has been configured as the root directory in the storage group.
<port>The port number to be used (220). This value is typically left out to use the NetStorage default port.
<dest>This is the complete path to the desired destination for the operation. This value can apply to a directory in your NetStorage storage group (a destination for an upload), or to one on your local system (for a download).

Supported Rsync command options

This table shows rsync commands supported with NetStorage. Rsync may offer additional options, but they're not supported.

Rsync accepts both long (double-dash + word “--option”) and short (single-dash + letter “-o”) options, but some options only have a long variant, not a short. If an option can be specified in more than one way, the choices are comma-separated. If the option takes a parameter, the parameter is only listed after the long variant, even though it must also be specified for the short. When specifying a parameter, you can either use the form --option=<param> or replace the = with a whitespace.

Support key

Yes. The option is supported.
No. The option is not supported.

Option DescriptionSupport
-v, --verboseincrease verbosityYes
--info=FLAGSfine-grained informational verbosityYes
--debug=FLAGSfine-grained debug verbosityYes
--msgs2stderrspecial output handling for debuggingYes
-q, --quietsuppress non-error messagesYes
--no-motdsuppress daemon-mode MOTD (see manpage caveat)Yes
-c, --checksumskip based on checksum, not mod-time &amp; sizeYes
-a, --archivearchive mode; equals -rlptgoD (no support for -H,,-A,,-X)Yes
--no-OPTIONturn off an implied OPTION (e.g. --no-D)Yes
-r, --recursiverecurse into directoriesYes
-R, --relativeuse relative path namesYes
--no-relativeUsed to cancel an implied --relativeNo
--no-implied-dirsdon't send implied dirs with --relativeYes
-b, --backupmake backups (see --suffix &amp; --backup-dir)Yes
--backup-dir=DIRmake backups into hierarchy based in DIRYes
--suffix=SUFFIXset backup suffix (default %s w/o --backup-dir)Yes
-u, --updateskip files that are newer on the receiverYes
-d, --dirstransfer directories without recursingYes
-l, --linkscopy symlinks as symlinksYes
-L, --copy-linkstransform symlink into referent file/dirYes
--copy-unsafe-linksonly "unsafe" symlinks are transformedYes
--safe-linksignore symlinks that point outside the source treeYes
-k, --copy-dirlinkstransform symlink to a dir into referent dirYes
-K, --keep-dirlinkstreat symlinked dir on receiver as dirYes
-p, --permspreserve permissionsNo
-E, --executabilitypreserve the file's executabilityYes
-o, --ownerpreserve owner (super-user only)No
-t, --timespreserve modification timesYes
-O, --omit-dir-timesomit directories from --timesYes
-J, --omit-link-timesomit symlinks from --timesYes
-S, --sparsehandle sparse files efficientlyNo
-n, --dry-runperform a trial run with no changes madeYes
-W, --whole-filecopy files whole (without delta-xfer algorithm)Yes
--no-whole-filealways use incremental Rsync algorithmNo
-x, --one-file-systemdon't cross filesystem boundariesYes
-B, --block-size=SIZEforce a fixed checksum block-sizeYes
-e, --rsh=COMMANDspecify the remote shell to useYes
--rsync-path=PROGRAMspecify the Rsync to run on the remote machineNo
--existingskip creating new files on receiverYes
--ignore-existingskip updating files that already exist on receiverYes
--remove-sent-filessender removes files that are sent (non-dirs)No
--remove-source-filessender removes synchronized files if they exist or are copied to remote end (non-dirs)Yes
--delan alias for --delete-duringYes
--deletedelete extraneous files from destination dirsYes
--delete-beforereceiver deletes before transfer, not duringYes
--delete-duringreceiver deletes during the transferYes
--delete-delayfind deletions during, delete afterYes
--delete-afterreceiver deletes after transfer, not duringYes
--delete-excludedalso delete excluded files from destination dirsYes
--ignore-missing-argsignore missing source args without errorYes
--delete-missing-argsdelete missing source args from destinationYes
--ignore-errorsdelete even if there are I/O errorsYes
--forceforce deletion of directories even if not emptyYes
--max-delete=NUMdon't delete more than NUM filesYes
--max-size=SIZEdon't transfer any file larger than SIZEYes
--min-size=SIZEdon't transfer any file smaller than SIZEYes
-m, --prune-empty-dirsprune empty directory chains from the file-listYes
--timeout=SECONDSset I/O timeout in secondsYes
--contimeout=SECONDSset daemon connection timeout in secondsYes
-I, --ignore-timesdon't skip files that match in size and mod-timeYes
--size-onlyskip files that match in sizeYes
-T, --temp-dir=DIRcreate temporary files in directory DIRYes
-y, --fuzzyfind similar file for basis if no dest fileYes
--compare-dest=DIRalso compare destination files relative to DIRYes
--copy-dest=DIR... and include copies of unchanged filesYes
--link-dest=DIRhardlink to files in DIR when unchangedYes
-z, --compresscompress file data during the transferYes
--compress-level=NUMexplicitly set compression levelYes
--skip-compress=LISTskip compressing files with a suffix in LISTYes
-C, --cvs-excludeauto-ignore files the same way CVS doesYes
-f, --filter=RULEadd a file-filtering RULEYes
-Fsame as --filter='dir-merge /.rsync-filter'Yes
--exclude=PATTERNexclude files matching PATTERNYes
--exclude-from=FILEread exclude patterns from FILEYes
--include=PATTERNdon't exclude files matching PATTERNYes
--include-from=FILEread include patterns from FILEYes
--files-from=FILEread list of source-file names from FILEYes
-0, --from0all *-from/filter files are delimited by 0sYes
--address=ADDRESSbind address for outgoing socket to daemonNo
--port=PORTspecify double-colon alternate port numberNo
--sockopts=OPTIONSspecify custom TCP optionsYes
--blocking-iouse blocking I/O for the remote shellNo
--outbuf=N|L|Bset output buffering to None, Line, or BlockYes
--statsgive some file-transfer statsYes
-8, --8-bit-outputleave high-bit chars unescaped in outputYes
-h, --human-readableoutput numbers in a human-readable formatYes
--progressshow progress during transferYes
-i, --itemize-changesoutput a change-summary for all updatesYes
-M, --remote-option=OPTIONsend OPTION to the remote side onlyYes
--out-format=FORMAToutput updates using the specified FORMATYes
--log-file=FILElog what we're doing to the specified FILEYes
--log-file-format=FMTlog updates using the specified FMTYes
--password-file=FILEread daemon-access password from FILEYes
--list-onlylist the files instead of copying themYes
--bwlimit=RATElimit socket I/O bandwidthYes
--write-batch=FILEwrite a batched update to FILEYes
--only-write-batch=FILElike --write-batch but w/o updating destinationYes
--read-batch=FILEread a batched update from FILEYes
--protocol=NUMforce an older protocol version to be usedYes
--checksum-seed=NUMset block/file checksum seed (advanced)Yes
--versionprint version numberYes
(-h) --helpshow this help (-h is --help only if used alone)Yes

Did this page help you?