Use Rsync
The “Rsync” program is a UNIX system-based tool that allows you to copy content from a local machine to a remote machine—your system to a NetStorage storage group. This tool is particularly useful because it quickly synchronizes content by transferring only the changed files. It does not overwrite standing files you’ve already transferred. This makes it easier to simply transfer what has changed from a local copy, at the expense of extra remote read operations.
NetStorage offers support for two “versions” of Rsync:
- Secure Rsync. With this version, you connect by resolving a Secure Shell (SSH) key pair. You then issue Rsync commands via this connection.
- Non-secure Rsync. With this version, you create a unique password and include it for authentication when issuing Rsync commands.
Rsync-specific guidelines and requirements
Use the same Rsync version as NetStorage
For best results, we recommend that the Rsync client installed the system that will perform the transfers (the “client”) be as close in version as possible to the Rsync server used by NetStorage. This ensures compatibility with supported rsync command options. Older versions can be problematic, and newer ones may offer functionality that is not supported.
NetStorage currently supports version: 3.1.2
Rsync file total limitations
There's no specific limit to the number of Rsync operations that target a NetStorage storage group. However, the storage group itself is subject to technical limits.
Recommendations when targeting a directory
The nature of Rsync is to scan directories and only update content it finds to be new or updated. This is the benefit of Rsync—you can be lenient on tracking your file content, because there is reduced fear of complete overwrite. We recommend that you be specific in targeting subdirectory trees you know to contain new or changed content. Rsync under-performs if you target a large directory structure, because it continually scans until it finds this content.
The more directories and files Rsync has to scan, the longer an operation will take.
Know your source environment
Be mindful of your local source environment and protect your directory tree configuration. This helps to avoid sudden large target changes as a result of accidental source changes. For example, there could be an accidental configuration change on your end that selects a new, empty source directory. Rsync could inadvertently use it to synchronize content in NetStorage. Since the new source is empty, Rsync will faithfully attempt to delete everything in the storage group to make it conform with this empty directory.
Rsync and smaller quantity transfers
Due to the nature of Rsync and its typical configuration, we recommended you use Rsync to transfer larger quantities of files. Since Rsync must scan the entirety of a directory tree to determine what to transfer, it may not be suited for
the immediate transfer of files. For example, if you are looking to quickly transfer five or six files, you would be better off uploading them via the CMShell system (if SSH security is desired), or an FTP connection.
Using Rsync on the Sun Microsystems Solaris operating system
The Solaris operating system truncates any Rsync password greater than eight (8) characters (i.e., its getpass()
function call in Rsync only takes the first eight characters). If your password is longer than this, you can work around the problem by using the --password-file
option, or by setting the environment variable, “RSYNC_PASSWORD”
.
The “--delete” command and the Force Case feature
Use this combination of features with caution
Misconfiguration of your system may have undesired results that leads to additional processing of files and additional latency due to geographic replication.
NetStorage’s Force Case functionality only affects individual file lookup and creation, not file enumeration (that is, any call that results in a list.) An end user’s file requests are converted to the desired case, as are uploaded file names.
Force Case will likely behave unexpectedly if used with Rsync's --delete
option. This option removes files from the destination directories being synchronized if they are no longer present on the source.
Example unsupported scenario
This example demonstrates an incorrect use of Force Case with Rsync and the --delete
option. The symptoms will appear as slow processing of files, excessive processing, origin churn, and geographic replication latency.
- Force Case: Set to force file names to lowercase.
- Initial upload of a local file called TextFile.txt.
- Force Case processes the file and stores it on NetStorage as textfile.txt.
- When you attempt a new Rsync operation using the
--delete
option:- Your client notes that it does not have a local file named textfile.txt.
- Your client directs NetStorage to delete textfile.txt.
- Your client then re-uploads the local file TextFile.txt.
- Force Case subsequently renames to textfile.txt.
Two work-around options are currently available:
- Don't use the
--delete
option. Instead, use Rsync to update any changed files first. Then perform a local scan for deleted files and use CMShell or another mechanism to delete those files from NetStorage. - Rename all local files to match your Force Case settings. In this case,
--delete
will work as expected.
Secure Rsync
Use SSH with rsync for secure transfers.
With this method, you define an SSH key pair—a “private” and “public” key set. You apply the public instance of the key to an upload account that has been configured to access the desired storage group. You then tell Rsync which Secure Shell (SSH) client to use so content is encrypted as it traverses the public Internet. The private key is incorporated within commands to resolve with the public instance to grant access to the target storage group. Other requirements apply, and are discussed here.
Before you begin using secure Rsync
You must generate and apply an SSH key for use.
- If you have already applied an SSH key to an applicable upload account, this key can also be used. However, for added security, it is recommended that a unique key be used.
- You must know the upload domain name associated with the target storage group.
- You must have an Rsync client installed on your local system.
- The Rsync tab in the Access Methods section of the NetStorage Groups UI is not used if you wish to use secure Rsync. That functionality only applies to the use of non-secure Rsync.
- If you intend to use an existing SSH key for connection, you will need to know the “passphrase” associated with it, if one has been applied to it. If you don’t know the passphrase, you’ll need to create a new SSH key pair.
Additional requirements for secure Rsync
If you are using secure Rsync, you should also meet the following requirements:
- OpenSSH version 6.7 or later. For multi-threading and higher window sizes. See additional information on the OpenSSH.
- High Performance Network Patches are recommended. These patches are applied to remove a networking bottleneck that exists in the base OpenSSH code. Additional details are available via an FAQ.
- If your client CPU maxing out. This is unlikely if you are using a wide-area network, but if it does happen, you can use the multi-threaded AES-CTR cipher if that meets your security needs.
How secure Rsync commands are issued
Rsync commands are issued via a Terminal Session (Linux/UNIX-based OS) or a terminal session emulator such as Cygwin (Windows OS).
Regardless of the type of command used, various values must be set in the syntax when using secure Rsync:
- Call the Rsync
SSH
Command Option: The Rsync command option-e ssh
must be included in the syntax. - Call Out the “Private Key” - You will need to include the
-i
option along with the Rsync SSH command option (above), and include the path to the private key on your local system. Additionally, the entire “ssh” segment must be
enclosed in quotes:-e "ssh -i /<path>/<Private Key File>"
. - Username - When providing the username, the value
sshacs
must be used. This is the case for all Secure Access Methods in NetStorage.
Secure Rsync usage examples
Here are examples to securely view directory lists, uploading, and downloading files.
Upload a single file with secure Rsync
In this example, a single file is to be uploaded. Here, the <src>
variable is included first, to target the file for upload, and the <dest>
variable exists as the path in the target storage group.
rsync <option> -e "ssh -i <private key>" <src> sshacs@<Upload Domain Name>:/<CPCode>/<dest>/
rsync -av -e "ssh -i usr/local/scripts/keys/private_1.ppk" /uploads/footage_03-13.mp4 sshacs@baseball.rsync.upload.akamai.com:/123456/baseball/
Ensure that the path included for the destination ends in a trailing slash (“/”). Otherwise the operation will rename the file. In the example above, if the trailing slash was left out, “footage_ 03-13.mp4” would be renamed to “baseball.”
Download a single file with Secure Rsync
In this example, a single file is targeted for download. Here, the <src>
variable represents the path to the target file in the storage group, and the <dest>
is included last to specify where on the local system the file is to be downloaded.
rsync <option> -e "ssh -i <private key>" sshacs@<Upload Domain Name>:/<CPCode>/<src> <dest>
rsync -av -e "ssh -i usr/local/scripts/keys/private_1.ppk" sshacs@baseball.rsync.upload.akamai.com:/123456/baseball/footage_03-13.mp4 /downloads/baseball/
View a directory contents with Secure Rsync
In this example, a single directory is called out to view its contents. Here, a <src>
variable is not included because a specific file isn't being moved. The <dest>
variable serves to call out the specific directory to be viewed.
rsync <option> -e "ssh -i /<private key>" sshacs@<Upload Domain Name>:/<CPCode>/<dest>
rsync -av -e "ssh -i usr/local/scripts/keys/ssh/private.ppk" sshacs@baseball.rsync.upload.akamai.com:/123456/baseball/
Secure Rsync connection syntax
Syntax variable definitions and their use in example commands.
Variable | Description |
---|---|
... | This indicates that multiple instances of the variable can be used. For example, <option>... indicates that multiple options can be applied). |
<option> | This indicates that you can specify at least one command option. |
<private key> (optional) | This represents the complete path to the private instance of the SSH key on your local system (including its extension, if applicable). After execution of the command, this is resolved with the public instance of the SSH key you have applied to the applicable upload account for access. |
sshacs@<Upload Domain Name> | The required secure access username is sshacs , followed by your domain name prefix and upload domain.For example, if your domain name prefix is |
<src> | This is the complete path to one or more source files to be targeted. This value can apply to content on your local system (to be uploaded), or content in your NetStorage storage group. (For example, to target it for download or view stats). |
<CP Code> | This is a Content Provider (CP) code that has been provisioned for use with your instance of NetStorage, and has been configured as the root directory in the storage group. |
<dest> | This is the complete path to the desired destination for the operation. This value can apply to a directory in your NetStorage storage group (a destination for an upload), or to one on your local system (for a download). |
Non-secure Rsync
With "non-secure Rsync," you develop a unique password and associate it with an upload account you use to access a specific storage group.
When you issue commands via your Rsync client to this storage group, you include this password value for use in authentication.
In difference to FTP, the password is protected by a challenge-response mechanism. However, content is still sent “in the clear” via the public Internet. It is not encrypted, as is the case with Rsync via SSH.
Before you begin using non-secure Rsync
You should use secure connections when possible, but if there's a need for non-secure connections, here's what you need to know.
- You must generate a unique Rsync in an upload account that has been configured to access the desired storage group.
- You must also know the username ("Id") of the upload account in which you set up the password.
- You must know the Upload domain associated with the target storage group.
- You must have an Rsync client installed on your local system.
Non-secure Rsync (password) and Solaris
The Solaris operating system truncates any Rsync password greater than eight (8) characters. (Its getpass()
function call in Rsync only takes the first eight characters). If your password is longer than this, you can work around the problem by using the --password-file
option, or by setting the environment variable, “RSYNC_PASSWORD”.
Non-secure Rsync usage examples
Here are examples to non-securely view directory lists, uploading, and downloading files.
Each NetStorage access method has an optimized upload domain. Use <Domain name prefix>.rsync.upload.akamai.com
for rsync uploads.
Upload a single file with non-secure Rsync
In this example, a single file is to be uploaded. Here, the <src>
variable is included first, to target the file for upload, and the <dest>
variable exists as the path in the target storage group.
rsync <option> <src> <user@upload_domain_name>::<Username>/<CPCode>/<dest>/
rsync -av /uploads/footage_08-17.mp4 editor@sports.rsync.upload.akamai.com::editor/123456/baseball2017/September/
Ensure that the path included for the destination ends in a trailing slash (“/”). Otherwise the operation will rename the file. In the example above, if the trailing slash was left out, “footage_ 08-17.mp4” would be renamed to “September.”
Download a file with non-secure Rsync
In this example, a single file is targeted for download. Here, the <src>
variable represents the path to the target file in the storage group, and the <dest>
is included last to specify where on the local system the file is to be downloaded.
rsync <option> <src> <user@upload_domain_name>::<Username>/<CPCode>/<src> <dest>
rsync -av editor@sports.rsync.upload.akamai.com::editor/123456/baseball2017/September/footage_08-17.mp4 C:/Downloads/Baseball/
View a directory listing with non-secure Rsync
In this example, a single directory is called out to view its contents. Here, a <src>
variable is not included because a specific file isn't being moved. The <dest>
variable serves to call out the specific directory to be viewed.
rsync <option> <user@upload_domain_name>::<Username>/<CPCode>/<dest>
rsync -v editor@sports.rsync.upload.akamai.com::editor/123456/baseball2017/
Non-secure Rsync connection syntax
Syntax variable definitions and their use in example commands.
Variable | Description |
---|---|
... | This indicates that multiple instances of the variable can be used. For example, <option>... indicates that multiple options can be applied). |
<option> | This indicates that you can specify at least one command option. |
<user@upload_domain_name> | This is the Username ("Id") of the upload account that has access to the desired storage group (and in which you have set up an Rsync password), followed by the Upload Domain Name for that storage group. For example, if your upload account Id was “editor” and your Domain Name prefix was “sports,” you would look up the correct Upload Domain Name for your storage group, this value would be:editor@sports.rsync.upload.akamai.com |
::<Username> | This is a duplicate instance of the upload account Id. Using the example above, you would input editor for this value. This is always preceded by two colons (“:: ”) as its separator. |
<CP Code> | This is a Content provider (CP) code that has been provisioned for use with your instance of NetStorage, and has been configured as the root directory in the storage group. |
<port> | The port number to be used (220). This value is typically left out to use the NetStorage default port. |
<dest> | This is the complete path to the desired destination for the operation. This value can apply to a directory in your NetStorage storage group (a destination for an upload), or to one on your local system (for a download). |
Supported Rsync command options
This table shows rsync commands supported with NetStorage. Rsync may offer additional options, but they're not supported.
Rsync accepts both long (double-dash + word “--option”) and short (single-dash + letter “-o”) options, but some options only have a long variant, not a short. If an option can be specified in more than one way, the choices are comma-separated. If the option takes a parameter, the parameter is only listed after the long variant, even though it must also be specified for the short. When specifying a parameter, you can either use the form --option=<param>
or replace the =
with a whitespace.
Support key
Yes. The option is supported.
No. The option is not supported.
Option | Description | Support |
---|---|---|
-v, --verbose | increase verbosity | Yes |
--info=FLAGS | fine-grained informational verbosity | Yes |
--debug=FLAGS | fine-grained debug verbosity | Yes |
--msgs2stderr | special output handling for debugging | Yes |
-q, --quiet | suppress non-error messages | Yes |
--no-motd | suppress daemon-mode MOTD (see manpage caveat) | Yes |
-c, --checksum | skip based on checksum, not mod-time & size | Yes |
-a, --archive | archive mode; equals -rlptgoD (no support for -H,,-A,,-X) | Yes |
--no-OPTION | turn off an implied OPTION (e.g. --no-D) | Yes |
-r, --recursive | recurse into directories | Yes |
-R, --relative | use relative path names | Yes |
--no-relative | Used to cancel an implied --relative | No |
--no-implied-dirs | don't send implied dirs with --relative | Yes |
-b, --backup | make backups (see --suffix & --backup-dir) | Yes |
--backup-dir=DIR | make backups into hierarchy based in DIR | Yes |
--suffix=SUFFIX | set backup suffix (default %s w/o --backup-dir) | Yes |
-u, --update | skip files that are newer on the receiver | Yes |
-d, --dirs | transfer directories without recursing | Yes |
-l, --links | copy symlinks as symlinks | Yes |
-L, --copy-links | transform symlink into referent file/dir | Yes |
--copy-unsafe-links | only "unsafe" symlinks are transformed | Yes |
--safe-links | ignore symlinks that point outside the source tree | Yes |
-k, --copy-dirlinks | transform symlink to a dir into referent dir | Yes |
-K, --keep-dirlinks | treat symlinked dir on receiver as dir | Yes |
-p, --perms | preserve permissions | No |
-E, --executability | preserve the file's executability | Yes |
-o, --owner | preserve owner (super-user only) | No |
-t, --times | preserve modification times | Yes |
-O, --omit-dir-times | omit directories from --times | Yes |
-J, --omit-link-times | omit symlinks from --times | Yes |
-S, --sparse | handle sparse files efficiently | No |
-n, --dry-run | perform a trial run with no changes made | Yes |
-W, --whole-file | copy files whole (without delta-xfer algorithm) | Yes |
--no-whole-file | always use incremental Rsync algorithm | No |
-x, --one-file-system | don't cross filesystem boundaries | Yes |
-B, --block-size=SIZE | force a fixed checksum block-size | Yes |
-e, --rsh=COMMAND | specify the remote shell to use | Yes |
--rsync-path=PROGRAM | specify the Rsync to run on the remote machine | No |
--existing | skip creating new files on receiver | Yes |
--ignore-existing | skip updating files that already exist on receiver | Yes |
--remove-sent-files | sender removes files that are sent (non-dirs) | No |
--remove-source-files | sender removes synchronized files if they exist or are copied to remote end (non-dirs) | Yes |
--del | an alias for --delete-during | Yes |
--delete | delete extraneous files from destination dirs | Yes |
--delete-before | receiver deletes before transfer, not during | Yes |
--delete-during | receiver deletes during the transfer | Yes |
--delete-delay | find deletions during, delete after | Yes |
--delete-after | receiver deletes after transfer, not during | Yes |
--delete-excluded | also delete excluded files from destination dirs | Yes |
--ignore-missing-args | ignore missing source args without error | Yes |
--delete-missing-args | delete missing source args from destination | Yes |
--ignore-errors | delete even if there are I/O errors | Yes |
--force | force deletion of directories even if not empty | Yes |
--max-delete=NUM | don't delete more than NUM files | Yes |
--max-size=SIZE | don't transfer any file larger than SIZE | Yes |
--min-size=SIZE | don't transfer any file smaller than SIZE | Yes |
-m, --prune-empty-dirs | prune empty directory chains from the file-list | Yes |
--timeout=SECONDS | set I/O timeout in seconds | Yes |
--contimeout=SECONDS | set daemon connection timeout in seconds | Yes |
-I, --ignore-times | don't skip files that match in size and mod-time | Yes |
--size-only | skip files that match in size | Yes |
-T, --temp-dir=DIR | create temporary files in directory DIR | Yes |
-y, --fuzzy | find similar file for basis if no dest file | Yes |
--compare-dest=DIR | also compare destination files relative to DIR | Yes |
--copy-dest=DIR | ... and include copies of unchanged files | Yes |
--link-dest=DIR | hardlink to files in DIR when unchanged | Yes |
-z, --compress | compress file data during the transfer | Yes |
--compress-level=NUM | explicitly set compression level | Yes |
--skip-compress=LIST | skip compressing files with a suffix in LIST | Yes |
-C, --cvs-exclude | auto-ignore files the same way CVS does | Yes |
-f, --filter=RULE | add a file-filtering RULE | Yes |
-F | same as --filter='dir-merge /.rsync-filter' | Yes |
--exclude=PATTERN | exclude files matching PATTERN | Yes |
--exclude-from=FILE | read exclude patterns from FILE | Yes |
--include=PATTERN | don't exclude files matching PATTERN | Yes |
--include-from=FILE | read include patterns from FILE | Yes |
--files-from=FILE | read list of source-file names from FILE | Yes |
-0, --from0 | all *-from/filter files are delimited by 0s | Yes |
--address=ADDRESS | bind address for outgoing socket to daemon | No |
--port=PORT | specify double-colon alternate port number | No |
--sockopts=OPTIONS | specify custom TCP options | Yes |
--blocking-io | use blocking I/O for the remote shell | No |
--outbuf=N|L|B | set output buffering to None, Line, or Block | Yes |
--stats | give some file-transfer stats | Yes |
-8, --8-bit-output | leave high-bit chars unescaped in output | Yes |
-h, --human-readable | output numbers in a human-readable format | Yes |
--progress | show progress during transfer | Yes |
-i, --itemize-changes | output a change-summary for all updates | Yes |
-M, --remote-option=OPTION | send OPTION to the remote side only | Yes |
--out-format=FORMAT | output updates using the specified FORMAT | Yes |
--log-file=FILE | log what we're doing to the specified FILE | Yes |
--log-file-format=FMT | log updates using the specified FMT | Yes |
--password-file=FILE | read daemon-access password from FILE | Yes |
--list-only | list the files instead of copying them | Yes |
--bwlimit=RATE | limit socket I/O bandwidth | Yes |
--write-batch=FILE | write a batched update to FILE | Yes |
--only-write-batch=FILE | like --write-batch but w/o updating destination | Yes |
--read-batch=FILE | read a batched update from FILE | Yes |
--protocol=NUM | force an older protocol version to be used | Yes |
--checksum-seed=NUM | set block/file checksum seed (advanced) | Yes |
--version | print version number | Yes |
(-h) --help | show this help (-h is --help only if used alone) | Yes |
Updated over 1 year ago