Suggestions for large files

When uploading large files, the various buffers, window sizes and packet loss are important.

Window size. On the client, a window size that rate limits throughput may be in place in the upload application, operating system TCP configuration, or TLS implementations.
Packet loss. You can test packet loss using mtr. (This is a network diagnostic tool which can be used to identify packet loss and RTT/latency at each hop from the source to the destination.) To test for packet loss during transfers, use mtr and use large packets:

mtr -s 1024 foo.sftp.upload.akamai.com

👍
As a best practice for debugging, we recommend that you also incorporate the -T tag (for TCP) with the above command, for all protocols except Aspera Upload Acceleration.

Address last mile bottlenecks

You can also incorporate mtr during a transfer to see if you're experiencing high packet loss from your local up-link, onward to determine if the bottleneck is at the last mile.

Suggestions for large quantities of small files

When uploading many small files, overhead from the commit and round-trip times for single objects might dominate.

If this is the case, consider the following:

Determine if you have scope to add parallelism, while still staying below the noted storage group request rate limits.
Look into our Serve content from zip files functionality. With it, you can convert your many small uploads into a single large file upload.

Avoid starting a new TCP connection for every small object transferred.

This is to avoid frequent delays from TCP/TLS re-connection round trip times and needing to regrow the TCP window size.
When using command line utility programs (for example, scp, rsync) supply many small files to one invocation of the client utility, rather than starting the utility once per file.
A reasonable approach is to transfer a logical group of content that takes more than about ten seconds to transfer. But no more than a few minutes worth, to allow DNS a chance to reconnect you to a better server if the current one has become overloaded or better ones become available. Ten seconds should be enough to get most of the gains.

How to handle large upload rates

There are different throughput rates for single and multiple TCP connections.

Assuming that you encounter no "first mile" issues, you're following all of the recommended best practices, and you're geographically close to the storage location, your large file uploads typically experience the following rate of transfer:

You can see a peak of 0.5 Gbps over a single TCP connection.
You can achieve a one (1) Gbps average, sustained over multiple TCP connections to a storage group.

📘
"Geographically close" assumes that you're at least in the same half of the continent in which your storage resides.

Try using Parallelism

Performing uploads of different content in parallel ("Parallelism") may help with upload performance. You can implement parallel uploads in many ways without using multiple machines.

For example, you can use an upload client that offers built-in parallelism such as rclone (https://rclone.org/netstorage/).

You can also run multiple scripts in the background, with each script using its own instance of the upload client, and acting over a separate list of files or distinct directory trees.

While more client machines can help, this only applies if a bottleneck is the result of things on the client such as a faulty hardware (a bad ethernet nic), local disk I/O (Rsync md5s or directory scanning), or CPU usage for an expensive cryptographic algorithm.

The benefits of parallelism with small and large files

Large file transfers: Parallelism effectively makes the size of the TCP window the sum of all connections. It also lets you to take advantage of multiple resources (machines and uplinks) at either end of the request.
Small file transfers: Parallelism allows some uploads to use more available bandwidth while other uploads are blocked awaiting NetStorage confirmation and network turnaround times.

The limits to parallelism with small and large files

Large file transfers. Parallelism helps with bottleneck and limit issues, but only until packet loss occurs or the limits of the client machine are reached.
Small file transfers. Parallelism can help with these issues, but only up until NetStorage storage group write request limits are reached.

Upload approaches can differ in performance, flexibility, and effort

rclone is a high-performing and modern approach to managing cloud content. It supports direct use with NetStorage and is a mature content management client with extensive flexibility to optimize upload performance for specific client environments and content requirements.
Aspera Upload Acceleration (Aspera) is a high-performing protocol due to its built-in parallelism and its custom congestion control algorithm.
The NetStorage Usage API is a high-performing solution that provides a low-overhead option to upload content to NetStorage. When using the NetStorage API's you are responsible for managing parallelism. Too many concurrent uploads can result in TCP packet loss or insignificant improvement. In extreme cases too many parallel uploads can result in 429 errors. (The latter is a rarity, as NetStorage has throttling and request queuing mechanisms in place to address this. However, a 429 can occur if you're implementing parallelism, and for example, exceed set rate limits by sending very large quantities of small files via HTTP.)

Shard content to multiple storage groups

Technical limits with NetStorage exist for how much content and performance is allocated to a single storage group to help spread the load to different physical resources. To avoid reaching these limits, you can divide your content up into multiple storage groups.

Sharding is the concept of a "super" storage group in NetStorage. It essentially combines multiple storage groups—each considered a single shard—into a single entity that you access for uploads and downloads of your content.

Benefits of sharding

Increased footprint limits are the primary benefit. Additionally, sharding content between multiple storage groups may increase performance with larger libraries. However, smaller libraries (and those with an average file size of less than 0.5 MB) may still be clustered into the same hardware resources.

The sharding workflow

This workflow typically works as follows:

You need to perform both of the following, for each object (directory or file) that is to be changed:
- Extract or calculate the storage shard from the target object’s existing path name.
- Change the object using the storage upload hostname that matches the shard. (An "object" can be an actual file, a symlink or an explicit directory; a "change" is any write operation, including an upload, delete, timestamp or other metadata change to the object.) It helps if:
  - The storage group CNAME prefix matches the shard name.
  - CP code directories are hidden in uploads. (Or, alternately if the CNAME prefix is set to "cpNNN" where "NNN" matches the CP code.)
In the Content Delivery Network (CDN) delivery configuration, when objects need to be requested from the origin, you must:
- Extract or calculate the storage shard from the request for the object. Typically, only a portion of the path is used, but any HTTP header in the request could also provide the needed input.
- Request the object from the computed storage group’s hostname and CP code.

📘
This is a non-self service operation that requires the use of an advanced metadata configuration. You'll need assistance from your Account Representative to implement this.

Understand the sharding algorithm

This algorithm should be based on either:

Some portion of the content path name.
The delivery request headers that can be turned into the storage group upload and download hostnames.

Popular options are to use a portion of the path name directory (for example, the year or department), or a hash derived from this portion.

👍
Use consecutive storage CP codes for the shards, and select a CNAME prefix so that the storage group hostnames are computable from the these numbers. For example, cp123456.upload.scp.akamai.com for uploads to the CP code 123456.

Limits and drawbacks to sharding

Even though we have increased the limits per storage group via improvements in technology, a best practice is to double your current minimum requirements. It is also a good idea to plan and determine what your workflow might demand in two years and adjust this value, accordingly. You may want to use more than double, if you think your needs might demand it.

Increasing your requirement in the future requires that you copy existing content across to new storage groups that a new hashing algorithm will expect the content.

Here are some of the known, potential drawbacks to sharding:

All CDN delivery configurations and content management clients need to be instrumented to select the right shard hostname and CP code path. This includes any of these configurations that you may have actively working. For NetStorage HTTP API uploads, the complication could be hidden behind a custom upload configuration.
You can't rename across shards.
A directory listing (for example, "dir" via the NetStorage HTTP API) only returns results from an individual shard, not all of the storage groups included.
Support for growing or re-balancing shards isn't provided.

Operating system and client software performance

Use modern versions of operating systems and client software to get the latest TCP algorithms, and have access to functionality that allows you to increase things like buffer or window sizes.

We can't ensure peak performance if you use older versions.