Downloads of huge objects
When downloading huge objects (typically any file in excess of 2 GB in size), be prepared to retry the HTTP request if the TCP connection fails. With a retry, ensure that you perform the following:
- Update the authentication signature header. The previous one may have expired.
- Incorporate a byte range request that begins from the point of the failure. This will ensure forward progress. This is especially helpful when the token associated with Authentication Signature Header has expired, and NetStorage is trying to transparently retrieve the latter part of the object from a different system in the network.
- Only construct full objects from partial byte range responses that use the same “ETag” response header value. This will help if the object has changed between requests. Otherwise, at least check that the full object reports the same “md5” value in one of the Etag response headers.
While these points predominantly address issues that occur with huge objects, they can also occur with smaller files, as the probability of TCP connection loss increases. (For example, they can occur due to higher packet loss, or a slower connection which may take longer.)
Downloads and byte range requests
Byte Range requests are honored as long as the HTTP “Range” header consists of a single, closed range (number-number).
Large quantities of files may affect your downloads
While there's no set maximum on the number of files you can download using this action, it is important to note that it's not intended for use as a “mass download” resource. If used to download a large quantity of files in a single call, you may notice performance issues and failures.
If you intend to download a large quantity of files, incorporate a Delivery Configuration and incorporate Large File Optimization.
You may also experience the same performance issues if attempting repeated “download” action calls for content, over a short period of time.