Technical limits with NetStorage
NetStorage has active rate-limiting in place to recognize unreasonable bursts. This is not solely determined by your specific usage.
The Usage API isn't aware of content replication
This API doesn't redirect ('chase') requests to other NetStorage replicas. This is especially important if your NetStorage configuration uses replica-specific hostnames, because you may be interacting with content that's outdated or missing. Interacting with outdated content may occur in these situations:
- If your content isn't in the initial replica region.
- If the content you're accessing isn't the most recent version.
We recommend using a delivery product, as those requests will redirect across all NetStorage replicas of the target storage group. This ensures delivery of the freshest content even if the first delivery request goes to a replica which doesn’t yet have the data.
Network availability
Network availability is subject to unreasonable bursts or temporary overloads. This is likely to affect you if your usage is contributing to the problem, or if other customers are performing similar actions to the shared resource at the same time.
NetStorage supports both short- and long-term models:
- Short term. Via DNS load-balancing, we try to spread reasonable bursts to fit the capacity of online locations that are holding relevant content.
- Long term. We use automated content movement to re-balance content locations.
View throttled operations in a Read & Write Operations Report. Use this report to determine what types of operations are exceeding the rate limit thresholds. Contact your account team to adjust your threshold limits.
Operational and rate limits
NetStorage doesn't support a "fixed" number of operations per second. Multiple factors can impact this total, so we don't guarantee a static number of operations per second, per storage group. Instead, these are the recommended “best practice” maximums:
Limits per storage group
| Type of limit | Description |
|---|---|
| Storage group size | 1 PB An alert is sent when an 80% threshold is met. NetStorage will reject uploads that exceed 100%. |
| Number of objects | 200 million An alert is sent when a 75% threshold is met. NetStorage will deny uploads and the creation of new objects that exceed 100%. |
| Objects per directory | No limit.Client timeouts may occur if a directory list contains an excessive amount of objects. |
| Maximum CP codes | 20 If you need more CP codes to separate permissions or usage within a storage group (for example, resale aggregators with large numbers of sub-customers), put together a viable business case and contact your account representative. Keep in mind that user generated content is different. |
| Write operations per second | 50 |
| Read operations per second | 1,000 |
| Maximum upload accounts | 100 |
| Maximum connections | 90 concurrent connections per upload account. |
| Creating or updating a storage group configuration | 10 operations per hour. |
| Getting storage group configuration details | 50 operations per hour. |
| File path length | 800 characters per object. This is the total length, including file names and directories. |
| File size | 150 GB single object maximum. |
| Naming of files and directory paths | - All characters used in filenames and path names must be valid ASCII characters in the range of 32-255 (%20-%FF). - Double-byte character sets are supported (via the default iso-8859 encoding). UTF-8 encoding is also available. - Use of a dot or slash character as a complete path component (eg /foo/./bar/ or /foo/../bar/ or /foo//bar/) is only supported for operations performed via the NetStorage HTTP API. - A dot is not supported for use as a path component when using content management protocols such as SSH, FTP, etc. |
CMShell limits with the “unzip” (zip32) operation
| Type of limit | Description |
|---|---|
| Maximum archive size | 4 GB archive size. |
| Maximum number of files in a “zip” archive | 65,000 files per archive. |
| Unsupported capabilities | Capabilities beyond the basic UNIX "unzip" are not supported, such as: _ bzip compression _ encryption * zip64 |
You should try Serve from Zip
You might be able to serve your files from within an uploaded zip archive on NetStorage without unzipping it, by using Serve from Zip. Serve from Zip supports much larger files and the zip64 file format. If you can use it, you won't need to use this unzip command. See if the Serve from Zip considerations are compatible with your workflow.
Serve from Zip limits
| Type of limit | Description |
|---|---|
| Maximum file size | 100 GB |
| Maximum number of files in an archive | 1,000,000 |
Read and write requests defined
| Delivery Method | Type | Request type |
|---|---|---|
| Content Management | HTTP-API | Write requests: Any HTTP-API request that changes content. E.g., uploads, deletes and metadata changes like mkdir, rmdir, renames, symlinks and mtime are write requests. Read requests: An HTTP-API dir or list command counts as 10 read requests for rate-limiting purposes. Any other HTTP-API request is a read request, including file downloads and even just checks to see if an object is present. |
| Content Management | Session-Based protocols | When using the non-HTTP content management protocols (Aspera, rsync, ftp, scp, sftp, cmshell) their actions are converted into corresponding HTTP-API requests by Akamai. As well as a request per object, there are often extra read requests to adhere to POSIX semantics or to scan directories. |
| Delivery | Origin for content delivery | There is a read request for every attempted object access. These are often the cause of most read requests. |
Which requests are counted?
Requests counted include those with these HTTP response codes:
| Response code | Description |
|---|---|
| 2xx | Successful, including 204 "Partial Content". |
| 3xx | Redirections, including 304 "Not Modified". |
| 4xx | Client errors, including 404 "Not Found" (and 429 "Rate Limited"?), but excluding some errors which we can't attribute to the right customer. E.g., some 400s, 401s, and 403s. |
Storage Group and CP code total limits
If you require more than 600 total storage CP codes on your contract, establish different storage groups to use different geographic locations. A single storage group cannot span multiple geographic regions.
Set up individual upload accounts to access a specific CP code (upload directory), and limit directory access via Subdirectory Restrictions. This way, the individual users associated with a specific upload account will only be able to access that directory. You can then use the NetStorage HTTP API or the CMShell (via the "du" command) to report the total footprint for the directory. While it offers a method of tracking, it doesn't offer the robust reporting and graphs that you have access to via Control Center Reports and Analytics that are driven by CP codes.
Upload Account and user session limits
Typically, you can get by with a just a few total upload accounts. (For example, ten accounts, plus one more for each additional storage group.)
This page provides situational information, and some possible suggestions to accommodate those situations. Also see how User login limits are enforced for specific recommendations on avoiding login failures for non-NetStorage HTTP API upload protocols.
| Upload account needs | Description |
|---|---|
| If you need more than 100 upload accounts | |
| If you need more than 10 concurrent connections, and are uploading large files in each login session | Note: The average sustained change rate across the storage group still matters, regardless of the upload protocol you're using. |
Overall footprint limits
| Use-case | Recommendation |
|---|---|
| If you have more than 1,000 TB stored in one storage group | Look into "Sharding" content to multiple storage groups. |
| If you have more than 200 million objects in one storage group | Look into using Serve from Zip to implement indexed zip files. Look into "Sharding" content to multiple storage groups. |
| If you are working with files larger than one (1) GB in size | Enable large file delivery support in the CDN, when single objects are over 1.8 GB. Be aware that if a connection fails during upload, you can only "retry from the start." (NetStorage does not support "resuming an upload.") So, the quality of the last-mile during a transfer is more important. Avoid downloading content via the NetStorage Usage API. If you do for occasional content management, see Working with Huge Objects in the NetStorage Usage API. Review content in the next point, If you are working with files larger than 100 GB in size. |
| If you are working with files larger than 100 GB in size | Split these files into smaller parts before uploading, and deal with the split parts on the client-side. This can also allow for faster uploads of these single huge files if you upload the parts in parallel. (Use parallelism.) For example, when uploading mezzanines to be transcoded by the Media Services On Demand: Content Preparation, the Akamai Secure Ingest Tool can be used to accomplish this. |
Rate limit use cases
High levels of read/write requests ("bursts") or temporary overloads will result in rate limits, for example, delays in responses. If the clients are not moderated by this (e.g., there are many clients acting in parallel) then further requests may be denied. (For example, a 429 response code is sent on HTTP protocols.)
There is no guaranteed level before these rate limits kick in, but they are applied first to storage groups that are most likely causing the overload. So, we recommend that you stay under the "suggested requests per second" values. When evaluating your workflow, the following apply:
About these rates
These numbers consider all requests to single storage group, even if one replica is distant or unavailable. On average, you are likely to see higher upload and download request rates to a storage group. However, on a single TCP connection you should expect lower request rates. For example, including common network delays, 10 write operations per second on a single connection is normal for many customers uploading small files.
These rates also assume that requesting clients are adhering to the normal DNS best practices for use regarding TTL settings, and assume that applications are not "sticking" to the IP addresses returned in the past. Otherwise, you reduce the chances we have to map your requesting clients away from overloaded servers.
| Use case | Description |
|---|---|
| Sending more than 1,000 read requests per second to a single storage group | You can look into "Sharding" content to multiple storage groups. This applies if this is a large library. (If you have many small storage groups, they may still cluster to the same hardware, which would negate any benefit sharding would offer.) Reduce the number of requests to NetStorage. If most are for downloads with no response body: Check for unnecessary requests. This includes 404 errors or other failures. Avoid the use of NetStorage unless 90% of the objects requested are likely to be present. For use cases with high numbers of 404 errors, contact your Account Representative for optimization suggestions. Increase Content Delivery Network (CDN) Time to Live (TTL) settings. This can help to reduce the number of 304 responses. Limit low Time to Live (TTL) settings, to only those objects that need them. If most are for downloads with a small response body: Increase the size of response bodies by merging objects always requested together if each object is less than two megabytes in size. For example, increase the size of range-requests or the video streaming segment length. * Look into client and workflow changes. For example, you can use sprites for the thumbnail images in a video scrub-bar. |
| Sending more than 50 write requests per second (sustained rates averaged over 10 seconds) to a storage group | Reduce the burst demanded by your workflow. (For example, use less parallelism in your uploads.) Look into using Serve from Zip to implement indexed zip files. Look into "Sharding" content to multiple storage groups. This may help with large libraries. Contact your Account Representative for help with large onboarding situations. |
| Your replication configuration will impact rate limits | Storage groups with a default of two Geographic Replications ("replicas"): 50 write requests per second. 1,000 read requests per second. Storage groups with restrictions which replicas can download/upload: Read/write rates are halved. _ Storage groups with three replicas: 15 write requests per second. No change to read rates. Note: The following configurations require additional configuration before they are enabled. They are not available by default due to potential performance issues: _ Three-way replication (three replicas). * Restricting replicas from being used for upload/downloads (including use as a “Failover” backup). |
User login limits are enforced
Concurrent login sessions use shared resources. We enforce fairness policies to limit the impact that each customer, upload user account and storage group may have on each other. Increasing the number of concurrent logins for one customer, one upload user account and one storage group raise the probability that a login denial will occur. Note these recommendations:
- Large File Transfers: In this case, rate of progress is limited by data transfer throughput. More connections via additional Upload User Accounts are unlikely to improve things.
- Small File Transfers: In this case, rate of progress is limited by turnaround time to get responses. Additional Upload User Account connections can be seen as a reasonable way to improve things.
What can I do if I get a login denial?
You can attempt a retry strategy that involves the following:
- Try re-resolving the Upload Domain Name and creating a new network connection on every login attempt. This allows your application to avoid overloaded upload servers, by choosing alternate IP address answers from your client’s past cached DNS responses; or new IP addresses as we identify overloaded servers and the DNS time to live (TTL) expires.
- Implement an increasing "back-off time" to ensure your application is not the cause of the overload. For example, back-off your attempts at increasing intervals, such as zero, five, 10 and 20 seconds apart, and then attempt every 30 seconds.
Potentially problematic use cases
Be aware of the design implementation constraints of NetStorage.
NetStorage provides complete redundancy with wide geographical separations, as well as redundancy within each data center. This allows high performance and availability for uploads and downloads, because requests are DNS load-balanced to the best location for a client. This gives you high durability for your content, even in the face of natural disasters of regional scope. To maintain this redundancy, geographical replication is performed after an upload or failure, and the last request from the customer to change the same object eventually wins.
NetStorage is optimized for large media object delivery workloads to clients across the world. (For example, this includes tools that deliver video on demand media or perform software downloads.) Additionally, NetStorage can be used for other high availability and durability applications that fit its system limits, and can also cope with eventual consistency.
What constitutes a potentially problematic use case
These use cases are problematic and require a more in-depth understanding of how NetStorage works:
- You have a large total number of GB to store.
- You have a large number of files to store.
- You work with very large files.
- You deal with files that change frequently.
- You require high bandwidth uploads.
- Your files are uploaded from large numbers of geographically diverse sources.
- You need to avoid delivering stale content after a change.
- You have many delivery requests for large numbers of files that have not yet been uploaded. (These return a 404 error.)
Relevant aspects for these use cases
With each of the above use cases, there are specific aspects you need to keep in mind.
- System Limits
- Rate Limits
- Upload Account and User Connection Limits
- Geographic Replication and the "Eventual Consistency Model"
Known NetStorage limitations
Hard link creation is not supported
However, symbolic links ("symlinks") are fully supported.
User-specified scratch directories are not supported
Control of all scratch directory parameters and behaviors are retained internally by Akamai.
File Manager capabilities are limited
File Manager provides a basic UI that you can use to browse and manage content stored in NetStorage via Control Center. It is meant to be used for small and quick operations.
You should not use File Manager and instead, implement a production file upload system, if any of the following apply:
- You need to preserve directory structures.
- You need to traverse through complex directory trees.
- You need to initiate large data transfers, or upload large files.
- You need to perform other "mass operations," such a deleting or moving a large number of files.
File Manager does not preserve nested sub-directory structures on upload, and nested files will reside in the top-level directory.
A "production file upload system" is one you develop using the various protocols available with NetStorage, or the NetStorage Usage API.
Some content management operations are not allowed
Renaming directories isn't supported
Because of replication and certain other features involved in NetStorage, you cannot rename directories. Some workarounds include the following:
- You can create symbolic links between directories. You can then use a desired, new name for the symbolic link.
- You can "rewrite" a path via an associated media delivery product. You can set up the configuration for an associated delivery product to rewrite the directory path. For example, using Adaptive Media Delivery you can set this up using the Modify Outgoing Request Path behavior in Property Manager.
With this in mind, take care when naming your directories in a storage group. Ensure the name you've chosen will fit your needs in the future.
About file attribute settings
A limited number of file attributes (for example, setting up "read only" status) can be set up in an upload account using the NetStorage Groups UI in Control Center. However, most file attributes such as read, write, ownership and execute (attributes other than the time stamp) are not self-service. Speak with your account representative for details on setting these attributes.
Updated about 2 years ago