Technical limits with NetStorage
NetStorage has active rate-limiting in place to recognize unreasonable bursts. This is not solely determined by your specific usage.
The Usage API isn't aware of content replication
This API doesn't redirect ('chase') requests to other NetStorage replicas. This is especially important if your NetStorage configuration uses replica-specific hostnames, because you may be interacting with content that's outdated or missing. Interacting with outdated content may occur in these situations:
- If your content isn't in the initial replica region.
- If the content you're accessing isn't the most recent version.
We recommend using a delivery product, as those requests will redirect across all NetStorage replicas of the target storage group. This ensures delivery of the freshest content even if the first delivery request goes to a replica which doesn’t yet have the data.
Network availability
Network availability is subject to unreasonable bursts or temporary overloads. This is likely to affect you if your usage is contributing to the problem, or if other customers are performing similar actions to the shared resource at the same time.
NetStorage supports both short- and long-term models:
- Short term. Via DNS load-balancing, we try to spread reasonable bursts to fit the capacity of online locations that are holding relevant content.
- Long term. We use automated content movement to re-balance content locations.
View throttled operations in a Read & Write Operations Report. Use this report to determine what types of operations are exceeding the rate limit thresholds. Contact your account team to adjust your threshold limits.
Operational and rate limits
NetStorage doesn't support a "fixed" number of operations per second. Multiple factors can impact this total, so we don't guarantee a static number of operations per second, per storage group. Instead, these are the recommended “best practice” maximums:
Limits per storage group
Type of limit | Description |
|---|---|
Storage group size | 1 PB |
Number of objects | 200 million |
Objects per directory | No limit. Client timeouts may occur if a directory list contains an excessive amount of objects. |
Maximum CP codes | 20 |
Write operations per second | 50 |
Read operations per second | 1,000 |
Maximum upload accounts | 100 |
Maximum connections | 90 concurrent connections per upload account. |
Creating or updating a storage group configuration | 10 operations per hour. |
Getting storage group configuration details | 50 operations per hour. |
File path length | 800 characters per object. |
File size | 150 GB single object maximum. |
Naming of files and directory paths |
|
CMShell limits with the “unzip” (zip32) operation
Type of limit | Description |
|---|---|
Maximum archive size | 4 GB archive size. |
Maximum number of files in a “zip” archive | 65,000 files per archive. |
Unsupported capabilities | Capabilities beyond the basic UNIX "unzip" are not supported, such as:
|
You should try Serve from Zip
You might be able to serve your files from within an uploaded zip archive on NetStorage without unzipping it, by using Serve from Zip. Serve from Zip supports much larger files and the zip64 file format. If you can use it, you won't need to use this unzip command. See if the Serve from Zip considerations are compatible with your workflow.
Serve from Zip limits
Type of limit | Description |
|---|---|
Maximum file size | 100 GB |
Maximum number of files in an archive | 1,000,000 |
Read and write requests defined
Delivery Method | Type | Request type |
|---|---|---|
Content Management | HTTP-API |
|
Content Management | Session-Based protocols | When using the non-HTTP content management protocols (Aspera, rsync, ftp, scp, sftp, cmshell) their actions are converted into corresponding HTTP-API requests by Akamai. |
Delivery | Origin for content delivery | There is a read request for every attempted object access. These are often the cause of most read requests. |
Which requests are counted?
Requests counted include those with these HTTP response codes:
Response code | Description |
|---|---|
2xx | Successful, including 204 "Partial Content". |
3xx | Redirections, including 304 "Not Modified". |
4xx | Client errors, including 404 "Not Found" (and 429 "Rate Limited"?), but excluding some errors which we can't attribute to the right customer. E.g., some 400s, 401s, and 403s. |
Storage Group and CP code total limits
If you require more than 600 total storage CP codes on your contract, establish different storage groups to use different geographic locations. A single storage group cannot span multiple geographic regions.
Set up individual upload accounts to access a specific CP code (upload directory), and limit directory access via Subdirectory Restrictions. This way, the individual users associated with a specific upload account will only be able to access that directory. You can then use the NetStorage HTTP API or the CMShell (via the "du" command) to report the total footprint for the directory. While it offers a method of tracking, it doesn't offer the robust reporting and graphs that you have access to via Control Center Reports and Analytics that are driven by CP codes.
Upload Account and user session limits
Typically, you can get by with a just a few total upload accounts. (For example, ten accounts, plus one more for each additional storage group.)
This page provides situational information, and some possible suggestions to accommodate those situations. Also see how User login limits are enforced for specific recommendations on avoiding login failures for non-NetStorage HTTP API upload protocols.
Upload account needs | Description |
|---|---|
If you need more than 100 upload accounts | |
If you need more than 10 concurrent connections, and are uploading large files in each login session | Note: The average sustained change rate across the storage group still matters, regardless of the upload protocol you're using. |
Overall footprint limits
Use-case | Recommendation |
|---|---|
If you have more than 1,000 TB stored in one storage group | Look into "Sharding" content to multiple storage groups. |
If you have more than 200 million objects in one storage group |
|
If you are working with files larger than one (1) GB in size |
|
If you are working with files larger than 100 GB in size |
|
Rate limit use cases
High levels of read/write requests ("bursts") or temporary overloads will result in rate limits, for example, delays in responses. If the clients are not moderated by this (e.g., there are many clients acting in parallel) then further requests may be denied. (For example, a 429 response code is sent on HTTP protocols.)
There is no guaranteed level before these rate limits kick in, but they are applied first to storage groups that are most likely causing the overload. So, we recommend that you stay under the "suggested requests per second" values. When evaluating your workflow, the following apply:
About these rates
These numbers consider all requests to single storage group, even if one replica is distant or unavailable. On average, you are likely to see higher upload and download request rates to a storage group. However, on a single TCP connection you should expect lower request rates. For example, including common network delays, 10 write operations per second on a single connection is normal for many customers uploading small files.
These rates also assume that requesting clients are adhering to the normal DNS best practices for use regarding TTL settings, and assume that applications are not "sticking" to the IP addresses returned in the past. Otherwise, you reduce the chances we have to map your requesting clients away from overloaded servers.
Use case | Description |
|---|---|
Sending more than 1,000 read requests per second to a single storage group |
|
Sending more than 50 write requests per second (sustained rates averaged over 10 seconds) to a storage group |
|
Your replication configuration will impact rate limits |
Note: The following configurations require additional configuration before they are enabled. They are not available by default due to potential performance issues:
|
User login limits are enforced
Concurrent login sessions use shared resources. We enforce fairness policies to limit the impact that each customer, upload user account and storage group may have on each other. Increasing the number of concurrent logins for one customer, one upload user account and one storage group raise the probability that a login denial will occur. Note these recommendations:
- Large File Transfers: In this case, rate of progress is limited by data transfer throughput. More connections via additional Upload User Accounts are unlikely to improve things.
- Small File Transfers: In this case, rate of progress is limited by turnaround time to get responses. Additional Upload User Account connections can be seen as a reasonable way to improve things.
What can I do if I get a login denial?
You can attempt a retry strategy that involves the following:
- Try re-resolving the Upload Domain Name and creating a new network connection on every login attempt. This allows your application to avoid overloaded upload servers, by choosing alternate IP address answers from your client’s past cached DNS responses; or new IP addresses as we identify overloaded servers and the DNS time to live (TTL) expires.
- Implement an increasing "back-off time" to ensure your application is not the cause of the overload. For example, back-off your attempts at increasing intervals, such as zero, five, 10 and 20 seconds apart, and then attempt every 30 seconds.
Potentially problematic use cases
Be aware of the design implementation constraints of NetStorage.
NetStorage provides complete redundancy with wide geographical separations, as well as redundancy within each data center. This allows high performance and availability for uploads and downloads, because requests are DNS load-balanced to the best location for a client. This gives you high durability for your content, even in the face of natural disasters of regional scope. To maintain this redundancy, geographical replication is performed after an upload or failure, and the last request from the customer to change the same object eventually wins.
NetStorage is optimized for large media object delivery workloads to clients across the world. (For example, this includes tools that deliver video on demand media or perform software downloads.) Additionally, NetStorage can be used for other high availability and durability applications that fit its system limits, and can also cope with eventual consistency.
What constitutes a potentially problematic use case
These use cases are problematic and require a more in-depth understanding of how NetStorage works:
- You have a large total number of GB to store.
- You have a large number of files to store.
- You work with very large files.
- You deal with files that change frequently.
- You require high bandwidth uploads.
- Your files are uploaded from large numbers of geographically diverse sources.
- You need to avoid delivering stale content after a change.
- You have many delivery requests for large numbers of files that have not yet been uploaded. (These return a 404 error.)
Relevant aspects for these use cases
With each of the above use cases, there are specific aspects you need to keep in mind.
- System Limits
- Rate Limits
- Upload Account and User Connection Limits
- Geographic Replication and the "Eventual Consistency Model"
Known NetStorage limitations
Hard link creation is not supported
However, symbolic links ("symlinks") are fully supported.
User-specified scratch directories are not supported
Control of all scratch directory parameters and behaviors are retained internally by Akamai.
File Manager capabilities are limited
File Manager provides a basic UI that you can use to browse and manage content stored in NetStorage via Control Center. It is meant to be used for small and quick operations.
You should not use File Manager and instead, implement a production file upload system, if any of the following apply:
- You need to preserve directory structures.
- You need to traverse through complex directory trees.
- You need to initiate large data transfers, or upload large files.
- You need to perform other "mass operations," such a deleting or moving a large number of files.
File Manager does not preserve nested sub-directory structures on upload, and nested files will reside in the top-level directory.
A "production file upload system" is one you develop using the various protocols available with NetStorage, or the NetStorage Usage API.
Some content management operations are not allowed
Renaming directories isn't supported
Because of replication and certain other features involved in NetStorage, you cannot rename directories. Some workarounds include the following:
- You can create symbolic links between directories. You can then use a desired, new name for the symbolic link.
- You can "rewrite" a path via an associated media delivery product. You can set up the configuration for an associated delivery product to rewrite the directory path. For example, using Adaptive Media Delivery you can set this up using the Modify Outgoing Request Path behavior in Property Manager.
With this in mind, take care when naming your directories in a storage group. Ensure the name you've chosen will fit your needs in the future.
About file attribute settings
A limited number of file attributes (for example, setting up "read only" status) can be set up in an upload account using the NetStorage Groups UI in Control Center. However, most file attributes such as read, write, ownership and execute (attributes other than the time stamp) are not self-service. Speak with your account representative for details on setting these attributes.
Updated about 1 month ago