Add a storage group

Use the NetStorage wizard or Configuration API to create a new storage group. To begin, access the Storage Groups entity and click the + icon to access a wizard.

1. Open the application. Go to ☰ > ORIGIN SERVICES > NetStorage
2. Access the Storage Groups entity and click the + icon to access the Create a new storage group wizard.
   

Here's a short summary. You can find more details below.

1. Basic Information. Here you define the Storage group name and the Domain name information for the storage group.
   • Storage group name. Enter a name for the storage group. This is how it's identified throughout the UI and API.
   • Domain name prefix. This value serves as a unique prefix for each of the domain names used to access this storage group. For example, it's included in your upload domain names used to access this storage group.
2. Geo Replication Settings. These allow you to determine where your content will be replicated for added stability and increased availability. You must select two different geographic regions.
3. Additional Domains (optional). You can optionally add domains to target specific regions.
4. Directory Settings. These settings establish a root directory (an "Upload Directory") for the storage group. Your storage group needs at least one upload directory. If you don't already have one, you can generate one in the UI. You can also apply optional Security and Advanced Settings.
5. Automatic Purge Routines (optional). Define a routine to target directories in your storage group and have content automatically deleted once certain conditions have been met.
6. Finally, you need to review the Summary. All of the settings you've applied for the group are revealed, and you use this window to save them.

📘

You can edit an existing storage group to modify settings as necessary. However, you must wait for a new storage group to complete provisioning before you can edit it.

---

Basic Information

Create a Name for the storage group to serve as an identifier, and define a unique Domain Name value for use in identifying points of access for the group.

• Contract. An Access Control Group (ACG) is a grouping of products associated with a specific contract. (NetStorage is the product, in this case.) ACGs determine how service usage is logged and billed (if applicable), so make sure that the appropriate ACG is selected. If only one ACG is associated with your account, it will be default selected here.
• Storage group name. Input a desired name for the storage group. This is how it is identified throughout the UI. You should use a unique value that's easy to remember. For example, name the group based on the type or classification of media you wish to store in it.
• Domain name prefix. This value serves as a unique prefix for each of the access method upload domains. For example, if you provide the value baseball, the upload domain for FTP would appear as baseball.ftp.upload.akamai.com.

Domain name prefix value requirements

These rules apply to the prefix value:

• Special characters are not supported. Only alphanumeric characters ("a-z", "0-9") and the hyphen ("-") can be used.
• Uppercase characters are not supported. All alphabetic characters used in a domain name must be in lowercase.

🚧

You can't change this value after you create the storage group. Ensure that the prefix you choose is the one you wish to use with a storage group.

Differences between the storage group name and the domain name

When setting up a storage group, you define both of these values, you should understand the difference between the two:

• The Storage group name. This is a value you apply to the group to uniquely identify it.
• The Domain name. This is a prefix value that you define (the "domain name prefix") that is pre-pended to unique domain name(s) you use to access the storage group.

You can use the same value for both your Storage group name and Domain name (prefix), and this is a common practice. However, both must meet the value requirements discussed above.

---

Geo Replication Settings

Replicate your data in at least two different geographic regions (zones) for added stability. For best performance, ensure that both regions are geographically close. In general replica location should be tailored to where uploads come from, and where delivery performance is most important.

📘

The default No capacity action will temporarily Spill outside of your zone if there's no capacity available. This is the default action. Contact your account representative if you would like to Deny Upload to other zones.

Three-way Replication and NetStorage

Storage groups default to the use of two replicas, or "two-way replication." NetStorage offers support for three-way replication, but it's not a standard offering. It can be provided after a valid business case consideration. If you have implemented it, be aware you'll have lower limits on the number of write operations per hour than standard two-way replication. This may delay serving content when requested.

Contact your account representative for additional details and availability information.

📘

The term "write operation" includes any successful change to content or metadata stored in a storage group. This includes uploads, explicit directory creation, symlinks, renames, timestamp changes and deletes.

Supported Geo Replication Zones

Replicas should be nearby for best performance and stability. Available zones may change over time. Contact your account representative to make changes to existing replication configurations.

---

Additional Domains

You can mitigate temporary connectivity issues by using additional domains to target specific regions to have specific domains access a specific region. You can add additional domains during the creation or editing of a storage group. Each additional domain is appended to your primary domain name.

In this example, we append each region to the domain, and upload traffic is allowed and restricted as needed.

Upload Directory

An upload directory is a root directory in a storage group, represented by a CP code that has been provisioned for use with NetStorage. This is also referred to as a "CP code root." The CP code is used to track billing usage as well as a reference for your use in reporting and gathering statistical data. When adding or editing an upload directory, you use the CP code field to select a CP code for use.

• If only a single CP code is available it will be default selected. In this case, a single CP code has been provisioned and is available for use as an upload directory.
• If multiple CP codes are available, select the appropriate one. In this case, multiple CP codes have been provisioned for use as an upload directory. Ensure that you select the appropriate one for use. Speak to your NetStorage administrator to determine the proper one, or contact your account representative.
• If no CP codes are available, you can create a new CP code. Click the Create new button, provide the CP code name and then click the Create button.

Upload Directory Settings

You use these settings to define and configure an upload directory for use with NetStorage.

An upload directory is a root directory in a storage group, represented by a CP code that has been provisioned for use with NetStorage. You later associate this upload directory with an upload account, to grant it access to this storage group.

Several settings can be applied and are broken down into various categories:

The settings here are used to select, or create the CP code to serve as the upload directory.

1. Click the + icon to add a new upload directory.

2. Do you have a CP code provisioned and available for use?

   • Yes. Select the applicable one. If you have only one, it's selected by default. You can't create new CP codes using the UI if you already have one. If you want another CP code, try using the API to create a CP code.

   • No. You're given an option to create a new one. A CP code name can be up to 128 alphanumeric characters, but can't include _ , # ' " % or non-printable characters. It can take up to two hours for a new CP code to provision.

     

3. Additional settings are revealed that allow you to configure the upload directory. We discuss them in the sections that follow.

📘

Some options may be unavailable

Contact your account representative for assistance in applying settings that are shown but grayed-out.

Security Settings

These settings allow you to determine how various requests will be handled by the storage group.

Advanced Settings

To view or change these settings for an existing upload directory:

1. Open the application. Go to ☰ > ORIGIN SERVICES > NetStorage
2. Access the Storage Groups entity and select the storage group you want to view.
3. Click Edit to edit the storage group.
4. Scroll down to the Upload directory you want to edit and click "..." followed by Edit.

Force Case

Your environment may have specific file-naming conventions pertaining to the use of upper vs. lowercase text. Some web services require that filenames use a specific case usage model. For example, a web-based database application may require that all file names be lowercase, but your environment requires the use of capital letters.

Enable the Force Case feature to change case-usage of your files during upload to match the needs of your workflow.

• This feature doesn't change the case during file access or requests, it changes the actual filename stored on NetStorage during the upload process.
• This feature doesn't automatically create subdirectories or apply Force Case rules to them.

With Force Case enabled, access requests to objects stored on NetStorage can impact how you Use Rsync.

👍

NetStorage as your origin and the ignore-case-cache-key behavior

To use the ignore-case-cache-key behavior with NetStorage as the origin server, you need to configure NetStorage with the Force Case enabled to match the ignore-case-cache-key setting. Make sure you name the original files consistently as either upper or lowercase.

📘

NetStorage doesn't automatically apply the Force Case setting to Index requests

If using the Force Case option, you must manually name your index.htm file accordingly. For example, if the Force Case setting is set to Convert all requests to uppercase, then you would change the Index Name to INDEX.HTM instead of index.htm.

🚧

Before modifying case settings

Ensure any files that currently exist in the targeted storage group conform with the selected setting. For example, if Convert all requests to lowercase is selected, all existing files in the storage group must currently be all lowercase. If an option is selected, but existing files do not conform to the selected case setting, a denial of service will result for your end users.

Path Name Mode

NetStorage doesn't adhere to a traditional directory-based file system. As a result, characters such as “/” can be used within a directory name. For example, you could actually name a directory /files, in which case, NetStorage would establish the path to the directory as //files (adding an additional "/" as a path separator). Use this drop-down to define how path name requests made to the selected upload directory should be interpreted.

• Don't check on incoming paths. With this selected, paths are not checked. The path set in an upload or download request is acknowledged exactly as input.
• Disallow improper paths on upload: With this mode selected, paths using naming conventions outside of what is supported for use as an explicit or implicit directory name are blocked, and an error message is displayed on upload.
• Translate all incoming path names to a canonical directory-friendly format. With this option selected, a / is seen as a path delimiter and repeat instances are treated as a single instance in an upload path. For example, if an upload request is sent to target the directory, //files/new///mp4/movie1.mp4, the file is uploaded to /files/new/mp4/movie1.mp4, and non-existent directories within the path are auto-generated to exist as explicit ones.
• Disallow improper paths on upload, but perform translations on download requests. This disallows improperly formatted path names on an upload request, but translates improper paths called out in download requests, similar to what is discussed for “Translate all incoming path names....”. For example, if a download request is issued for the path, //files/new///mp4/movie1.mp4, NetStorage looks to the path, /files/mp4/movie1.mp4 to find the file.

Send content MD5 header

1. On. A content item's MD5 digest values are sent in the HTTP “Content-MD5” response header.
2. Off. If you are not concerned about sending this header data.

Query string mode

If a request targeting the upload directory has a query string value appended to it, determine how the string should be handled. This applies to both upload and download operations targeting the upload directory.

• Strip all incoming query strings. With this selected, the string is removed from a request.
• Leave incoming query string as is. If this is selected, the string is left alone and interpreted, as applicable.
• Apply ACS query string conversion. With this mode enabled, as content is uploaded, its query string content is reviewed. A new, “stripped down” instance of the string is applied to it, based on what is defined in the accompanying fields (see below). Finally, a hash of this new string is generated and appended to the stripped down string. Once a download is requested for the content, the query string within that request is processed using the same conversion and the hashes are compared. If they match, access is granted. The accompanying fields within the NetStorage Configuration API are mutually exclusive and used as follows:
  • Key order. Input individual query string values that should be included, in the specific order they should be interpreted. Separate multiple entries with a single whitespace. Any string left out of the this field that exists in an actual query string is automatically excluded.
  • Exclude. Input specific query string values that should be excluded from the conversion, separating multiple entries with a single whitespace.

🚧

If the final path component for an uploaded file is in excess of 200 characters after conversion, some of the stripped down query string will be truncated to meet the 200 character maximum.

Quick Delete

Set this switch to "On" to enable use of the "quick-delete" operation available in the NetStorage Usage API and the CMShell. This allows you to target a specific directory in this storage group and recursively delete all of its content. Specific usage requirements and caveats apply.

📘

Enabling this option changes the default CMShell behavior of the "rm -r" command. Enabling this option changes the command to operate as a “quick-delete” rather than its default setting.

Encoding, key order and exclude

These options allow you to select and enforce the type of encoding to be used when displaying paths in XML-aware contexts. For example, this applies to call response output for the dir, list and stat calls via the NetStorage Usage API, as well as for file names and paths displayed in the NetStorage Groups UI. Via the Encoding drop-down, you can select from the following:

• iso-8859-1. This is the default. With this selected, file names and paths are encoded and displayed using iso-8859-1, 8-bit, single-byte coded graphic character sets. (This is the “Latin alphabet no. 1,” that consists of 191 characters from the Latin script.) This method treats all byte sequences as valid.
• UTF-8. With this selected, file names and paths are encoded and displayed using variable length, 8-bit code units via UTF-8 character encoding. UTF-8 sees some sequences as invalid. In this case, the XML encoding cannot represent file names that do not comply with the encoding. Therefore, the XML output changes the attribute from “X” to “X_base64” (and provides the raw octet stream as the base64-encoded value).

Enforce encoding. When set to On, upload operations will verify that all path values defined within the URL for target content are properly formatted using the selected Encoding method. If the path is not properly formatted to meet the selected method, an error will be returned.

📘

This only applies to the use of UTF-8 as an encoding method. If you have selected iso-8859-1, the Enforce Encoding slider is grayed-out and unavailable.

Serve from Zip

1. On. NetStorage will dynamically examine archived files to directly serve individual files from within the archive.
2. Off. Archive files will need to be completely decompressed to access and serve its contents. Specific usage conditions and requirements apply.

Index name, Directory listing, Index limit, and Search on 404

These options, as well as the related, Search on 404 option account for the unique file system supported by NetStorage. They are used to define an "Index File" that should be accessed for requests that do not end in a specific file name and extension. (For example, a request for an object that ends in a trailing slash -- "/".) The "Index File" is similar in functionality to that of Apaches "directory index" auto-index feature. Use these fields as follows:

Automatic Purge Routines

Automatic purge routines perform a routine scan of a target directory, and once a specified capacity is reached, the system will begin to purge files that are older than a date you specify.

📘

Automatic Purge functionality is not supported for storage groups used by iPhone Live configurations. If the group is associated with this type of configuration, this option will be grayed-out, and you will need to set up automatic purge at the stream level. Please contact your account representative for additional assistance.

Set up a purge routine

To set up an automatic purge routine, click the + icon and set these options as necessary:

Purge routines don't delete empty, explicit directories

If you create an explicit subdirectory in a root that you've targeted for an Automatic Purge Routine, the subdirectory itself will not be deleted, even if you have enabled "Recursive purge." You will need to manually delete the subdirectory. In the case of an implicit subdirectory in NetStorage, this is not an issue. Since these subdirectories are "virtual," once the content within the target root is purged, these directories just cease to exist. For example, the path <CP code root>/uploads/a/b/c/ exists in a storage group in one of two ways:

• The entire path is explicit. This indicates that you manually created each subdirectory in the path explicitly, and are uploading content to it. If you target /uploads for a recursive purge, you will need to manually delete the subdirectories /a, /b and /c.

• Only "/uploads" is explicit. This indicates that you performed an upload operation, and set the path, /a/b/c in the request, thereby making those subdirectories implicit. When you target /uploads for purge, all of its contents will be deleted and the implicit path, /a/b/c will just cease to exist.

Purge routines take time

If the primary purpose of using auto-purge is to manage costs please be aware that this service runs best effort and may take days to run if the number of files to be scanned within the rule is greater than 1M files.

👍

Improve purge performance

We recommend reducing the scope of the rule to scan the smallest set of files possible.