Create a storage group
Add a storage group
Use the NetStorage wizard or Configuration API to create a new storage group. To begin, access the Storage Groups entity and click + Add New Storage Group to access a wizard.
- Open the application. Go to ☰ > ORIGIN SERVICES > NetStorage
- Access the Storage Groups entity and click + Add New Storage Group to access a wizard.
Here's a short summary. You can find more details below.
- Storage Group Details. Here you define the Storage Group Name and the Domain Name information for the storage group.
- Storage Group Name. Enter a name for the storage group. This is how it's identified throughout the UI and API.
- Domain Name. This value serves as a unique prefix for each of the domain names used to access this storage group. For example, it's included in your upload domain names used to access this storage group.
- Geo Replication Settings. These allow you to determine where your content will be replicated for added stability and increased availability. You must select two different geographic regions.
- Additional Domains (optional). You can optionally add domains to target specific regions.
- Directory Settings. These settings establish a root directory (an "Upload Directory") for the storage group. Your storage group needs at least one upload directory. If you don't already have one, you can generate one in the UI. You can also apply optional Security and Advanced Settings.
- Automatic Purge Routines (optional). Define a routine to target directories in your storage group and have content automatically deleted once certain conditions have been met.
- Finally, you need to review the Summary. All of the settings you've applied for the group are revealed, and you use this window to save them.
You can edit an existing storage group to modify settings as necessary. However, you must wait for a new storage group to complete provisioning before you can edit it.
Storage Group Details
Create a Name for the storage group to serve as an identifier, and define a unique Domain Name value for use in identifying points of access for the group.
- Access Control Group. An Access Control Group (ACG) is a grouping of products associated with a specific contract. (NetStorage is the product, in this case.) ACGs determine how service usage is logged and billed (if applicable), so make sure that the appropriate ACG is selected. If only one ACG is associated with your account, it will be default selected here.
- Storage Group Name. Input a desired name for the storage group. This is how it is identified throughout the UI. You should use a unique value that's easy to remember. For example, name the group based on the type or classification of media you wish to store in it.
- Domain Name. This value serves as a unique prefix for each of the access method upload domains. For example, if you provide the value
baseball, the upload domain for FTP would appear as
Domain name prefix value requirements
These rules apply to the prefix value:
- Special characters are not supported. Only alphanumeric characters ("a-z", "0-9") and the hyphen ("-") can be used.
- Uppercase characters are not supported. All alphabetic characters used in a domain name must be in lowercase.
You can't change this value after you create the storage group. Ensure that the prefix you choose is the one you wish to use with a storage group.
Differences between the storage group name and the domain name
When setting up a storage group, you define both of these values, you should understand the difference between the two:
- The Storage Group Name. This is a value you apply to the group to uniquely identify it.
- The Domain Name. This is a prefix value that you define (the "domain name prefix") that is pre-pended to unique domain name(s) you use to access the storage group.
You can use the same value for both your Storage Group Name and Domain Name (prefix), and this is a common practice. However, both must meet the value requirements discussed above.
Geo Replication Settings
Replicate your data in at least two different geographic regions (zones) for added stability. For best performance, ensure that both regions are geographically close. In general replica location should be tailored to where uploads come from, and where delivery performance is most important.
The default No Capacity Action will temporarily Spill Outside of your zone if there's no capacity available. This is the default action. Contact your account representative if you would like to Deny Upload to other zones.
You should see a message stating "Valid zone combination" if you have selected appropriately. By default, only two replicas are supported.
Three-way Replication and NetStorage
Storage groups default to the use of two replicas, or "two-way replication." NetStorage offers support for three-way replication, but it's not a standard offering. It can be provided after a valid business case consideration. If you have implemented it, be aware you'll have lower limits on the number of write operations per hour than standard two-way replication. This may delay serving content when requested.
Contact your account representative for additional details and availability information.
The term "write operation" includes any successful change to content or metadata stored in a storage group. This includes uploads, explicit directory creation, symlinks, renames, timestamp changes and deletes.
Supported Geo Replication Zones
Replicas should be nearby for best performance and stability. Available zones may change over time. Contact your account representative to make changes to existing replication configurations.
|global||Not recommended. Choosing this zone can't guarantee where your replica will reside.|
|useast||United States - East|
|uswest||United States - West|
You can mitigate temporary connectivity issues by using additional domains to target specific regions to have specific domains access a specific region. You can add additional domains during the creation or editing of a storage group. Each additional domain is appended to your primary domain name.
In this example, we append each region to the domain, and upload traffic is allowed and restricted as needed.
An upload directory is a root directory in a storage group, represented by a CP code that has been provisioned for use with NetStorage. This is also referred to as a "CP code root." The CP code is used to track billing usage as well as a reference for your use in reporting and gathering statistical data. When adding or editing a storage group, you use the Upload Directory field to select a CP code for use.
- If only a single CP code is available it will be default selected. In this case, a single CP code has been provisioned and is available for use as an upload directory.
- If multiple CP codes are available, select the appropriate one. In this case, multiple CP codes have been provisioned for use as an upload directory. Ensure that you select the appropriate one for use. Speak to your NetStorage administrator to determine the proper one, or contact your account representative.
- If no CP codes are available, you have multiple options. If there is nothing available in this drop-down, you currently have no CP codes provisioned. Perform either of the following:
- Contact your account representative. You can work with your representative to get one or more new codes provisioned. Once provisioning completes, you can restart this process and select the new CP code for use.
- Continue with creation of the storage group.. Leave this field blank. After you finish and save the storage group, a CP code will be generated and applied to it for use as the upload directory. (The code is added internally during the storage group's provisioning. This provisioning can take roughly 60-120 minutes.)
Upload Directory Settings
You use these settings to define and configure an upload directory for use with NetStorage.
An upload directory is a root directory in a storage group, represented by a CP code that has been provisioned for use with NetStorage. You later associate this upload directory with an upload account, to grant it access to this storage group.
Several settings can be applied and are broken down into various categories:
The settings here are used to select, or create the CP code to serve as the upload directory.
Click + Upload Directory to add a new upload directory.
Do you have a CP code provisioned and available for use?
Yes. Select the applicable one. If you have only one, it's selected by default. You can't create new CP codes using the UI if you already have one. If you want another CP code, try using the API to create a CP code.
No. You're given an option to create a new one. A CP code name can be up to 128 alphanumeric characters, but can't include
_ , # ' " %or non-printable characters. It can take up to two hours for a new CP code to provision.
Additional settings are revealed that allow you to configure the upload directory. We discuss them in the sections that follow.
Some options may be unavailable
Contact your account representative for assistance in applying settings that are shown but grayed-out.
These settings allow you to determine how various requests will be handled by the storage group.
|Download Security||These options are used to determine the level of security that should be applied to incoming download requests.|
As download requests come for content from your selected upload directory via Edge Servers, you can define the level of security that must be met to obtain access:
While this offers added protection for your content, it can take slightly longer for replication to process.
|SSL Replication||Content in all of your storage groups is replicated to other persistent systems within the NetStorage network to ensure accessibility (via Geo Replication settings applied in the storage group).|
When this option is set to “ON”, contents within the selected upload directory will be replicated securely via Secure Sockets Layer (SSL).
To view or change these settings for an existing upload directory:
- Open the application. Go to ☰ > ORIGIN SERVICES > NetStorage
- Access the Storage Groups entity and select the storage group you want to view.
- Click Edit to edit the storage group.
- Scroll down to the Upload Directory you want to edit and click "..." followed by Edit.
Your environment may have specific file-naming conventions pertaining to the use of upper vs. lowercase text. Some web services require that filenames use a specific case usage model. For example, a web-based database application may require that all file names be lowercase, but your environment requires the use of capital letters.
Enable the Force Case feature to change case-usage of your files during upload to match the needs of your workflow.
- This feature doesn't change the case during file access or requests, it changes the actual filename stored on NetStorage during the upload process.
- This feature doesn't automatically create subdirectories or apply Force Case rules to them.
With Force Case enabled, access requests to objects stored on NetStorage can impact how you Use Rsync.
NetStorage as your origin and the ignore-case-cache-key behavior
To use the ignore-case-cache-key behavior with NetStorage as the origin server, you need to configure NetStorage with the Force Case enabled to match the
ignore-case-cache-keysetting. Make sure you name the original files consistently as either upper or lowercase.
|Pass through all requests without modifying||With this option selected, filenames will remain as is, in regards to case usage when uploaded to the upload directory.|
|Convert all uploads to lowercase||This forces lowercase character use for all filenames when content is uploaded to the upload directory. |
|Convert all uploads to uppercase||This forces uppercase character use for all filenames when content is uploaded to the upload directory. |
NetStorage doesn't automatically apply the Force Case setting to Index requests
If using the Force Case option, you must manually name your index.htm file accordingly. For example, if the Force Case setting is set to Convert all requests to uppercase, then you would change the Index Name to INDEX.HTM instead of index.htm.
Before modifying case settings
Ensure any files that currently exist in the targeted storage group conform with the selected setting. For example, if Convert all requests to lowercase is selected, all existing files in the storage group must currently be all lowercase. If an option is selected, but existing files do not conform to the selected case setting, a denial of service will result for your end users.
Path Name Mode
NetStorage doesn't adhere to a traditional directory-based file system. As a result, characters such as “/” can be used within a directory name. For example, you could actually name a directory
/files, in which case, NetStorage would establish the path to the directory as
//files (adding an additional "/" as a path separator). Use this drop-down to define how path name requests made to the selected upload directory should be interpreted.
- Don't check on incoming paths. With this selected, paths are not checked. The path set in an upload or download request is acknowledged exactly as input.
- Disallow improper paths on upload: With this mode selected, paths using naming conventions outside of what is supported for use as an explicit or implicit directory name are blocked, and an error message is displayed on upload.
- Translate all incoming path names to a canonical directory-friendly format. With this option selected, a
/is seen as a path delimiter and repeat instances are treated as a single instance in an upload path. For example, if an upload request is sent to target the directory,
//files/new///mp4/movie1.mp4, the file is uploaded to
/files/new/mp4/movie1.mp4, and non-existent directories within the path are auto-generated to exist as explicit ones.
- Disallow improper paths on upload, but perform translations on download requests. This disallows improperly formatted path names on an upload request, but translates improper paths called out in download requests, similar to what is discussed for “Translate all incoming path names....”. For example, if a download request is issued for the path,
//files/new///mp4/movie1.mp4, NetStorage looks to the path,
/files/mp4/movie1.mp4to find the file.
Send Content MD5 Header
- On. A content item's MD5 digest values are sent in the HTTP “Content-MD5” response header.
- Off. If you are not concerned about sending this header data.
Query String Mode
If a request targeting the upload directory has a query string value appended to it, determine how the string should be handled. This applies to both upload and download operations targeting the upload directory.
- Strip all incoming query strings. With this selected, the string is removed from a request.
- Leave incoming query string as is. If this is selected, the string is left alone and interpreted, as applicable.
- Apply ACS query string conversion. With this mode enabled, as content is uploaded, its query string content is reviewed. A new, “stripped down” instance of the string is applied to it, based on what is defined in the accompanying fields (see below). Finally, a hash of this new string is generated and appended to the stripped down string. Once a download is requested for the content, the query string within that request is processed using the same conversion and the hashes are compared. If they match, access is granted. The accompanying fields within the NetStorage Configuration API are mutually exclusive and used as follows:
- Key Order. Input individual query string values that should be included, in the specific order they should be interpreted. Separate multiple entries with a single whitespace. Any string left out of the this field that exists in an actual query string is automatically excluded.
- Exclude. Input specific query string values that should be excluded from the conversion, separating multiple entries with a single whitespace.
If the final path component for an uploaded file is in excess of 200 characters after conversion, some of the stripped down query string will be truncated to meet the 200 character maximum.
Set this switch to "On" to enable use of the "quick-delete" operation available in the NetStorage Usage API and the CMShell. This allows you to target a specific directory in this storage group and recursively delete all of its content. Specific usage requirements and caveats apply.
Enabling this option changes the default CMShell behavior of the "rm -r" command. Enabling this option changes the command to operate as a “quick-delete” rather than its default setting.
Encoding, Key Order and Exclude
These options allow you to select and enforce the type of encoding to be used when displaying paths in XML-aware contexts. For example, this applies to call response output for the
stat calls via the NetStorage Usage API, as well as for file names and paths displayed in the NetStorage Groups UI. Via the Encoding drop-down, you can select from the following:
- iso-8859-1. This is the default. With this selected, file names and paths are encoded and displayed using iso-8859-1, 8-bit, single-byte coded graphic character sets. (This is the “Latin alphabet no. 1,” that consists of 191 characters from the Latin script.) This method treats all byte sequences as valid.
- UTF-8. With this selected, file names and paths are encoded and displayed using variable length, 8-bit code units via UTF-8 character encoding. UTF-8 sees some sequences as invalid. In this case, the XML encoding cannot represent file names that do not comply with the encoding. Therefore, the XML output changes the attribute from “X” to “X_base64” (and provides the raw octet stream as the base64-encoded value).
Enforce Encoding. When set to On, upload operations will verify that all path values defined within the URL for target content are properly formatted using the selected Encoding method. If the path is not properly formatted to meet the selected method, an error will be returned.
This only applies to the use of UTF-8 as an encoding method. If you have selected iso-8859-1, the Enforce Encoding slider is grayed-out and unavailable.
Serve from Zip
- On. NetStorage will dynamically examine archived files to directly serve individual files from within the archive.
- Off. Archive files will need to be completely decompressed to access and serve its contents. Specific usage conditions and requirements apply.
Index Name, Limit and Search on 404
These options, as well as the related, Search on 404 option account for the unique file system supported by NetStorage. They are used to define an "Index File" that should be accessed for requests that do not end in a specific file name and extension. (For example, a request for an object that ends in a trailing slash -- "/".) The "Index File" is similar in functionality to that of Apaches "directory index" auto-index feature. Use these fields as follows:
|Index Name||A file specified here will be served if the request does not end in a specific file name and extension. Use single white-spaces to include multiple entries, the system will use the first entry found.|
The file(s) named here is served for HTTP download requests that end with a trailing slash (“/”), as well as those that actually end in the value input here. NetStorage doesn't automatically apply the Force Case setting to Index requests. If using the Force Case option, you must manually name your index.htm file accordingly. For example, if the Force case setting is set to Convert all requests to uppercase, then you must set to the filename to INDEX.HTM instead of index.htm.
|Index Limit||If desired you can truncate or hide the file path displayed in the HTML directory listing for the Index Name file. Input an integer value here that represents the number of directories in a path that should be revealed:|
|Search on 404||Select the action that should be taken in the event of a 404 error. (The client was able to access NetStorage, but the requested content or directory could not be found.)|
Automatic Purge Routines
Automatic purge routines perform a routine scan of a target directory, and once a specified capacity is reached, the system will begin to purge files that are older than a date you specify.
Automatic Purge functionality is not supported for storage groups used by iPhone Live configurations. If the group is associated with this type of configuration, this option will be grayed-out, and you will need to set up automatic purge at the stream level. Please contact your account representative for additional assistance.
Set up a purge routine
To set up a routine, click the + Automatic Purge button and set these options as necessary:
|Upload Account||This is the upload account that will use this purge. The upload accounts listed have been associated to, and are used to access a specific upload directory within a storage group.|
|Upload Directory||This defaults to the root of the upload directory associated with the selected upload account. You can target pre-existing subdirectories to be purged as needed.|
|Purge When The Directory Reaches||Input a non-zero, positive decimal and select the appropriate file size type from the drop-down (Bytes, KB, MB or GB). When the specified value is reached, the purge will be initiated.
|Recursive Purge||With this option enabled, subdirectories within the specified upload directory will also be targeted, and their contents will be purged. If left disabled, only files within the actual Upload Directory will be purged. Purge routines don't delete empty, explicit directories and must be deleted manually.|
|Only Purge Content Older Than (in days)||Input a non-zero, positive integer to represent the desired number of days. (For example, insert “10” to represent files 10 days older than the current date.) Once the value set in the "Purge When The Directory Reaches" field is reached, files older than this date will be purged.|
If no content is older than what you’ve specified here, nothing is purged--even if the directory size exceeds what is defined in the Purge When The Directory Reaches field.
|File Exclusion Regular Expression||Using POSIX Extended Regular Expression syntax, input file names/types that should be excluded from the purge. Some basic examples of use include the following:
|File Selection Regular Expression||Using POSIX Extended Regular Expression syntax, input file names/types that should explicitly be included in the purge. These file types/names will be targeted first in the list of applicable files for the purge. The same examples cited above apply with this option as well, except those files would be included.
Purge routines don't delete empty, explicit directories
If you create an explicit subdirectory in a root that you've targeted for an Automatic Purge Routine, the subdirectory itself will not be deleted, even if you have enabled "Recursive Purge." You will need to manually delete the subdirectory. In the case of an implicit subdirectory in NetStorage, this is not an issue. Since these subdirectories are "virtual," once the content within the target root is purged, these directories just cease to exist. For example, the path
<CP code root>/uploads/a/b/c/ exists in a storage group in one of two ways:
The entire path is explicit. This indicates that you manually created each subdirectory in the path explicitly, and are uploading content to it. If you target
/uploadsfor a recursive purge, you will need to manually delete the subdirectories
Only "/uploads" is explicit. This indicates that you performed an upload operation, and set the path,
/a/b/cin the request, thereby making those subdirectories implicit. When you target
/uploadsfor purge, all of its contents will be deleted and the implicit path,
/a/b/cwill just cease to exist.
Purge routines take time
If the primary purpose of using auto-purge is to manage costs please be aware that this service runs best effort and may take days to run if the number of files to be scanned within the rule is greater than 1M files.
Improve purge performance
We recommend reducing the scope of the rule to scan the smallest set of files possible.
Updated 2 months ago