Use the Content Management Shell (CMShell)

The Content Management Shell (CMShell) allows you to log in to a shell-like interface to manage content in NetStorage, using a secure environment.

CMShell configuration requirements

CMShell requires a client that supports SSH protocol 2 (DSA/RSA). Use any client that provides secure, persistent sessions to securely connect to a storage group. CMShell supports both interactive login and single command execution. This includes applications such as “PuTTY” and “WinSCP.” You can use CMShell after you add an SSH key to your upload account.


Configure “PuTTY” for CMShell

To configure an SSH client, you'll need various items and information:

  • The “private” instance of your SSH key. This key resolves with the “public” key that you applied to the upload account. This file must be locally accessible.
  • The storage group "domain name prefix". This is the storage group you will be accessing.
  • The secure login username. This is always “sshacs” for secure NetStorage access.
  • The client must support “remote commands”. Your client needs to append cms as a remote command to the connection call.
  • The password assigned to the private SSH key. If you assigned a password during key generation, you will need to know it.

Apply an SSH key to the PuTTY application

  1. Launch PuTTY
  2. Ensure the Category is set to Session, and input the following in the Host Name (or IP address) field: <Domain Name Prefix>.ssh.upload.akamai.com
  3. Set the Connection Type to SSH.
  4. Select the Connection > Data sub-category, and set the Auto-login username to “sshacs” (minus the quotes). This value must be used.
  5. Select the Connection > SSH sub-category and set the following:
    1. Remote command - Input “cms” (minus the quotes).
    2. Preferred SSH protocol version - Ensure that “2” is selected.
  6. Open the Connection > SSH sub-category and select its Auth sub-category to set following:
    1. Private key file for authentication - Click the Browse button to navigate to the associated Private key.
  7. Recommended: Return to the Session category, input a desired name in Saved Sessions (“CMShell Access,” for example) and click the Save button. This allows you to quickly load the session for connection in the future.
  8. Click Open to launch the session in a separate session window.
  9. If you have established a Key Passphrase for the SSH Key pair, input it in the session window.
    With the connection open and resolved, you can now issue CMShell commands.


Access CMShell with an SSH client

Here is an example of an interactive login

In this example shows how to access the CMShell by including specific login information. After the login is resolved, you will be able to issue multiple commands.

    ssh -t -i <Private-key> <Username>@<Domain Name Prefix>.ssh.upload.akamai.com cms

These variable values apply:

  • <Private-key>: This is the full path to your private key.
  • <Username>: You must use sshacs as the username. This is the case for all secure access methods in NetStorage.
  • <Domain Name>: This value followed by .ssh.upload.akamai.com represents your full Upload Domain Name. You set up your domain name during creation of the target storage group.

Once this login is accepted, you will be free to launch multiple CMShell commands, as desired.

Here is an example of single command execution

In this example, we show you how to access the CMShell by including specific login information within an individual command. (This allows you to login and issue a single command.)

ssh <Username>@<Domain Name Prefix>.ssh.upload.akamai.com <command line>

The following variable values apply:

  • <Username>: When providing the Username, the value sshacs must be used. (This is the case for all Secure Access Methods in NetStorage.)
  • <Domain Name Prefix>: This value followed by .ssh.upload.akamai.com represents your full Upload Domain Name. You set up your Domain Name during creation of the target storage group.
  • <command line>: The single command you wish to execute.

CMShell command syntax

Syntax basics

Syntax, Objects, and VariablesDescription
Elipses (“...”)This indicates that the component that precedes it can be repeated. For example, multiple instances of the component can be provided to, include multiple sources for moving, copying or deleting.
Square Brackets ( [ ] )The content within must be included in square brackets. In some instances, an individual component must exist in a single set of brackets ([-a]), while in others, multiple components can be included in a single set ([-aldc]). Unless specifically noted in a command’s description, only a single option/component can be included in a set of square brackets.
Angle Brackets ( < > )Unless specifically noted in the description of a command, angle brackets indicate a variable value that you must provide. In these cases, the < > characters are not included in the syntax.
PathsAny path in the syntax targeting a storage group must be prefaced by the appropriate numeric “CP Code Root.” This is represented as “123456” in these examples.

🚧

Operations are restricted to existing directory trees

CMShell requires that all operations use pre-existing paths. CMShell doesn't create a non-existent directory during command operations.

Regarding Option FormatsWith some options, you can achieve the same result by inputting the short or long format. A command may be revealed options as follows:
  • -a, --action- When you see this as an Option type in a table, you can include either -a or --action to accomplish the same result.

  • -a, --action=<variable> - You can use either option format to accomplish the same result, but both must be accompanied by an appropriate variable (For example, -a=<variable> or --action=<variable> achieve the same result.)

Wildcard SupportCMShell supports the following wildcards:
  • *: Asterisk ( * )
  • ?: Question mark ( ? )

Command responses

Once a CMShell command is issued, one of three responses will be returned after processing:

ResponseDescription
“0”This indicates the command was executed successfully.
“1”An error has occurred. Check that the syntax of the command was formatted properly, and re-issue it. If issues still arise, contact Technical Support.
“255”The SSH connection has failed. Verify that you are properly connected via SSH, and try again.

CMShell commands

CommandDescription
az2zThis command creates a Serve from Zip compatible zip archive.
catUse this command to concatenate a file or files and display the result; or concatenate multiple files into a single output file.
cdUse this command to change to a different remote directory.

The az2z command

This command creates a Serve from Zip compatible zip archive.

Use this command to rewrite a ZIP archive file to Serve from Zip file format (which will verify the file’s integrity and create a hash table for it). The “<FILE>” variable should be set to the complete path and filename for the target file.

 az2z <OPTION> <FILE>
OptionDescription
-h, --helpDisplay help information for this command.

The “cat” command

Use this command to concatenate a file or files and display the result; or concatenate multiple files into a single output file.

cat <OPTION> <FILE>...
OptionDescription
-h, --helpDisplay help information for this command.
Example: Concatenate one or more files

Include one or more instances of the <FILE> parameter, setting each to the complete path and file name to the desired file for concatenation. Resulting concatenated files will be housed in the current active directory.

cat <FILE> <FILE>...
cat /123456/file01 /123456/file02
Example: Concatenate multiple files into a single output file

If you need to process several files as one and save the results of such processing to a single output file, this command can be applied. Include one or more instances of the <FILE> variable, setting each to the complete path and filename for the desired file, and set the <OUTPUTFILE> variable to a desired path and filename for the resulting concatenated output file.

cat <OPTION> <FILE1> <FILE2> <FILE3>... > <OUTPUTFILE>
cat /123456/file01 /123456/file02 /123456/file03 > /123456/completed/output_file

🚧

With this instance of the command, you must use the right angle bracket (>) within the syntax. The final path/file to be concatenated must be followed by a single whitespace, the >, a second whitespace, and finally the filename of the concatenated file. All other instances of < > in the example that follows indicate a variable value.


The “cd” command

Use this command to change to a different remote directory.
Specify the full path to the desired directory as the “<DIRECTORY>” variable.

    cd <DIRECTORY>

While not specifically “options,” these features are also available:

  • Include cd standalone to return to your established root directory.
  • Include a whitespace followed by two periods to jump up a level in the directory tree (For example, cd ..)
cd <DIRECTORY>
cd /123456/secondary

The “cp” command

Use this command to copy one or more files or directories.

To copy a file, include any “<OPTION” variables along with the “<SOURCE>” path and filename of the file to copy. You can include multiple “<SOURCE>” file entries with a whitespace. Include the "<DIRECTORY>” for the file destination.

    cp <OPTION> <SOURCE> <DIRECTORY>
OptionDescription
-i, --interactiveAsk before overwriting existing files.
-p, --preserveCopy the attributes for each file, as well.
-rUse recursion, creating files from non-directories.
-v, --verboseDisplay the operation’s execution step by step.
-h, --helpDisplay help information for this command.

📘

You can use absolute or relative paths, so long as they resolve within the same top-level directory.

Example: Copy two files from a test directory into a release directory while preserving their attributes

cp <OPTION> <SOURCE> <DIRECTORY>
cp -p /123456/test/movie1.mp4 /123456/test/movie2.mp4 /123456/release/

🚧

Operations are restricted to existing directory trees

Your destination <DIRECTORY> must use a pre-existing path, as this command doesn't create a non-existent directory. For example, because the “non-existent” directory doesn't exist, this command will fail: cp /123456/movie.mp4 /123456/non-existent/


The “debug” command

While shown as available via the command interface, this command is currently not supported for use.


The “du” command

Use this command to estimate how much disk space each specified file or subdirectory uses.

Specify one or more individual files or sub-directories by including the complete path to the desired item as a “<FILE or SUBDIRECTORY>” variable. Separate each entry with a whitespace. Output will summarize each “<FILE or SUBDIRECTORY>” variable entry as disk usage in bytes.

    du <OPTION>... <FILE or SUBDIRECTORY>...
OptionDescription
--block-size=<size>Use a block size other than the default.
-fInclude the number of files inside subdirectories.
-S, --separate-dirsOmit subdirectory sizes.
-s, --summarizeDisplay totals for the specified file(s) and/or subdirectory(ies) only.
--max-depth=<#>Display totals for the specified file(s) and/or subdirectory(ies), and also for any subdirectories if they are within <#> sublevels of any specified directory.
-h, --helpDisplay help information for this command.

Example: Target a subdirectory and file

The --max-depth=<#> option is incorporated to set the maximum sub-directory depth to four.

du <OPTION>... <FILE or SUBDIRECTORY>...
du --max-depth=4 /123456/test/movie1.mp4 /123456/test/primary

📘

NetStorage supports a limited set of options for this command. NetStorage doesn’t support command options not listed here.


The “exit” command

Use this command to terminate the CMShell session.

exit

There are no options available with this command.


The “help” command

Use this command to display CMShell help.

Issued standalone (help), a list of available CMShell commands will be displayed. Include the name of a specific CMShell command as the “<COMMAND>” variable (for example, help unzip), and any available help for the specified command will be revealed.

    help <COMMAND>

There are no additional options available for use with this command.


The “hostname” command

Use this command to display the IP address of the remote machine.

hostname

There are no additional options available for use with this command.

Example: Display the IP of the remote machine

hostname
[writingteam /]$ hostname
72.246.17.108

The “ln” command

Use this command to create a symbolic link to a file or a directory.

NetStorage doesn't support the creation of hard links. When using the ln command in CMShell, creation of a symlink is implied. (The -s option is applied by default.) A symbolic link is a file that points to another file, much like a shortcut. You can point a symlink to a file, directory, or both.

    ln <OPTION> <TARGET> <LINKNAME>
OptionDescription
-f, --forceUnlink the target file if it already exists, so that the link may occur. (The -f option overrides any previous -i options.)
-n, --no-dereferenceTreat LINKNAME as a normal file if it's a symbolic link to a directory.
-s, --symbolicCreate symbolic links instead of hard links. This is applied by default.
-v, --verboseDisplay the operation’s execution step by step.
-h, --helpDisplay help information for this command.

Example: Target a single file and directory for symlink creation

ln <OPTION> <TARGET> <LINKNAME>
ln -v /123456/videos/movie.mp4 /123456/videos/subtitles /123456/symlinks

Example: Create a symlink to a single file

Include a single “<TARGET>” variable, defining it as the complete path to a desired file. The system will create a symbolic link using the value defined as the "<LINKNAME>" variable.

To create a symlink in your current directory for a file located at /123456/videos/movie.mp4 called symlink.mp4, view the Example CMShell command tab.

The targeted /123456/videos/movie.mp4 file is accessible by using the new symlink located at /123456/symlink.mp4. Performing a list (ls -l) shows the symlink relationship to the target as shown in the Example list output tab.

ln <OPTION> <TARGET> <LINKNAME>
ln -v /123456/videos/movie.mp4 symlink.mp4
[123456]$ ln -v /123456/videos /123456/symlinks
create symbolic link /123456/symlinks to /123456/videos
[123456]$ ls -l
    lrwxrwxrwx storageGroup  storage           24 Mar 26 20:43 symlink.mp4 -> /123456/videos/movie.mp4
    drwxrwxr-x storageGroup  storage            0 Mar 26 20:21 videos

Example: Create a symlink to a single directory

Include a single “<TARGET>” variable, defining it as the complete path to a desired directory. The system will create a symbolic link using the value defined as the "<LINKNAME>" variable.

When you create a symlink to a directory, it will display all content of the target directory. To create a symlink in your current directory for a directory located at /123456/videos, view the Example CMShell command tab.

You can access all files in the targeted directory by using the new symlink directory.

ln <OPTION> <TARGET> <LINKNAME>
ln -v /123456/videos /123456/symlinks
[123456]$ ln -v /123456/videos /123456/symlinks
    create symbolic link /123456/symlinks to /123456/videos
[123456]$ ls -l
    lrwxrwxrwx storageGroup  storage           14 Mar 26 21:07 symlinks -> /123456/videos
    drwxrwxr-x storageGroup  storage            0 Mar 26 20:21 videos
[symlinks]$ ls -l
    -rw-rw-r-- storageGroup  storage        78612 Mar 26 20:08 movie.mp4

Example: Create a symlink to multiple targets

Include multiple “<TARGET>” variables — defining each as the complete path to a desired file or directory, for which individual symbolic links will be created, and placed in the path named for the “<DIRECTORY>” variable.

ln <OPTION> <TARGET> <TARGET> <DIRECTORY>
ln -v /123456/videos/movie.mp4 /123456/videos/subtitles /123456/symlinks
123456]$ ln -v /123456/videos/movie.mp4 /123456/videos/subtitles /123456/symlinks
create symbolic link /123456/symlinks/movie.mp4 to /123456/videos/movie.mp4
create symbolic link /123456/symlinks/subtitles to /123456/videos/subtitles
/123456/symlinks$ ls -l
lrwxrwxrwx storageGroup  storage           24 Mar 26 21:41 movie.mp4 -> /123456/videos/movie.mp4
lrwxrwxrwx storageGroup  storage           24 Mar 26 21:41 subtitles -> /123456/videos/subtitles

The “ls” command

Display the contents of a directory.

This command is used to display the contents of a directory, where the`” variable should be populated with the complete path to the desired directory.

    ls <OPTION>... <PATH>...
OptionDescription
-l, --listInclude additional file information (e.g., permissions and ownership).
-Q, --quote-namePlace entry names in double quotes.
-r, --reverseSort in reverse order.
-SUse file size to sort the list.
-tUse modification time to sort the list.
-ULeave the list unsorted and display the contents in directory order.
-XUse entry extension to sort alphabetically.
-h, --helpDisplay help information for this command.
ls <OPTION>... <PATH>...
ls --list -S /123456/test /123456/new

The “md5sum” command

The md5sum command performs a raw byte analysis to display the MD5 digest for one or more files.

The “ <FILE>” variable should include the complete path and filename for the target file. This command doesn't provide a hash digest for symlink directories.

    md5sum <OPTION> <FILE>...
OptionDescription
-rExecute the command recursively.
-h, --helpDisplay help information for this command.

Example: Display the MD5 digest for a single file

md5sum <OPTION> <FILE>...
[123456 /]$ md5sum /videos/movie.mp4
[123456 /]$ md5sum /videos/movie.mp4
d81f21f9d0ac51621cad3de05ff734c6  /videos/movie.mp4

📘

If you run this command after [The “az2z” Command] (This command creates a Serve from Zip compatible zip archive."), its hash will differ from that of your local version.


The "mkdir" command

Use this command to create one or more new directories.

Include one or more instances of the <DIRECTORY> variable (separating each with a whitespace), and set each to the complete path to the new directory to create.

mkdir <OPTION> <DIRECTORY>...
OptionDescription
-pIf parent directories in the specified path do not exist, automatically generate them to accommodate (and do not generate an error).
-vDisplay the operation's execution step by step.
-h, --helpDisplay help information for this command.

📘

The -p and -v options can't be used together.

Your CMShell directory structure

Your upload account name and default upload directory CP code determine your root directory. When you connect to CMShell, your base directory will appear as [uploadAccountName], displaying the upload account that's connected. On the back-end, it translates to the associated upload directory code.

If directories within a named path don't exist, an error will be generated. To have CMShell automatically create non-existent directories in a path, be sure to include the -p or --parents option.

Example: Create a directory

In this example, the verbose -v option is included, and [writingteam /] is the root folder for the upload account named writingteam.

mkdir <OPTION> <DIRECTORY>...
[writingteam /]$ mkdir -v films
mkdir: created directory `films'

Example: Create a nested directory

Example creation of nested directory structure. The -p option will create any folders that don't already exist. This example combines mkdir with the -p option.

[writingteam /]$ mkdir -p /tv/shows/

The "mv" command

Use this command to rename a single file, or move a single/multiple files.

With this command these points apply:

  • Any directory named within a path for a rename or move must preexist. CMShell will not automatically create non-existent directories. If applicable, see The "mkdir" command for details on creating a directory.
  • Only files can be targeted as a <SOURCE> for a Rename/Move. Directories and sub-directories cann't be renamed or moved.
mv <OPTION>... <SOURCE> <DESTINATION>
OptionDescription
-i, --interactiveAsk before overwriting existing files.
-vDisplay the operation's execution step by step.
-h, --helpDisplay help information for this command.

Example: Rename a file

To rename a file, set the complete path and filename of the target file as the <SOURCE> variable, and include the desired path and new filename as the <DESTINATION> variable. The path set within the <DESTINATION> variable can either be the existing path, to keep the file in the same directory; or a new path, in order to also move the renamed file.

In this example the file movie.mp4 is renamed to movie_old.mp4 with the mv command.

mv <OPTION>... <SOURCE> <DESTINATION>
mv -i /123456/test/movie.mp4 /123456/test/movie_old.mp4

Example: Move a file

To move a file, set the <SOURCE> variable to the complete path and filename of the file to be moved, and set the <DIRECTORY> variable to the path to which the file is to be moved. Multiple <SOURCE> file entries can be included to move multiple files (separating each with a whitespace), with the final path in the syntax existing as the <DIRECTORY> to which the files will be moved.

In this example, two files, movie.mp4 and movie1.mp4 are moved to a new directory, preserving their filenames.

mv <OPTION>... <SOURCE>... <DIRECTORY>
mv -v /123456/test/movie.mp4 /123456/test/movie1.mp4 /123456/new/

The “pwd” command

Use this command to display the path of the current remote directory.

pwd
OptionDescription
N/AThere are no options for this command.

Example: Show the present working directory

This example displays the present working directory, when the current directory is movies.

pwd
[writingteam movies]$ pwd
/movies

The "rm" command

Use this command to delete one or more files or directories.

Include one or more instances of the <FILE> variable, separating each with a whitespace, and set it to the complete path to be deleted. Each <FILE> variable can be either of these:

  • End with a directory. This will delete it, as well as all of its associated contents. This includes all sub-directories and file contents.
  • End with a specific filename. This will delete an individual file.
    rm <OPTION>... <FILE>...
OptionDescription
-f, --forceThis is applicable if you have defined a list of separate <FILE> entries and one (or more) among them does not exist. The removal will continue without prompting you. To use in conjunction with -r, quick-delete must be enabled.
-i, --interactiveAsk before removing files. If targeting a complete directory for deletion it is recommended that you incorporate “interactive” mode as an option, to avoid inadvertently deleting a wanted file. Additionally, when enabled, your confirmation response must begin with “ y” or “ Y”, otherwise the file will be skipped.
-r, --recursiveThis option performs a "quick-delete" to remove content recursively. Review quick-delete considerations when using this option.
-v, --verboseDisplay the operation’s execution step by step.
-h, --helpDisplay help information for this command.

Example: Delete a single file and directory

This example targets a single file and an empty directory for deletion.

rm <OPTION>... <FILE>...
rm --interactive /123456/test/movie_old.mp4 /123456/old/

The “rmdir” command

Use this command to delete empty directories.

Include one or more instances of the <DIRECTORY> variable (separating each with a whitespace), and set each to the complete path to the directory to be removed.

    rmdir <OPTION> <DIRECTORY>...
OptionDescription
-v, --verboseDisplay the operation’s execution step by step.
-h, --helpDisplay help information for this command.
--quick_delete=imreallyreallysure1Remove a directory and all of its contents.

📘

Targeted directories must be empty to use this command

To remove a directory and all of its contents, you must enable quick-delete and use the --quick_delete=imreallyreallysure <path> option.

Example: Remove two empty directories

This example removes the movies and tv directories.

rmdir <OPTION> <DIRECTORY>...
rmdir /123456/movies 123456/tv
rmdir --quick_delete=imreallyreallysure /123456/movies
[writingteam /]$ rmdir --quick_delete=imreallyreallysure movies
quick_delete for "movies" Error: Permission denied

The “stat” command

Use this command to display information on one or more files, or a file system.

stat <OPTION> <FILENAME> [<FILENAMES>...]
OptionDescription
-h, --helpDisplay help information for this command.

Example: Stat of a single file

This example performs a stat of the movie.mp4 file.

stat <OPTION> <FILENAME>
[writingteam movies]$ stat movie.mp4
File: movie.mp4
Size: 41392
Filetype: File
Mode: (0664/-rw-rw-r--)
Uid: writingteam
Gid: storage
Modify: Tue Jun  9 23:28:59 2020
Md5sum: d81f21f9d0ac51621cad3de05ff734c6

Example: Stat of a directory

This example performs a stat for the films directory.

stat <OPTION> <DIRECTORY>
[writingteam]$ stat films
File: films
Size: 0
Filetype: Directory
Mode: (0775/drwxrwxr-x)
Uid: writingteam
Gid: storage
Modify: Fri Dec 11 17:36:28 2020
Md5sum: -

The “wget” command

Use this command to download files from remote servers.

This command allows you to perform an immediate download of content from one or more specified <URL> entries, to the currently active directory. Prior to initiating this call, use either the cd command to navigate to an existing directory; or use the mkdir command to create a new directory and then use the cd command to access it.

wget <OPTION>... <URL>...

Available options

The <OPTION> variable displayed in the above example can be populated with one or more of the following options:

OptionDescription
StartupUse these options for the startup phase.
-B, --base=<URL>When a wget download is initiated using both the -F and -i options, a specific file of URLs is targeted, and the format of that file is to be read as HTML. Include this option and specify a desired <URL> to be pre-pended to relative links found in this file, if applicable.
-h, --helpDisplay in-line Help for the call.
Logging and Input FileUse these options to specify log and input files.
-F, --force-htmlInclude this in addition to the -i option to denote that its targeted <FILE> should be treated as HTML format.
-i, --input-file=<FILE>Download from a list of URLs specified in a file input as the <FILE> variable. This file must exist on the local system. Ensure that the complete path to the this file, including file name and extension are included.
-o, --output-file=<FILE>Include to have log messages saved to the named <FILE>. (Ensure that you include the appropriate extension.)
-q, --quietWhen included, no output will be generated.
-v, --verboseDisplay the operation’s execution step by step.
DownloadThese options are for download operations.
--connect-timeout=<#>Set the <#> variable to a value, in seconds that should serve as a connection timeout.
--dns-timeout=<#>Set the <#> variable to a value, in seconds that should serve as a DNS timeout.
--escape-filename-ctrlInclude this option to escape control characters in targeted filenames.
--limit-rate=<RATE>To set a download rate limit, include this option and set the <RATE> variable to a desired rate (in Mbps)
-N, --timestampingWhen this option is included, files will not be retrieved if their timestamp is older than that of an existing instance of the file.
--no-clobberDo not overwrite existing files.
-O, --output-document=<FILE>Include output information in a file specified as the <FILE> variable.
-Q, --quota=<#>Include this to set a file retrieval quota to a specific number, defined as the <#> variable.
--password=<PASSWORD>Include this (along with the --user option) and set its variable to the appropriate password if username/password values are required to access content at the specified URL.
--random-waitInclude to set a random wait time between retrievals -- from 0-2 seconds.
--read-timeout=<#>Set the <#> variable to a value, in seconds that should serve as a read timeout.
--retry-connrefusedInclude this option to attempt a retry if the connection is refused.
-S, --server-responseInclude this option to have the server response displayed for each retrieval.
--spiderInclude this option do block all downloading (i.e., if you wish to use other options to review information).
-T, --timeout=<#>Include this option to define a read timeout. Set the <#> variable to the desired time, in seconds.
-t, --tries=<#>Set the <#> variable to the number of retries that should be attempted before reporting a failure (“0” = unlimited)
--user=<USER>Include this option and set its variable to the appropriate user name if username/password values are required to access content at the specified URL.
--upload-quota=<#>Include this option to define a <#> to serve as the maximum quota of uploads. (The default = 1,000,000.)
-w, --wait=<#>Include this option and set the <#> variable to define a wait time between retrievals, in seconds.
--waitretry=<#>Include this option and set the <#> variable to define a wait time between retries of a retrieval, in seconds.
DirectoriesThese options control directory handling.
-nd, --no-directoriesWhen included, directories will not be created. All files captured in the wget will be copied directly in to the active directory.
-nH, --no-host-directoriesWhen included, host directories will not be created.
-P, --directory-prefix-<PREFIX>When included, files will be saved to a subdirectory of the active directory. Define this sub-directory using the <PREFIX> variable).
-x, --force-directoriesInclude to force the creation of directories. Directory paths within the target URL will be automatically created in the active directory.
--protocol-directoriesUse the protocol name as a directory component of local file names. For example, with this option, ‘wget -r http://host’ will save locally as ‘http/host/<content>’ rather than just to ‘host/<content>
--cut-dirs=numberIgnore number directory components. This can be combined with -nH to simplify directory depth.
HTTP optionsThese options set how to handle various HTTP operations.
--header-<STRING>Include this value and define a <STRING> value to have this content included in headers located via the targeted URL.
--http-passwd=<PASSWORD>Set the HTTP password to the value set as the <PASSWORD> variable.
--http-user=<USER>Set the HTTP username to the value set as the <USER> variable.
-E, --html-extensionSave HTML documents with an ".html" extension.
--ignore-lengthIgnore the Content-Length header.
--ignore-robotsInclude this option to ignore any “robots.txt” files and the “metatag”.
--load-cookies=<FILE>Load cookies from the target <FILE> before initiating any retrievals
--load-regex=<FILE>Follow URLs within the specified <FILE> that match regular expression.
--no-cacheInclude to block references to the local cache as a result of accessing the target URL.
--no-check-certificateInclude to block the checking of local certificate files as a result of accessing the target URL.
--no-cookiesInclude to block the loading of any cookies as a result of accessing the target URL.
--no-http-keep-aliveInclude this option to disable persistent connections)
--no-redirect-linkInclude this option to disable the use of redirect links.
--referer=<URL>Include this and set the <URL> variable for use as an applicable referer site
--save-cookies=<FILE>Save all cookies to a specific file, named as the <FILE> variable. This file will be saved in the active directory along with the wget content.
--save-headersInclude this option to have headers in target content saved. They will be saved in the active directory along with the wget content.
FTP optionsThese options are available for FTP connections.
--ftp-password=<PASSWORD>To utilize FTP to process the wget include the applicable FTP password necessary to access the content.
--ftp-user=<USER>To utilize FTP to process the wget include the applicable FTP password necessary to access the content.
-g, --glob=<ON/OFF>Include and set the variable to ON to turn on file name globbing. By default, “globbing” with wget is set to OFF.
Note: While this is supported, be aware the sequence of the resulting output is not guaranteed to be in an expected (alphanumeric) order.
--passive-ftpInclude this option to set the transfer mode to “passive”. Default FTP mode is “active”.
--retr-symlinksIf incorporating any of the Recursive Retrieval options, include this option for FTP to download linked-to files rather than directories.
Recursive RetrievalThese options specify recursive traversal.
-r, --recursiveWhen included, the wget will recursively traverse subdirectories in order to obtain all content.
Note: By default, the maximum recursion depth for this option is ten levels from the target directory ( /<TARGET>/1/2/3/4/5/6/7/8/9/10). To deviate from this default, include the -L, --LEVEL=<#> option to specify the desired depth.
-k, --convert-linksInclude this option to convert non-relative links to relative.
-K, --backup-convertedWhen converting a file, back up the original version with a ‘.orig’ suffix. Affects the behavior of ‘-N’ Time-Stamping.
-l, --level=<#>Limit recursion depth to a specific number of levels, by setting the <#> variable to the desired number.
-m, --mirrorThis is used to perform a “mirror” copy of the target directory. This option turns on recursion and time-stamping, defaults to a recursion depth of ten levels ( <target directory>/1/2/3/4/5/6/7/8/9/10) and keeps FTP directory listings.
Note: To deviate from the default ten level recursion depth, include the -L, --LEVEL=<#> option to specify the desired depth.
-p, --page-requisitesInclude this option to ensure that all files required to properly display a targeted HTML file are also included (images, scripts, etc.).
-z, --convert-absoluteInclude this option to convert non-absolute links to absolute.
Recursive Accept/RequestAll of the options discussed here only apply if the -r(or --recursive) Recursive Retrieval option has been included with this command.
-A, --accept=<LIST>When using recursion, you can specify specific file extensions for file formats you wish included in the wget and all other file formats will be ignored. Set the <LIST> variable to a comma-separated list of desired extensions, with no whitespaces between each entry. For example, -A=htm, jpg,mp4 could be input to include only these file formats.
-R, --reject=<LIST>When using recursion, you can specify specific file extensions for file formats you wish excluded from the wget and all other file formats will be included. Set the <LIST> variable to a comma-separated list of desired extensions, with no whitespaces between each entry. For example, -A=htm, jpg,mp4 could be input to specifically exclude these file formats.
-D, --domains-<LIST>Set the <LIST> variable to a comma separated list of domains that should be included in the wget , and only these domains will be included.
--exclude-domains=<LIST>Set the <LIST> variable to a comma separated list of domains that should be excluded from the wget.
--follow-tags=<LIST>Set the <LIST> variable to a comma separated list of HTML tags that correspond to content that should be included in the wget.
--follow-ftpWhen included, FTP links within targeted HTML content will be found (and the content there will be included in the wget as well).
-G, --ignore-tags=<LIST>Set the <LIST>variable to a comma separated list of HTML tags that correspond to content that should be excluded from the wget.
-H, --span-hostsWhen included, recursion will traverse links to foreign locations (and include content from those locations in the wget).
-I, --include-directories=<LIST>Set the <LIST> variable to a comma-separated list of specific directories that should be allowed for inclusion.
-L, --relativeInclude this option to have recursion only include content from relative links.
--no-parentWhen included, the recursion of directories will not ascend to the parent directory.
-X, --exclude-directories=<LIST>Set the <LIST> variable to a comma-separated list of specific directories that should be blocked from inclusion.

The “unzip” command

Use this command to perform various operations on a ZIP archive file’s contents.

The “<ZIPFILE>” variable is the complete path and filename of the Zip file to be targeted, while the “ <FILE-or-DIRECTORY>” variable should be the file or directory that will be the target of the operation. This is all based on the selected “ <OPTIONS>” and “ <MODIFIERS>” set.

unzip <OPTIONS> [MODIFIER] <ZIPFILE> [<OPTIONS>] <FILE_or_DIRECTORY>

📘

Only a single hyphen (-) is required when calling out more than one option (e.g., -fldv would be set to include the -f, -l, -d and -v options in a single command.

OptionDescription
-fUncompress only files that pre-exist on the disk and that are updated, use them to refresh the existing files, and if the files do not pre-exist, do not uncompress them and create them on the disk.
-lDisplay the ZIP archive file’s contents using a shortened format.
-dUncompress files to directory.
-pDisplay the contents of all files contained in the ZIP archive file using binary format.
-tCheck the ZIP archive file’s contents for errors.
-uUncompress all files, updating existing ones, and creating new ones as necessary.
-vDisplay the ZIP archive file’s contents using the verbose (long) format.
-xDo not extract the file or files that follow.
-zShow the comment in the ZIP archive file, if present.
-h, --helpDisplay help information for this command.

Available modifiers

The <MODIFIER> variable displayed in the above example can be populated with one of the following modifiers:

ModifierDescription
-aIf files are recognized as text, extract them as such, not as binary
-aaExtract all files in the archive as text files, regardless of whether they actually are
-bExtract all files as binary, regardless of whether they are text
-CDo not apply case-sensitivity when extracting files
-jDo not re-create the ZIP archive file’s directory structure when extracting files; extract all files to the working directory
-LIf a file was created on an uppercase-only system, convert its file name to lowercase
-LLConvert all uppercase file names to lowercase
-nIf files already exist, do not overwrite them
-oDo not prompt before overwriting files
-qPerform the operation quietly; do not display the operation’s step-by-step execution
-qqPerform the operation even more quietly than with -q
-vDisplay diagnostic information (used as unzip -v )
-XPreserve user and group IDs

Considerations when using "unzip"

  • The unzip will take some time proportional to the number of files created.
  • The original zip archive will remain uploaded to NetStorage.
  • The standard zip32 archives supported are limited to 4,294,967,295 (2^32^-1) bytes and 65,535 (2^16^-1) files.
  • Capabilities beyond the basic UNIX "unzip" are not supported.

For example, zip archives with encryption, bzip compression, and zip64 format are not supported.

You should try Serve from Zip

You might be able to serve your files from within an uploaded zip archive on NetStorage without unzipping it, by using Serve from Zip. Serve from Zip supports much larger files and the zip64 file format. If you can use it, you won't need to use this unzip command. See if the Serve from Zip considerations are compatible with your workflow.


Perform a "quick-delete"

A quick-delete allows you to target a specific directory in a storage group to swiftly and recursively delete all of its contents.

Before you use quick-delete

  • Quick-delete needs to be activated before it's used.
  • You use quick-delete with CMShell and the NetStorage Usage API

To perform a quick-delete you can use either the rmdir command or therm command with these options:

  • rmdir: rmdir --quick_delete=imreallyreallysure <path>
  • rm: rm -r

🚧

Once used, these operations will perform a recursive deletion of all content in the target directory. Review your target content before use.

Activate “quick-delete” for your storage group

The quick-delete setting is disabled by default to protect against inadvertent deletions of potentially unknown content. This setting is enabled in ​Control Center​ Control Center, on a per-storage group basis. It can be enabled during creation of a group or by editing an existing one.

The Quick Delete switch is available in the upload directories settings for a storage group

📘

Accessing "quick-delete" via CMShell and the NetStorage Usage API are both configured here.

Considerations when using quick-delete

Review these points before enabling and using quick-delete:

  • Newly uploaded content will not be deleted. If you have targeted a file with a quick-delete that hasn’t been deleted yet, and you upload a new version of the file that overwrites it, this new version will not be deleted.
  • This feature is designed for large deletes. While it can be used to perform a recursive delete of any number of files, it is actually meant to be used when you have millions of objects to delete. You will see the best performance in this case.
  • The rmdir and rm -r options are the same with ObjectStore: Both methods can be used to perform a quick-delete in ObjectStore.
  • Quick delete must be activated.
  • The actual deletion may take some time: The actual deletion occurs in the background with ObjectStore storage groups. If a large number of files have been targeted, the deletions may take a significant amount of time to complete.
  • Objects targeted for quick delete may still be accessible. Taking the previous point into consideration, a targeted file or directory will exist until the deletion of that object can fully complete. Until the actual delete, it will still be accessible, and served if requested.
  • Currently, there is no alert that a quick-delete has completed. Quick deletes take time, and this may be an issue. You can use the “du” command or the “ls” command to view directories to check status.
  • “undeletion” is not supported. When performing a quick-delete from an ObjectStore storage group, targeted content cannot be recovered. Once issued, content will be permanently removed.

Did this page help you?