Serve content from zip files

NetStorage offers a “Serve from Zip” feature that allows it to dynamically examine archive files and directly serve individual files from within them. Without this feature, an archive file would have to be completely unzipped to serve its contents, which could be a lengthy process if it contained a large number of files. Serve from Zip replaces the unzip process with an indexing command that verifies the file’s integrity and adds a hash table to facilitate locating specific files within it.

Important: Serve from Zip requires that the archive contain an index, and the filename use a “.zip” extension.

  • It eliminates the “unzip” process. Individual files are available almost immediately.
  • File installation bottlenecks can be bypassed. Since many individual files are compressed into a single archive, this undesired slow-down can be avoided.
  • It offers single file management capability. Contents that are updated and deleted in groups can be managed as one file rather than many separate files.

How to enable Serve from Zip

The Serve from Zip functionality must be enabled on a per-upload directory basis for each applicable storage group.

Enable Serve from Zip using the NetStorage UI

  1. Open the application. Go to ORIGIN SERVICESNetStorage .
  2. Select the Storage Groups entity.
  3. Create or edit a storage group.
  4. Create or edit an upload directory.
  5. Enable the Serve From Zip option.

Enable Serve from Zip using the NetStorage Configuration API

Set serveFromZip=true for your intended storage group and upload directory (cpcodeId). See the NetStorage Configuration API for full configuration details.

Example upload directory modification

{
        "contractId": "5-555V556",
        "storageGroupName": "aka_storage",
        "domainPrefix": "baseball",
        "cpcodes": [
            {
                "cpcodeId": "456789",
                "serveFromZip": true
                }
        ]
    }

Index your archives to use with Serve from Zip

To use the Serve from Zip feature, the archive needs an index applied to create a hash table of its contents. The hash table is then added to the “.zip” archive as a comment.

🚧

You should enable Serve from Zip for your storage group or the indexed zip files will become corrupt and unusable.

Indexing an archive requires that the filename use a “.zip” extension. You can add an index using any of these methods:

Apply auto-indexing to your upload account

Auto-indexing is available for Aspera, Rsync, SCP, SFTP, and FTP protocols. Apply these steps to enable auto-indexing on your upload account:

  1. Access the Upload Accounts entity from the NetStorage UI in Control Center.
  2. Edit the upload account you want to apply auto-indexing.
  3. Navigate to the Advanced Settings section of the upload account.
  4. Select Append serve-from-zip index from the On Zip File Upload section.

📘

Auto-index will always index zip files for that upload account. This limits the use of Rsync or Aspera sync to upload only, as auto-indexing will change MD5 signatures and sizes that prevent the use of synchronization features. Auto-indexing doesn’t apply to File Manager or the HTTP Usage API.

Use the az2z command with FTP

Apply a zip index with FTP by performing these steps:

  1. Establish an FTP connection to the applicable NetStorage storage group.
  2. Issue the site az2z command from the FTP terminal.
  3. Upload the file as necessary.

🚧

  • The site az2z command needs to be re-entered prior to each archive file upload, and the subsequent upload must be initiated from the same FTP session.
  • An error will be generated if the file uploaded isn't an archive file.
  • FTP is an insecure protocol. Some GUI FTP clients don't support site commands.

Use the az2z command with CMShell

Apply a zip index with CMShell by performing these steps:

  1. Upload the archive file(s) to NetStorage using whatever means you prefer. This can be any of the protocols covered in this document or even via the FileManager tool available in ​Akamai Control Center​.
  2. Make note of the complete path to all uploaded archive files.
  3. Access CMShell.
  4. Run the az2z <archive_file> command for each archive.

🚧

  • If you unzip an indexed archive file with a tool that displays comments (for example, the unzip command available for use with CMShell), the table hash is listed among the archive’s contents, appearing as garbled text.
  • Using CMShell adds an extra step that reads and writes the whole file instead of adding just the index while uploading.

Use the NetStorage Usage API

Use the API's upload action with the index-zip=1 option to enable az2z processing. This will index an uploaded “.zip” archive for the “Serve from Zip” feature. The API allows full control over whether of not to index zip files without having to switch upload accounts.

Additional details on the use of the upload action can be found in the NetStorage Usage API.


Serve from Zip considerations

Recommendations and limitations for indexed zip files

Consider storing your zip files in different upload directories if serving indexed content impacts your performance. These limitations apply to indexed zip files:

  • Don't use NetStorage as your master copy. Indexed zip files stored on NetStorage has a different MD5 than what was uploaded. You won't be able to verify going back.
  • Don't check the MD5 or files size of indexed files on NetStorage. Use the return response from most protocol clients to confirm a successful transfer.
  • Don't use Aspera sync, Rclone, or Rsync with indexed zip files. All sync operations require matching file sizes. Indexing a NetStorage zip file will always change the file size, causing sync operations to continuously upload files which will result in a lot of extra bandwidth.

Review this table of frequently asked questions and considerations:

ConsiderationDescription
NetStorage translates backslashes when indexingWhen indexing archive files, NetStorage translates backward slashes ( \ ) to forward slashes ( / ).

This can be problematic if the archive contains any UNIX system-based files with that character in their name. You can overcome this problem by uploading the file external to the archive (i.e., do not include it in the archive, and upload it separately).

Archive file size or object limitationsArchive files are limited as follows:
  • Archive size: 100 gigabytes
  • Object limit: 1,000,000 objects

📘

An object constitutes a file, symlink or directory.

As a workaround, you can upload multiple archive files that fit within these restrictions.

Multiple disk archive files are not supportedYou cannot use multiple disk archive files with Serve from Zip.
No Support with “FTP Download”If you are using NetStorage as your origin for the FTP Download product, Serve from Zip is not supported for use.
Delivery issues with HTTP byte-range requestsUsing HTTP byte-range requests to a file within a zip archive could impede performance. This is because these requests need to decompress the bytes for that file up to the starting offset. A bad use case example is asking for byte 299,999,999 of a 300 megabyte file.

Observe the potential drawbacks:

  • Range-requests are not supported with Content Management protocols to objects inside a zip file. (Content management protocols are the HTTP-API and any session-based protocol.)
  • A 404 error results in a larger performance penalty.
  • Performance of range-request functionality to objects within a NetStorage indexed zip file will be penalized when the delivery product cache doesn't have the range requested, the full object will be requested from the NetStorage origin.

📘

Putting objects larger than 100 MB into an indexed zip archive isn't recommended. Large File Optimization (LFO) delivery options won't be usable for those objects, and high midgress traffic is the likely result.
The indexing operation adds information to the archive fileIf the CMShell command, md5sum is run after indexing, the hash differs from that of your local archive version. For the same reason, running FTP’s site az2z command in tandem with its site chkhash <MD5Digest> command produces an error.
Can be problematic for NetStorage’s “MD5 Sum” feature

This setting is set via the Upload Directory Attribute interface in Control Center.

When enabled, the feature sends an MD5 hash in the response header of each file served from NetStorage. However, in the case of Serve from Zip content, the MD5 hash is from the archive file, not of the content file itself. So, if the end user’s client (including Download Manager) checks the response header, the hash will not match that of the file received, and the file will be rejected.

This behavior also prevents content caching at Edge servers, if you have asked your account representative to enable the Edge’s hash-checking feature. In this case, it does not interfere with the actual serving of content to end users, but it does require Edge Servers to retrieve content from NetStorage each time content is requested. In the future, Serve from Zip will override the MD5 Sum feature and NetStorage will not send an MD5 hash with Serve from Zip content.

Only basic zip format features are supportedFor example, encryption and bzip compression are not supported.
Emergency directories or stale content protection are not supportedZip logic does not support these features.
Aspera does not support zip indexing as part of the uploadAfter a zip archive is uploaded via Aspera, the indexing must be done separately using the CMShell az2z command.
Double indexing an archive is not supportedOnce a file has been indexed, any attempt to re-index the same file will result in failure.
You can use these archive utilities
  • Mac Finder
  • Command-line (Mac)
  • Windows Explorer
  • WinRar
  • 7-Zip
Serve from Zip uses relative pathsUploaded zip files should contain relative paths that do not start with a slash ( / ). Absolute paths starting with a slash are not supported.
What does auto-indexing do to a zip file pre-indexed?Error 451 will occur after uploading pre-indexed zip files with auto-index enabled, and the file will not transfer to NetStorage.
What does the CMShell “az2z” cmd do to a zip file that was auto-indexed?Once a file has been indexed, any attempt to re-index the same file will result in failure.
How do I find out which zip files are, or are not, indexed?Indexed zip file MD5s and file sizes stored on NetStorage will not match your local copy:
  1. Get the list of all zip files by scanning a local copy of the content tree. For example, using Linux: find . -name '*.zip' -printf '%s %p\n'
  2. Use the HTTP-API "stat" action (or equivalent on another protocol) for each path and compare the size of the zip file. If the uploaded file size is the same, the uploaded zip file is not indexed.

Otherwise you'll need to know the name of a sub-object within each zip file (e.g., a video manifest name.) and be able to download via a delivery config using serve-from-zip.

  1. Use the HTTP-API list action to scan all objects in NetStorage.
  2. Test if a sub-object can be retrieved using the HTTP Delivery with the HEAD method.
How do I remove the Akamai index from a zip file on NetStorage?Removing an index isn't supported. Re-upload the original zip file without auto-indexing to replace an indexed zip file.
How do I upload zip files without auto-indexing?You can choose any of these options:
  • Create a new, or use another upload account that does not auto-index.
  • Use the HTTP-API with the zip-index=0 parameter, if using an upload account with auto-indexing enabled.
  • Temporarily disable the auto-index option on the upload account.

Did this page help you?