Guide
  1. Object Storage

Lifecycle policies

Deleting a few objects in an Object Storage bucket doesn't take too long. But, when the objects number in the thousands or even millions, delete operations can easily become unmanageable. When deleting a substantial amount of objects, you should use lifecycle policies.

A policy is an XML-format set of rules that you apply to a specific bucket. It governs the management of objects after they've reached a certain age. For example, you can create a lifecycle policy that deletes objects once a week, or every thirty days. You can apply a lifecycle policy to handle bucket objects that have become outdated, such as when collecting activity logs.

When policies are enforced

Lifecycle policies are triggered at midnight, in the Object Storage bucket's local time. If you set a lifecycle policy of one day, objects are deleted the midnight after they reach 24 hours old. For example, assume you created an object at 5:00 PM on January 1. It will reach 24 hours in age at 5:00 PM on January 2. As a result, Object Storage enforces your policy on the object at 12:00 AM on January 3.

📘

A lifecycle policy may not delete all of the files in a bucket, the first time it's triggered. This is especially true for buckets with upwards of a million objects. In cases like these, most of the objects are deleted, and any remaining objects are typically deleted during the next iteration of the lifecycle policy's rules.

Create a lifecycle policy

Here's how to create lifecycle policies, using various tools:

s3cmd lets you set and manage lifecycle policies from the command line. Here, we cover how to create and manage lifecycle policies to delete:

  • Objects
  • Previous versions of objects
  • Failed multipart uploads
📘

If you don't have s3cmd set up on your computer, see Using s3cmd with Object Storage.

Create the policy file

Use your preferred text editor to create an XML file to serve as your policy file. Consider the following lifecycle policy file as a template:

<LifecycleConfiguration>
    <Rule>
        <ID>delete-all-objects</ID>
        <Filter>
        <Prefix></Prefix>
        </Filter>
        <Status>Enabled</Status>
        <Expiration>
            <Days>1</Days>
        </Expiration>
    </Rule>
</LifecycleConfiguration>

This deletes all objects in the bucket after one day. Each lifecycle policy file needs a LifecycleConfiguration block and a nested Rule block. The Rule block needs to contain Prefix and Status, and at least one action, like the Expiration block. It's also a good idea to include an ID block:

BlockDescription
IDThe name of the lifecycle policy rule. If your lifecycle policy contains multiple rules, then you need to include a unique ID for each. If you don't specify one in your policy file, a random alphanumeric ID will be assigned to your policy when the policy is applied to a bucket.
FilterA lifecycle rule can apply to all, or a subset of objects in a bucket. This depends on the <Filter> element that you specify in the lifecycle rule. You can filter objects by key prefix, object tags, or a combination of both.
PrefixThis string is used to select objects for deletion with the same matching prefix. For example, if you set this to error_report-, all files that begin with this would be targeted for delete. Leave this empty if you want a rule to apply to all files in a bucket.
StatusWhether this policy is Enabled.
ExpirationInclude the Days block to set the number of days before this rule is enforced. For example, if you set Days to 1, the objects in the bucket will be deleted after one day.

Additional actions

You can apply other actions in a rule:

  • NoncurrentVersionExpiration block, and its child, NoncurrentDays. Use these to control the lifecycle of objects with multiple, older versions. Only include it if you're configuring bucket versioning. Use it to delete objects that are not the newest, most current version. Here's an example of its use:

    <LifecycleConfiguration>
    <Rule>
        <ID>delete-prior-versions</ID>
        <Filter><Prefix></Prefix></Filter>
        <Status>Enabled</Status>
        <NoncurrentVersionExpiration>
            <NoncurrentDays>1</NoncurrentDays>
        </NoncurrentVersionExpiration>
    </Rule>
</LifecycleConfiguration>
    📘

    If a versioned object is deleted in a bucket with the NoncurrentVersionExpiration policy, only the DeleteMarker for that object (not the actual object itself) will be retained, after the number of days specified by NoncurrentDays have passed since the object was deleted.

  • AbortIncompleteMultipartUpload, and its child, DaysAfterInitiation. These work similarly to NoncurrentVersionExpiration, but instead of deleting previous versions of objects, they delete failed multipart uploads. The following will delete failed multipart uploads three days after they were initiated:

    <LifecycleConfiguration>
    <Rule>
        <ID>delete-incomplete-multipart-uploads</ID>
        <Filter><Prefix></Prefix></Filter>
        <Status>Enabled</Status>
        <AbortIncompleteMultipartUpload>
            <DaysAfterInitiation>3</DaysAfterInitiation>
        </AbortIncompleteMultipartUpload>
    </Rule>
</LifecycleConfiguration>
    📘

    About multipart uploads

    Objects that are part of failed multipart uploads stay in Object Storage buckets, counting toward your total Object Storage costs. The s3cmd tool automatically initiates a multipart upload when a file is larger than 15MB. Lifecycle policies are a great way to clear out stale multipart uploads.

❗️

The NewerNoncurrentVersions policy isn't supported with Object Storage.

Multiple actions in one rule

You can specify more than one action in a single rule. For example, you may want to expire the current version of an object after a set number of days, and also remove older versions of it after another period of time.

Here's an example policy that deletes the current version of an object after 10 days, and removes any non-current versions of an object three days after they're demoted from the current version:

<LifecycleConfiguration>
    <Rule>
        <ID>delete-prior-versions</ID>
        <Filter><Prefix></Prefix></Filter>
        <Status>Enabled</Status>
        <Expiration>
            <Days>10</Days>
        </Expiration>
        <NoncurrentVersionExpiration>
            <NoncurrentDays>3</NoncurrentDays>
        </NoncurrentVersionExpiration>
    </Rule>
</LifecycleConfiguration>

Multiple rules

A lifecycle policy file can only contain one LifecycleConfiguration block, but the LifecycleConfiguration block can contain more than one Rule. For instance, if you had a bucket that contained both error and general output logs, you could set a lifecycle policy that saves error logs for a week, but deletes standard logs at the end of every day:

<LifecycleConfiguration>
    <Rule>
        <ID>delete-error-logs</ID>
        <Filter><Prefix>error</Prefix></Filter>
        <Status>Enabled</Status>
        <Expiration>
            <Days>7</Days>
        </Expiration>
    </Rule>
    <Rule>
        <ID>delete-standard-logs</ID>
        <Prefix>logs</Prefix>
        <Status>Enabled</Status>
        <Expiration>
            <Days>1</Days>
        </Expiration>
    </Rule>
</LifecycleConfiguration>

Upload the policy to a bucket

To apply a lifecycle policy to a bucket with s3cmd, you need to upload the policy's XML file to that bucket. This isn't a normal PUT operation. Instead, you need to use the setlifecycle command, followed by the name of the lifecycle policy file, and the name of the bucket.

You can do this by running this command, replacing example-bucket with the name of the target bucket:

s3cmd setlifecycle lifecycle_policy.xml s3://example-bucket

You should see output like the following:

s3://example-bucket/: Lifecycle Policy updated

After successful upload, the system will read your policy and delete objects, based on how you've configured it.

View a bucket's policy

To view a lifecycle policy, use the getlifecycle command, replacing example-bucket with the name of the target bucket:

s3cmd getlifecycle s3://example-bucket

If the bucket has a policy applied, you should see an XML response that looks like this:

<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>delete-all</ID>
    <Prefix/>
    <Status>Enabled</Status>
    <Expiration>
      <Days>1</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

Back to top

Disable a lifecycle policy

Here's how to disable lifecycle policies, using various tools:

To disable a lifecycle policy with s3cmd, you can simply delete it, using the dellifecycle command. Replace example-bucket with the name of your bucket:

s3cmd dellifecycle s3://example-bucket

You'll see a confirmation that the lifecycle policy was deleted:

s3://example-bucket: Lifecycle Policy deleted

Use lifecycle policy filters

You can use more complex filters with a lifecycle policy, to help identify objects based on specific rules that you define.

You can define a specific tag to assign to your objects. Then, you can use a lifecycle policy filter to delete tagged objects, after X days. Your application can declare a lifecycle by automatically deleting tagged objects when they reach the age you define.

For example, you can create tags named, 1day, 10day, and 30day and apply them to delete objects, one, 10, and 30 days after they were created.

Tag example

This example shows you how to add a tag to a lifecycle policy to delete a tagged object from a bucket after one day.

  1. Create a policy xml file, using settings for tags:

    <LifecycleConfiguration>
   <Rule>
       <ID>delete-tagged-objects</ID>
       <Filter>
           <Tag>
               <Key>retention</Key>
               <Value>1day</Value>
           </Tag>
       </Filter>
       <Status>Enabled</Status>
       <Expiration>
           <Days>1</Days>
       </Expiration>
   </Rule>
</LifecycleConfiguration>

  2. Set the policy for the bucket:

    $ s3cmd setlifecycle delete_all_after_1day_wtags.xml  s3://example-bucket 
s3://example-bucket/: Lifecycle Policy updated

  3. Check the policy for the bucket:

    $ s3cmd getlifecycle s3://example-bucket                    
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
	<ID>delete-tagged-objects</ID>
	<Filter>
		<Tag>
			<Key>retention</Key>
			<Value>1day</Value>
		</Tag>
	</Filter>
	<Status>Enabled</Status>
	<Expiration>
		<Days>1</Days>
	</Expiration>
</Rule>
</LifecycleConfiguration>

  4. Set the tag on your objects:

    $ s3cmd settagging s3://example-bucket/test_file.txt retention=1day
s3://example-bucket/test_20241128.txt: Tagging updated
    s3cmd gettagging s3://example-bucket/test_20241128.txt
s3://example-bucket/test_20241128.txt (object):
retention:	1day

Any objects with the tag will be deleted after one day.

retention:	1day

Limits

These limits apply when using tags to apply a lifecycle policy.

DescriptionLimit
Number of tags that can be set at a time per objectMaximum of 10 tags.
Rate limit for setting Object Storage tagsFor E2/E3 rate limits, GetObjectTagging counts as a GET and PutObjectTagging counts as a PUT.
Maximum file size for the number of tags supported by the APITag keys are limited to 128 Unicode characters each, tag values are 256 characters. Every Unicode character counts as one character against the 128 or 256 character limit. There is no relationship or restriction between tags and object size.

Back to top

Updated about 3 hours ago

Did this page help you?