Configure AMD property for redundant MSL5 origins

Configuring the end-to-end MSL5 Origin with the Akamai CDN AMD (Adaptive Media Delivery) property is essential for ensuring high service availability and reliable content delivery. This document outlines the key components and best practices for implementing a redundant origin architecture.

Service Availability Architecture

High service availability requires redundancy at multiple levels:

Multiple Ingest Endpoints: Use primary and backup ingest points located in different geographic locations.
Encoder Failover: Configure publishers to deliver content simultaneously to both endpoints.
Player Failover: Ensure that OTT players can seamlessly switch between primary and backup streams.
CDN Routing: Set up the AMD property to route requests to the appropriate origin based on the URL path.

Primary Architecture Components

Redundant Ingest Endpoints
MSL5 Origin deployments must have multiple ingest endpoints to ensure continuous content ingestion even if one endpoint fails.
- Configuration Requirements:
  - Primary Ingest Endpoint: Located in the primary Linode Region.
    - Region: Primary geographic location (e.g., us-east-1)
    - Endpoint: <https://p-ep50001.i.harmonicincentrypoint.net/50001/event1/master.m3u8>
  - Backup Ingest Endpoint: Located in different Linode Region.
    - Region: Geographically distinct location (e.g., us-west-1)
    - Endpoint: <https://b-ep50002.i.harmonicincentrypoint.net/50002-b/event1/master.m3u8>
    - Critical: The backup endpoint must be in a different region than primary to ensure geographic redundancy.
- Benefits of Multi-Region Deployment:
  - Protects against regional outages.
  - Provides geographic load distribution.
  - Enables local encoding near content source.
  - Reduces latency for encoder-to-origin communication.
Encoder Configuration for Dual Publishing
The encoder/publisher must be configured to simultaneously publish to both primary and backup ingest endpoints with perfectly aligned content.
- Encoder Requirements:
  - Dual Output Streams: Configure the encoder to publish to both the primary and backup endpoints simultaneously.
  - Content Alignment: Ensure both streams conform to the following:
    - Identical video codec and bitrate
    - Identical audio codec and bitrate
    - Synchronized segment boundaries
    - Aligned IDR (Instantaneous Decoder Refresh) frame positioning
- Critical Configuration:
  - Segment Naming: Both primary and backup streams must use identical segment filenames.
    - Primary segments: segment-0000.ts, segment-0001.ts, etc.
    - Backup segments: segment-0000.ts, segment-0001.ts (same naming)
  - IDR Alignment: IDR frames must occur at the same presentation time in both streams.
    - Ensures smooth playback when switching between streams
    - Prevents glitches during failover scenarios
  - Playlist Synchronization: The master playlist and variant playlists must be updated simultaneously on both endpoints.
  - Example Encoder Configuration:
    Primary Stream Publishing: URL: https://p-ep50002.i.harmonicincentrypoint.net/50002/event1/master.m3u8 Stream Key: {stream-id} Backup Stream Publishing: URL: https://b-ep50002.i.harmonicincentrypoint.net/50002-b/event1/master.m3u8 Stream Key: {stream-id} Both streams configured with: - Bitrate: 5000 kbps - Framerate: 30 fps - Segment Duration: 6 seconds - IDR Interval: 2 segments
Playlist Strategy for Player Failover
The OTT player requires a mechanism to fail over from primary to backup stream URLs when the primary stream becomes unavailable.
- Playlist Distribution Options:
  - Option A: Merged Playlist (Recommended)
    The origin publishes a single merged/master playlist containing both primary and backup stream variants:
    #EXTM3U #EXT-X-VERSION:3 #EXT-X-TARGETDURATION:6 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080 /hls/live/{stream-id}/{event_name}/variant.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080 /hls/live/{stream-id}-b/{event_name}/variant.m3u8
    - Behavior:
      - The player receives both URLs in master playlist.
      - The player initially selects the primary stream.
      - The player automatically switches to backup stream on error/timeout.
      - Requires player support for automatic variant failover.
        Reference: Create a merged variant playlist
  - Option B: Player-Controlled Failover
    The OTT player application logic handles failover:
    Try: /hls/live/{stream-id}/{event_name}/variant.m3u8 ↓ (on failure/timeout) Switch to: /hls/live/{stream-id}-b/{event_name}/variant.m3u8
    - URL Path Naming Convention:
      - Primary Stream Path: /hls/live/{stream-id}/{event_name}/variant.m3u8
        
        Standard naming without suffix
        
        Routes to primary MSL5 origin
        
        Example: /hls/live/channel-1/news-broadcast-20231124/variant.m3u8
      - Backup Stream Path: /hls/live/{stream-id}-b/{event_name}/variant.m3u8
        
        Suffix -b appended to stream-id
        
        Routes to backup MSL5 origin
        
        Example: /hls/live/channel-1-b/news-broadcast-20231124/variant.m3u8
      - Critical Path Characteristics:
        
        The backup path must be distinguishable by URL pattern for CDN routing.
        
        The -b suffix is essential for AMD property path matching rules.
        
        Both paths must reference the same underlying event/content.
Akamai AMD Property Configuration
The AMD property serves as the traffic router, directing requests to the appropriate origin based on the request URL path.
- AMD Property Setup:
  Reference:

Match Rule for Primary Origin:

Rule Name: Route to Primary MSL5 Origin
Condition: 
  no condition, default rule
    
Origin Target: Primary MSL5 Origin Hostname
  (us-sea-myoriginname.mslorigin.net)

Match Rule for Backup Origin:

Rule Name: Route to Backup MSL5 Origin
Condition:
  Request Path matches: */*-b/*
  
Origin Target: Backup MSL5 Origin Hostname
  (us-ord-myoriginname.mslorigin.net)

Regex Pattern Explanation for Backup Rule:

Pattern: */*-b/*

Examples matching this pattern:
  ✓ /hls/live/500002-b/event/variant.m3u8
  ✓ /hls/live/2023123-b/news/playlist.m3u8
  ✓ /path-b/file
  
Examples NOT matching:
  ✗ /hls/live/500002-1/event/variant.m3u8 (no -b)
  ✗ /hls/live/500002-b-1/event/variant.m3u8 (-b in middle)
  ✗ /hls/live/500002b/event/variant.m3u8 (no dash before b)

Failover Behavior

Expected Failover Flow

Normal Operation (Primary Active):

Player Request → /hls/live/{stream-id}/{event_name}/variant.m3u8
                ↓
          AMD Property
                ↓
    Primary MSL5 Origin (us-sea-myoriginname.mslorigin.net)
                ↓
         Segment Delivery

Failover Scenario (Primary Unavailable):

Player Request → /hls/live/{stream-id}/{event_name}/variant.m3u8
                ↓
          AMD Property
                ↓
    Primary MSL5 Origin [TIMEOUT/ERROR]
                ↓
     Player Switches to Backup URL
                ↓
/hls/live/{stream-id}-b/{event_name}/variant.m3u8
                ↓
          AMD Property
                ↓
    Backup MSL5 Origin (us-ord-myoriginname.mslorigin.net)
                ↓
         Segment Delivery

Failover Considerations

Failover Detection Time: Depends on player timeout settings and HTTP read timeouts.
Content Continuity: Content must be aligned to ensure smooth handoff.
Segment Availability: Both origins maintain identical segment libraries.
Manifest Freshness: Playlists must be current on both origins.

Monitoring and Troubleshooting

Key Metrics to Monitor

Ingest Health: Monitor the segment arrival rate on both primary and backup endpoints to ensure consistent ingestion.
CDN Performance: Track request latency and cache hit rates for both paths to assess CDN efficiency.
Player Connectivity: Monitor successful requests to both primary and backup URLs to ensure robust player functionality.
Failover Events: Set up alerts for sudden switches to the backup stream, indicating potential issues.

Common Issues and Resolution

Issue: Misaligned IDR Frames
- Symptom: Video glitches or freezing during failover.
- Cause: IDR frames are not synchronized between primary and backup streams.
- Resolution: Verify that the encoder IDR interval settings are identical; adjust segment duration if needed.
Issue: Segment Naming Mismatch
- Symptom: 404 errors or playback gaps when switching streams.
- Cause: Different segment filenames on primary vs. backup.
- Resolution: Configure the encoder to use identical segment naming on both endpoints.
Issue: Playlist Stale on One Origin
- Symptom: Backup stream has delayed or missing segments.
- Cause: Asynchronous playlist updates.
- Resolution: Ensure simultaneous playlist updates to both origins
Issue: AMD Rule Not Matching Backup Path
- Symptom: Backup requests routing to primary origin.
- Cause: AMD match rule pattern incorrect.
- Resolution: Verify */*-b/* pattern is correctly configured in the property.