Welcome to the 224 new subscribers who have joined us since last week.
If you aren’t subscribed yet, join 5000+ engineers and technical managers dedicated to learn System Design.
Today, we’re going to design a Live Video Streaming Platform and cover the following components:
Real time video ingesting
Routing
Transcoding
Video Delivery
Distribution
Playback
Below is a high level diagram of the whole system:
Video Ingestion
The first stage in the live streaming process involves the capture and ingestion of live video content. At the production site, live video feeds are captured using cameras and then encoded into a digital format suitable for streaming over the internet or direct connection to the data center.
To support real-time streaming of 1-second video fragments, the encoder must be configured to segment the video into 1-second chunks.
The encoded video fragments are then pushed to the core datacenter responsible for transcoding of the video. This push can be facilitated using RTMP (Real-Time Messaging Protocol)
Routing
The Media Proxy acts as the data gateway. It operates across all Points of Presence (PoPs). It processes live video streams from broadcasters by extracting key stream properties and directing them to appropriate core regions.
The routing service is a configurable, stateful service designed for rule-based routing. For instance, it can ensure streams from specific channels are always sent to a designated origin, catering to unique processing needs, or direct multiple related streams to a single origin for scenarios like premium broadcasts with primary and backup feeds.
Redundant Network
Popular events use private, guaranteed bandwidth paths with geographic redundancy, employing dual fiber-optic routes or a combination of fiber-optic and satellite backup.
To support high-quality broadcasts, we have to use a dedicated broadcast hardware with managed encoders, connected to our data centers via dedicated links.
We should also have a system to seamlessly integrate primary and secondary streams into our infrastructure
Processing and Transcoding
Once we receive 1-second video fragments, the core datacenter's primary responsibility is to transcode these fragments into multiple resolutions and codecs.
This step allows users to enjoy the best possible viewing experience regardless of their device or bandwidth limitations. The transcoding process involves converting the original video fragments into various formats, such as 1080p, 720p, 480p, and 360p resolutions, and using codecs like AVC (Advanced Video Coding), HLS, and DASH.
This transcoding process is resource-intensive and requires dedicated transcoding cluster within the datacenter, equipped with powerful GPUs. It should support auto-scaling to handle varying loads dynamically, ensuring that the transcoding of 1-second video fragments is completed swiftly and made available for distribution without delay.
Video Distribution
Once transcoded, the video fragments are then prepared for distribution to the end-users. This involves packaging the fragments into adaptive bitrate streaming formats like HLS or MPEG-DASH, which allows the video player on the user's device to select the appropriate stream based on their current network conditions.
The packaged video streams are distributed through a Content Delivery Network (CDN), which caches the content at edge locations closer to the users, reducing latency and improving the streaming experience.
The CDN insures the scalability of the video streaming platform, as it offloads the traffic from the core datacenter and provides a geographically distributed network to serve the video content from locations nearest to the end-users. This setup minimizes the distance the data travels, reducing latency and buffering times.
Playback
On the user's side, a video player capable of handling adaptive bitrate streaming formats (HLS or MPEG-DASH) is required for playback. This player adapts to changing network conditions, seamlessly switches between different resolutions, and maintains smooth playback without interruptions.
Check out a more detailed video coverage on my Youtube.
📣 Shout-outs of the week
How Disney+ Hotstar Delivered 5 Billion Emojis in Real Time on System Design Newsletter by NK- How Disney scaled Hotstar with Golang routines and Erlang mnesia.
Engineer's Guide to LinkedIn on
by About the importance of the Linkedin for the engineersDesign context is more important than a detailed solution on
by Great read on the importance of the context for any system designHow to start, grow and monetize your engineering newsletter on
by Great post on how and why to start an engineering newsletterFacebook's Memcache Breakdown on
by Facebook Cache Architecture
Thank you for your continued support of my newsletter and the growth to a 5k+ members community 🙏
You can also hit the like ❤️ button at the bottom of this email or share this post with a friend. It really helps!
Thanks for the mention Eugene!
And a really nice explanation of how videos are streamed.
Could you please upload the video for SD questions.