People tend to think of video-streaming giants like Netflix and Hulu as entertainment providers, but they're also global tech companies. Underneath the new season of a show you're binging or a live-streamed March Madness game with millions of concurrent viewers are evolving user interfaces, high-powered backend data operations, and complex streaming-video pipelines.
Our cover story in this month's PCMag Digital Edition breaks down the increasingly competitive video-streaming landscape and how new services rolled out by big tech companies such as Apple and media conglomerates including AT&T, Comcast, and Disney are creating a content-industrial complex for consumers.
We spoke to execs from Amazon Prime Video, CBS, Disney, and Hulu (Netflix declined multiple interview requests), as well as experts and analysts, about the content, technology, and market forces shaping this fast-evolving industry melding the tech and entertainment worlds. This companion story dives into the tech side of streaming.
At the heart of the technical complexity is one simple truth: The internet wasn't built to stream high-quality video to millions of people.
"People talk about the technology as if we just push all the TV over the internet. It's not set up to do that," said streaming media consultant and expert Dan Rayburn. "The internet was not built to deliver video at great quality in large scale; it can't physically handle it."
How Video Streams Over the Internet
The top streaming players have spent years and millions of dollars building out their streaming infrastructure to beam on-demand content across the web and support live streams of sports and other events. They've also had to figure out how to distribute that video to an increasingly connected landscape of varying devices and screens.
"Just in the last five to 10 years, we've gotten into this world with significantly better connectivity between devices, both mobile and in the living room, where you are able to deliver much higher streams to the consumers," said Rafael Soltanovich, VP of Software Development at Hulu; Soltanovich oversees everything from video delivery and metadata to the tech powering payments and subscriptions for the streaming company.
Streaming isn't a static medium, like TV. In our on-demand culture, viewers consume content in their own way and on their own time. To get that video to whichever device a user is watching on—a browser on a laptop or desktop, a smartphone or tablet, a media-streaming device, a game console, or a smart TV—that video has to travel through a lot of steps.
Over-the-top (OTT) streaming, or distributing standalone media over the internet, can be a convoluted process to explain. Dan Rayburn breaks it into five steps: video ingestion, transcoding (converting a video from one file format to another), management, delivery, and playback.
"So if a live event is taking place in Atlanta, that signal from Atlanta is the ingestion," said Rayburn. "Then you encode it to work on the internet; management, which means things like content protection, DRM, dynamic ad insertion if you have an ad model; and then delivery through the internet; and playback on a platform or device."
In reality, it's a lot more complicated. There's no standard for encoding, so video files need to be "wrapped" differently for every platform they're delivered to; files are wrapped differently for Roku than they are for the PlayStation 4, a desktop browser, a smart TV, or an Android or iOS device. A single video file could be wrapped 20 times or more depending on the devices to which it's being delivere, Rayburn said.
"We have individuals who will be enjoying Hulu on a powerful gaming device: let's say an Xbox or maybe a PlayStation, or something with far less computational power and memory, such as a streaming stick," said Soltanovich. "We've learned to vary both the size of the [video file] segments and the size of the metadata payload to these devices in order to dynamically adjust, no matter where you're watching."
What Are CDNs?
There's also the matter of delivering HD video at 720, 1080p, or even 4K resolution quickly and reliably to anywhere in the world. That's where content delivery networks (CDNs) come in, giving streaming platforms a distributed array of data centers and servers to cache and stream content locally to nearby end-users.
Netflix spent years building out Open Connect, its globally distributed CDN, so as not to have to rely on Akamai, CenturyLink (Level 3), or Limelight, which are three third-party CDN companies. Netflix is the only standalone streaming platform to build out its own CDN network, though Amazon has its own CDN network through AWS CloudFront, and tech companies including Apple, Facebook, Google, and Microsoft have their own CDNs.
Streaming apps also need cloud providers for the rest of the pipeline. Netflix completed a massive cloud migration in 2016 from its data centers to Amazon Web Services (AWS).
"Netflix built out OpenConnect specifically for video; 100 percent of Netflix's video for the last three and a half years has gone over a network they built out. They were using third-party CDNs before that," said Rayburn. "Netflix doesn't use Amazon for video delivery at all but still relies on AWS for storage, cloud-compute, databases, and its own transcoding engine built on top of AWS."
Hulu's Soltanovich said relying on third-party CDNs has allowed the platform to focus its efforts elsewhere rather than having to build out its own expensive in-house network.
"We're transferring petabytes of data on a monthly basis," said Soltanovich. "We're working with an on-demand library that's over 85,000 titles. So just the sheer scale, from storage to data connectivity and transfer, makes having a multi-CDN strategy important. Storing it at our origin and then pushing it out to a number of major CDN providers ultimately allows the quickest delivery regardless of whether you are in a metropolitan location or on-the-go across the United States."
For Amazon, having AWS on the back end powering its video streaming makes things easier. Girish Bajaj, VP of Software Engineering for Amazon Prime Video, said Amazon employs its global cloud infrastructure combined with machine learning and real-time data processing to dynamically adjust its viewing experience. The goal is to maintain quality using the least amount of bandwidth. Amazon was also the first video service to offer HDR streaming, in 2015.
"Streaming top-quality video over the internet is complex. The closer you get to deploying your infrastructure to customers, the better your availability and performance and quality are going to be," said Bajaj.
How to Build a Streaming App From Scratch
Creating original content for new streaming services takes time, but so does building the infrastructure to ensure seamless delivery.
Joe Inzerillo, CTO of Disney Streaming Services, talked to us about building the entertainment giant's Disney+ service, which is scheduled to launch at the end of this year.
Disney+ has been in the works officially since 2017 when Disney took a controlling stake in streaming technology firm BAMTech, where Inzerillo previously served as CTO. He got his start as the first professional video coordinator for Major League Baseball and helped found MLB Advanced Media, which ultimately spun out into BAMTech and now underpins Disney's entire streaming operation.
"We're charged with building basically everything that runs off Disney client-device services; the service layer, video distribution, video encoding, all that stuff—the tactical execution of every Disney streaming product that the world will get to see," said Inzerillo.
More high-profile streaming apps, from WarnerMedia (AT&T's newly formed combination of HBO, Turner, and Warner Bros.) and NBCUniversal, are expected to launch in late 2019 and early 2020, respectively. Just as Apple and Amazon have become entertainment companies, these media companies are becoming technology providers.
CBS Interactive CEO Jim Lanzone, who is also the Chief Digital Officer of CBS, said it's important to think as a multi-platform company. Content is still the most important part of the offering, but you can't monetize it without the product and technology.
CBS has built out its own streaming infrastructure over the course of more than a decade, atop its acquisitions of SportsLine and CNET. The corporation's video stack now spans live sports, premium cable with Showtime, and a standalone streaming app in CBS All Access, along with its traditional broadcast channel.
"Think about what the next 10 years will look like for video stacks; think about machine learning and artificial intelligence, think about it from a content perspective and from an overall data perspective in what that can do for both content and advertising," said Lanzone. "That technology expertise is becoming a core competency, and it really changes the entire complexion of traditional content companies."
Fixing Broken Interfaces
One of the most consistent issues for users is how hard it can be to navigate a seemingly infinite selection of content. Even as Netflix, Amazon, Hulu, and the rest invest billions in original content and spend millions to license syndicated shows and movies, viewers are browsing streaming UIs with the same, tired scrolling interfaces that look like digital library shelves.
There's a fundamental disconnect between the discovery and navigation mechanisms users are given versus the need to develop intuitive UI tools for a new generation of connected entertainment. Streaming companies talk a lot about using artificial intelligence and machine learning to personalize the experience for each user, but those buzzword-laden solutions only go so far.
"These services like using the word 'personalized' to describe streaming experiences compared to live TV," said Dan Rayburn. "Watch a comedy, and you get recommended another comedy; that's not personalization to me. You can talk all about AI and machine learning, but on TV it's not hard to just look at a guide and see what's on."
I asked several providers how they're evolving streaming-service UIs. Hulu's Soltanovich said the platform isn't just crunching numbers to come up with recommendations; it's about developing individual user profiles and asking people what they want to see.
He added that you have to ask the right questions and add curated content on top of that. Only then can you start to layer in machine learning to recommend things programmatically, using metadata to surface what users are interested in watching. Netflix and Prime Video also have user profiles to tailor the interface to each user.
"Having that grid of the hundreds of videos that I've seen on a number of services is probably not the best way to get to the content anymore," said Soltanovich. "We need to get to know you as a user. It starts with the onboarding experience we've invested very heavily in and our taste picker, which allows you to tailor your preferences and experiences before any major event is coming, such as the Olympics or March Madness...but we still want you as a user to have the ultimate control, to be able to say, 'You know what? Stop recommending this to me. You actually misread me. That's not what I'm looking for.'"
CBS also uses a combination of human curation and automation. Marc DeBevoise, President and COO of CBS Interactive, said the company has an AI and machine-learning group building algorithms to target users with specific content. But a lot of the content highlighted on apps such as CBS All Access is manually slotted in, he added.
CBS uses automation more in how it markets shows, sending out emails and notifications to users to keep them engaged with new shows based on content preferences. CBS also tweaks its main interface to suit user preferences, DeBevoise said.
"If you click on the shows page, you'll get the same shows catalog as everyone else. But if you're in the flow of what is served up in the marquee at the top of the [homepage], sometimes that's manually curated, and sometimes it's specific to you," he said.
Amazon's user experience approach is a bit different. The actual Prime Video interface is a library-style homepage with tile-based rows of originals, movies, add-on channels, and licensed TV shows. Yet when you click into a section, you see the same search results page as a product search in the main Amazon e-commerce portal would produce.
Girish Bajaj said Amazon runs hundreds of experiments on its navigation and UI to increase engagement and improve discovery, down to selecting the right images to associate with titles. Amazon also uses machine learning to process vast amounts of customer viewing data and maps behavior patterns to predict what users may want to watch next.
As for improving the UI, Amazon is focused on the viewing experience. Bajaj touted X-Ray, a longstanding feature integrated with (Amazon-owned) IMDb that surfaces actor information in any given scene. This process used to be crowdsourced, but over time Amazon automated it with computer vision to scan every face for an IMDb match. The company has invested more in IMDb lately, even rolling out a free, ad-supported IMDb streaming service called Freedive.
Recommended by Our Editors
Recently, Prime Video has also experimented with creating additional behind-the-scenes X-Ray content to augment the viewing experience. And the platform launched X-Ray for sports, offering on-screen, play-by-play updates, live stats, and player information. It was available in Amazon's Thursday Night Football live-streamed broadcasts this past season and is also live for NBA League Pass games streamed through Prime Video.
"We have X-Ray on tens of thousands of titles in our catalog and growing, and now we're not only aggregating from IMDb but working with creators to produce our own X-Ray content," said Bajaj. "For example, with Jack Ryan and Man in the High Castle, we filmed interviews and roundtables with cast members to get inside into those characters and story development, and we offered those videos and those clips to customers at no charge."
We're Going Live
Live events add more complexity to the streaming process—and the potential for disaster. The more viewers expected to view a stream, the more it costs streaming platforms to prepare for that influx of concurrent users hitting the service at once.
In the heyday of broadcast TV, tens of millions of viewers would tune into the top-rated sitcoms. More than 105 million viewers watched the M*A*S*H finale, over 76 million watched the final episode of Seinfeld, and almost 66 million tuned into the finale of Friends. Sports are still the biggest draw on television; more than 100 million viewers watch the Super Bowl each year.
TV broadcasts have never had to worry about the scale of concurrent viewers watching a show or sporting event affecting the quality of the broadcast. CBS broadcasts its channel, and whether one person or 5 million people watch, there's no additional cost to CBS.
"The internet is the exact opposite," said Dan Rayburn. "The moment you have more people, you have more cost. The way live events work on the internet is that companies pay an RSVP fee to third-party CDNs, guessing how many concurrent viewers they expect to get to the live stream. Once you estimate the total terabits of capacity, you reserve that with CDNs and pay for it even if you don't use it."
As more streaming platforms start broadcasting live sports, including NFL and NBA games, the March Madness tournament, and the 2020 Tokyo Olympics, the number of concurrent viewers logging onto the same stream at the same time will push the boundaries of what live streaming can handle. And the scale isn't even close to traditional TV. The most popular live-streamed sporting events, including the Super Bowl, attract between 1 and 3 million viewers. The CBS live stream of the most recent Super Bowl averaged 2.6 million viewers at any given time and peaked at 3.1 million concurrent streams.
At that scale, some streaming providers prepare for months. Hulu streamed the Super Bowl on its live TV service and prepared by doing extensive load testing, beefing up its systems to quickly scale, putting failsafes and redundancies in place in case of disaster, and running war-game simulations for worst-case scenarios during the big game. (For some markets, Hulu's 2018 Super Bowl stream cut out close to the end of the game. The company didn't want a repeat this year.)
Hulu with Live TV
"When the Super Bowl starts or when overtime starts, millions of people will hit a specific endpoint on your service in an instant," said Soltanovich. "You need to be able to anticipate when that demand is going to hit and then scale very quickly. The cloud was invented to provide the type of elasticity we're talking about, but it doesn't come for free. You can read about all kinds of fancy technology like auto-scaling, but you need to build up that additional caching and capacity before the users actually click."
In his MLB Advanced Media days, Disney's Joe Inzerillo likened trying to deliver flawless live video simultaneously to millions of people to trying to live on Mercury: "The planet is completely inhospitable. Every day all you're doing is [fighting] a battle for survival in a place that really does not want you. Streaming video on the internet is sort of like that," he told The Verge in 2015.
I asked Inzerillo about that quote, and he said it still holds up. He's grappled with this catch-22 for years: The more successful a product is—in his current role, that might be a stream on the WatchESPN cable subscription app or the standalone ESPN+ app—and the more concurrent viewers watching, the tougher it is to maintain quality.
"Cable or satellite by its very nature is one-dimensional communication. You take a signal and distribute it out. The internet, by its very nature, is duplex," said Inzerillo. "What I would say now versus when I said that quote originally is that the internet is a living, breathing thing, so it does adapt. There are a lot of people working really hard to try to make it more suitable for the delivery of massive amounts of streaming media because that's the world that we live in."
Chaos Engineering
Streaming platforms need to prepare for every eventuality. As concurrent streams increase, the tech teams keeping these services running smoothly have turned to a process called chaos engineering.
The term was developed by Netflix to describe deliberately running "chaos experiments" on a production system in order to find small problems before they escalated into ones that could take a service offline. It's a similar concept to cybersecurity penetration testing or even bug bounty programs: Push a system to its limits and find the cracks to ultimately make it stronger.
Netflix built an open-source program called Chaos Monkey that randomly causes small-scale catastrophes by terminating instances in production during work hours, when engineers are available to track and fix issues. It's designed not only to catch errors but also to help engineers figure out what the most critical parts of the system are and to build in redundancies to make them more resilient.
Hulu uses chaos engineering too, specifically around its live TV service. Soltanovich said that for events like the Super Bowl or March Madness, there's no way to predict with complete certainty how many people will tune in. Chaos engineering keeps Hulu + Live TV ahead of the curve.
"We have a continuous process we've evolved over the past couple of years as we launched Live. On a daily or weekly basis, we're pushing our servers to the boundaries," said Soltanovich. "We want to understand where the weakest link is and improve our architecture and infrastructure."
During Super Bowl 2019, the race to monetize Instagram's World Record Egg—the photo of a brown egg that dethroned a Kylie Jenner pic as the most liked Instagram post of all time—culminated in a 30-second ad promoting mental health awareness. The ad was sponsored by Hulu.
On the Friday before Super Bowl Sunday, the egg's Instagram account put up a teaser post: "The wait is over… All will be revealed this Sunday following the Super Bowl. Watch it first, only on @hulu." Rafael Soltanovich said Hulu's engineering team then had to prepare for viral Armageddon.
"The campaign went viral. It was announced that you would be able to watch the egg crack on Hulu exclusively after the Super Bowl, so we had to deal with a potential demand of over 50 million people hitting Hulu at once," said Soltanovich. "There's the engineering part of being prepared, and there's the user experience part of how navigating [users through] a certain flow that makes smart use of our system. We absolutely chaos-tested for that one."
People Are Using Streaming Services More Than Cable
The way live events work on the internet is that companies pay an RSVP fee to third-party CDNs, guessing how many concurrent viewers they expect to get to the live stream. Once you estimate the total terabits of capacity, you reserve that with CDNs and pay for it even if you don't use it."
Get Our Best Stories!
Sign up for What's New Now to get our top stories delivered to your inbox every morning.
This newsletter may contain advertising, deals, or affiliate links. Subscribing to a newsletter indicates your consent to our Terms of Use and Privacy Policy. You may unsubscribe from the newsletters at any time.
Thanks for signing up!
Your subscription has been confirmed. Keep an eye on your inbox!
Sign up for other newsletters
