The New World of Hybrid and Virtual Production, SVCE04.Digital - November - 2022
The New World of Hybrid and Virtual Production, SVCE04.Digital - November - 2022
svconline.com
SOUND&VIDEO
CONTRACTOR
R E V I E W S | T E C H N O LO GY | A P P L I C AT I O N S
www
ww
02
SVCONLINE.COM
Temporary or Installed
Planar Venue Pro VX Series LED video walls have
mechanical features to suit temporary (set up)
applications or fixed installation. Cabinets and
modules are removable from the front or rear for
easy service, and quick-install features like locking
pins, magnetic points an embedded quick-lock
system support fast setup and teardown for
mobile studios.
www.planar.com/venue-pro-vx �
�
VIRTUAL AND HYBRID PRODUCTION
EXPAND THE
MESSAGE
W
hen Ford wanted to communicate in a fresh way with inves-
tors, disguise partner Meptik created a presentation that not
only impressed the intended audience with the new technol-
ogy inside Ford’s new line-up, it immersed investors in custom-designed,
fully branded virtual sets that took the audience on a visual journey.
The event kicked off with a high-level keynote in which the speaker
welcomed the audience and presented the newest developments on a vir-
tual stage. The virtually created reflections coming off the LED wall
immersed the speaker in a fully virtual environment and placed specta-
tors of a realistic setting that could shift realities to support the content.
With the press of a button, the location changes from a virtual stage
setup to the inside of a virtual innovation lab. The 90’ wide LED Vol-
ume was extended to infinity by virtually extending the digital content
beyond the volume.
Inside the virtual innovation design lab, Ford’s real-life keynote
speaker was able to show the full line-up of cars, inside and out without
ever having to physically take apart or re-assemble an actual vehicle.
A detailed, virtual replica of the technology behind the vehicles vir-
tualized the product and illustrated key pieces in detail to support the
speaker’s explanation of the technology.
Set extension around the 90ft. wide LED stage, realistic reflections
generated in software, and skilful natural lighting, allowed this extended
reality production to be truly immersive.
04
SVCONLINE.COM
VIRTUAL AND HYBRID PRODUCTION
06
SVCONLINE.COM
VIRTUAL AND HYBRID PRODUCTION
n the wake of the coronavirus pandemic there is new enthusiasm for hybrid live events, in which
I presenters and a live audience are joined by remote participants, and usually a streaming audience
as well. While meetings in conference rooms might have some of the same aspects, in order to run
smoothly a hybrid event may require some of the complexity of live TV production.
The reality of hybrid live events is so new, with so much constant audience, perhaps running cameras, etc. And all of these people need
change, that every setup is likely to be different. So this is not a simple to communicate. This is typical for any live event or show, but adding
“how-to” article. The aim is to provide food for thought about what func- remote guests is a new twist.
tionality complex events may require, and what kinds of technologies
might be used. These concepts can apply to fixed or portable systems, PRODUCTION CONTROL
and to other situations, such as classrooms. At this point let’s assume that there’s a central location where all the
incoming and outgoing signals meet, where there are operators running
BUILDING UP THE SYSTEM equipment, and someone is directing the event. This could be the back
Imagine, if you will, a large conference space or small auditorium. of the hall, a control room, or a remote truck. If the venue is built for live
There’s a stage area in front with a presenter, and audience seating in events there might already be a control area and typical operator posi-
rows or tables. An A/V system handles sound reinforcement for the tions such as house audio and lighting. For the hybrid scenario described
presenter in the room, and projection or video screens to show the pre- here, you might also need:
senter’s computer content. There might also be mics placed for audience • Producer who oversees the action and process of the live event.
members to ask questions via the room PA. Pretty standard stuff. • Possibly an Assistant Producer to handle communications with
Even before the pandemic, live streaming of events was becoming inside and outside personnel.
common, so there might be one or more cameras, and other production • Director for the “broadcast” stream, to call camera shots and video
equipment, that feed a streaming encoder for viewers to watch online via sources, call up graphics, etc.
a CDN, Youtube, etc. That’s a basic one-way stream--viewing only--not • Technical Director to run the stream production equipment (or this
a video conference. might be the Director).
This scenario also applies to concerts, house of worship, etc. Before • Camera Control operator for shading or robotic control (or this
internet streaming this would have been considered the “broadcast” out- might be the Director or TD).
put of the event and, depending on the client, might still feed other deliv- • Playback operator to roll in videos and graphics (could also run
ery paths, such as fiber or satellite transport. lower-third titles if used).
Now we’ll add a remote guest, connecting from home using a UCC • One or more VTC operators to manage the computers and remote
video conferencing app such as Zoom or Teams. We’ll call this video guests.
teleconference (VTC) generically. At the venue, the app runs on a com- • Someone to control what’s on the screens in the venue (or could be
puter and connects to the rest of the system via USB devices using UVC shared with another job).
A
and UAC protocols (generally built into the operating system). The
remote guest must be seen and heard by the live audience and presenter
in the venue and must be able to see and hear what’s going on in the
• A Production Audio operator to handle levels, mix-minuses, play-
back sound, etc.
• Prompter operator (could be shared if the demand is minimal)
venue. The presenter and guest might engage in conversation. If the live • Someone to monitor the live stream.
audience is going to ask questions the remote guest may need to hear
them. And if there is more than one remote guest all of this multiplies. Some of the job titles are kind of traditional from broadcast TV, but
In addition, depending on the “production values” desired by the cli- the bottom line is that all of these functions need to happen, whether
ent doing the event, there might be a need to play video clips or graphic assigned to individuals or shared. Some choices may depend on what
elements, such as show opens, intro and break slides, music, or back- equipment is being used for certain functions, and how particular
ground videos. These may be seen by the streaming audience and/or the devices interact. For example, if video clip or graphics playback is built
in-room audience. into the production switcher the TD might end up running playback--
Let’s also add teleprompter for the live presenter, or perhaps an event even for what goes to screens in the venue.
host. The prompter output is intended only for the person reading (using One role that should not be shared is managing remote guests (VTC
a stage-facing display of some sort) but it may also be useful to see, Op). This is because the remote guests need to be kept “in the loop” as if
or possibly control, the prompter in the central control room for the they were at the venue. Nobody wants to be hanging on a call, wondering
production. when they’ll be live. Plus VTC platforms tend to require some attention,
Finally, if the event is to run smoothly it’s likely that a crew will be such as muting audio, watching chat boxes, and simply making sure the
needed for stage management, wrangling mics, keeping an eye on the connection is solid.
07
NOVEMBER 2022
VIRTUAL AND HYBRID PRODUCTION
Local
Cameras Recording
VTC
Computer
In-Room
Display Proc
In-Room
Displays
Producer
Talkback
Production Mixer
In-Room
Speakers
Lab Tech Systems
17413 Monitor Drive Olney, MD 20832
301-438-8270
Title:
Sound & Video
Contractor
Hybrid Event Article Illustrations
Revise EHW
Refresh EHW
Dwg No.
Sheet 1 of 1
How much equipment is required, and what types, can vary greatly. content they will show. Prompting will have a computer, whether it’s run
If the outgoing stream is supposed to have the polish of a professional in the control room or in the venue.
TV show there should be a production switcher, character generator (for Something to consider with all-in-one production products (such as
titles) and playback device, or a production platform (such as the New- Tricaster, VMix, etc.) is that just because the platform CAN do every-
tek Tricaster) that does a combination of these functions. If the event is thing doesn’t mean it’s a good idea! For one, the operator will have to
being recorded on site that might require another device or be done in divide attention between running the outgoing stream and other func-
the platform. Robotic cameras should have a physical control panel, and tions. Secondly, there are cases where a particular signal needs to go to
of course there’s an audio mixer. several places. This can be a challenge if the production platform has
There will be computers for bringing in the remote guests--generally limited input and output capacity, and also creates complicated delay
one for each (although new options are appearing that allow separate scenarios (coming up). For example, some platforms include the ability
video of each guest from a single UCC computer). There might be a com- to connect with Zoom, Skype or other VTC options internally. But using
puter or wall processor feeding in-room displays, depending on what this feature may not be practical in a hybrid production when the remote
08
SVCONLINE.COM
VIRTUAL AND HYBRID PRODUCTION
guest feeds are also sent to in-room displays (and makes it difficult to The issue now is keeping people’s picture in sync with their sound
offer good guest management). (known as “lip-sync” in TV), because of processing delays in the pro-
duction equipment. We are talking here about delay in video frames,
SPEAKING OF TIMING which is 33mS/frame for 30fps video. I would suggest that a 2-frame
By now everyone understands that the live stream of an event is always lip-sync error is on the edge of acceptable; beyond that things start to
delayed. The processing required to encode at the venue, decode at the look “off” to the average viewer. A self-contained production platform
CDN, re-encode for different viewing streams, etc. takes time (far more might have 6-12 frames of latency--input to final output--which must be
than sending through the internet). In my experience Youtube Live is accounted for in some cases. The most obvious is if audio is being mixed
generally 20-30 seconds behind. But that doesn’t usually matter because outside the platform. If the audio is sent into the platform, and combined
the viewers have no knowledge of the real time event--they only know with video, the delay will be handled internally. But if audio and video
what they get at their time. are combined downstream the audio will need to be delayed to match.
When two-way communication is added, however, there are different This is also true for video through a conventional production switcher,
timing issues. As we’ve all found doing virtual meetings, the latency but the latency is often a frame or less.
of VTC platforms such as Zoom is surprisingly low, so not usually a Lip-sync issues also arise because remote guests are being seen in
big problem--even with real-time conversations between presenters and several places at the same time. When guest audio and video leave the
remote guests. This can vary unpredictably, and with changes in video UCC computer, via a USB converter of some sort, they should be (nomi-
quality, but still is usually acceptable. (Ain’t nothin’ you can do about nally) in sync. That a/v goes into the production system and becomes
it anyway. For best control over remote guest participation, and highest part of the outgoing stream.
quality, you must move away from popular UCC apps and into more But the remote guest also has to appear on in-room screens, with
proprietary signal transport. That’s another story.) matching audio coming from the house PA. The guest audio coming
09
NOVEMBER 2022
VIRTUAL AND HYBRID PRODUCTION
through the production mixer will be real time. But if the video gets to ing, UCC platforms using USB converters assume one audio input and
the screens by way of an output from the production system, it will be one audio output (“stereo” notwithstanding). The remote guest computer
late and the audience will notice a lip-sync problem. With a digital mixer gets a mix that contains the on-stage presenter, possibly audience mics,
it may be possible to add delay to particular inputs or outputs to line the and playback audio. This should be a mix-minus feed, meaning the
a/v back up in the room. remote guests get a full mix minus themselves, to avoid echoes. Here
Another scenario is if the in-room screens are fed from their own again, if the producer needs to speak to the guest, that audio must be
computer, let’s say to give the audience a finished-looking image, or added to their mix-minus, or must interrupt the mix-minus somewhere
show several guests together, TV-news style. Again, the remote guest between the mixer and the remote guest computer.
audio must be delayed to match the video that finally gets to the screens. Audio and comms is one of the hardest areas to implement with UCC
Fortunately, when it comes to lip-sync, there’s only one variable: conferencing because these platforms were not designed for individual
Latency in video paths. Nothing in any audio path--not preamps, mixers, communication. Everyone who joins the conference hears all the audio,
converters, cable or any processing--will delay audio significantly. If and what comes out of the UCC computer is a mix of all participants
your audio is behind your video there’s a delay dialed into some piece of (although that is slowly changing). So even when done well, the back-
equipment, or something is broken. (Yes, wireless mics can sometimes channel communication between a producer and guest is actually heard
take 10-20 mS, but that is still much less than a video frame.) by anyone else on the same call. In a case where remote guests should
Also keep in mind that evaluating lip-sync can be tricky with all the not interact with each other it’s necessary to connect each guest to a
types of displays and signal paths. Assume any flat-screen display might separate call and combine them in the control room to keep their audio
be a frame behind all the time (an issue CRT monitors did not have). isolated. Another comms option comes in the form of internet-based ser-
Also assume a basic video production switcher (not an all-in-one plat- vices, such as Unity Intercom, that allow cell phones to be part of an
form) will take a frame for processing non-genlocked sources. And the intercom network. The process is administered from a central applica-
“multiview” output of any switching device is at least a frame behind, tion running on a computer, perhaps in the control room, or can be cloud-
probably more. So even under the best conditions monitoring lip-sync based. It can also be interfaced to conventional wired intercoms. An app
in the control room may be sloppy by a few frames--generally not a big like Unity can add functionality when personnel need to participate in
deal. As with any TV production, the ideal is to have a “reference” dis- the production from outside the venue but can’t take the place of a dedi-
play that shows the final output signal--unadulterated by any additional cated intercom system for handling specific individual communication.
processing--to use for judging image quality and lip sync. A fully functional intercom that covers all the needs may require more
equipment and expertise than one might expect.
COMMS
Communication within the control room, and to venue crew, is critically IMPORTANT POINTS TO NOTE
important and can be handled fairly easily with any number of wired or • A full-bore live-streamed event, with live audience and remote
wireless intercom systems. A single “party-line” (where everyone is on guests, is more like a television broadcast production than a
one channel) may be fine. Sometimes it’s handy to have two channels, teleconference.
so that tech and producer conversations can be separate. Communicating • There are essentially three audiences being served simultaneously:
with presenters on stage and remote guests is where it gets complicated. The live audience in the venue, the live stream or broadcast view-
A live presenter may wear an earpiece so that IFB (interrupted fold- ers, and the remote guests. All three require somewhat different
back) can be used. In this case they will get a feed from the production content and communication.
mixer (often known as “IFB Program”) that includes whatever they need • Don’t underestimate the need for managing remote guest(s), includ-
to hear--typically the remote guests. The producer or director inter- ing the ability for people in the control room to talk to them.
rupts (or talks over) that audio when necessary. IFB with true interrupt • The production mixer will need to create several different mixes for
requires a more sophisticated intercom, while simply talking over the different purposes: Final show output to stream, IFB for live pre-
Program audio can be done via the mixer. (This does not address how the senter, mix-minus(es) for remote guests, possibly room PA. Some
producer or director audio gets into the mixer or intercom.) inputs or outputs may require delays added to fix lip-sync issues.
The ability to talk to a remote guest from the control room (not • As convenient and ubiquitous as they are, UCC platforms create
through a mic out in the venue) is tricky, but quite important as part of specific limitations for including remote guests in a production.
guest management. They need to know what’s going on, and high-profile Other approaches (such as using a platform like VMix as a remote
guests, who are used to doing TV news interviews, will certainly expect guest gateway) may offer more flexibility and control, but will
communication from the producer or director. In general, as of this writ- require guests to use different applications or equipment.
10
SVCONLINE.COM
VIRTUAL AND HYBRID PRODUCTION
VIRTUAL
PRODUCTION
FOR PRO AV
Not just for Hollywood
any more
By Cynthia Wisehart
12
SVCONLINE.COM
VIRTUAL AND HYBRID PRODUCTION
through the camera and to the presenter on set, it’s possible to avoid
“green-screen fatigue,” a common problem with actors or broadcasters
who perform against little more than a mark on a green scrim. This can
occur even more so for users in corporate, worship, and even live events
settings–a flat, unchanging screen can be too disconnected. The right
live LED video wall gives people clear visual context, keeping them as
engaged as they would be in a real place.
Best Practice: Layer the visuals on stage to add to depth of field.
The screen carries the load as the dynamic background, but adding
physical elements like a table or potted plant bring additional reality
to the visual.
Cameras, Lighting & Camera Tracking: Recent years have seen con-
siderable advancement in tracking cameras; depending on the desired
complexity of a virtual studio there are automation levels and manu-
ally driven options for any budget or application. Studio lighting doesn’t
have to be expensive but it does have to be optimally specified. The
incorporation of motion capture camera tracking in virtual sets signifi-
cantly improves the perspective of the presenter, making the set move
appropriately to convey accurate environments.
Best Practice: Have enough lights and flexible options. Light the fore-
ground and the people. Storyboard the shoot to understand how cam-
era movements (if any) and different lighting scenarios might be needed.
Allow studio time to tweak the stage and rehearse, and to deploy more or
different lighting. This is one of the benefits of a permanent virtual stage
where you can refine lighting needs and lock them in through repeat use.
Media Engines: The media engine is the heart of any virtual studio. The
Unreal Engine has grabbed the consumer spotlight and raised awareness
of the incredible capabilities of these processors and their accompanying
“Virtual production is software. However, closer to home for Pro AV, familiar media servers
provide a range of capabilities. Those manufacturers are natural partners
in specifying a virtual studio.
where the physical and Best Practice: Include your display and media engine partners early.
They can help you optimize a small, highly functional studio or go big
digital worlds meet.” with a brand-defining virtual stage. Either way they can help deliver
accessible, professional, repeatable infrastructure for an ROI-friendly
studio.
—Weta Digital
KEY EXPERTISE
capture. Many of these features reflect feedback from users. Today’s Content: Just as the right technology partners can help you design a
screens are more flexible; cameras and displays work better together and studio that is right-sized for your client, content experts can help set the
excel for the specific demands of on-camera performance. all-important dynamic content strategy. As virtual production expands
LED screens have been used for this purpose for a long time in broad- into Pro AV, so does networking. Relationships–among manufacturers,
cast; moire has been vanquished. But there are other artifacts that can integrators, and content experts–set the (virtual) stage for success. As
affect a virtual studio set up: communication between display and cam- just one example, disguise recently purchased Meptik studios, a leading
era, and attention to the optimal synchronization matters. Modern LEDs provider of virtual content, XR, and VR. Look for more of that kind of
can deliver curved XR and VR environments as well as floor and ceiling cross-pollination. Virtual production does not exist in a vacuum. Under-
screens for truly immersive applications. These days you can choose stand if your client is going to do in-house or third-party content and
displays that have been designed for VP and tested for it. media management. Engage with your client’s content needs and be a
By creating an entire scene live on an LED screen, visible both resource for other experts.
13
NOVEMBER 2022
VIRTUAL AND HYBRID PRODUCTION
14
SVCONLINE.COM
VIRTUAL AND HYBRID PRODUCTION
market. The cornerstone is the company’s in-house fine pitch LED dis- Unreal Engine, disguise, Planar’s sister company OptiTrack, Brompton
play and motion capture expertise through sister company OptiTrak, as Technology, Colorlight and more.
well as relationships with key industry partners on the processing side. “We are proud and honored to work with the world’s leading creative
Planar solutions for VP and XR are being deployed worldwide, includ- and technological professionals as we set out to perfect the art of VP,”
ing at Orbital Virtual Studios, a Los Angeles-based VP studio dedicated said Executive Vice President Adam Schmidt. “Recent developments
to producing movies, music videos and commercials from pre-produc- in VP and XR are already unlocking wonderful capabilities for story-
tion design and development, to production, photography and all the way tellers, but we’re just scratching the surface. We recognize advanced
through post-production.” technological development, productive industry partnerships and a dedi-
Through the Planar Studios initiative, the company leverages Planar’s cated team of experts to be key in helping customers overcome creative
established leadership in on-camera broadcast studio installations. The limitations. Planar Studios is our response and will be a driving force in
company has worked directly with creative and technical professionals unlocking the future of VP and XR.”
to design LED solutions that deliver unmatched in-camera visual perfor-
mance and deployment versatility needed to develop life-like recorded, WHERE DO I START?
streamed or broadcast video content. Virtual production is on the rise and rapid advancements that make it
Planar Studios is supported by a dedicated VP and XR team and better, easier, and more efficient for Hollywood will continue to adapt
strengthened by collaboration with industry-leading partners who spe- into Pro AV. It’s shaping up to be a collaborative future–look for pro-
cialize in digital production and developing next-generation visual viders who are attracting expertise, building connections, working with
storytelling tools. While LED display technology is a significant compo- studios, universities, and agencies. Like many advancements in Pro AV,
nent of VP and XR environments, successful studio implementation also this is not about selling boxes or screens, it’s about building a new AV
requires a broad range of hardware, software and services. Planar works platform together and adapting one of the most exciting trends in tech-
closely with leaders delivering in these areas, including Epic Games nology to our industry.
15
NOVEMBER 2022
VIRTUAL AND HYBRID PRODUCTION
VIRTUAL PRODUCTION
IN THE REAL WORLD
Tying it all together with PTZ cameras and LED Volumes
By Erik “Wolfie” Wolford
16
SVCONLINE.COM
VIRTUAL AND HYBRID PRODUCTION
SMPTE STRATEGY
n response to requests from members to speed up its work on an area that could benefit from SMPTE’s expertise and because the tim-
18
SVCONLINE.COM
VIRTUAL AND HYBRID PRODUCTION
VIRTUAL ESPORTS
M
o-Sys Engineering has successfully supplied its LED content an LED ceiling to increase the production value and to assist the cinema-
server solution, VP Pro XR, via regional partner Cgangs for tographer by providing real-time ‘scene spill’ on the talent. Mo-Sys also
a major esports championship production in Asia. introduced Garden Studios who provided expert on-site technical support.
Cgangs was selected by American games developer Valve to handle “We had limited time on-set so needed to work quickly,” Lim added.
the production of the opening sequence and top-four team player intro- “We were able to do this after ample pre-production planning and select-
duction videos for the International Dota 2 Championships, a major ing products that would enable rapid ICVFX production. StarTracker and
esports competition held in Singapore during October 2022, with a VP Pro XR were perfect for this project, both in terms of set-up, work-
reported prize pool of $18.9 million. flow and reliable performance. It was also a great pleasure to work with
The integrator was introduced to the Dota 2 project in July 2022 and the DOP from Valve, who was using LED virtual production for the first
shortly after welcomed the Valve software team to its studios in Singa- time. He had a very clear vision and worked collaboratively with our
pore. After shooting a sample video based on an animatic demo provided technical team to achieve it, and this was essential to making the project
using green screen virtual production, Valve was able to decide on the a successful one, with very happy clients.”
same day to work with Cgangs. “It has been a great pleasure to support Cgangs, and it’s fantastic to see
“The project made extensive use of Live AR, and it all needed to be virtual production together with StarTracker and VP Pro XR utilized for
shot in a single day without post-production,” Alvin Lim, Cgangs stra- such a high-profile eSports project,” said Stephen Gallagher, Mo-Sys’
tegic partnership and business development director explained. We only marketing director. “VP Pro XR is enabling more and more projects to
had about 1.5 hours per team to shoot and capture the opening and intro achieve exceptionally high production value through its cost-effective
videos, which meant we were shooting real-time ICVFX and needed and dedicated feature set for film and broadcast.”
rock-solid camera tracking and a powerful LED content server solution Following the DOTA 2 project, Cgangs has announced plans to posi-
that wouldn’t let us down.” tion itself as a technology enabler to support and train media students
Cgangs utilized VP Pro XR, its LED content server solution designed and industry professionals. The company plans to set up a permanent
specifically for film and broadcast production, and combined with Mo- LED VP Studio to enable local production companies to have the oppor-
Sys StarTracker camera tracking, delivered high-quality Live AR. The tunity to learn and apply VP for their own projects, as well as to become
entire system was integrated with real-time DMX controlled lighting and the leading VP technology production company in the Asia region.
19
NOVEMBER 2022