The Importance of Reliability in Live Television Broadcasts: A Personal Experience

Reliability engineering and powerlifting.  Those who know me understand that I deep dive into both.  Although they seem diverse, there are a lot of things in common with powerlifting being a microcosm of safety, reliability and communication as humans and equipment are pushed to their limits.  In a powerlifting meet there are three lifts with three attempts each.  The athlete’s total is based upon the highest squat attempt, bench press attempt and deadlift attempt.  Missing three attempts in any one of these lifts disqualifies the athlete.

Setting up for the next lifter

My background is in electrical and mechanical physical asset reliability in both design and physical asset management as a consultant and practitioner.  My hobby, turned second business, is a sport known as powerlifting, which has been an underground sport for almost 60 years.  With a partner, Eric Stone, we operate as a powerlifting club and events company known as 2XL Powerlifting LLC.  To date, we operate up to a dozen high volume powerlifting meets per year and compete/World Judge a number of others.  This requires a combination of strategic and tactical planning to ensure great meets.

On October 28, 2019, 2XL Powerlifting, following a full weekend World Powerlifting Congress international competition (2019 WPC CanAm), we had organized, provided equipment, most of the staff and safety personnel for the first ever internationally televised powerlifting meet live on ESPN.  The event, called the WPO Finals, involved the 30 strongest men and women on the planet.  A number of us ended up being directly part of the production staff as the ESPN director wanted to ensure experience was in key areas.

To put this into perspective, the ESPN crew had no practical experience in powerlifting and had planned the event to run exactly five hours.  Through many years operating powerlifting meets, we usually had a rough idea of when things would end and would usually compensate for technical and other challenges.  In this case, the wireless scoring light system was brand new and only used as part of the weekend CanAm competition, the lighting and broadcast personnel had not worked together in the past, and there was less than 16 hours to connect everything and verify functionality.  Part of the broadcast also required a leaderboard using a common scoring system and the software for tracking, requiring double entries on every attempt, was developed by the ESPN staff, but untested in operation.  That required the person in the truck and an experienced judge/software/staff person to operate the other end.

WPO CEO Wayne Pullum (left) and Eric Stone of 2XL Powerlifting discussing the platform during build-out

In the end, the event kicked off at exactly 12:05pm and the last lifter completed at 5:05pm flawlessly.  OK, there were a few minor bumps, but nothing that most of the audience would notice.

In order to ensure that everything went well, we had to rely upon partially tested electronics, battery powered wireless systems, computers and laptops, data entry personnel, safety and communications, communication between the lighting company and table staff, and the streaming part.  This required cooperation and coordination at all levels, which we did start with challenges, but ended very well.

View from the table of the spotters and loaders in action

The first challenge was the platform.  With some of the athletes squatting up to 1300 lbs, the floor and equipment was critical, especially with the concrete floor not being perfectly flat.  This required the combined use of ‘horse matts’ (rubber mats) and two layers of ¾ inch plywood covered by carpet for friction and a frame.  The plywood was screwed together and crossed in order to remove any warp so that the surface was stable, especially during squats and deadlifts where the lifters have a wide stance and ‘spread the floor’ by putting pressure towards the outside of their feet and on the heels.  This means that the combination of heavy weight and the weight of the lifter is concentrated in a very small area, in addition to five spotter/loader personnel and equipment.  Improper setup of the platform could end up with serious injury or, as has occurred on rare occasions, death.  2XL Powerlifting personnel assembled the platform as personnel without experience could put the athletes in jeopardy.  Unfortunately, we have experienced events where such inexperience has caused problems with the result that it is left to the lifters to adjust their performance to meet the environment.

Howard Penrose working with the leaderboard software during the ESPN live broadcast

Overall, the electronics setup was interesting.  We had a brand new scoring system, speakers, radios, laptops, Wi-Fi, lighting structure, pyrotechnics, cameras, a broadcast vehicle, power everywhere, electronic order boards, projectors showing the bar loading (which weights to place on the bar), and even software that was developed overnight for the event.  There was the meet director (Wayne Pullum) and a documentary crew (led by Michael Fahey), four different gyms provided personnel and staff coordinated by 2XL, the ESPN crew, a separate lighting company, and event center personnel.  While judges are independent, they also provide an quality assurance that each lift is safe, and they are experienced lifters.

While most of the cabling systems, lighting structure, pipe and drape, platform and equipment were placed and tested the day before, much of the fine-tuning and testing of communications was performed within a few hours of the event.  The leaderboard software required a double entry where next attempts were entered into the scoring and bar loading system, which was a freeware system used for years in the powerlifting community, and a software developed the evening before the WPO started.  While the bar load software has warnings in place to prevent errors and changes lifting order based upon the rules, the leaderboard software did not.  Yours truly was put at the scoring table to operate the leaderboard software and maintain communication with the broadcast truck.  The lack of built-in software protection did result in one embarrassing inter-posed number during the final set of lifts (227.5kg vs 272.5kg), but that only showed on the leaderboard displayed on the broadcast and did not affect the actual results, which are tabulated on the scoring software.  In effect, a redundancy prevented a human error defect that could have had significant negative results with the difference literally separating first and second place.

Announcer side of the table (gratuitous selfie) my direct neighbor is Donnie Thompson, first human being to break the 3000 lb barrier

Within 15 minutes of the broadcast, a splitter was used to put the announcer narration onto the audience speaker so that the live audience could hear what was going on.  Unfortunately, it was the first minutes into the broadcast that it was noted that there was no narration and the backup local announcer had been cancelled.  This was a combined technical and human error condition that left the audience trying to follow along with the projector scoreboard.  The additional side-effect was that the light system personnel were unable to light off pyrotechnics for each World Record achieved.  It was also discovered that their radios did not work with the ESPN radios.  In the end, Eric Hubbs, another powerlifter, brought up the records on his cell phone after the first 15 minutes and communicated across the floor with hand gestures a process for identifying when World Record attempts were up.  It was also discovered that the announcers did not have information on World Records so we came up with a system to let them know within the first 30 minutes and one of the spotters, Robert Bain, who is a logistics manager by day, picked up on the situation.  For the remainder of the meet, he would shout out to the audience when the lifts were potential World Records, plus kept up a cheerleading narration to engage the crowd.  Overall, this was the human interaction when technology failed, and one of the reasons why automation will rarely fully replace human interaction.

ESPN personnel communicated with an expeditor who held the athletes behind a curtain until it was their turn to lift.  The two breaks were timed between squats and bench press and then deadlift such that the event timed out at exactly five hours.  The associated communication and system was arranged the final moments before the event started with coordination by an ESPN site director who had experience with running broadcast CrossFit events.

Scoring Table Amy Jackson and Eric Hubbs (tracking World Records)

The spotters, loaders and equipment on the platform and in the warm-up area were of greatest concern.  Human error and equipment failure in these areas are related to safety.  We’ve noted in some other organization meets that spotter/loader capabilities are less than needed, which has resulted in severe injury.  In the case of this event, we placed experienced powerlifters who have worked together on the stage.  The equipment was inspected, tested and load bearing components tested with one replacement on the platform monolift for clearance.  During the meet, several of the lifters lost consciousness during their attempts and were caught, including the weight, before any damage could be done.  In other cases, the lifters lost control and were caught before the weight could harm them.  This requires simultaneous action by all of the spotters in the form of communications between personnel who know each other and the activity.

Overall, in this microcosm there were numerous reliability, safety and communication lessons learned.  For one of the most relevant ones, related to automation versus human interaction, when software goes awry, personnel with a high level of experience are necessary to step in.  The present narrative from IoT vendors is that software will replace employee knowledge.  This can be a catastrophic recipe.  If software experts who were not knowledgeable were involved at the meet, then corrections on-the-fly would not have been possible.  The safety and communication aspects were just as important, requiring constant and immediate adjustment.  The human element, while introducing some error, is required for making non-linear adjustments to complex systems of interaction.  This is where software, regardless of what marketing concerns call it, fails.  Software relies upon a programmer’s understanding of a rigid process with tolerances and, in effect, if-then decisions.  They are decision trees with somewhat adjustable variables.  Only human beings are not linear in thinking.  True reliability, communications and safety in real life variable situations require human beings who can react to chaotic situations.

The event, so long as ESPN3 keeps it online: http://www.espn.com/watch?id=f3940194-b0b3-4da4-9eda-f3b97cc385ab

The IronAuthority Berserker Strength Radio podcast Strength and Anger coverage of the event can be found here: https://anchor.fm/strengthanger/episodes/Episode-7-2019-WPO-Recap-e8eelc/a-av799b

Please share our blog