Podcast link – https://anchor.fm/steven-landes/episodes/3-Surprising-Building-Blocks-for-OpRes-e14q65d

BLUF – Developing operational resilience – an organisation’s ability to continue functioning through failure – requires reconsidering what brings the effect of shock and volatility closer to zero. Looking to the relative resilience in place twenty or more years ago, the need for analogue solutions in digital spaces becomes apparent.

Background

Before Colonial Pipeline, many inquiries were made about potential hacks of ICS/SCADA (to be fair, most of us didn’t see the billing portion as capable of shutting down infrastructure, but I digress). In examining the potential for interruption of refueling capabilities for forces deploying to the Pacific theater, I recall talking with ICS management for the fuel depots on Guam. At the time cyber wasn’t well understood; it was prior to having a domain, certainly before we had more than DDoS and website defacement as a known threat. Having heard me outline potential threats, the manager, who was easily thirty years my senior and half a world away, laughed.

‘What’s so funny?,’ I asked.

‘Son,’ he said, bemused. ‘We don’t need to worry about the systems or sensors going down, we’ve manual operations for override and people who know what is and isn’t right in the system. Won’t be easy, but we can take care of it.’

It was nearly ten years ago. After Colonial Pipeline went down, I wondered if his words still held true.

Ocado’s robotic crash last week set the stage for delivery delays in the short supply chain. With thousands of orders cancelled and delivery dates unavailable, the 1% of Ocado’s infrastructure affected by the crash has second and third order effects spreading far wider than originally anticipated. There was a fairly quick recovery time (depending on whom you ask) and relatively speaking, the impacts weren’t as bad as they could have been. Regarding OpRes, questions are raised:

We automate to take human error out of the equation, but how do we inject human solutions when automation fails?

We have a generation who understand optimal use of a platform, but do we still retain how the system was built?

Older cars have advantages versus new cars – they may not self-drive, but there aren’t cascading sensor failures either. As we consider the market/ region/ industry wide effects of organisations failing, three strategic considerations emerge to ensure operational resilience, that are otherwise not considered.

Organisational memory: Just as was pointed out for FIs, soft cues and memory retention passed from the experienced to the newcomers helps reduce the organisation’s potential for getting caught in risk. In Guam they ‘felt the pulse’ of the fuel systems, regardless of sensor data. To know how it feels (before it breaks) one has to learn interpretation of sensory data otherwise not captured – requiring someone who knows to help identify what senses to employ and what to look for.

Manual overrides of automated systems: If you catch the first pebbles starting down a mountain, you may avert the ensuing landslide. Implementing manual overrides to be employed – swiftly – when a failure occurs requires knowing not only what to and when, but the cultural encouragement to make the call to do it. In many cases the fear of incurring operational costs for a false positive provides hesitation where tenacity is salvation.

Need for experiential and age diversity in the workplace: The perspective and drive of youth (and the relative cost savings) is often preferable to older, experienced hands in the industry. We cycle people out at a certain age, losing knowledge depth as their experience isn’t carried forward or relayed. Age-based inclusion efforts offer the analogue skills of years past seen with Norsk Hydro – offering means to better understand the environments we seek to build.

How does this relate to the remote work challenges recently found?

The push to lose the office environment and greater work/ home balance found possible lose the connections between people otherwise found in co-location. This doesn’t mean it has to be lost, but efforts must be made to connect past Zoom calls.

Odd as it sounds, water-cooler and side-hall conversations are where soft-skills are relayed and war-stories are told.

Consider them organisational Aesop’s fables – creating operational resilience over time.

-scl