Welcome to Disaster.Stream Bringing hard-won Lessons-Learned from Disaster Recovery Responders
Jan. 23, 2023

S1E5 Bank ATMs Down on Holiday Weekend

S1E5 Bank ATMs Down on Holiday Weekend

Bill Alderson and Guest Bill Genovese discuss how a bank's ATM's go down on a holiday weekend disturbing an entire country. 
Bill Genovese's team "parachuted" in to perform root cause analysis and workshops for the client team to improve resilience t...

Bill Alderson and Guest Bill Genovese discuss how a bank's ATM's go down on a holiday weekend disturbing an entire country. 

https://kyndryl.com

Bill Genovese Linkedin Profile 

Bill Genovese's team "parachuted" in to perform root cause analysis and workshops for the client team to improve resilience to prevent future disruptive events.  

Here is a direct link to play the video for this Podcast https://customer-7ycaiddtws6ryn8x.cloudflarestream.com/4c54d33cac7e8c486acf4efd1a532aa4/watch 

Link to Guest Introduction Videos and Alternate Distributions

Guest: Nick Leghorn Director of Application Security at The New York Times offers a workshop on building effective Cyber Policies that aren't Miserable for Everyone. 

https://customer-7ycaiddtws6ryn8x.cloudflarestream.com/9076eee7151e12ef0d223272333e5c49/watch 

Podcast Website: https://Disaster.Stream 

Transcript

[00:00:00] Hello, I'm Bill Alderson and we're talking here with a leader at Kyndryl, and he's going to introduce himself. Bill. Hi. Thanks, Bill. My name is Bill Genovese. I live in St. Augustine, Florida, but I've lived and worked all over the world.
In terms of full stack architecture and technology at the intersection of primarily one major vertical, which I would say financial services in that umbrella, banking, capital markets, securities, investment [00:00:30] banking and insurance, and diversified healthcare to a second extent, and then telco.
And mainly from the provider side. So I'm a second generation IBM er. I worked in four divisions in IBM on five continents. I've been all over the world with big blue ex Big4 consultant. So I was with KPMG, two tours of duty with KPMG as a senior consultant and then a director on a contract basis in their technology M&A group.
Doing due diligence for acquisitions for [00:01:00] clients. So I've worked in a number of areas and I've never really escaped risk. And everything that comes under that umbrella. I've worked in a number of business units in IBM, high availability Center of competency as part of a lab services executive consulting firm all over the world to their technology in major Fortune 50 banks in enterprises.
So this is a topic near and dear to my heart. look forward to the discussion and [00:01:30] seeing where this kind of takes us over the next few minutes, an hour or so. Bill, it's really a pleasure to have met you and started to engage hearing some of the incredible high visibility, high stakes stories.
Of response and problem resolution for yourself and your organizations. Really looking forward to some of these stories. Now, I'm not really exactly sure how much we're gonna be able to get done in the [00:02:00] next hour or so, because you have a prolific career. If you go and look at your LinkedIn profile .
It's just littered with large organizations with critical problems. Let's get right into it. Bill, what do you think are some of the stories, and we can go into more detail, but just give me a synopsis of some of the type of issues that you've dealt with in the area of critical problem resolution, disaster [00:02:30] recovery, unanticipated risks that have become actual risks today.
So help me understand some of the things that you might be able to help us with in this particular area. Yeah, my, my experience and , what I've encountered throughout my career, primarily, contextually this is mostly relevant to my work with IBM.
And Kyndryl is a division of IBM or was a division of IBM Global Technology Services, where I worked for almost six [00:03:00] years in two countries. So I know this space quite well and I am a CIO executive advisor as part of our advisory services practice in Kyndryl now working with CXOs. So resiliency is still very much top in mind.
It's part of implementing digital transformation. But when I was with IBM Global Technology Services gts, which is now Kyndryl, a lot of the content in where my career took off [00:03:30] was in the result of outages, a nd stability issues. So I followed the career path from consultant and I moved more towards the technology engineering side of the house in IBM and architecture and number of profession certifications.
And as you cover more and more architecture and technology from an enterprise perspective, obviously you're working in different camps, different layers of that architecture. So what may have started [00:04:00] earlier career as an application or software architecture engineer move more into infrastructure and data center.
And up and down the platform. So invariably when there is an outage in a major enterprise, where does it occur first? It's usually a cross-platform service or it's unknown to the firefighters. Is it the application? Is it the database? Is it the infrastructure? Usually the investigative discovery process starts at the [00:04:30] infrastructure in the data center.
And that's where a lot of the focus is in terms of remediation teams and support teams. Which is fine if it was 15, 20, 25 years ago where one application sat on one platform, but in digital transformation in a major enterprise in a major industry, vertical and international bank, internet banking, .
Or ATMs don't all sit on one hardware platform. So it creates required [00:05:00] correlation across support teams to see exactly where the issue is. In my career in IBM, as I became more and more of a senior architect and chief engineer, chief architect client technical leader, so these were the folks that were actually advising CIOs and banks.
And, I was the most senior technical leader on the account for selected Fortune 20 accounts. If there was a major issue like an outage, we would get called in and work with the teams over the contract period to [00:05:30] remediate what that is. So I've seen a number of situations and issues ATMs going down right before a holiday weekend.
Performance degradation issues. It could be as rudimentary as annual or biannual disaster recovery testing. That goes fine in terms of flipping over to the DR site, but the client or customer can't come back . And it's good to see that your DR is working in the event of a smoke hole in the ground [00:06:00] situation.
But how do you come back to restore everything? And if that can't be done, that's a challenge as well. So everything in between. Another highlight in my career after I left GTS Global Technology Services and IBM, I moved to systems and technology groups, high availability center of competency. So with that team of.
Experts. The best of the best. We were a very elite SWAT team that would parachute into [00:06:30] anywhere in the country, a comp world, I should say, on a moment's notice to remediate outages and have week long discovery sessions to get to the real root cause of what was going on. And more often than not, there's a familiar pie chart that is always in burned in my mind in memory.
Outages are not caused primarily by infrastructure problem. It's usually service process management. Or applications first and foremost. Then [00:07:00] infrastructure. So these workshops and the remediation efforts that we would get into would be carved into technology days and then service or process management days.
because we would want to see exactly what's going on that's contributing to the outage in future remediation. I think that gives you a good cross composite in more recent years in between working for big IT providers, I've done work in the M&A space. I've worked with smaller companies and [00:07:30] startups and tier two, tier three companies.
And my knowledge and expertise has helped me do due diligence in terms of acquisitions. So if a private equity firm was going to buy a company, a smaller company for their portfolio, what types of things should they look for in terms of a risky investment in terms of stability in the infrastructure and cloud provider, as an example.
Yep, I'm very lucky, honored and humbled to be here and I've had a very good career, I [00:08:00] think and hope to keep contributing. Very nice. Bill. It's really interesting to, to hear some of those stories, especially on the international level, that it wasn't merely in one market, but a myriad of markets across the globe.
So in, in our initial discussions in talking, I remember a few different scenarios that you spoke of in detail. Are there some of those that you'd like to [00:08:30] highlight today? Yeah, one, one long-term contract that I was involved with IBM, it was a very important account for us.
It was very high up in terms of account focus, our relationship with the customer and the client. We had a 10 year managed service strategic outsource deal. It was a bank in Southeast Asia. I had finished up another engagement in the same division in IBM for another bank in Europe.
And I [00:09:00] was in between assignments and I was due to return back to my home country, the United States. But then I found out, I was contacted about this other opportunity and actually there were two banks that were experiencing some level of stability issues, both in Southeast Asia.
And both were managed service accounts. And it turns out I was supposed to go to Thailand but I got rerouted to a higher situation crit, sit. For a bank that experienced island wide [00:09:30] ATM network outage the day before major holiday weekend, public holiday weekend.
Now Bill when ATMs go out whether it's a weekend or not, but in particular holidays, what happens to the community? And then how is that high visibility, high stakes kind of issue, how does that get pushed back onto a system provider or somebody who's providing [00:10:00] assistance or services? How does that affect.
the company who's experiencing the problem, and then yourself on the other end trying to help. Yeah. So to frame that a bit, it's good to time box it in terms of when this happened. Because that should give some context in terms of where we are in the industry and overall as a planet in that timeframe.
The question I would throw out was mobile banking. And the ability to consume [00:10:30] digital financial services from a payment. Payment transfer, moving money around between accounts, paying from your phone, paying from your face, ordering stuff remotely, whatever you want to do.
Was it the same as it is now post pandemic? I think we all know the answer to that. No. It wasn't So if you experience a mainframe outage, , with the lack of mobile banking and mobile financial services, that's widely pervasive and [00:11:00] used almost as a default mechanism as it is today, there's gonna be an impact to a society, ?
People need to get money. Out of their ATMs. Before holiday weekend before they travel somewhere. Or I can think of a myriad of situations why you need to get to the atm. And so very highly impactful situation. In terms of the core livelihood of a banking institution.
I can only imagine if, like [00:11:30] today a lot of us have reduced the size of our physical wallet and we put one bank card inside a sleeve of our cell phone to take with us. And that's the only one that we have. And now, if that were the chosen bank for our a hundred percent dependency and we're on a motor trip from one location to another location, The ability to get [00:12:00] petrol, the ability to get food, the ability to stay in a hotel is now highly impacted.
So no doubt customers of the bank are screaming bloody murder at this point. Yeah, exactly. And, you hit the nail on the head in 2010, the concept of a digital wallet and essentially living from a digital financial services consumption model from the perspective of the brick in your hand, didn't [00:12:30] exist.
or it was just starting. , so you were tied to that atm, you're tied to that debit card more so than you are now. 13. 13 years later. Yeah. You can't just add another card to your Apple Pay or your Google Pay wallet and change cards easily. Right on, on the run, you're pretty much stuck with a physical card that it either works or it doesn't work, and when it doesn't work it's essentially catastrophic.
It's a [00:13:00] catastrophic disaster for that person in that situation out of town. So did the companies that you were working with, The level of urgency that their customers were pressing them with? Absolutely. Once again this was a very successful account up to that point, with IBM. Who I was working with at the time and it was year eight of the first 10 year contract.
So we were entering, when you go into a [00:13:30] renewal for a managed service outsource contract, you're not waiting until year 10 or year four. You're starting the discussions year seven and eight, and you're positioning what's gonna be in that renewal. So for this to happen in year eight, , 
Is catastrophic potentially. So very high visibility from the provider perspective. The other perspective is, I'm not gonna get into naming any clients or customers here, but in smaller countries, ? In Southeast Asia, other parts of the [00:14:00] world, the most successful banks are the poster children in terms of their visibility with the regulatory bodies.
? So the smaller institutions can be a little bit waffly, they can be unstable. They don't have the wallet share of the population, they don't have the visibility. They're not the media darlings. So if one of the big anchor banks goes down, , that is the wallet share of the country. That's a major ordeal.
[00:14:30] So it's no longer just the customer of a particular brand of bank, but now it has national significance within the geopolitical organization that they're a part of. And of course that goes on the nightly news, yeah, you can continue that thread of thinking, .
This is a representative pillar of industry representing where that country is going in terms of technology innovation, and it has a failure.[00:15:00] That's not a good thing. So that's proverbial the black eye that we talk about in industry, right? Exactly. Exactly. So you get a pretty clear picture of the back backdrop and the context of what I was facing.
So how did that end up coming to, to closure? How did you navigate your way through? That critical problem? It was a holistic approach. All roads led me to that as a enterprise architect in my [00:15:30] career developing. So I didn't go into that situation being a web architect or an Oracle application architect or a DB2 database architect.
At that point, I had worked across all layers of the architecture in a number of banks globally. And then also I have a patent in terms of automation and provisioning and cloud environments with IBM I am certified as a technology consultant and [00:16:00] architect in terms of systems management and service management.
So when I went into that context and why I think I was brought in, I know why I was brought in. It was that full comprehensive diagnostic that I would need to do. People, process and technology in terms of, going into that dark room and flipping on the light. Where is everybody scattering from?
And it's not one, one situation. We've gotta look at everything, ? And [00:16:30] rebuild the estate, the culture, the people, the process, the organization, the technology, the infrastructure of the data center, and raise it from three nines, availability to six nines. And it was a complete, that's all I did for two years to help my company and the client and the regulator.
I was involved in discussions with the regulator. How do we turn this situation around and in a preventative sense, make sure it never happens again. And [00:17:00] looking back at that exact type of situation, what are some of the lessons learned that you brought forward to help the organization improve their resiliency?
Good. Very good question. And this is why I brought up this example, because, none of us is infallible. We're always learning, I don't care if I have 27 years of experience. You have 40, somebody else has 60. A couple of key points that have stuck with me in my career. Every single client [00:17:30] situation that I've been in, any country all over the world, I could be in the same city, in the same state in the United States on the other side of this, and I'll come across a different situation.
The context or the symptoms may be very similar, but the solution is never a hundred percent repeatable. I There's always a wrinkle. There's always something new that pops up. Bill, are you basically saying that there's no cookie cutter solution? So essentially if, let's just [00:18:00] say, of course IBM has a lot of major clients around the world and almost that one time or another, almost every company, but taking the solution from bank A and simply applying it to bank B, that doesn't seem from what you're saying to be the way things work.
Saying it another way, there's no silver bullet. I will say even another way, if you have a 95% silver bullet that you implemented in the United States. [00:18:30] And then you went to Europe and maybe in, in one or two countries it was a 92 or a 93% silver bullet due to other reasons. And then you said you based your assumption.
I've lived and worked all over the United States in small, medium, large environments. It's worked everywhere here. It's worked in two or three countries with some differences. Minor it's gonna work in any eight country in Asia. I learned the hard way. Culturally that's not the case. This is where I got [00:19:00] slapped a bit in the face with cultural and people differences.
 In terms about technology and services are delivered. And in terms of risk appetite. And approach and thinking. Approaches to enterprise architecture approaches to DevOps. Approaches to teams working together in terms of rigor and testing. I can go on and on.
And I had to learn that certain ways, mindsets of [00:19:30] thinking in the United States and the West and Europe. I had to jettison and adjusts on the fly from my experiences in Asia. So Bill, can it be said that regardless whether we're using the same technology, i e mainframe or certain types of systems, that almost every implementation of a company's architecture has a unique fingerprint that [00:20:00] requires specialization and theorists who can really look at the true underlying technology architecture, that it's not simply.
the same fingerprint that company A has, and you can simply apply that to company B. I haven't found that to be the case in almost anything. You have three banks who have mergers and all three of them have completely different technology. Even [00:20:30] if they're using the same IBM mainframes, their communications architectures, everything requires some type of specific planning to approach their architecture in the way that their architecture works and their fingerprint of technology, so to speak for that particular organization, which makes it a lot more complex problem.
And like you said, you can't simply use a paint by the number or a cookie [00:21:00] cutter plan to take disaster recovery for company A and apply it to company B. Absolutely. So as architects, as engineers we're all familiar with reference architectures for industry. What does an internet banking reference architecture look like?
What does a core banking deposit systems architecture look like? And its deployment patterns. But to your point, as everything around it has transformed those reference architectures are a point in time or a specific point in an organization from a pattern [00:21:30] perspective. What we need to be more adept at is identifying outliers and anti patterns as architects.
Those anti patterns that pop up right now in the next outage or tomorrow become inputs to the next reference architecture. And that's how I would best describe what you're framing up there. And that's what I've loved about my career. I've been exposed to not only looking for and being hung up on this is the reference architecture [00:22:00] I know and how it should be, but I look for the anti patterns.
Now first, can you explain to us a little bit about what you mean by those anti-patterns so that we can get our arms around that a little bit more? Yeah, once again, going back in time you had basic client server architecture based technology, ? And, you had an application with a call to a database, a client, a thin, a thick client in a call back to a database, via client server.
The application database could be on a mainframe, [00:22:30] but it was a thick client on an application that was installed on a workstation, then you went into thin client, ? Then you started to get involved with internet banking, transforming in terms of a anti pattern where you had front ends on web server X 86 farms thin client.
Then you had some type of compilation. Logic, computational logic, mid-tier engine, which could be on Unix risk-based systems. And then you have, in terms of messaging and connectivity, MQ back to the [00:23:00] mainframe database. Yes. And yeah, the, all of those technologies can be used in a different pattern.
The message queuing systems, of course, that IBM's famous for the database thin, thick clients and all those various architectures, even though they're the same technologies, they're implemented differently, perhaps different vendor interfaces, different vendor computers. So each one of those represents [00:23:30] an institution that has their own fingerprint of technology that you have to, as a theoretical expert, you need to be able to look at.
Holistically, like you said, and look at the exact problem situation that they have. And, teams in a support mechanism, troubleshooting firefighter context have not necessarily changed. Along with those anti patterns. In a holistic sense that says, I am the internet banking service support guru, [00:24:00] and I'm gonna look across all three platforms.
No, it's more often that they're broken out by infrastructure, platform and data centers, and you've got three separate individuals each looking at their monitor. Representing their tier of the service and them trying to figure out where's the bottleneck, where's the outage occurring? The front end's fine.
It must be you in the middle . Yeah, exactly. Exactly. Bill, one of the things that I was hoping that you might be able to share with [00:24:30] us is some of the messaging that you find to be cogent and relative that you're presenting now inside of your advisory services, that you might wanna take a couple of those anecdotal places and show us a little bit or talk to us about some of the messaging you're helping large institutions understand from a particular experience viewpoint.
Do you have anything to share with us that you'd like to discuss? [00:25:00] Yeah, I mean there's a, there's, this whole concept of multidimensional risk, and, catastrophic events, we've all heard the term black swan, ? And that more or less can, be framed up of a smoking hole in the ground scenario, ?
That comes around once every hundred years. What, if you have two or three of these events that are not necessarily smoking hole in the ground, lights out events, [00:25:30] but they're problematic enough to disrupt operations, and if you have two or three of them happening at the same time in different areas of the world the.
Combined aggregated result can be even worse than a black swan. And some of these are being characterized or defined as gray swans. So covid and the pandemic, not necessarily a black swan a transformative enough to it and the [00:26:00] business that major adjustments had to be made.
In terms of digital transformation I've been speaking for a number of years on what I call next generation digital transformation with the advent, for example, of mobile financial services. This Gray Swan event of Covid. The pandemic with some environmental hurricanes, typhoons, regional war.
May have kept us cooped up in our houses more. And may have [00:26:30] directly and indirectly fueled the focus in institutions to move further and faster with digital transformation, including mobile financial services. Obviously we had to do things more from a mobile position. Just a case in point, bank of America in my Austin community closed probably a dozen bank branches two of which are quite near to my home that I [00:27:00] used to avail myself to, and they were gone.
I drive over there and they're completely closed. Now, they kept the ATMs open at those locations, but subsequently, two of those are now completely closed down. They even removed the ATMs. So vast changes. I think when you talk about this gray swan, which is an interesting concept that I'm definitely gonna wanna study and keep my ears to the ground on, [00:27:30] because you're right, exactly what you just said has major repercussions.
But it wasn't a black swan catastrophic zero day event that brought everything down. It was kinda like putting the lobster in the pot and turning up the heat. It changed a little bit by little bit and fundamentally affected how we do business. I'll share a couple of slides here if I [00:28:00] can, to of frame up a bit more than a bit more about what I'm talking about further here.
Let me know when you can see my screen. I can see it and I'm broadcasting it. All right. From a kind of framing up exactly what I'm I was saying here, you know what the background and the problem, and this is not only, impacting financial services it's multiple industries in terms of a trend.
? So you have risk that's compounding, due to [00:28:30] multidimensional events. Obviously as technologists we focus on that middle pier pillar on the bottom half of the diagram, operational risk, but you also have a run on credit. Based on the confluance of multi-dimensional events, you have a run on market risk of pay.
People are selling securities, they need to become more liquid. If you can't get into your broker to make an appointment to sell securities, and some still operate that way, you wanna be able to sell and liquidate your stock holdings or options [00:29:00] from the palm of your hand. So there's other factors culturally too that's further compounding this and fueling from a consumer perspective the need for the institution to be more agile. If these events do come up and they are compounding. So interesting statistics across the top. From a consumer experience perspective, if these incidents and situations are going to occur, 50% of customers will give their [00:29:30] bank only two chances to fail before considering a change in banks.
? That's somewhat dated as a statistic a few years ago at least right now, due to everything that's been going on in the world for a number of years, I don't have all my money in one institution. I don't invest with one institution, ? A number of the institutions that I invest with, I've never had a discussion with a broker,
I do my trades via Robinhood or other institutions like that, acorns or Stash. So I'm [00:30:00] spreading my money around from a risk avoidance perspective for the very reason that, a regulator or bank would not go with one technology provider. And the more that you have correlation of risk events, , the more that you're exposed from a consumer experience perspective, by being with one player. So the traditional icons and titans of the industry in terms of market share, need to be aware of that. And that's what's fueling investment in adoption with fintechs and [00:30:30] smaller institutions with consumers, and some of which, have a lower risk appetite to get those consumers, but those risks have not gone away.
So it's a double-edged sword there. And then you see some other statistics across the top. Cybersecurity attacks, 93% still focus mainly on the finance sector. And then just the sheer volume in terms of performance and capacity degradation. Global trading systems and transaction systems.[00:31:00] 
The constant discussions that I get involved in mainframe is costing too much for us, Kyndryl or IBM. How do we move to a distributed environment? Can that distributed environment process like visa, , and what types of technologies in terms of containerization in cloud can compete with a mainframe environment in terms of its stability?
Very complex picture, very complex problem. That tied back to what I said earlier, there's no silver bullet solution [00:31:30] especially go. And it looks like these type of statistics and drawing our attention to this type of a problem the type of responses that customers have to these type of problems and what people do as a result of experiencing these things are a key part of decision making in these large institutions.
From a technology by Location or IBM. In solution development it usually falls into [00:32:00] fiefdoms or camps in terms of ownership. Being a bit colloquial and colorful and how I'm saying that, I just heard a recent saying, if it's not my pasture, it's not my BS!
If it's not my operational platform, if it's not my mainframe, it's X 86 or somewhere else. It's not my database, it's not my P&L it's not my problem. So even within each of these pillars, you've got silos, ? And all it takes is two or three correlated gray [00:32:30] swans, some morphin to a once in a hundred year black swan event that blows the walls of those silos completely down.
Very interesting. Thank you. Thank you Bill for that. So what are we what are we looking at in, in some of these textual things? I know that there's probably some key components that you can talk to or bullet points that you can talk to about some of these materials that you have created for your customers that are using your [00:33:00] advisory services.
What kind of things are you helping industry trends and state of the industry are you helping people understand so that they can make the better decisions within the environment that we're in today? Very good point. And we're always we've always been a in a catch up mentality or mindset as humans, regardless of what country we're in, especially from a reactionary regulatory perspective.
So if it's not broke, don't fix it or how it was broke the [00:33:30] past becomes the road for the future. What if we haven't encountered new ways that things have gotten broke or how they're even measured. So in terms of, reserves for risk protection and banks, and determining its level of risk in terms of society, traditionally all along it's been the size of the bank is based on, its in terms of assets.
So how much is it holding? [00:34:00] But that necessarily doesn't get into how interconnected it is on a world basis with other ecosystems. So that needs to change to reflect where we are going in terms of multi factorization. And assessing and preparing for risk events. So an alternative approach, .
In global regulators such as the Basel committee are looking to tweak this further, are carrying it past size of the bake institution and asset [00:34:30] holdings alone. So how interconnected is it? There's a great chart, another chart I have that shows all of the cloud providers working with the major banks and how interconnected this landscape is.
Between aws, Azure, G C P. and the foreign bank providers in China, the foreign cloud providers. And when you look at the interconnectedness picture there, from a potential risk issue perspective, and if one, [00:35:00] one piece goes down, how it affects everybody that's connected. That's critical.
So it's no longer independently how much money each of those is holding. Then you get into other factors. Of suitability. This is component, component failure, but on a more macro sense, how can things be swapped in and out? So the ecosystem keeps going.
How complex is the interconnectedness? Is there cross jurisdictional activity? So it seems like Bill what you're talking about and trying to help us all understand is [00:35:30] that there's a high degree of dependence. And what I just heard you say is not only on. A particular mainframe technology or a particular cloud technology or a brand of cloud, but there are national inst ances of cloud capabilities that are not even AWS's Azure or Google.
They're another localities type of cloud or different institutions or nation [00:36:00] state cloud. My, my company works in multi-cloud management. We recognize this issue. Many of our c lients are not in one country in terms of data centers. And based on that, whoever they're working with in terms of a cloud provider in one country, they may not be able to, in another country, the world where they're.
So how do from a risk management perspective, get your arms and head around that to see exactly what's going on? The dependency matrix must be absolutely, [00:36:30] incredibly complex to consider. Yeah. And is that something that you help people with, is to look at that complexity and those dependencies and factor that into the way that they have to respond and what they have to manage?
Absolutely. We have a number of solutions and capabilities. There's one we launched this year called Kyndryl Bridge for IT operations and multi-cloud management that we're using internally in our managed service context, but it's completely open in [00:37:00] terms of the architecture to work with multi-cloud vendors for this purpose.
So we can gather that information and provide better visibility into a complete estate for those reasons. Absolutely. Getting into a bit of I'm showing a chart now with, some terminology again in the past, and based on how we approach things, myopically, once in a hundred year storms.
The financial crisis of 2008. But what happens if you have Covid and then a major regional war like Russia and Ukraine? [00:37:30] And the impact on commodity and energy markets. And supply chains that were already crippled by Covid that are even more so now. With the regional war in Russia and Ukraine.
So these are independent black swans that are of morphing into they could be the next generation black swan, but they're gray swans right now that are boiling up. And I'll explain more on what I mean on that, on the next chart. But there's also new types of black swans.
As digital [00:38:00] transformation, decentralized finance, the fueling of the everyday layman getting into investing in trading from the palm of his hand without getting professional investor guidance. So there's a great movie everybody should go take a look at when you have time, if you haven't seen it called Money Monster.
And this is all about hedge funds Jerry rigging outcomes and commodity markets to get a short position and it benefits the [00:38:30] hedge fund. But everybody else that's invested in that, and everybody that's participating in those shells and concentric circles in that hedge fund gets screwed and lose everything.
What is there out there that prevents something like that. FTX is an example, in the crypto space. That's what I was just gonna, I was just gonna ask, talk about a current event. We're talking exactly about that type of occurrence and we've seen it in the past with some of the [00:39:00] other big organizations that have been subject to various scams both small and large.
How. How does blockchain enter into this equation to some degree? What are you looking at solutions that use blockchain technology and I'm making the assumption that we can use blockchain technology for more than simply crypto. Yeah. So a number of points before I get into that a bit [00:39:30] further, but that's part of general trend in technology overall.
That the technologies are not legacy technologies anymore born and bred and developed in the enterprise of the institution. As the economy has moved more towards a consumer economy out of necessity. Less to a market economy in terms of technology, people transacting from their mobile devices.
Using digital wallets that are supported by blockchain. In a [00:40:00] decentralized sense, public blockchains Ethereum outside the enterprise walls. To move money around payment transfers, remittances. Via the blockchain Bitcoin transfer. You can pay and PayPal in cryptocurrency now.
As an example. So a lot of disruption going on there from a consumption base. That, that is definitely occurring. Let me stop sharing for a minute here. Go back to the Sure. While you're getting set up there, I'll [00:40:30] just talk a little bit about the fact that our world is moving at a very rapid rate.
With technology and what you talked about it with digital transformation and disaster recovery because of covid is having to change and transform along with all the various digital transformations. And it seems that I hear people talking about digital or digital transformation and a disaster recovery, [00:41:00] having to keep up with those type of things and.
Make sure that their plans are still relevant and meet the needs of the new digital transformation. And then you talked about the nationalization, or the internationalization or the globalization of all of these things for so many of the organizations that you are serving. And of course, blockchain is a big issue as well as some of these other things.
And [00:41:30] so if you wanna queue up the next slide there, you go ahead and I'll bring it on board for us. Yeah. So the a as you bring up that point, the individual institutions from a regular regulator perspective, and what I'm specifically talking about are the central banks.
? They're the ones in each country and each of them across the world. This chart is showing. supervisory agencies that have the capability holistically [00:42:00] to look at black and gray swan events and what needs to be in place to remediate in a preventive sense, any bad things from really happening.
Now, across the top you may recognize some of those acronyms. SEC obviously is the United States Security Exchange Commission. Monetary Authority Society of Singapore d n b. , so these are all big institutions. Bank of India, Boi, in powerhouse [00:42:30] countries that are the policemen of the banking institutions.
? What you see here is on the left side, supervisory. And then you see color coding in terms of if there's a solution in development and experimental stage in development or operational production. Now this came from a report from the Central Bank of Central Banks, the BIS.
Bank of International Settlement. So that's the police dog [00:43:00] watch organization over the central banks. And they took a look at this a number of years ago. There's white space all over this. First of all, as we're moving into blockchain, decentralized finance, emerging tech, these supervisory areas are not silos.
They're more and more correlated, especially with interconnectivity with big tech in the cloud providers. You'll see. That even if you pick one of the [00:43:30] supervisory areas and go horizontally across the chart, not one is fully operational for all of the big central banks. That's illuminating
Number two. Going vertically down. From a comprehensive, holistic correlated perspective, from a supervisory risk perspective for remediation, nobody's got 'em fully operational either. So those are the ones that are all [00:44:00] yellow in that second line there where the realtime monitoring and all the various efforts, and some of 'em are not even yellow.
They're blank. Yeah. What does that say? Haven't started.
So this is largely reflecting too, if you trace back. So like I said, this is a bit dated. It it's at least three, four years old, so there could be some improvements on this, but for the most part it, the picture is still the same. But if you trace back in terms of the big black swan [00:44:30] events, and I mentioned some of them, .
In the past 10, 15 years, you'll see that the color codes that are operational that are put in place, especially in certain jurisdictions. Those are the red ones, I presume. Yeah. You'll see that those reflect a reaction to the major event. There's the pattern . Got it. So if you went, so if you went back and looked at a timeline of this, , [00:45:00] you would basic a chronological timeline of when these things started to appear to be regulating or to work.
They would correspond to, for instance, some of the big world events of 2008 and other such situations. And that they didn't really do that for that kind of correlating regulation until after those type of events. Is that what you're saying? Yeah. And there's been [00:45:30] some intentions, for example, to modernize this view based on how the world is changing.
So the best example I think I have on this chart is macro financial risks and emerging risk signaling. So some of the major players of joining forces and saying, look, we need to be more holistic on an international basis, on a, how we're all interconnected on a macro level. And start developing solutions in that context instead of our own sandboxes.[00:46:00] 
So basically at this point, any of those areas that are not yet operational are pretty much wide open to black swan events and to all sorts of disruption in international commerce and banking that might Absolutely, bill, let me just throw a complete wild card scenario out at you.
Everybody right now, in the past 60 days or so, maybe a bit longer is all [00:46:30] excited about this open AI chat G P T initiative that was fueled by Microsoft and natural language processing. I've read now that and what it can do. So you can ask it a question. and the response accuracy it comes back with is phenomenal in many cases.
In terms of how fast. So let me add some almost instantaneous feedback on that. , over the weekend I was on chat and I was [00:47:00] exercising the system and I'm a computer network analyst and so I asked it how to write a computer network analyzer called Wire Shark, how to put in a filter for a certain TCP application port and if it would write that for me.
And sure enough it came right back and it delivered the exact appropriate syntax for that. And then I started asking it some questions about disaster [00:47:30] recovery plans or disaster recovery surveys, and it was amazingly accurate. Now it's not going to do a comprehensive. Amount of work for us, but in a small amount of work that you ask it to do, write me a disaster recovery plan for the major risks in Austin, Texas.
I actually asked it that, [00:48:00] and it came back and it told me the natural disaster type specific problems that Austin would have that others wouldn't have. So it's an incredibly accurate, albeit very specific and to compare the chat AI with something like Google Google makes money when there's clicks.
So when you stop clicking, Google stops making money. Yeah. So Google makes us click. This technology, [00:48:30] you ask it a question and it gives you the exact specific answer. With context that you're looking for in almost pretty much every area of technology or information that I could quiz it with.
Amazing. But that's based on a certain context, like you said in dataset that it's available and has access to. And it's not gonna cover the unknowns. How can it, what I'm [00:49:00] saying is there is an inherent risk there as it evolves further. And how do we know that what it provides as an answer .
Is the best solution in, in terms of its input and output. It's still binary to some degree, right? Oh, absolutely. It's very specific and it's also it warns you that it's not accurate in all cases. And it also says I haven't really been taught anything prior to 2001 or after [00:49:30] 2001. So my, it gives you these pieces of information.
The other interesting anecdote on this is that it remembers everything that you asked. So I asked it to create a Cisco router configuration for five VLANs, and then I asked it a second question, which is a very technical thing. It drew out and it gave me the exact syntax for that Cisco router config.
And then I said, now put Ether channel connections [00:50:00] between the various. Switches and it gave me the exact Cisco syntax to do all of those sort of things. So it's actually quite capable, but it, like you said, it, it doesn't know or understand or anticipate other things. So it's very good for very specific tasks with very specific outputs.
But like you said, it does not know the future, but it is pretty amazing [00:50:30] to, to utilize the tool and to get some experience with it. I just did that this morning and over the weekend. So I wanted to let you know that is something that's happening today, although I could not imagine anybody depending upon that for any type of mission critical system.
Yeah I bring this up as an example because there's the row in this chart here, and then I'll stop sharing machine readable regulations. In this [00:51:00] context, I can see how that type of solution may want to fill in some of that white space. But it's still in its current way, shape, and form is open to bias in a limited dataset perspective.
So there, there's still some inherent risk in that solution. It's just very interesting as AI develops and takes a life on its own. I You've read about, AI being able to program itself or code itself. There's gotta be a [00:51:30] base starting point.
For all of that. So yes. Now as we start to wind down, I'm wondering if you could summarize for us some of the lessons that we have learned in today's session and prepare us for some future sessions that we might do on some of this very complex global dependencies on technology. And as digital transformation takes us [00:52:00] forward, what are some of the lessons learned, you think that we have gained and what things like you just mentioned, do we need to take care of in the.
Yeah. It goes back to basic hygiene and, should be baked into dna. So make your core solid number one. If you look at the picture of the concentric circles when you cut down a tree, ? And innermost circle is the oldest part of the tree as it, and then it, grows outward.
You get newer pieces of the bark and the layers added [00:52:30] into the tree trunk. That's the liquid ecosystem, ? As you're growing that tree outwards and you add more circles around it, ? And the way that I visualize that from an analogy perspective, that's more disruption to your core business.
? So you've gotta make sure you've got a solid core, ? And, in terms of frameworks and methodologies that have further evolved, zero trust architecture, ? Not only protecting you from. outside in threats, but inside out threats. Looking at your application and [00:53:00] service estate.
Not just applications, because it's not applications anymore, serving one function or one service. There's interconnectedness. So do you have a proper view and inventory from a categorization in terms of criticality perspective along confidentiality, data integrity and availability. So for example, a, B, C, 1, 2, 3.
So if an [00:53:30] application or more appropriately a function or service is rated one for confidentiality, integrity, and availability, that is lifeblood to the enterprise. That's like oxygen, right? You can't live without it. So that means from an outside in perspective, strengthening the core, you need high availability six, nine s and full DR for that service.
? And, most institutions don't even do what I just described. And as you [00:54:00] go further out from that core and add those additional concentric rings, there's gonna be different permutations of the 1 23 ratings. You could get into 1.5, 2.5, and don't you see don't, doesn't it seem like we are also transforming some of our technologies?
We're still moving additional resources and systems to the cloud. Those cloud systems are still moving into microservices. They're moving into various [00:54:30] containers. That, again, shift the paradigm in being able to build those systems up and scale them rapidly, but at the same time, it increases the complexity and changes the technology in those like you said, it beautifully said is the concentric rings on the outside that continue to affect even though we have a good core, we're still changing as we grow.
Yeah. And to, to some extent, [00:55:00] it goes back to the piece of slogan that I mentioned earlier. It's not my pasture in the core ring. It's not my bs I'm shifting it off to a container to, to an outside ring, but that doesn't necessarily reduce the risk. Exactly. And then disaster recovery.
In some of the things that you discuss with your customers are you finding that the ability to have disaster recovery is more [00:55:30] synchronous obviously than it is asynchronous? In other words, it has to people cannot tolerate any downtime or minimal downtime.
And do organizations who spend a lot of energy, money, and budget building out realtime capabilities, do they also sit back and look at the potential for catastrophic situations where they have to accept that they may have to [00:56:00] triage for something that's unforeseeable? My experience, it's usually been reactionary, so there's been some type of incident, there's been some type of monetary loss, reputational loss.
And, the amount of focus that they go back and look at this varies depending in some correlated sense to what that loss was. And it really depends. Some of them still treat it like a speed bump. It was an [00:56:30] annoyance. We learn from it, they look at it in context to the same or similar things happening to their competitors
and they more or less chalk it up to the cost of doing business. And I don't necessarily agree with that, especially with that statistic I showed earlier. 50% of customers are gonna give the bank two chances to get it right. You can get it right now. A hundred percent. I'm still taking my money elsewhere because in terms of consumer experience, I won't name my bank, but I used to work for them.
I've been in and out of [00:57:00] them for projects through two companies, and I'm still with 'em over 20 years. Do you think they send me unsolicited offers to make my life better? Hell no. Meanwhile, I, meanwhile, I've been with PayPal for two years, two and a half, three years. I've got credit lines with them. I can buy crypto with them every six months.
They're offering me some new way to improve my financial situation, unsolicited. And that's all it took for me to move some of my [00:57:30] business to PayPal. Yes. So it's interesting because today and I know we've been talking about large corporate banking global and other such things, but have you take into account.
Companies that are utilizing things of like Facebook for all their marketing. They get all of their sales from Facebook marketing large companies who are utilizing that for all their new business. And last October 4th, 2021, they went [00:58:00] down for six straight hours catastrophic outage, a black swan, a zero day, so to speak.
And, it's not so easy to just say, , I'm moving all my marketing for to my Twitter right folks, or I'm moving everything over to some other LinkedIn or something of that nature. It that, that requires not just. A disaster recovery capability, but something that has to be baked in for, many years to move all your marketing from one [00:58:30] social media platform to the other.
And of course the lessons learned that Facebook had that, that cost them 25 to 50 billion in that one day. And of course it went, this talk went back up, but somebody on that day lost between 25 and 50 billion of value and may have made decisions. Like you said, banks are not very, forgiving of customers are not very forgiving.
They see this happen once and they say it might happen again. They'll give you that one, but they [00:59:00] probably wouldn't give you a second one. Is there any allegory to the banking world and, revenue production or nonstop systems that we can take away from that type of an event? . The disruption of the traditional industry, the traditional bank has not moved other to other providers that quickly. For those reasons, they just don't have the industrial strength capability in context with the volume that they need to protect yet.[00:59:30] And it's the not, it's not the same type of data.
As well. Which we all know. Even like your example using PayPal. Yeah. You can't move between Stripe and PayPal and banking, traditional banking . Very rapidly, probably more rapidly than you could move from Facebook to Twitter. But nevertheless it's a macroeconomic change that and it's hierarchical based or tiered based in terms of risk appetite.
So the consumption [01:00:00] model that I look at and use and present, in, in terms of the bastion of what's being disrupted, payments has already left the building. That's fair Game to a number of providers. The bank doesn't own that anymore. Anybody can do payments. these days.
That's why Facebook and Apple and Google, they've all gotten into this space. When you the next piece is really lending and credit. And some of these other alternative providers have moved into that PayPal credit. So they know what you're spending your money on. [01:00:30] It's a natural extension to offer you credit vehicles in financing.
Not much more risky. There's a credit risk scoring algorithm. You've gotta have reserves in place to protect. But there's all kinds of buy now pay later schemes as well. Yeah. So that's being disrupted. The key part that's staying away from the disruptors has been really asset preservation i e deposits.
To a certain extent. And where there's more regulation. You need a more [01:01:00] intense banking charter to hold wallet share of a customer in your system. Got it. And same with investments or insurance. Because if there's a, if there's a total disaster or loss and you're holding people's money when you're promising some type of return, or you're ensuring it, if they lose it, then your risk quotient is much higher than if you're offering 'em the credit or just processing payments from point A to point B.
Yeah. Yeah. Good point. [01:01:30] Yes. So in closing I wanna give you the last word and let you just talk to our audience and discuss some of these lessons learned and where you think things are going and how you and your organization might be able to help people that are struggling with these exact type of issues.
Yeah. Number one, this is a holistic perspective. And it's number of analogies I used in terms of peeling back the layers of the onion or the concentric trees in the circle. And the other point is it's [01:02:00] people processing technology not to use, a common term that's been bandied about for decades, but it still is definitely all about that.
Digital transformation is not a technology play. Only it covers your organization. It covers how you're interacting with your target customer and who that really is to improve their experience. Whether you're B2C, B2B, or B2B to C. And we primarily plug ourselves in terms of B2B and to a second extent [01:02:30] B2B, B2C context to help clients.
But our approach is really digital transformation, not only from a technology perspective, but business strategy enabled by technology as well. Very good. Thank you so much. We've been talking with bill Genovese and he is the CIO advisory partner and c t O of technology strategy. Kyndryl, a former IBM technical services company. [01:03:00] So I just want to say thank you so much, bill, for joining us today. We look forward to having you on a future broadcast, and thank you so much for joining us.
And folks, if you want to get in contact with bill, we'll give you his contact information in the down, in the show notes so that you can contact Bill or ask him for some type of a presentation to talk about your particular issues in your environment. So now [01:03:30] thank you Bill.
Really been a pleasure to talk with you and to get to know you. Look forward to additional times on disaster stream, disaster recovery responder stories. Thank you. Thank you. Thanks for having me, Bill. Thanks. 

William GenoveseProfile Photo

William Genovese

William Genovese

Innovative IT professional with over 20 years of experience driving revenue and profit increases through value enabling technologies for employers and clients in U.S., Europe, Asia and South Africa. Skilled in designing, building, and implementing high-ROI strategic architecture, platforms, applications, and programs, delivering and creating business and technical strategy resulting in next generation digital services, establishing and executing strategic plans, and providing strong innovation, digital transformation and operational leadership.

Utilizing IT as a transformative tool, created cost-effective solutions that provided security, functionality, scalability, performance and reliability in enterprise IT environments. Leveraged emerging technologies and best-in-class software with state-of-the-art development methodologies, architecting scalable software and system solutions, driving transformation building next generation digital services and implementing business improvements. Can contribute immediately by:

* Translating and converting business strategy into technology innovation to solve business problems
* Business and delivery model innovation - definition, development and execution
* Innovation and technical leadership for exceptional strategy to execution outcomes
* Risk and Compliance transformations that capture new opportunities and drive revenue and profit
* Forging profitable partnerships among vendors, customers and key stakeholders
* Aligning personnel, services, processes, and technology to positively impact the bottom… Read More