Disaster Recovery in Depth

Disaster Recovery in Depth

January 29, 2013 2:00 pm

(Save to cal)


Online Tech's Systems Support Manager Steve Aiello leads a three-part webinar series on the topic of disaster recovery.

Transition from theory and thought processes into practical application of disaster recovery.
Covers various disaster response options including:

  • Hidden Benefits of BC / DR planning
  • Disaster Case Studies
  • Staffing / Facilities Recovery Strategies
  • Processes organizational design
  • Facilities design that facilities DR
  • IT tools available to increase availability

View slides (PDF).




Additional Resources:

Bruce Schneier

Strategy + business (autumn 2012)

ABN AMRO Chicago Fire

The Motor Vehicle Supply Chain: Effects of
the Japanese Earthquake and Tsunami

April: Hi everyone, and thanks for joining us this afternoon. We are on our second of three webinars focusing on Business Continuity and Disaster Recovery. I am happy to welcome back Steven Aiello. For those of you who don’t know about Steve, he is Online Tech’s systems support manager and he also is a certified information systems security professional (CISSP). He also has a certificate as an incident responder, among a dozen others, including VMware and Cisco and some other security ones. I’ll let Steve talk about that in more detail.

Two weeks ago we started this webinar series and we started at the high-level overview focusing on a business impact analysis. For those of you who were not able to attend it, it is posted on our website and we’ll share a link with you. (Business Continuity in Lean Times - http://onlinetech.com/events/business-continuity-in-lean-times) Today we’re going to focus on different business organizational structures and how those different ways of organizing business units impact the risks that they save and how they may want to plan for and how they react to disasters. In two weeks from today, Steve will conclude the series by talking about more of the technical aspects of disaster recovery and dive into some of those technical details.

For now, let’s focus on the second part and how the organizational structures might impact risks. Welcome back, Steve.

Steve: Thank you everybody for joining us again. I was actually surprised how many people signed up. Generally I think this is a dry topic that only I enjoy, but that may not be the case.

April: It’s just your personality.

Steve: Yes. I don’t know about all that. Thank you very much for coming back. I know there were some folks who weren’t able to make it to previous webinar, so what I want to do is to take a quick recap about what we were looking at in our last webinar and just cover some of the very, very high levels, the points that were really key, so that if you weren’t here for the last one you won’t be completely lost.

Getting into it here, essentially we talked about a couple key objectives or key terms rather.

We have an RTO, recovery time objective. Basically just in simple language, how long can something be down before it impacts your business.

We talked about RPO or recovery point objective, and that is how much time can you lose if the system is down before it starts impacting your business. If you had an email server and the email server was down, let’s say only for an hour, but a large portion of your database was wiped out and you lost 12 hours worth of email. How would that impact your business?

Then we talked about this third one, which may be what most people are not familiar with, and that’s the concept of backlog or of backlog trap. That’s where every business has a certain amount of things or a certain amount of parts that they need to fulfill or services that they need to fulfill over a given period, and what happens is when you’re down these things just start to accumulate in a backlog. That’s what basically backlog or backlog trap is. If you get so far behind, you’re just never able to recover all those services that you were delivering your customers.

Those are three real big points. I just want to point out the big key is what is the threshold where those things will start to negatively impact your business. As we spoke about in the last webinar, this is not … Business continuity and disaster recovery planning is not about IT, it’s about the business. When you think about these things, think about it in terms of it impacting your business, your profit, things like that.

Steve: The second thing that we spoke about is this is a very long-term project. This is not something we’re going to do in a month. This is not something we’re going to do a quarter. This is a 12-month to 18-month project.

Really, I don’t like the term project because that almost indicates an ending. Projects start and projects finish. This should really be an ongoing process in the organization that really, under best practices, should be visited preferably quarterly. The minimum on a lot of the compliance initiatives are annually. Just remember when you’re undergoing and you’re designing a business continuity and DR strategy, it takes a lot of effort, it takes a lot of time, but it is very well worth it.

We were talking about risk management. A lot of times you hear people when they talk about risk, they’ll just toss out the term, “We want to mitigate risk.” Mitigation is not the only feather that you want to have or the only arrow that you want to have in your quiver.

There’s four different ways that you can manage risk. Whenever you’re dealing with a threat to a business you can never completely avoid risk unless you’re not in business. That’s why certain companies will choose or not to choose to do or conduct business in certain geographic areas. They may not conduct a certain line of business because it’s too risky, but avoidance, choosing not to get into a line of business or not to do business in a geographic region. Mitigating, which is really what we will be focusing on today, it does get a large, the lion’s share of attention when talking about risk management. Transferring risk, we spoke about insurance, and the act of acceptance.

These are all ways that as you are looking at from how to protect your businesses and your companies and your assets, these are all different tools that you have. It’s not just about risk mitigation, it’s about risk management. In some ways you might not be able to mitigate a risk, you might have to accept it, but just remember that you have all of these tools available to you. You don’t have to just accept it if you’ve done the due diligence and really thought about what that means for your business.

We have a couple of critical questions that we want to ask every single step as we go through this process. We want to ask questions. How do we know what to recover? We know what we want to recover when the loss of that process, that asset, that service impacts the business in some financial way. Being that everybody has fairly limited resources, you’re going to want to align your strategies with the things that impact your business the most. How do we make wise decisions on how much money and time?

If you remember when we spoke in the last webinar, there were very specific formulas that you could use. I won’t cover them all now, but if you go back and read the last webinar or re-watch the last webinar, we talked about annual loss expectancy or single loss expectancy. There’s just a real quick and dirty formula that you can use to help you get a solid base for what you should be spending.

Again, do you have the full support of your executive body? That’s really one of the most critical questions. If you cannot say yes to that third question then it can be a very, very frustrating road that you are going to have to travel.

Steve: Any kind of questions – I’ll try to go through the chat – on the subjects that we covered last week.

April: Please feel free to state your questions as they come up and we’ll find a chance to answer those throughout the course of the webinar.

Steve: Like last week’s topic, I had the quote from Dolly I really liked. I really like this quote for this piece of the webinar: “Know your enemy and know yourself and you can fight 100 battles without disaster.” That’s from the great general, Sun Tzu.

As we go through this webinar, every single thing that we talk about will go back and relate to that concept. It’s knowing your business and it’s knowing the threats that are out there that could impact your business. Just keep this in mind. Know yourself and know your enemy. That is how you become very prepared in a situation where your business might come into some sort of harm.

Last week we really focused on the top two areas in this Seven Steps that were defined by NIST for business continuity planning. We spent not too much time on the continuity planning statement or the policy statement, we really spent a large portion of the webinar on the business impact analysis because this is something that a lot of companies don’t do but it actually provides the meat and bones and the skeletal structure for your business continuity and disaster recovery plan.

Even when I talk to people that do this full-time for a living, I say, “Hey, tell me about the steps that you go through for your business impact analysis.” A lot of people will go, “Well, we just made rough guess at where things were coming from and we just loosely broke out where our profits were coming from and what tools people were using.”

That’s why we wanted to talk about the business impact analysis really in-depth. Because if you don’t have that, if you don’t understand where your profit’s coming from, what the processes are that support your customers, you’re really not going to know what the key things are to make sure that they’re online.

Steve: In this particular webinar we’re really going to start you identify preventive controls and recovery strategies. How we are going to look at these two pieces, and we’ll just very, very lightly talk conceptually about the IT contingency plan, we want to start looking at this from the business first and then from the IT perspective.

Next week we’re going to get really down and dirty about all the IT things that you can do, but none of those IT pieces will work until you have gone through and thought about points three and four and how to integrate those into your business. These are the two points that we’re really going to focus on, on these seven steps.

Some of you might remember this slide. If you’re a small business maybe you have some sort of organizational structure that looks like this. It’s obviously very simplified. I can only spend so much time in Microsoft Paint. Of course there are external factors. We used this simple model when we were talking about our fictional company, the bed and breakfast that we were trying to do a business continuity and disaster recovery plan for.

Basically what we want to first to get into is some of the benefits of a business continuity plan that you may not really think about day to day. When we can sit and we can look at the business like this, we start to unveil some certain things that we want to accomplish for the business’ sake. This may be a slide that you remember too.

Of all these three things, you want to consolidate IT infrastructure. Especially today with virtualization and Internet bandwidth being as inexpensive as it is today, consolidating IT infrastructure doesn’t necessarily mean putting ourselves at risk if it’s designed properly. That’s a real big disclaimer: If it’s designed properly. There are specific technologies that we’re going to talk about that will do exactly what you want to do, it’ll consolidate infrastructure.

I’ll just give you an example, there’s one of our customers. Essentially they had a medium-sized data center and they had all of this overhead of maintaining the data center. We worked on a project with them and we were able to consolidate this medium-sized data center down to two racks of equipment. You went from a fairly sizeable room in a building somewhere to two 4x4 racks of networking equipment. That’s a very, very high level of consolidation. We’ll get into talking about VMware and how you can consolidate your infrastructure and be more cost effective and all those things.

It doesn’t run in contrast to upgrading disaster recovery and business continuity capabilities or upgrading the security of your environment. These are really, really important things to know. If you have an IT guy that really loves business continuity and disaster recovery like I do, I’m just going to say you should hold on to him or her because – that’s my shameless plug there - because this is something that when you design environment like that you reap many, many benefits. It’s not just about cost savings. It’s about protecting your assets and it’s about improving the security of your environment overall.

Steve: One of the things that we’re looking at when we talk about benefits, any time that you really dig into an organization you’re going to find a lot of things that just grew organically. Maybe they were effective when it was a company of 10 or 15 or 20, but if you’ve grown beyond that the processes and the procedural steps that you have in place, maybe they’re not optimized for an organization of that scale. Maybe you find that you’re not being as flexible or as agile as you want to be. If you’re interested, I told April that I will provide some links.

Even sales and marketing strategy, if you’re a company that’s dispersed regionally, having marketing and sales people in specific regional locations can actually be a godsend to your company. To do a specific example, I’m sure I am not the only one that knows Tang, the orange drink. I don’t know who their manufacturer is, but Tang has really had a stark decline in the US. The owner of that company, they used a lot of the concepts that we are going to be talking about and what they found is that when they stepped outside of their shell and when they stepped outside of their comfort zone and they allowed sales and marketing to work in a fairly economist way, having delivered specific results, they got great improvement. I would never have ever thought of it, but if you look up Tang, one of the big flavors of Tang in South America is coffee.

It sounds disgusting, actually. Now I don’t really know what coffee-flavored Tang would taste like, but it’s working for them. So when you release some of that control and you allow people to get closer to market and become a little less centralized, there is a lot of benefit that you can get out of that. These are all the hidden benefits.

Also when you talk about the fourth bullet point, when you realize that something is very, very valuable to your business, maybe it’s a key process, maybe it’s a piece of machinery, maybe it’s an employee if you’re a consulting firm; you will generally use those resources much more wisely. If there’s only one of Bob and Bob is your chief financial analyst, if you’re a bank or some other financial piece of information I know nothing about, you’re going to be very, very jealous of Bob’s time because you’re going to want to make sure that Bob is working on a type of project that you want him to work on so he’s most profitable for you.

These are really some of the big benefits that you can get out of having a business continuity plan in place. Of course, my personal favorite – security.

I wanted to bring this up. If you don’t know who Bruce Schneier is, he is probably one of the world’s foremost security experts. I’ll let you Google him, please. That’s all I’ll say. Bruce Schneier is a very prolific speaker in the area of security, and he made back in 2000 a really wonderful quote. I put the link here so that you could read the rest of the article. It says, “Security is a process, not a product.”

I am going to say that again because that is really, really important. “Security is a process, not a product. Products provide some protection but the only way to effectively do business in an insecure world…” Insecure could mean hackers, it could mean fire, it could be corporate espionage, it could be whatever. “The only way to do business in an insecure world is to put processes in place that recognize the inherent insecurity in the product. The trick is to reduce your risk of exposure regardless of…” That could be fire, hackers, there’s a whole long list of things that Bruce Schneier is talking about mitigating.

Hopefully, one thing, as we spoke in our last webinar and that you are going to get out of this one, before we ever talk about technology we need to talk about the process, we need to talk about the business, the structure of the business and everything that that involves.

Look at the summary. A solid business continuity plan when executed well can help an organization accomplish critical goals across the company. This is what we’re building for. If, like me, you are IT folks out there or you are a manager somewhere, these concepts and what we’re going to talk about specifically today hopefully will give you fuel to go back and talk to your boss or talk to finance or talk to whoever it is that you need to talk to and say, “We have this idea. We see that there are these business objectives. We want to be more secure. We want to be more resilient. We want to validate these things.” Hopefully, the thoughts you’ll get out of this discussion will give you ambition to go back and to talk to those people.

Are there any questions right now before we go through and start talking about the concept here? No? Okay. Feel free, I very much enjoy an interactive discussion.

Steve: This is a quote from Jack Smith, the manager of Global IT and Business Continuity at ABN AMRO, and we’re going to specifically talk about his company. ABN AMRO was, I believe, mixed with LaSalle Bank at that time. This is a quote and he has done an excellent example for these organizations in developing their business continuity.

He says, “Far too often business continuity is thought of an expensive overhead or something we have to do to please auditors.” That’s absolutely true. Maybe that’s a way that a lot of people look at auditors or auditing.

The biggest thing and what I’ve recommended to people that I’ve worked with, especially IT people, look at your auditors as your friend. Look at your auditors as a way to call out issues that you have concerns about in your environment, real legitimate issues, and use that leverage to make sure that things are being done in a constructive, solid, safe way. What we can do is transition and stop looking at these things as a hassle but look at it as a business opportunity and a competitive advantage.

We’re going to look at some companies that followed these strategies and we’re going to see what happened to them when they had a pretty major incident.

The biggest thing that we want to talk about today is this concept of micro-sufficiency versus macro-efficiency. Basically in real simple terms, think of macro-efficiency as economy of scale. For everybody who has taken macro economics, macro of scale: The bigger you are as a company, the more buying power you have, the larger equipment you can buy and more widgets are produced per minute.

Just another really good reading, I recommend that if you’re interested in what you’re hearing here, conceptually the concept of micro-sufficiency versus macro-efficiency, I took a lot of this information from a really wonderful article in Strategy in Business. It’s in Autumn 2012 you can go, the link is on in my PowerPoint here. The article’s really around operational models.

Why that’s important and why I chose to use this is because if you can make the case to modify operational models in your business, then you’ve already won 90% of the battle for business continuity and disaster recovery. These specific concepts are so, so critical and will make your life, if you’re trying to develop a business continuity and disaster recovery plan, much, much easier.

Steve: What we’re going to look at is something called the regional cluster model. What this group of researchers came up with is the regional cluster model. It is helping companies break down where they need to spend their time. What things are best at scale? What things are best de-centralized? What benefits do you get out of, let’s say, your sales and marketing team being in very close proximity to your customers? What benefits do you get to manufacturing in one place versus another place?

The article really addresses those types of concerns at a business level, so for those of you who are in IT maybe you may find this a little dry. For those of you who are business managers, you will probably find this article that we were talking about here, very, very informative and very, very helpful. I recommend that you take a look and read it.

Macro-efficiency. My better half happens to work at Ford Motor Company; a very large, multi-national company. She was relaying as the close of last year came in, there was the VP of Ford; they were having their conference at the world headquarters and the VP of Ford came down and said, “We are going to use the same exact strategy and do the same exact thing everywhere in the world. Our strategy is phenomenal and that is how we will succeed as a company.”

I immediately started to laugh because I thought that was a very arrogant thing … Because if you’ve traveled at all, you understand that human beings all over the world are very, very different; their desires, their wants, what they have as far as business expectations are very, very different. The last company that I worked at, I worked in IT and they had some outsourcing in India. There was a large point of contention when all of our East Indian friends would go take tea breaks three times a day for a half hour. It was a very, very sore spot.

This whole concept that macro-efficiency and economy of scale is great, up until a point. I think we see a lot of evidence in that in our economy, thinking about companies that are too big to fail. You have companies like AIG and … well, Merrill Lynch did fail, Bears and Stern. When they run into trouble, when they run into a situation where the corporate IT has different needs than the regional IT, when corporate customer service has different ideas than maybe a customer service model if you were doing business in Australia, maybe there’s a lot of different HR policies depending on what state or what country you’re doing business in, and the differences in your customers.

For a small-scale company, maybe the sense of macro-efficiency until you can reach a critical mass is very important. Once you do, when you get into a very large organization – and this is actually still probably a very simple model for a large organization – you end up with all of these really complex business interdependencies.

If you remember back from the first week, if you have a hard time mapping out the dependencies within your business you’re going to have a really, really hard time accurately calculating what is actually generating your revenue, what these processes and what departments are actually generating that revenue and in what order do you need to get those things back online. It can be very, very difficult if you have a big, messy organizational structure. Can I name companies that I’ve worked for in the past?

April: I don’t see why not.

Steve: Okay. Again, I worked for ADP. ADP, when we were going through and practicing our business continuity and disaster recovery planning, it was a logistical nightmare. ADP, as a corporation touched 74% of the world’s payroll. They’re an absolutely enormous company.

They were going through an immense phase of corporate consolidation. They wanted all decisions to be made out of New Jersey, but the executives there didn’t fully understand the business model in all the different regions. When we were tasked with understanding and trying to give insight and input into our business continuity and disaster recovery model, they didn’t understand the need because they didn’t understand the relationship that every department had to the customers, to the different regions, how they interacted. This can be an example where macro-efficiency really goes, horribly, horribly wrong.

Like I stated, if you are not able to easily map out what the interdependencies are, you are really most likely not going to be successful in analyzing what you need to bring back up first.

Steve: What is the opposite? Micro-sufficiency. Micro-sufficiency is a very simple concept where you have your core function, let’s say HR or legal, or some very specific business-type functions in your headquarters. In this case, the headquarters is represented by the red … What is that? Six-sided, that’s a hexagon? Yes, the hexagon in the middle.

Then you have various operational units that you would have in maybe different regions. Let’s say even if you are let’s say a local bank, you have your headquarters and then you have different branches. That would be a very applicable model. One of the big business units at ADP was an auto dealership. Let’s say you have a couple of auto dealerships. Anything where you have multiple geographic locations, you see the benefit of getting sales, marketing, maybe some light IT into those operational units and only consolidating critical core functions in HQ.

What does it do for you? Essentially, if you have all of your core functions like HR, IT and finance in your headquarters, then you know that you can mitigate your risk by spreading around all of those other functions between your operational units, which are much closer to your customers. What that allows you to do is say, “All right, we’re going to consolidate,” let’s say, IT infrastructure and HQ.

We also know that we have some critical functions and let’s say we’re going to use VMware just because I’m a VMware fan, not that other virtualization platforms are not almost as adequate.

We have all of our core functions at HQ and we’ve gone through the due diligence and the processes of identifying those core functions. We did the business impact analysis, we looked at what is most critical and what is most applicable to the organization as a whole, and then we’re going to consolidate that based on some virtualization or redundant IT infrastructure and have that so that we can achieve that goal of consolidating IT infrastructure.

Once you’ve made that decision and once you’ve identified all of those critical pieces of your infrastructure, you have any one of these operational units that you can fail over to. It doesn’t necessarily mean that you have to have a whole other data center or something like that, but what it does mean is that you have a better control over what you have to look at day to day. That’s a way to mitigate the scale issue. You’re really identifying critical processes and then if push comes to shove, you can distribute these core functions into your various operational units.

Steve: There are some drawbacks to that when you’re first starting out, and I will speak to that because it speaks to the question of how much can you use at any given operational unit. What this really would look like, let’s say each of your operational units, as seen here, function as a simple business but each operational unit is responsible for generating a certain amount of leads. They each have a small management structure. They have sales goals that they are responsible for. All that data is really funneled up to HQ. If any one unit was down, you’re really only down a smaller percentage versus if all of these things were consolidated in one location.

What does this do for us? As mentioned, it really helps to mitigate risks. There’s a couple of things that I would say about this. What I would say is that you should definitely bring in your IT person at this stage in the game. This is just my personal bias, because I come from an IT background. IT people think about this stuff all the time. We think about how to make our networks faster. We want things to be speedy and a lot of these things are very, very similar to how we want our business to operate. You want them to be fast. You want them to have a high volume of throughput. You want them to be resilient and we want them to be optimized for quality. This is why a lot of IT people are very interested this field, because now we’re playing the network game but with business processes and with organizational structures within the business.

Basically, let’s just say we’ve got a cluster of servers. We’ll talk a little bit of tech here. In this particular model we’ll just say that we have seven servers. We know that our capacity and our failover, we could stand to lose three of those servers and we’re letting this operational model and technology here just so I make sure everybody follows.

Let’s say we know that the capacity plan, we could run all of our business in the gray, in these gray boxes here. Yet, we have extra units for failover. Maybe one is a primary controller for our cluster. This is the way that IT people generally think. If our firewall ... if we only have one and it fails, “Oh my goodness. We won’t be able to conduct business. My boss is probably going to fire me so I should definitely push for at least two firewalls.”

If my switches die; the same thing. If this one server goes down people aren’t going to get their email. By nature and in our job description, we think about this redundant and resilient type of being all the time. When you start thinking about your organizational structure and how you want to fail things over, that may be when you want to bring in somebody who is very, very seasoned and say, “Listen, we need you to think now if each one of our business units was a piece of an IT system. How do we get these things to failover gracefully?” That’s basically the goal. All businesses are going to experience some sort of failure in a piece of hardware, but how do you fail gracefully and fail safely? That is really the goal of business continuity and disaster recovery. A failure is going to hit you at some time. How do you handle it gracefully?

This doesn’t work very well in a small environment. I will have to admit that. Talking a little bit about IT, imagine if you have a hard drive in your computer and you want to keep all of that data on another hard drive exactly like the first one. That’s kind of crappy because now you only get 50% of the capacity of what you’re paying for in any given unit.

What this really allows us to do at Online Tech is be a partner for you where maybe you don’t have to pay for a whole other data center, you don’t have to pay for a whole other group of IT staff. Because of our scale and the people we have on staff, we can be a partner to you. Or if you wanted to bring some colo equipment or you wanted to do some DR here or any of those pieces that you might be looking to improve about your business, that’s where external partnerships really come in handy, because it will be much cheaper for you to come in, get some space here in our data center than having to make all this redundant equipment in one of your alternate facilities.

If you do want to be fully redundant, it’s less capital intensive for you to say, “We have 10 servers at our location in our HQ location, we need to buy 10 more in our operational site.” Maybe it’s a regional site. It can be very, very expensive and that is definitely a limitation of this micro-sufficiency model. I, 100% recognize that. When you look at it from a scaling perspective, this is where things start to shine.

Let’s say if you are a really large company and maybe if you are looking at this from an operational process perspective, let’s say you have five locations and you’re talking about, let’s say, my ATM tellers, to go back to using a bank. If I have five locations and we need to handle a certain amount of volume of business in a geographic location, if one of our locations comes down, as long as we’re not running at over 80% capacity a person could choose to go to another branch location in that geographic area and we could handle all of that overflow from location five, let’s say, if it went down.

Initially there are some definite problems with this micro-sufficiency, macro-efficiency model but in the long-run, when businesses really need it, once they start to get to that position they say, “Hey, we’re a $100 million business,” when you start getting into larger volumes of money in a bigger business, you are scaling like this anyway.

Take the time, go through, analyze your business. Go through the whole business impact analysis process. Redefine those processes and those business structures that you have in place and make sure that you’re leaving an appropriate amount of slack space in all of your locations to compensate if you do have a disaster.

If I’m telling you to leave some slack space I probably should have a good example why. This is the good example why. Again, I’ll put this link I thought I had in this slide, but I’ll make sure the link to this congressional research report is out for the webinar.

Steve: Congress wanted to do a survey of the impact that Sandy had on suppliers and manufacturers over in Northeast Japan. Two companies that everybody knows very, very well, Toyota and Honda. This is something that we find. These are both profitable companies. I am pretty sure anybody on the phone listening to me speak would be very happy if their companies were doing as well as Toyota and Honda.

Toyota built about 45% – I just grabbed a screenshot from the article here – 45% of its vehicles in Japan. They consolidated twice as heavily as Honda and Nissan. As you can see, I won’t read this whole spiel here but you can read it, but Toyota lost 77% of its profit that year. That is a big blow for any company to bear, especially for a smaller business. Ask yourself, critically, could your company afford to lose 77% of its net profit and continue to operate?

Honda, on the other hand, had a much more distributive supply line, a much more distributive supply base. They were geographically disbursed so they weren’t anywhere near in the risk profile like Toyota, and they lost 38% of their profit. Not great, but when you have a once-in-a-decade tsunami which hit a nuclear reactor which takes out a large portion of the nation, I would imagine that your shareholders, that your CIOs, that your board of executives would be much happier with a 38% profit loss than a 77% profit loss.

Again, for all the business managers out there I highly recommend that you go through and you read this congressional report that was put together. Because, again, it goes back to the organizational structure and how these two companies conducted their business. Again, business continuity is not about IT. It’s about the business. I can’t say that enough. So, enough of the doom and gloom.

Steve: Next ABN AMRO and LaSalle Bank. I don’t know how many people remember this. This was before I was really interested in business continuity and disaster recovery. What happened, it just goes to show that a disaster can really come from anywhere.

If you go through and you look up the research that happened and you look what happened to ABN AMRO, they had a simple miswiring of a light ballast in the high rise where they were located. This could happen to anyone. It doesn’t require a tornado. It doesn’t require a tsunami. It doesn’t require a nuclear reactor going ballistic. It was a simple electrical wiring fault.

This was the second largest high-rise fire and I believe the second largest fire that Chicago ever had to deal with. Essentially what happened is it burned 12 floors, more than 12 floors of the LaSalle Bank complex in Chicago. It started about 6:30 p.m. and the result was that this gentleman, I’ll have to go and look back on his name, the gentleman that was in charge of managing the business continuity and disaster recovery plan had done a phenomenal, phenomenal job. His name was Jack Smith.

Essentially, 6:30 pm when the fire started [inaudible 00:44:31] due to the efforts of the bank’s business continuity team, the coordinators were at the location by 7:30. Within an hour all the people knew what they had to do to address this emergency. By 7:30 in the morning, the fire started at 6:30 pm, 13 hours later, this is 12 floors of a critical business, every single critical function is back online, every single one.

What does that mean? It means that they first went through and defined what their critical functions were. That’s like getting back to that whole business impact analysis point, that’s key.

They had 750 employees; 400 of them knew that, “Our jobs are going to work from home. Our jobs don’t require us to be in the office.” They had phone lists that they could reach out and contact all these employees and 350 of these other employees knew exactly where they were supposed to show up in the case of an emergency. They showed up at other banks that were geographically close enough for them to show up to and continue doing their work by 7:30 the next morning.

There are a couple of things to think about. If this were to happen to your business, what would it say to your customers if you show up on the front page of the X Gazette or the X News, maybe it’s the Detroit News or whatever your hometown is, that you had this huge, massive fire that takes out 12 floors of a high-rise building? You can’t get back in. What sort of confidence in you as an organization did that inspire in your customers?

What level of planning and resilience is that saying about your company? How much money would you lose if you weren’t prepared for that type of disaster? My personal selfish point of view is, “Will I still have a job?” Everybody has selfish motivators. I would like to be employed in the case of a fire.

These are the things that as you look at your company organizationally, this is how you model and design the company first before you start looking at IT solutions to facilitate the safe recovery, the gentle recovery, and inspire confidence from your customers.

April: That really goes back to your point about disaster recovery and business continuity being an issue that’s about processes, not about products. I don’t know how many of us could look at all of our colleagues with a degree of confidence that if a disaster happened we would know exactly where to go, exactly what to do the next day.

Steve: I am very thankful … I did a presentation similar to this for a supply chain organization. We in IT, for all of those of you who are a service-based business or an IT-based business, we could take a big, big lesson from manufacturing. Manufacturing has been mapping input and resources, how every beam of steel flows through a production line. If you look at the Malcolm Baldrige Quality initiative, Six Sigma Quality initiative, they have for a long time been looking at how to optimize their business processes. They know every single input that comes into that supply line. In IT we need to be a little humble and look to these more mature businesses to see what we can learn.

Just like in manufacturing, our resources are our people and our processes. If we’re not willing to step back and say, “Hey, this is important, we really need to understand what we do as a business and how that makes us successful,” maybe you need to clear some time up on your calendar.

That’s all I can really say because if you care about this and you’re passionate about this and you’re passionate keeping your business up and running, this is something you have to do. You have to understand your business. By doing this, by understanding your business it makes you more secure. It goes back to the quote that Bruce Schneier said. If you understand the business you understand the threats to the business and you can defend the business more effectively from whatever threat that is.

A quick review is business continuity, as I just said, it starts at understanding your business. Business continuity is a business issue. It is not an IT issue. It was very, very hard for me in developing a three-week webinar series as an IT guy to not talk about IT for two weeks.

April: I can just verify that point for everyone.

Steve: This is a very big personal struggle, but it’s for a reason. This is for a reason. There’s a lot of benefits that you can get from developing a business continuity plan. Don’t make it a chore. Look at it as the opportunity to re-invent your business.

I’ve sat in on meetings and we’ve talked about this subject with various people from my co-workers to CEOs and they said, “This is the way we do business,” and they said, “We’ve always done it this way.”

I said, “What does that matter? Just because you’ve always done something some way, it doesn’t matter.” A lot of times if your company’s evolving, if your people are evolving, their skill sets are evolving. You need to re-evaluate the processes. I’m not saying process design is something that should be taken on willy-nilly. Just because you’ve done something some way for X amount of years doesn’t mean that it has to stay that way. Just think about that. Step outside the comfort zone a little bit.

I talked about the concept of macro-efficiency versus micro-sufficiency. In case you haven’t guessed, I am in favor of micro-sufficiency.

Basically we reviewed two cases. I’ll make sure that these links to these articles and the research paper that Congress did, you have those available in case you want to review those and read those. This is great, great ammunition to take back to your supervisor, superiors and say, “This is why we need to do this.”

Week three. This is where the fun stuff is going to start for me. We’re going to talk about backup strategy. We’re going to talk about hot, warm and cold sites. Just to talk about backup strategies. We’re going to talk about virtualization. Virtualization absolutely will help not only consolidate your infrastructure, it can in fact make you more secure if you’re using a good hypervisor or a good virtualization product and it can help you with your business continuity and disaster recovery planning. Virtualization is great.

We need to very specifically talk about databases. A lot of corporate information is held in databases. There are two kinds of data; there’s structured and unstructured data. Unstructured data is things like Excel documents, Microsoft Word documents, generally things that go on a file share somewhere. Those are pretty easy things to back up.

Databases can be very, very sensitive, so you have to take care when thinking about a backup strategy for those.

We’re going to talk about some spam technologies and how that can facilitate RTO, RPO and really improve in some cases the amount of data that you can get off site very, very quickly.

We need to talk about the differences between system recovery plans, recovering complete systems, versus recovering individual files. Especially as servers are getting larger and larger – 1, 2, 3, 10 terabytes – we’re seeing it more and more frequently. There’s a tradeoff, and you have to be prepared to know what the tradeoffs are when you build very large systems versus recovering individual files.

How do all of these things fit together to protect your organization? That is what we will discuss next week. Thank you everybody. Does anybody have any questions or anything? I feel like I’ve been just talking a lot here.

April: We can see people are definitely listening. Assume that everyone is absorbing.

If you don’t have a question right away but do come up with one later, feel free to send that to us. Contactus@onlinetech.com

We’ll make sure that Steve gets that question and gets back with you.

In the meantime, while you’re thinking up questions or getting ready to sign off, I wanted to invite you to join an impromptu webinar we’re holding Thursday at 2:00 p.m. with Brian Balow, who is a specialist in Healthcare IT Law. He’s going to comment on the final rules released by Health and Human Services about the HIPAA Privacy and Security Rules. Please join us January 31st; that’s this week, Thursday at 2:00 p.m. Join us in two weeks; Tuesday at 2:00, we’ll have the third part of our Business Continuity and Disaster Recovery webinar series.

Thanks, Steve.

Steve: Thanks very much. Thanks for listening, everybody.

About Steven Aiello

Steven Aiello is a Systems Support Manager with Online Tech, the Midwest’s premier managed data center operator. His certifications include CISSP (Certified Information System Security Professional), ISACA CISA, VMware VCP ( VMware Certified Professional), Cisco CCNA ( Cisco Certified Network Associate), Comptia Security+, and Certified Incident Responder (New Mexico Tech).


Webinars    |    Online

Get started now. Exceptional service awaits.