Posted by Michael Bullock on Fri, Oct 02, 2009
With data center operations, equipment maintenance, staffing and licenses consuming the lion's share of today's IT budgets, companies can gain a great deal by optimizing their current operations.
Given the current recessionary environment, technical expertise, strong budgetary controls and procurement discipline are more critical than ever to maintaining operating margins.
Across the board we are seeing organizations taking on smaller, quick hit projects to balance spending and risk. Conversely, most companies are postponing or scaling back major transformational projects as managers look for rapid ROI either by better utilizing the technology assets they already have or by undertaking initiatives with shorter implementation times and faster paybacks. Not surprisingly, fewer companies are willing to fund multi-year projects that will not begin paying back their investment for 18 to 24 months or more.
But saving money is never easy. Many initiatives to reduce recurring long term costs are complex and expensive endeavors in themselves and especially difficult to implement during a recession. Even in the best of times, such projects warrant a thorough technical and financial analysis to make sure they align with the company's business direction and economic goals.
My advice for CIOs and their enterprises is to continue to invest in solutions that deliver well-defined results that at the same time minimize disruptions to employee productivity. You may be surprised how many projects can produce a healthy ROI in just nine months or less. These are the initiatives that will easily pass muster with your CFO and executive team.
Here are two examples of how taking a fresh look helped two companies save significantly on OPEX and CAPEX - delivering faster value to their organizations:
Example 1: How we saved $17M and moved one year early
A Fortune 1000 manufacturer planned to relocate to a new data center at a projected cost of $17 million. The company's facilities manager had spent months considering the move and was among the chief advocates for building a new data center over the next two years. He knew facilities (but not data centers) and his site selection was based on traditional real estate values -- size, location and cost -- which, unfortunately, do not always translate well into data center requirements.
Fortunately, with some outside help, the company took a closer look at the proposed relocation, set aside its internal preferences, and a $17 million mistake was avoided. The company discovered that it would need to install a fiber ring connecting the new data center at a recurring cost of $500,000 per year. On top of that, the new facility would not have sufficient cooling capacity for the anticipated power load.
By starting with the requirements, not the real estate, the company found suitable colocation/lease space with total annual costs equivalent to the fiber trunk expense alone. By going into a collocation facility with multi-carrier access, the fiber trunk requirement was eliminated, facility construction costs were avoided, and the cooling problem went away. On top of this, the client moved into the space one year ahead of schedule.
The moral of this story is that in-house employees, and even senior managers with lots of experience, can become cheerleaders for projects that keep them in their comfort zone or may even seem to guarantee long-term job stability. Companies need to spend some effort on understanding that which is less familar and looki in unfamilar places for fast payback rewards.
Example 2: How to save $300,000 per year and improve service
Large enterprises with big IT operations and large facilities staffs typically have multiple departments reporting to different parts of the organization. It's not uncommon for such groups to operate in silos, ignorant of each other's
For example, a major Boston-based investment firm's IT infrastructure had grown over the years through mergers and acquisitions and by decisions made by department heads who didn't communicate.
As the company became alarmed by its ballooning IT budget, it sought help from an independent IT consultant with no ties to a specific hardware vendor. After looking at all its processes, the company discovered that it was spending $2.7 million annually on a jumble of security systems with many overlapping features. Worse, these systems were installed in a cascading fashion that created management headaches and performance bottlenecks.
By redeploying the existing technology more effectively and retiring unnecessary systems, the company immediately cut annual fees by $300,000 per year while improving manageability and performance.
Stories of the left hand not knowing what the right hand is doing are rampant in business. The lesson here is that taking a full inventory of systems and services across all groups is an important first step in consolidation and cost containment.
As you search for new ways to deliver value, don't forget to take a hard look at areas that you may have forgotten during the boom years. Be open to questioning your internal biases and the ways you've always done things. There are many places you may be able to recover budget simply through smarter operations.
Posted by Steve Gunderson on Wed, Sep 16, 2009
If you'll be moving your servers to a new location during the next couple of years, I have some advice for you: Before you buy more servers and storage systems from your current vendor, find out what impact a relocation will have on your warranty and maintenance agreements. A little forethought in this area may save you a bundle down the road.
Now while most vendors work in a reasonable and collaborative manner, and take into account the heterogeneous nature of data centers, some do not. We've noticed that some well known service and storage vendors exert a lot of pressure when it comes to signing up customers to services they don't really need.
Make no mistake: Data center relocations are complex undertakings, full of business and technical risks, and every vendor would like to get a piece of the pie when you relocate. That's their business and there's nothing wrong with that. And many vendors provide reasonable value for the fees proposed. But you really owe it to yourself and to your enterprise to apply some due diligence and a measure of skepticism when they start tacking on services and fees that don't make business sense.
Here's a scenario I've seen repeated often:
1. After the vendor is notified that its systems will be moved, the customer is told that he can't use third-party movers for the vendor's gear. In fact, the customer even may be told he must use the vendor's own moving vans. If every vendor took this position, moving day would turn into a logistical nightmare as one would need to manage multiple vendors and multiple trucks at the loading docks on both sides of the move.
So if your vendor demands that you use its men and equipment to relocate, ask it show you where this is spelled out in your purchase and maintenance agreements. These agreements are highly customized but a requirement that you must use the vendor to move is rare. So if it's not there, just say no. But after you do, be prepared for the vendor's fallback position.
2. You're told that if you use a third-party mover, you will void the warranty/maintenance and you'll need to have your systems recertified. Again, check your agreement. The truth is most likely to be that if the system operates at the new facility for 30 days without failure, then it will be returned to warranty/maintenance status. Even if you do pay for recertification, you'll still be on the hook for hardware fees related to any failure during the 30-day window; recertification service only covers the labor, not the hardware components. If you dig into the details about recertification, you're likely to find that given the cost, the benefits are minimal. Most companies are better off self-insuring their servers or simply asking their relocation partner to take on this risk on their behalf.
3. You may also be told to expect a very high failure rate (as high as 20%) when you power on your servers at the new location. If you hear this from your vendor, you owe it to yourself to press for details. Is there any data to support the assertion -- which certainly should raise concerns about the system's fragility, no? But don't be alarmed; the failure rate caveat is pure moonshine. Having relocated thousands of servers over the last 12 months, I have only seen two failures -- and these were very old boxes caked in dust before the move. Now if you have some servers that have been in service for years without a power cycle, yes, there's risk. But there's also a simple solution to shake out the outliers. Prior to relocation day, you should power cycle all systems (and maybe clean them at the same time) to make sure they're not just a power cycle away from a failure. If some of the systems do fail, this can be fixed under your current maintenance plan before the relocation.
4. If this pressure doesn't work on you, then don't be surprised if the vendor goes to your CEO or CFO. But by doing your homework, you can explain to your executives how you can save the company a lot of money while avoiding unnecessary and non-value added service fees.
Certainly, some large, monolithic systems are best moved by the vendor, but applying that thinking to general purpose rack and blade servers is a mistake.
Finally, a well-executed data center relocation will improve reliability, not degrade it. Since most data center relocations are undertaken to solve underlying problems with power, space and cooling, moving into a properly designed space will eliminate hot spots, reduce power spikes and improve system reliability.
I welcome your comments and feedback. Have you had similar experiences with your server and storage vendors during a data center relocation?
Posted by Michael Bullock on Thu, Jan 29, 2009
If you're facing a data center relocation, one of the most important decisions you'll be making is site selection. This is true whether considering collocation space or building your own facility. Either way, site selection will have the biggest impact on your implementation schedule, your data center’s reliability and its lifecycle costs.
Perhaps your eye was caught by Google’s floating data center concept, a design recognized by Time as one of the best inventions of 2008. Google imagined (and patented) a data center on a raft, with electricity generated by wind turbines and wave power, and the ocean waters cooling the servers. And, of course, there’s no landlord on the high seas. Sounds nice. But before we can fully endorse the Google floating data center notion, let’s take a closer look at some of the issues this data-center-on-a-boat raises for site selection.
For a data center with Tier-III class redundancy (the Uptime Institute’s designation for centers that are “concurrently maintainable while in service”), here are some of the factors we need to take into account:
Power Density – What type of systems do you expect to house in the new facility? If you’re expecting to support high density systems (like blade servers requiring 10kW per rack or more), how much room will you need? And if you’re planning to move into an area that only supports 150 W/SF, you will need to space out your racks accordingly (for proper airflow), increasing the amount of raised floor space you thought you needed by a factor of 4.
Power Availability
– Is there sufficient power available in the grid to supply the site? If not, be prepared to foot the bill for power company infrastructure upgrades which may take 18-24 months to complete.
Power Redundancy – Is the space serviced by a redundant substation with independent power feeds to your location (or maybe even multiple substations)? You want that unless you’re prepared to deal with downtime.
Power Backup – If you’re looking at an existing facility, you have to ask if it has backup power to cover the center’s full load. It’s also wise to find out if there’s a clear plan in place to increase capacity as you grow. If you’re looking at a new facility, you have to find out if there’s sufficient space for backup power generators and their fuel storage tanks. And are there any zoning issues which may cause delays or add to your costs when you try to install these backup systems?
Cooling – Is there sufficient cooling capacity to maintain a proper operating temperature in the facility even when it’s operating on backup power? Is there sufficient floor to ceiling clearance to allow for adequate cool air supply and the removal of hot air exhaust? Quick test: If you’re moving into a facility with 200 W / SQ ft. capacity, are there at least 36-inches of raised floor space to assure adequate airflow for system cooling? If you’re planning on 400 W/SQ FT? Look for a floor raised 4 feet or more. Lack of sufficient height is one of several reasons why conventional office space is suboptimal for anything more than 100W / SQ ft.
Network Access – Ideally there should be multiple WAN providers capable of serving your facility, each with redundant fiber / network connections. This will assure long term competition on price and an alternative if one provider’s price or service levels become an issue.
Geographic Considerations – Are you planning on putting your data center someplace where earthquakes, hurricanes, tsunamis or floods occur from time to time? I wouldn’t. Will planes fly over it making their approach to a nearby airport? I’d think about that. Will it be easy to deliver replacement parts and get professionals there for needed repairs or maintenance? Is the location safe from terrorism or desperate profiteers like Somali pirates?
Perhaps the most costly and avoidable mistake I see companies make is deciding to use a space “because we have it.” Unless you get extremely lucky, it’s highly unlikely that the spare space you have will be suitable for your data center. More likely is that you’ll be paying for it over and over again through costly retrofits, increased infrastructure costs and much higher operating expenses than would otherwise be necessary. Fact is, even “free” real estate quickly can become a very costly proposition for a new data center. Anyway, as real estate represents a very small portion of data center costs,it should not be a primary driver in your decision.
Google’s Floating Data Center
So how well does Google’s floating data center match up to these requirements? It certainly offers an abundance of energy – provided you’re comfortable with harnessing and depending upon the waves, the tides, the sun and winds.
Ocean water indeed can be used to cool the servers through the use of seawater / fresh water heat exchangers, which in turn will cool the air. Of course, the environment better be airtight unless you think salty air is good for servers. (It’s not, by the way.)
While the floating data center does offer the promise of cheap real estate, it does so at the expense of increased complexity and risk. Security is on the ocean is—shall we say—problematic. Exposure to disaster is high. Access to multi-WAN providers . . . Who knows?
I’m not suggesting Google was entirely serious with its water-based data center patent, but it certainly won’t help organizations that face data center deficiencies today. Perhaps this is part of Google’s secret strategy to scale up its App Engine / Cloud Computing service. If so, good luck, guys!
Nonetheless, Google’s floating data center model has raised awareness about the very real and growing problem of increasing data center power and cooling demands. It also provides an excellent way to focus one’s thoughts on what data center site selection really requires and, more importantly, what potential problems could rise from the depths to bite you where you don’t want to get bitten.
As always, thank you for sending comments, tips and topic suggestions to me at CIOblog@TransitionalData.com.
Posted by Michael Bullock on Wed, Dec 17, 2008
I recently was interviewed by Francis Rose of WFED radio in Washington D.C. His show, “In Depth with Francis Rose,” focuses on topics of interest to federal IT leaders and professionals. (You can listen to the interview online.)
Now, Francis is not a technology guy so I was curious as to why he was interested in dedicating 20 minutes of his show to data center relocation. Why did he think that was newsworthy? Of course, folks in the know understand that aging facilities are forcing companies to consolidate and relocate data centers all over the country. But why did Francis care, and what did it have to do with the government?
I quickly saw where he was heading. Francis had read an article I wrote about the five pitfalls of data center relocation and how the State of Oregon managed (disastrously) to fall into every one. Francis wanted me to expand on how IT managers could avoid repeating Oregon’s mistakes.
You can read about all five pitfalls in the article, but I wanted to focus on one in particular because it’s a seductive trap that’s snaring a lot of organizations: thinking about the move as an opportunity to introduce new technologies (particularly virtualization) and methodologies. As I told Francis, we strongly advise against introducing any new moving parts during data center relocations. Doing is so like performing open heart surgery while throwing in a hip replacement on the side. It can be done, but it’s an enormous risk—and one not worth taking. Francis agreed; it seemed like a no-brainer to him.
So why do so many companies do it? They usually do so for two basic reasons:
1. Pent-up demand. When everything in the IT function (including the data center) is working smoothly, there aren’t a lot of CIOs willing to introduce infrastructure changes (or CFOs willing to fund them) unless the ROI is overwhelming. Especially in this economy, the guiding principle seems to be, if it ain’t broke, don’t fix it. Therefore, projects (including virtualization) tend to be pushed off even when they make business sense. Over time, this creates a project backlog—an itch that IT organizations want to scratch.
In the excitement of a data center relocation, the organization discovers a new enthusiasm for IT improvement and innovation. With all its resources allocated to the relocation, the organization convinces itself that the staff, energy, and budget is available to do anything and everything. In other words, it gets to scratch that backlog itch. Instead of “If it ain’t broke, don’t fix it,” the order of the day becomes, “Let’s take advantage of this focus on IT to fix everything!”
If you find yourself in a similar place, and the pressure to virtualize is mounting, at least make sure you have the time, money, staff, and expertise to do it right. If you’re not absolutely convinced, my advice is to finish the relocation first, satisfy yourself that your new data center is functioning optimally, and then think about virtualization.
2. Vendor hype. Let’s face it, most vendors know the best time to sell you anything new is when it aligns with changes that are already taking place. That’s the easiest and best time to capture your attention, not to mention your budget. Vendors specialize in taking advantage of any fears and uncertainties you may have to tell you that they have the tools and expertise to make them all go away. So if your virtualization software provider says virtualizing during a relocation is a good idea, and your hardware vendor says it’s a good idea, and the press and the analysts agree (as they often do as they spend a lot of time with and get a lot of their information from those same vendors), then it must be a good idea, right?
Well, not exactly. Performing two complex IT operations at the same time is almost always a recipe for disaster.
If you’re ready to make a move to a virtualized environment, you need to accept that fact that no matter what your vendors say, you will be beginning a complex, resource-intensive project. It’s important to be sure that all your systems will be able to support appropriate service levels throughout the transition. Unless those service levels can be maintained to the business’ satisfaction, it doesn’t matter how smoothly the virtualization initiative goes.
And if you’re also scheduled to relocate your data center at the same time, please, stop and reflect upon the risk you’ll be assuming by attempting to pull off two complex projects at the same time, essentially under a stop watch held by a watchful business.
Me, I wouldn’t want to do it.
Posted by Michael Bullock on Tue, Nov 18, 2008
Read this story as published in CIO.com 
No C-level executive, whether it's the CIO or CFO, wants to invest in their company's data center, especially not now when the economy is executing an almost perfect swan dive into an Olympic-sized recessionary pool. But an optimally (or even an adequately) functioning data center is not a luxury; it's a business necessity. If it ain't right, it's got to be fixed.
And chances are, your company's data center is not right. In fact, according to a study conducted by the AFCOM Data Center Institute, an organization for data center professionals, a majority of U.S. companies (53 percent) expect to relocate or expand their data centers during the next several years. Nearly one-third say they will need to move, while 45 percent expect to make major improvements to their existing facilities.
What's wrong with data centers today? What isn't? They're old (at a 2007 Gartner conference a third of the attendees said their data centers were seven years old or older, meaning that they weren't designed for the power and cooling needs of today's high-density servers); their TCO is growing at twice the rate of most companies' revenues, and due to the growing amount of data being collected, stored, and processed, they're often located in facilities that while perhaps suitable five years ago cannot be upgraded today.
It's no surprise that companies such as Alcatel-Lucent are doing radical makeovers of their data center strategy, as profiled by CIO.com earlier this year.
So, whether you want to or not, you're going to have to move or consolidate or redesign your data center sooner rather than later, and you want to do it as well and as cost-effectively as possible. You certainly don't want it to turn into a disaster like what happened to the state of Oregon.
Oregon's Data Center Nightmare
In 2004 the State of Oregon launched an initiative to consolidate the data centers of 12 state agencies and their approximately 1,700 servers into a single, new, Tier 3 facility. Oregon wanted to reduce the number of servers and operating systems it supported (thereby lowering hardware, licensing, and management costs), offer new and better service level agreements, improve the state's disaster recovery capability, allow for growth and technological advances, and ensure better data security.
The state's new site was completed in January, 2006 at a cost of $20 million. A year later, 11 agencies were migrated to the new facility, at a cost of $43 million. At $25,000 per relocated server or about $4 million per agency, the move's cost was astounding-and it gets worse.
In July, 2008, the state issued a report concluding that in fact only 70 out of the 1,700 servers had been eliminated, and new service level agreements had not been provided. Data security was so poor that the Department of Education could not move into the new facility due to its failure to meet federal privacy regulations, and another agency had to move back into its old center because the new center's power supply was inadequate. As for the projected cost savings, who knew?
What went wrong?
Oregon had tumbled into almost all of the pitfalls that can ruin a data center relocation and consolidation.
The Five Pitfalls and How to Avoid Them:
1. Poor Planning: Oregon's project technology administrator admitted that the relocation plan underestimated the number of servers the new facility would have to accommodate. Underestimating the complexity of data center moves-the time it will take, the skills required to do the job, the hardware needed-is more the rule than the exception when it comes to relocation and consolidation.
This is especially true when it comes to taking into account application dependencies. In the heterogeneous environments that characterize most IT application portfolios and IT infrastructures-with their many bolt-ons and homegrown systems developed over the years-no software tool exists today that can see all of the interdependencies. To account for them, the knowledge has to be collected from the people managing the applications-a time-consuming task. Indeed, moving a data center the right way places a large burden on IT departments that already are fully utilized and often over-booked.
In our experience, it's wise either to dedicate a full-time team to planning the move or to look outside the organization for professional help.
2. Underestimating Power Requirements: The Oregon project administrator allowed that the electrical power the facility was designed to provide-55 watts per square foot-was too low. Data centers built for today's equipment range from 150 to 300 watts per square foot.
IT professionals frequently underestimate power requirements, and power costs, particularly if facilities management pays the bills-as is typically the case.
In a recent survey, 68 percent of IT managers said they were not responsible for power bills related to their data center's IT equipment. It is important to make sure facilities and IT talk about their respective issues so that they gain an appreciation for their differing perspectives and areas of expertise. This is the only way to prevent their issues from turning into problems, and their problems from turning into data center relocation and consolidation disasters.
3. Failure to Establish Pre-Move Baselines:
It was difficult for Oregon to determine whether the agencies it had moved were realizing any of the cost reductions originally sought because "the baseline data provided by the agencies before the consolidation was either grossly understated or nonexistent."
It's an old saw that you can't improve what you can't measure. A corollary is that you can't compare one thing to another if you don't know what the first thing was. Know your current data center TCO and have the numbers in hand before moving into your new facility or risk opening yourself up to ceaseless fingerpointing and complaining.
4. Upgrading Systems During the Move: Oregon consolidated its facilities before "the underlying architecture, standards, and licensing issues had been worked out." In our experience, any change undertaken during a move adds risks and complicates the project. This is especially significant when it comes to today's popular practice of using a data center move or consolidation to drive server virtualization.
Although worthwhile, virtualization is a significant project in itself, and attempting to implement server virtualization during a move means trying to do two very difficult things at the same time-a sure recipe for disaster. In short, try to minimize changes during the move planning and execution periods: don't switch vendors, and certainly don't virtualize. The exception to this rule is that it often pays to re-IP and purchase new networking gear before the move. This will save the effort of reinstalling new gear in the new site during the move.
5. There's No Substitute for Experience: Because a data center move is generally a once in a career event for IT professionals, few companies have the expertise on-hand to do it well. Very high density power and cooling environments require specialized expertise and coordination. Unfortunately, IT knowledge does not translate into an understanding of how to move a data center, nor does a knowledge of facilities (and operations) translate into an understanding of the singular requirements of today's data centers, not to mention tomorrow's.
Experience counts. If your organization has someone with the requisite experience, get him or her on the moving team. If it doesn't, find someone who does.
In this economy especially, it is critical that data centers both facilitate current operations and provide the flexibility for future business growth. A botched move can stop an enterprise dead in its tracks; a poorly managed one can force an organization to incur the expense of moving again far too soon. Avoiding these five pitfalls won't ensure success, but it's a good way to start preventing disaster.