Posted by Michael Bullock on Thu, Jun 24, 2010
TDS Unified Monitoring Service
Before virtualization, understanding the inter-dependencies in a typical network environment was fairly straightforward. A connected with B connected with C.
With the proliferation of virtualization, compounded by clouds and SaaS applications, the challenge of maintaining continual visibility and control has put availability and performance of systems at risk.
While traditional monitoring tools provide availability and performance monitoring at the application, network and operating system layers,they do not provide sufficient visibility into virtualized and hybrid enterprise computing environments with multiple critical applications competing for shared resources. There is a clear need for a more holistic approach including unified monitoring, performance trending and unified alerting and event correlation.
TDS unified monitoring solves this need by providing a consolidated view of all internal and external services. Built for complex, enterprise deployments, TDS tools provide visibility into both the end-to-end application environment and storage level to quickly detect and diagnose the root cause issues which are affecting service delivery. Utilizing industry-standard monitoring tools, TDS provides a "single pane of glass" view of all internal and external services.
This unified approach identifies and isolates cross application impact (including network, storage, CPU,memory, etc.) that may not exist in the traditional environment, but do exist in a shared component, virtualized environment.
There are many considerations involved in managing a hybrid data center. TDS unified monitoring addresses the need for a single, centralized view of overall data center performance and availability.
Posted by Michael Bullock on Tue, May 18, 2010
AFCOM recently suveyed 436 member data center sites on eight of the hottest topics ranging from Greening and Data Center Consolidation to Performance Monitoring and Cyber Terrorism.
-
- 71% are actively engaged in greening
- Power Recovery - 60%
- Cooling Recovery - 51%
- Expansion
- 60.3% require additional data center capacity within five years
Please view the tabulated results and significant findings below:
Posted by Michael Bullock on Fri, Apr 30, 2010
Think IT and facilities operating in silos is last year's problem? In some cases, it may actually be getting worse.
Last year I wrote Four Reasons to Get Facilities and IT on the Same Page. At the time, I was surprised by how little IT leaders knew about their real data center costs, and I suggested that these leaders would do well to talk to their facilities people to understand better where their money was going. I even showed how IT could help facilities by understanding their problems. Since collaboration between IT and facilities can provide immediate financial benefits to the enterprise without layoffs or the risk of IT outages, there's a lot in it for both groups to work together.
At least one would think.
But when I bring the subject up, especially with the press, I find that most believe the problem has already been solved. "IT and facilities operating in silos is last year's problem, right?" Wrong. When I work with real world customers, I realize that the situation is bad and maybe even getting worse.
Why is there this disconnect between perception and reality? Perhaps it's because when the press looks for a story, it needs real-life case studies to make the story compelling and believable. And, of course, it is possible to find companies where, in fact, facilities and IT are on the same page. Naturally, they're willing to go public with their stories. But nobody is all that interested in going on record to talk about how they've made mistakes, how IT and facilities don't work together, or how they're pouring money down the drain by having both IT and facilities operate in a vacuum.
Don't believe there's still a problem? Here are some real life examples illustrating what can go wrong when facilities and IT don't collaborate:
• I saw a company purchase a building because it looked like an attractive value and, as a bonus, it already had a data center the company thought it could use. Because the facilities group knew IT needed a new data center, it followed the realtor's recommendation and purchased the building at a bargain. After learning that their new data center would require an additional $17 million in upgrades to make it usable for IT, plus another $500,000 a month in WAN charges, the company decided a collocation alternative would be a better approach. In the end, the company avoided that $17 million upgrade but it also ended up with a building it didn't really need. This could have been avoided with closer collaboration between facilities and IT.
• I recently visited a 9,000 square foot data center where something was amiss. IT had requested Tier 3 level (N+1) redundancy so the facility would be concurrently maintainable. Facilities actually delivered what they believed to be a full Tier 4 environment (2N) with no single point of failure. Besides costing the company over $500,000 more than it should have, the center actually did have a single point of failure, meaning it wasn't even a Tier 3 data center. Another investment of $150,000 was required to correct the problem.
• I've recently seen over 100 cabinets at one location ejecting hot exhaust right into the intake of other systems. Besides subjecting these systems to the risk of premature failure, this made it necessary to run the data center much cooler than it needed to be just to head off cooling-related problems. This unnecessarily raised the PUE, resulting in an electric bill that was 30 percent higher than it needed to be. By now everyone knows about hot aisle-cold aisle alignment, but many legacy data centers—such as this one—haven't been reconfigured to take advantage of efficiency improvements like that just because IT and facilities are not communicating and are instead holing up silently in their respective silos.
I have other stories like this; I see them every day.
Let's face it. Facilities typically reports to the CFO's office and IT typically reports to the CIO and when the CFO and CIO get together they have better things to do than discuss PUE, power density, cooling, ultrasonic humidification and data center tier ratings (although I'm sure they have a chat or two about budgets).
CFOs and CIOs expect their respective teams to do the right thing. We all understand that. But maybe these teams need some top-down guidance to help get them moving in the right direction, together.
So I open this blog up to learn about your own experiences with IT and facilities. Have you had any great successes or horror stories you'd like to share?
As always, I welcome feedback, questions and comments. And if you know of other companies effectively enabling cloud computing with an impact on the enterprise you believe similar to those listed above, I'd be interested in learning more. You may reach me at cioblog@transitionaldata.com.
Posted by Eric Kraieski on Wed, Apr 07, 2010
Markley Group Boston has invited over six thousand attendees from New England to their 2010 Data Center event at One Summer Street. TDS, Cisco, APC, Gilbane and some 60 other companies are sponsoring this event.
Principal consultants from TDS will be on hand discuss your IT and data center priorities as well as service providers and vendors in the following areas:
- Data Center Facililities: Critical Infrastructure
- IT Infrastructure
- IT Service Providers and Managed Operations
- Data Center Optimizations and Green Data Center Efficiency Improvements
- Service Providers and Network Access Companies
- Architects and Engineers
Event runs from 8:30 a.m. until 6:00 p.m. on April 28, 2010.
Posted by Michael Bullock on Thu, Mar 18, 2010
Computation Fluid Dynamics - Hot topic at Data Center World
Congrats to AFCOM for another fine event! More than 800 IT and facilities attendees converged in Opryland last week to learn the latest best practices for data center design and operations. Most seemed very keen on improving data center efficiency.
And like usual, there were plenty of vendors hawking their wares to scratch that itch. From floor fan tiles and cable racks - to containers and critical infrastructure, no stone seemed unturned anywhere there was a buck to be made.
So was it just me or did CFD modeling seem to take center stage at many of the vendor booths and presentations? And for good reason I might add. Computation Fluid Dynamics provides a visual illustration of your data center cooling system where you can see clear cause and effect of hot spots, inefficient design and suboptimal deployment.
Sophisticated CFD modeling has been used by Fortune 100 companies to design products ranging from automobiles to hypersonic spacecraft, and has been more recently adapted to data center cooling and airflow analysis. You can rotate a three dimensional rendering of your data center viewing it from all angles and are not limited to a fixed perspectives.
As you know, the main purpose of a data center cooling systems is to maintain equipment within acceptable operating temperature range. This helps to reduce downtime, extend equipment life and optimize energy costs associated with the air conditioning system. Achieving this requires an adequate supply of cooling air as well as effective distribution of the airflow to the inlets of all the racks.
Recent studies have shown that although most Data Centers have over twice the cooling capacity required, they still experience "hot spots" throughout the data center. This is a result of poor airflow management. And since airflow and pressure are invisible, it is difficult to develop a strategy for improving airflow management without the use of an airflow management modeling tool.
With CFD analysis, you can see these invisible temperature differences and airflow pathways. Here is a view that combines temperature and airflow in a single 3D perspective. Notice the red / hot IT racks and the cooler / blue CRACS.
CFD modeling also allows you to do "what if" planning if you want to optimize your current data center in concept, before physical implementation begins. You may believe that ultrasonic humidification will allow you to cool more efficiently, but will it go far enough in power and cooling recovery to postpone a pending data center migration?
And I caution you to keep in mind that while these tools do an excellent job in visualizing temperature and airflow, you still need to be diligent when factoring in power, efficiency and overall return on investment. For example, if you improve the efficiency of your critical infrastructure by 50% (i.e. improving PUE from 3.0 to 2.0), will you be able to immediately apply the recovered power to IT systems?
This partly depends on your infrastructure to deliver conditioned power - the load that can be sustained by your UPS and IT power distribution systems. But even if are not able to expand with this recovered power and cooling, you still will have reduced your data center electric bill by a whopping 33%!
As always, I welcome feedback, questions and comments. And if you know of other companies effectively enabling cloud computing with an impact on the enterprise you believe similar to those listed above, I'd be interested in learning more. You may reach me at mailto:cioblog@transitionaldata.com
Posted by Eric Kraieski on Mon, Feb 15, 2010
Latest PUE Standards News from Green Grid
The Green Grid recently announced agreement by global world governments on using PUE as the primary measurement of data center efficiency:
"A group of global leaders met on February 2, 2010 to agree on data center energy efficiency measurements, metrics, and reporting conventions. Organizations represented were the U.S. Department of Energy's Save Now and Federal Energy Management Programs, U.S. Environmental Protection Agency's ENERGY STAR Program, European Commission JRC Code of Conduct, Japan's Ministry of Economy, Trade and Industry, Japan's Green IT Promotion Council, and The Green Grid."
"The collective groups are in agreement on the following guiding principles, as an interim step toward the desired outcomes (1. b.). It is recommended that data centers begin to measure PUE according to these principles... "
Here is the announcement from Green Grid:
Posted by Michael Bullock on Tue, Dec 15, 2009
Welcome to the hybrid world.
I've hardly been alone in my skepticism about cloud computing and my belief that public cloud computing for the enterprise is not ready for prime time. But I’m sensing a general global warming on the subject. And if applied properly, I'm one of them.
Here’s what I said back in January 2009 in my top 5 predictions for 2009
(Hype Overload): "There will be a well publicized cloud computing disaster in 2009 – probably linked to security, reliability or service restoration delays."
Was I right? You be the judge.
• Google alone had 7 significant outages. Google outages damage cloud credibility (PC World).
• Amazon had at least 3 outages including taking down the Amazon retail site. Outage for Amazon web service (Data Center Knowledge).
• A Salesforce.com outage lasting nearly a day cut off access to critical business data for many of the company's customers on Tuesday in what appears to be Salesforce's most severe service disruption to date. Salesforce outage angers customers (cnet news).
• And of course security has been a concern. See The Cloud Security Survival Guide.
So there were over a dozen significant outages at Google, Amazon and Salesforce alone. The only surprising thing here is the widespread numbness to these failures that helps keep the hype coming.
And why do people gloss over these disasters? It's because, in truth, very little serious, mission critical, life-threatening, or deep financial work is going on in public clouds. So as long as nobody dies, loses millions, or breaches thousands or millions of personal records, these failures may be acceptable. The key is using the public option for a suitable purpose.
Some things will continue to be best left outside the cloud. The security issues alone are huge and may block deployment: PCI (payment card industry), HIPAA, information privacy laws like 201 CMR 17.00 in Massachusetts (and similar laws in other states) make it almost impossible to insure adequate security in the cloud.
These security concerns mean that it will be a long, long time before you see companies making a full transition to public cloud services. However, in the near term, these companies will continue to leverage stand-alone cloud applications where it makes sense (like Salesforce.com). They will continue to develop their internal virtualized infrastructure to look more and more like a private cloud. But this approach is evolutionary, not revolutionary.
Some vendors seem to get this evolutionary approach. They understand that the old must work with the new. They understand that applications and infrastructure are equally important to adopting cloud computing in the enterprise. They also understand that with the cloud’s increased infrastructure flexibility comes complexity that must be managed and monitored in order to assure the highest availability of applications to the users. After all, that's the only true measure of success.
So the companies to watch in 2010 that can enable these hybrid cloud/legacy applications include:
• Cisco, EMC and VMware which, through a joint venture called Acadia, have partnered to accelerate internal cloud development with unified management of the data center.
• Nimsoft, which has one of the most advanced platforms for monitoring and measuring end-to-end applications in a hybrid internal/public cloud environment. By tracking end-to-end application performance in addition to physical infrastructure Nimsoft enables application monitoring independent of the underlying infrastructure.
• IBM (with a good hybrid cloud vision) and Cordys, the latest Jan Baan company with a focus on enterprise cloud enablement.
As always, I welcome feedback, questions and comments. And if you know of other companies effectively enabling cloud computing with an impact on the enterprise you believe similar to those listed above, I’d be interested in learning more. You may reach me at cioblog@transitionaldata.com.
Posted by Michael Bullock on Fri, Dec 11, 2009
Let's you and I do a little math about the National Security Agency's planned $1.5 billion data center.
Reading about the National Security Agency’s (NSA's) plan to build a $1.5 billion cyber security data center (Information Week) at the Camp Williams National Guard training base in Utah, I became curious about just how much one could glean about the project from public NSA budget information. Here's what I took away:
At first, this looks like a pretty good deal for Uncle Sam. We seem to be talking about a 1.5 million square foot data center that’s going to cost $1.5 billion, or $1,000 per square foot. That's well below what one would expect to pay for a Tier 3 class data center facility. Or is it?
For me, the math just doesn't add up. According to the budget document, the power density will be "appropriate for current state-of-the-art high-performance computing devices and associated hardware architecture." Yet if you calculate the watts per square foot by dividing the center’s total watts (65MW) by total square feet (1.5 million), you come up with a power density estimate of about 43 watts per square foot. No way that's "state of the art."
So let's say you triple the power density to a relatively modest 130 watts per square foot. That means you could support the center's full load of 65MW in about 500,000 square feet of space. At this power density, you'd probably require another 500,000 square feet for support space (generators, UPS, cooling, etc.). That's 1 million square feet. So what happens with the remaining half million square feet? According to the U.S. Army Corps of Engineers (as reported in the Information Week story) this data center "will eventually employ between 100 and 200 workers." That translates into a whopping 2,500 square feet of office space per employee. Roomy, wouldn't you say?
Now let’s suppose this facility really did support current state-of-the-art power density. That would mean about 400 watts per square foot. Given that density, the total space the data center actually would require would be just 160,000 square feet (i.e. 160,000 square feet x 400 watts per square foot = 64MW). Even assuming the same 500,000 square feet for support, that leaves us with a data center that's 660,000 square feet. Consequently, when you do the math, a 1.5 million square foot facility seems nutty and hard to reconcile with "state-of-the-art" anything. So what's going on?
My guess is that either the NSA has grossly miscalculated their space and power requirements or, perhaps, the true purpose and scale of the facility is a secret. The NSA keeping something secret? That wouldn’t be unheard of, would it?
Just for kicks, I wanted to see how this would compare to Uptime Institute estimates for this type of facility. I assumed 200 watts per square foot and used the Uptime Institute’s guidelines for a Tier 3 data center: $23,000 per KW of load and $300 per square foot. Using this approach, I came curiously close to NSA's $1.5 billion budget number ($1.495 billion, to be precise), and that made me wonder if the data center's budget is being built from the bottom up, or is simply a number tossed out to the public based on Uptime Institute estimates.
One last thought. Do you know how much money it will cost to operate a data center like the proposed Camp Williams facility? Based on a 65MW IT load, a PUE of 1.3 and Utah's cost of $.07 per kilowatt of electricity delivered (a nice rate while it lasts), it will cost $40 million per year simply to pay the electric bill.
But guess what? The reason the electricity is so cheap is because 98 percent of Utah’s electricity is produced by coal and natural gas. That's not very carbon friendly and with pending cap and trade legislation Utah's electricity costs will most definitely increase. How much? Who knows?
But whatever it is, the taxpayer (that's you) will be paying for it.
As always, I welcome feedback, questions and comments. You may reach me at cioblog@transitionaldata.com.
Posted by Michael Bullock on Tue, Oct 27, 2009
What you don't know can hurt you...
If you're in the market for new or additional data center space, I have some advice that can help you make better decisions, save money, and avoid some lurking pitfalls.
For example, a company found what at first seemed like the one of the most affordable spaces around, but after I asked the colocation provider a few pointed questions (actually, it was more like 70), it turned out to be one of the most expensive. Once the provider was pinned down and the customer's true requirements were taken into account, it turned out that all the power charges were vastly underestimated and the simple, 1,000 square foot / 120KW facility would have cost the customer about 250% more than he originally thought, amounting to an extra $1.6 million over the first five years of tenancy.
That's a lot of money.
So my advice has two parts:
- Try to keep more than one target space in the mix until the deal is done. Even if you are actively negotiating with your first choice, it pays to have a backup option. At minimum, this will allow you to negotiate without feeling trapped and provides a solid plan ‘B' plan should the preferred space become unavailable - or too costly upon full discovery.
-
My second tip is that you should go into negotiations armed with a trusty sword of skepticism. As Ronald Reagan said many years ago, "Trust, but verify." And since you and your company will be living with this decision for at least 5-10 years, it's very important to understand the details and long term impact of your agreement with your data center provider. It's always a mistake to view them as IT consultants who are there to help you. Their goal is to get you to commit to their space and lock you in while they retain the flexibility to increase their rates over the long term. This is not because they're bad guys. This is simply their business. You'd do the same.
Here are some of the collocation gambits that will end up costing you:
The Shell Game -- As you know, total facility fees include rent, power (utility) and a power surcharge that covers cooling, humidification, UPS (to avoid spikes and dips), etc. An inefficient facility will look very expensive and risky based on this surcharge, which could amount to one-to-two times as much as the cost of the actual metered power usage. So as not to scare off prospective customers, some operators may try to hide a percentage of this cost (that comes from their own inefficiency) by burying it in your lease charge. This would not be an issue were this a fixed cost, but more often than not the lease is subject to annual escalations. So you need to make sure you understand your provider's true operating expenses, pass through charges and cost escalation terms in order to have some degree of cost predictability going forward.
Convenient Laziness -- I've seen this happen this way: You put out an RFP with very specific questions, such as, "Does your proposal include redundant power feeds?" The bidder provides a one page summary of its expected charges and then replies to your question about backup power by writing, "See quote." When you look at the quote, you see a line item for redundant power. So does that answer your question? Maybe yes, maybe no. The quote may refer only to the UPS battery backup on the primary line and not to the second power feed into the building that's required for Tier 3 facilities. So, at best, it takes a lot of digging and hard work on your side to interpret the provider's response when he writes "See quote," and there's always the risk that you won't get it right. On the other hand, this convenient laziness in the response to your RFP allows the provider to crank out many quotes for many RFPs using an Excel spreadsheet with fixed assumptions that have nothing to do with your unique business and its unique requirements, not to mention the specifications contained in your RFP-the only one you care about. So when issuing an RFP for data center space, you should make it clear that specific, detailed responses are required or the bid will be rejected as noncompliant.
The Old Bait and Switch -- In this case, the bidder may provide detailed responses to your questions but he may not have the capacity or architecture to live up to his promises or your requirements. Recently, I saw a colo provider assure a customer of Tier 3 redundancy even though its facility's backup power feed was unconditioned street power. This meant that it was unprotected by UPS power and therefore represented a single point of failure and vulnerability to power spikes whenever the primary went down for maintenance. The tenant was paying market rate (which seemed like a good deal) for a true Tier 3 facility but the truth was that it wasn't getting one. In other cases, I've seen mismatches in what a given facility is capable of providing and what is actually quoted and provisioned to the customer. In these cases, the provider hopes first to get the new tenant in the space and plans to address the requirements later, after the relocation has been completed. I wouldn't feel comfortable with that set-up. Would you?
Rounding Down -- In this scenario, the customer asks for a specific power capacity he requires to operate his IT equipment. The colocation bidder immediately discounts this by 20%, claiming, "You shouldn't run the system above 80% of design load." So the facility provides you with 120 KW of breakered service where only 80% (or 96KW) is available for IT use. Of course, it could have provided 150 KW of breakered service (120 IT usable), but then its price would look too high. All the power and cooling estimates you get after that are based on this assumed 20% reduction to your requirements. If you don't look at the details, you may end up with a space you thought was 20% more efficient than others but really has 20% less capacity than the alternatives.
The message is that when evaluating new data center space, make sure you're diligent, and be on your guard. Take advantage of the people in your organization (or outside experts) who have the experience to know how large data centers are operated and where to look for the traps and gaps that will cause your requirements to go underserved. And beware of real estate brokers who may not understand what data centers really require and simply try to sell you on sites that pay them a higher commission. I'm not saying all real estate brokers are like that . . . I'm just saying . . .
As always, I welcome feedback, questions and comments. You may reach me at cioblog@transitionaldata.com.
Posted by Michael Bullock on Fri, Oct 02, 2009
With data center operations, equipment maintenance, staffing and licenses consuming the lion's share of today's IT budgets, companies can gain a great deal by optimizing their current operations.
Given the current recessionary environment, technical expertise, strong budgetary controls and procurement discipline are more critical than ever to maintaining operating margins.
Across the board we are seeing organizations taking on smaller, quick hit projects to balance spending and risk. Conversely, most companies are postponing or scaling back major transformational projects as managers look for rapid ROI either by better utilizing the technology assets they already have or by undertaking initiatives with shorter implementation times and faster paybacks. Not surprisingly, fewer companies are willing to fund multi-year projects that will not begin paying back their investment for 18 to 24 months or more.
But saving money is never easy. Many initiatives to reduce recurring long term costs are complex and expensive endeavors in themselves and especially difficult to implement during a recession. Even in the best of times, such projects warrant a thorough technical and financial analysis to make sure they align with the company's business direction and economic goals.
My advice for CIOs and their enterprises is to continue to invest in solutions that deliver well-defined results that at the same time minimize disruptions to employee productivity. You may be surprised how many projects can produce a healthy ROI in just nine months or less. These are the initiatives that will easily pass muster with your CFO and executive team.
Here are two examples of how taking a fresh look helped two companies save significantly on OPEX and CAPEX - delivering faster value to their organizations:
Example 1: How we saved $17M and moved one year early
A Fortune 1000 manufacturer planned to relocate to a new data center at a projected cost of $17 million. The company's facilities manager had spent months considering the move and was among the chief advocates for building a new data center over the next two years. He knew facilities (but not data centers) and his site selection was based on traditional real estate values -- size, location and cost -- which, unfortunately, do not always translate well into data center requirements.
Fortunately, with some outside help, the company took a closer look at the proposed relocation, set aside its internal preferences, and a $17 million mistake was avoided. The company discovered that it would need to install a fiber ring connecting the new data center at a recurring cost of $500,000 per year. On top of that, the new facility would not have sufficient cooling capacity for the anticipated power load.
By starting with the requirements, not the real estate, the company found suitable colocation/lease space with total annual costs equivalent to the fiber trunk expense alone. By going into a collocation facility with multi-carrier access, the fiber trunk requirement was eliminated, facility construction costs were avoided, and the cooling problem went away. On top of this, the client moved into the space one year ahead of schedule.
The moral of this story is that in-house employees, and even senior managers with lots of experience, can become cheerleaders for projects that keep them in their comfort zone or may even seem to guarantee long-term job stability. Companies need to spend some effort on understanding that which is less familar and looki in unfamilar places for fast payback rewards.
Example 2: How to save $300,000 per year and improve service
Large enterprises with big IT operations and large facilities staffs typically have multiple departments reporting to different parts of the organization. It's not uncommon for such groups to operate in silos, ignorant of each other's
For example, a major Boston-based investment firm's IT infrastructure had grown over the years through mergers and acquisitions and by decisions made by department heads who didn't communicate.
As the company became alarmed by its ballooning IT budget, it sought help from an independent IT consultant with no ties to a specific hardware vendor. After looking at all its processes, the company discovered that it was spending $2.7 million annually on a jumble of security systems with many overlapping features. Worse, these systems were installed in a cascading fashion that created management headaches and performance bottlenecks.
By redeploying the existing technology more effectively and retiring unnecessary systems, the company immediately cut annual fees by $300,000 per year while improving manageability and performance.
Stories of the left hand not knowing what the right hand is doing are rampant in business. The lesson here is that taking a full inventory of systems and services across all groups is an important first step in consolidation and cost containment.
As you search for new ways to deliver value, don't forget to take a hard look at areas that you may have forgotten during the boom years. Be open to questioning your internal biases and the ways you've always done things. There are many places you may be able to recover budget simply through smarter operations.