The SANMAN

VI's Archie Hendryx Chinwags with Mike Laverick

2012-02-21T04:19:00.000-08:00

If you've ever been involved with VMware it's more than likely that you've heard of Mike Laverick. As well as being a VM Guru, vExpert, all-round cool guy and author of great books such as "Administering VMware Site Recovery Manager 5.0", Mike also has a popular site of blogs, info and videos, namely RTFM.

Occasionally Mike invites certain vendors or tech folk for a "chinwag"and publishes this on his site and this week I was lucky enough to have the opportunity to discuss Virtual Instruments' solution and how it enables the successful virtualization of Mission Critical Apps.
Here's a link to the video:

I hope you enjoy it!

Avoid Storage Oversubscription Ratio Problems

2012-02-15T21:17:00.000-08:00

Just finished up a quick two minutevideo presented by my colleague, Jim Bahn, on how to solve the common problemof SAN oversubscription ratio problems. Interestingly I've found theoversubscription problem with just about every end user I've been to this lastyear, so there clearly is a lack of awareness on the issue.

 The biggest limitation is that youcannot historically trend the ASIC or Blade utilisation of a SAN Director withits native tools. I've sadly had customers that have had oversubscriptionissues at different times of the day on different Blades and ASICs e.g. nightlybackups etc. One customer had a six month problem with their Oracle backups andwhen we deployed the VirtualWisdom software we found that this was due to theoversubscription. Every other night the backups were causing the Blades tomomentarily have link resets which in turn would reset each of their tapedrives causing 'shoe-shining' and the prolonged backups. There was no way wecould have identified this without the historically trending feature unless wephysically saw this happen on the switches at the exact moment itoccurred.

This is just one of many examples ofproblems that occur due to incorrect oversubscription.

Anyhow enough of the anecdotes here's the video:

vSphere 6.0 - What's Needed?

2012-02-14T17:04:00.000-08:00

Hi - Below are the details for what will be my final webcast for Virtual Instruments before I move on to another adventure.

I hope you can join and look forward to your contribution and feedback!

vSphere 6.0 - What's Needed? - Webinar 23/2

The new vSphere 5.0 storage features such as VAAI, Storage DRS, Site Recovery Manager 5 and VASA, highlight the fact that whether on a virtualized or non-virtualized platform, application performance is heavily affected by its underlying storage infrastructure. The silos that once existed between storage and VMware teams are now being challenged as vSphere 5 brings to the forefront the need for a common understanding and integration.So what is needed in the next version of vSphere to achieve the successful virtualization of Mission Critical Applications?

Join Archie Hendryx, Virtual Instruments Senior Solutions Consultant & vExpert, as he discusses the way to counter such challenges and ensure a successful virtualization initiative that eliminates risk, optimizes performance and enhances business continuity and availability of key applications.

Register for free now at:
http://info.virtualinstruments.com/Webinar-vSphere022312_WhatsNew.html
February 23, 2012 at 9:00 AM PST, 5:00 PM GMT, 12:00PM ET

VCE FastPath, Virtual Instruments' SAN Probe & Hadoop - 2012 Storage Predictions

2012-02-13T06:30:00.000-08:00

Yearly prediction blogs are so clichéd hence why I’ve always tried to avoid writing one. Despite this I’ve always made a mental note of technology, products or companies that I thought were going to really do well in the upcoming year. Back in 2008 I felt VMware were going to really take off after the release of 3.5. In 2009 I had a gut feeling DataDomain would explode just before they were bought by EMC. In 2010 I spoke to a friend about how 3PAR’s technology could no longer be ignored and in 2011 I still wasn’t convinced that FCoE would overtake FC in revenue despite all the analysts’ claims. But why believe me when I’d never put these thoughts on paper? So now at the beginning of 2012, I’ve decided to put my money where my mouth is, pull out my crystal ball and document my predictions.

First off I’m going with VCE’s Vblock and their new FastPath feature. VCE (or the company formerly known as Acadia) have always been an exciting prospect with their all-in-one Vblock solution. While other vendors such as HDS, HP and Dell all plot the launch of their own unified computing block, VCE have had the advantage of being the first on the market and consequently the first to learn and adapt their messaging and offering in accordance to customer needs. One such initiative is what is being coined as FastPath. In essence FastPath is a Wizard-GUI based deployment of a Vblock VDI infrastructure that’s based on best practice reference architecture that enables deployment to be accelerated from months to days. I’ve often blogged on the many benefits of VDI and the immense CAPEX and OPEX savings that come with it; to be honest it’s a no brainer. What I did fail to mention was the sometimes long drawn out and painful PoC process that would be required to prove out the value of a VDI deployment to a potential customer. Well, FastPath is the solution to that conundrum.

VCE Vblock - A solution not a 'box'

Available in pre-configured Vblocks, FastPath allows the customer to choose from a variety of products that scale according to their needs thus eliminating the risk of sizing errors and scaling out as needs grow. So if you have a requirement for 500, 1000 or 1500 desktop users, choose the appropriate preconfigured model and you’re ready to go as your VDI roll out is based on known capacities hence avoiding unnecessary pre-purchasing of hardware. Added to this the Vblocks are leveraging proven design and reference architectures via an installation wizard that focuses on performance and usage specific to your environment mitigating any risk to a VDI success. The Installation wizard immediately configures the VMware View components, as well as the connection broker and is completed in minutes, even creating and optimising the VDI storage layout that can be cloned as ‘Gold’ master images.

VCE VBlock FastPath - Preconfigured with a single SKU

The business benefits are obvious in that customers can now accelerate their turnaround time from order to installation and enjoy a seamless roll out from PoC to Production. Your TCO is easily quantifiable as you know exactly what you’re acquiring, how it will operate and perform and how much the whole package costs. While FastPath is for VDI deployments, it wouldn’t be surprising to see VCE adopt a FastPath strategy for other Vblock deployments such as Oracle or SAP P to V migrations, or primary and secondary Vblock DR set ups that leverage Site Recovery Manager. The possibilities are numerous and 2012 could well be the year when FastPath transforms an erroneous mindset of Vblocks being a unified hardware computing block to instead being an all in one, quick to deploy and essential solution to the business.

Secondly is obviously a technology that I hold close to my heart having worked for the product’s company Virtual Instruments, namely the SAN Performance Probe. Initially VI were depending on Finisar technology for their probe products and their unique ability to track millisecond latency across Fibre Channel SAN infrastructures. Now with last year’s launch of their own SAN Availability Probe they’ve seen hardware sales rocket as they’ve empowered Storage, Server & VMware administrators to master the once complex art of FC SAN optimization via an easy to use dashboard GUI. What initially was seen as a FC SAN troubleshooting tool, it’s quickly becoming apparent via customer use cases that the value of the platform extends far beyond the realms of the SAN administrator. Already customers have found the SAN Availability Probe provides them the ability to de-risk disaster recovery, optimize backups, safeguard virtualization of Tier 1 applications as well as optimize the performance of their existent infrastructure while offsetting future procurement.

Virtual Instruments' SAN Performance Probe - Redefining Infrastructure De-risking

One of the keys to success for any company in such a highly competitive start-up market is to have a 'Blue Ocean Strategy', an experienced and top class leadership and a vision for the future. The reality is the product has no competitor and while this may have irked some vendors into producing FUD that this is not the case, it speaks volumes that a company which has yet to reach the 200 employee mark could be rattling the cages of such big corporations. Add to the mix that you have an executive team that includes a legend of the industry such as former Symantec CEO John W.Thompson and veterans from EMC, McData and HDS as VPs of Sales, Pre-Sales, Marketing and Services, it’s not surprising that a new product from a relatively new start up can so easily walk into large enterprise accounts and justify their unique value. As VI’s customer base will inevitably grow in 2012, so too will the SAN Performance Probe's use cases and consequent business value.

Lastly is a technology that was named after a kid’s toy elephant - Hadoop. Like all great things in life this Java-based programming framework is free. Part of the Apache project and partly invented by Google to help them present back to their users meaningful results from all the information they were indexing and collecting, Hadoop is the solution to what will be the term of 2012 i.e. ‘Big Data’. The long standing problem that Google and their like faced i.e. lots of structured and unstructured data and the challenge of having to run process intensive analytics was always an expensive proposition when put in the context of a traditional centralized database system. So instead of being limited to a single disk mapped to eight processors, Hadoop simply breaks up an application into numerously small fragments which can then be run on any node in a cluster. Hence In a cluster of servers that each have eight CPUs, Hadoop will send your code across those numerous servers enabling you to run your indexing job with all those processors working in parallel, quickly and efficiently and still return your results as a single readable whole.

Silly logo aside, Hadoop is key to the serious market of Big Data

With the Hadoop framework being already adopted by the likes of Yahoo, IBM and Google, 2012 could well be the year when Hadoop moves beyond search engine sites and find more prominence in the retail and finance sector. That is not to say that current datawarehouses or transaction processing systems are about to be ripped out of these sectors. Instead when these traditional databases reach their peaks, running Hadoop will enable further analysis across multiple data feeds in a single platform at a relatively cost effective price. So for example in the finance sector Hadoop will easily find a useful space in the context of identifying transaction fraud where large data sets for modelling and backtesting need to be created. Other use cases could include supporting compliancy by using Hadoop for the daily processing of equity markets data or even utilizing it for the consolidation of datawarehouses that run loan, banking and credit card consumer products. As for the retail sector, their drive towards cost-effective solutions to deal with their growing amount of consumer and product information is another ideal for a Hadoop based solution. What retail outlet wouldn’t want to provide an online customer experience that provided a product search result comparable to that of Google’s? In fact such is Hadoop’s potential for the Enterprise that even EMC have taken note with the recent launch of their Isilon scale-out NAS that incorporates Hadoop's Distributed File System. This could just be the beginning for Hadoop as the big vendors start to also give their seal of approval.

So while there were a number of other technologies, products and vendors I feel are going to cause some waves this year such as HDS' HAM, Tintri, EMC's VFCache, IBM's SVC 6.2 support for VAAI and of course VMware's eventual move into PaaS with the acquisition of SpringSource, I'm putting my neck on the line with these three being a guarantee; VCE's Vblock FastPath, Virtual Instruments' SAN Availability Probe and the Open Source Hadoop. Either way 2012 looks to be another great year for technology and Storage innovation in particular.

vSphere's Virtual CPUs - Avoiding the vCPU to pCPU ratio trap

2011-12-30T19:47:00.000-08:00

2011 was a year where despite the economic constraintseverything Big was seemingly good; Big Data, Big Clouds, Big VMs etc. Caught inthe industry’s lust for this excess, 2011 was also the year I lost count of howmany overprovisioned resources to ‘Big’ Production VMs I witnessed. More oftenthan not this was a typical reaction from System Admins trying to alleviatetheir fears of potential performance problems to important VMs. It was the yearwhere I began to hear justifications such as “yes we are overprovisioning ourproduction VMs..but apart from the cost savings, overallocating our availableunderlying resources to a VM isn’t a bad thing, in fact it allows it to bescalable”. Despite this 2011 was also the year where I lost count of the amountof times I had to point out that sometimes overprovisioning a VM does lead toperformance problems - specifically when dealing with Virtual CPUs.

VMware refers to CPU as pCPU and vCPU. pCPU or ‘physical’ CPUin its simplest terms refers to a physical CPU corei.e. a physical hardware execution context (HEC)if hyper-threading is unavailable or disabled. If hyperthreading hasbeen enabled then a pCPU would consitute a logical CPU. This is because hyperthreadingenables a single processor core to act like two processors i.e. logicalprocessors. So for example, if an ESX 8-core server has hyper-threading enabledit would have 16 threads that appear as 16 logical processors and that wouldconstitute 16 pCPUs.

As for a virtual CPU (vCPU) this refers toa virtual machine’s virtual processor and can be thought of in the same vein asthe CPU in a traditional physical server. vCPUs run on pCPUs and by default, virtual machines are allocatedone vCPU each. However, VMware have an add-on software module named Virtual SMP(symmetric multi-processing) that allows virtual machines to have access tomore than one CPU and hence be allocated more than one vCPU. The greatadvantage of this is that virtualized multi-threaded applications can now bedeployed on multi vCPU VMs to support their numerous processes. So instead ofbeing constrained to a single vCPU, SMP enables an application to use multipleprocessors to execute multiple tasks concurrently, consequently increasingthroughput. So with such a feature and all the excitement of being ‘Big’ it waseasily assumed by many that taking advantage of such a feature by provisioningadditional vCPUs could only ever be beneficial – but if only it was that simple.

The typical examples I faced entailed performance problems thatwere either being blamed on the Storage or the SAN and not CPU constraintsespecially as overall CPU utilization for the ESXserver that hosted the VMs would be reported as low. Using VirtualInstruments’ VirtualWisdom I was able to quickly conclude that the problem wasnot at all related to the SAN or Storage but the hosts themselves. By beingable to historically trend and correlate the vCenter, SAN and Storage metricsof the problematic VMs on a single dashboard it wasapparent that the high number of vCPUs toeach VM was the cause. This was indicated by a high reading of what is termedthe 'CPU Ready' metric.

To elaborate, CPU Ready is a metric that measuresthe amount of time a VM is ready to run against the pCPU i.e. how long a vCPUhas to wait for an available core when it has work to perform. So while it’spossible that CPU utilization may not be reported as high, if the CPU Ready metricis high then your performance problem is most likely related to CPU. In theinstances that I saw, this was caused by customers assigning four vCPUs and insome cases eight to each Virtual Machine. So why was this happening?

VirtualWisdom Dashboard indicating high CPU Ready

Well firstly the hardware and itsphysical CPU resource is still shared. Coupled with this the ESX Serveritself also requires CPU to process storage requests and network traffic etc.Then add the situation that sadly most organizations still suffer from the‘silo syndrome’ and hence there still isn’t a clear dialogue between the SystemAdmin and the Application owner. The consequence being that while multiplevCPUs are great for workloads that support parallelization but this isnot the case for applications that don’t have built in multi-threadedstructures. So while a VM with 4 vCPUs will requirethe ESX server to wait for 4 pCPUs to become available, on a particularly busyESX server with other VMs this could take significantly longer than if the VMin question only had a single vCPU.

To explain this further let’s take an example of a four pCPU host that has four VMs, three with 1 vCPU andone with 4 vCPUs. At best only the three single vCPU VMs can be scheduledconcurrently. In such an instance the 4 vCPU VM would have to wait for all fourpCPUs to be idle. In this example the excess vCPUs actually imposescheduling constraints and consequently degrade the VM’s overall performance,typically indicated by low CPU utilization but a high CPU Ready figure. With the ESX server scheduling and prioritising workloadsaccording to what it deems most efficient to run, the consequence is thatsmaller VMs will tend to run on the pCPUs more frequently than the largeroverprovisioned ones. So in this instance overprovisioning was in fact provingto be detrimental to performance as opposed to beneficial. Now in more recent versions of vSphere the scheduling ofdifferent vCPUs and de-scheduling of idle vCPUs is not as contentious as itused to be. Despite this, the VMKernel still has to manage every vCPU, acomplete waste if the VM’s application doesn’t use them!

To ensure your vCPU to pCPU ratio is at its optimal level andthat you reap the benefits of this great feature there are some straightforwardconsiderations to make. Firstly there needs to be dialogue between the silos tofully understand the application’s workload prior to VM resource allocation. Inthe case of applications where the workload may not be known, it’s key to notoverprovision virtual CPUs but rather start with a single vCPU and scale out asand when is necessary. Having a monitoring platform that can historically trendthe performance and workloads of such VMs is also highly beneficial indetermining such factors. As mentioned earlier CPU Ready is a key metric toconsider as well as CPU utilization. Correlating this with Memory andNetwork statistics, as well as SAN I/O and Disk I/O metrics enables you toproactively avoid any bottlenecks and correctly size your VMs and hence avoidoverprovisioning. This can also be extended in considering how many VMs youallocate to an ESX Server and in ensuring that its physical CPU resources aresufficient to meet the needs of your VMs. As businesses’ key applicationsbecome virtualized it’s an imperative that whether they are old legacy singlethreaded workloads or new multi threaded workloads the correct vCPU to pCPUratio is allocated. In this instance size isn’t always everything it’s what youdo with your CPU that counts.

Exchange Completion Time - SAN Storage Performance Redefined

2011-12-21T06:30:00.000-08:00

Roll back several years and certain vendors had you believe that Fibre Channel was dead and that the future would be iSCSI. A few years later and certain vendors were then declaring that Fibre Channel was dead again and that the future was FCoE. So while this blog is not a iSCSI vs FC or FC vs FCoE comparison list (there’s plenty of good ones out there and both iSCSI or FCoE each have immense merit), the point being made here is that Fibre Channel unlike Elvis really is alive and well. Moreover Fibre Channel still remains the protocol of choice for most Mission Critical Applications despite the FUD that surrounds its cost, manageability and future existence. Most Storage folk who run Enterprise class infrastructures are advocates of Fibre Channel not only because of its high performance connectivity infrastructure but also due to its reliability, security and scalability. Incredibly this is all with the majority of Fibre Channel implementations being vastly under utilized, poorly managed (due to lack of visibility) and running at a far from optimized state due to the constant day to day operations of most SAN Storage administrators. Indeed if Storage folk were empowered with a metric that could enable them to gain a better insight and understanding of their SAN Storage’s performance and utilization the so called impending death of Fibre Channel may have to take an even further rain check. Well that metric does exist; cue what is termed the “Exchange Completion Time.”

It’s now common for me to visit customer environments that run Fibre Channel SANs yet have various factions that complain they are suffering performance issues due to lack of bandwidth or throughput, whether that's server, VM, Network or Storage teams. In every single instance FC utilization has actually been incredibly low with peaks of 10% at the most and that's with 4GB/s environments not 8GB/s! At worst there may be an extremely busy backup server that singlehandedly causes bottlenecks and creates the impression that the whole infrastructure is saturated but even these occasions are often rare. What seems to be the cause of this misconception is the lack of clarity between what is deemed throughput and what is an actual cause of bottlenecks and performance slow downs i.e. I/O latency.

Sadly (and I am the first to admit that I was also once duped), Storage folk have been hoodwinked into accepting metrics that just aren’t sufficient to meet their requirements. Much like the folklore and fables of Santa Claus that are told to children during Christmas, storage administrators, architects and engineers have also been spun a yarn that MB/s and IOPS are somehow an accurate determination of performance and design considerations. In a world where application owners, server and VM admins are busily speaking the language of response times, Storage folk are engrossed in a foreign vocabulary that revolves around RAID levels, IOPS and MB/s and then numerous calculations to try and correlate the two languages together. But what if an application owner requested Storage with a 10ms response time that the Storage Administrator could then allocate with a guarantee of that performance? That would entail the Storage engineer not just looking at a one dimensional view from the back end of the Storage Array but one that incorporated the comprehensive transaction time i.e. from the Server to the Switch port to the LUN. That would mean considering the Exchange Completion Time.

To elaborate, using MB/s as a measurement of performance is almost akin to how people used to count cars as a measurement of road traffic. Harking back to my days as a student and before all of the high tech cameras and satellites that now monitor road traffic, I was ‘lucky’ enough to have a job of counting the amount of cars that went through Trafalgar Square at lunchtime. It was an easy job, I'd see five cars and I'd click five times but this was hardly accurate as when there was a traffic jam and all of the lanes were occupied I was still clicking five cars. Here also lies the problem with relying on MB/s as a measurement of performance. As with the counting car situation a more accurate way would have been to instead watch each single car and measure it's time from its origin to its destination. In the same vein, to truly measure performance in a SAN Storage infrastructure you need to measure how long a transaction takes from being initiated by the host, received by the storage and acknowledged back by the host in real-time as opposed to averages. This is what is termed the Exchange Completion Time.

While many storage arrays have tools that provide information on IOPS and MB/s to get a better picture of a SAN Storage environment and it’s underlying latency it's also key to consider the amount of Frames per second. In Fibre Channel a Frame is comparable to a word, a Sequence a sentence and an Exchange the conversation. A Standard FC Frame has a Data Payload of 2112 bytes i.e. a 2K payload. So for example an application that has an 8K I/O will require 4 FC Frames to carry that data portion. In this instance this would equate to 1 IOP being 4 Frames and subsequently 100 IOPS of the same size equating to 400 Frames. Hence to get a true picture of utilization looking at IOPS alone is not sufficient because there exists a magnitude of difference between particular applications and their I/O size with some ranging from 2K to even 256K. With backup applications the I/O sizes can be even larger. Hence it's a mistake to not take into consideration the amount of Frames/sec when trying to measure SAN performance or if trying to identify whether data is being passed efficiently. For example even if you are witnessing a high throughput in MB/s you may be missing the fact that there is a minimum payload of data and the Exchange (conversation) is failing to complete. This is often the case when there’s a slow draining device, flapping SFP etc. in the FC SAN network where instead of data frames causing the traffic you have a number of management frames dealing with issues such as logins and logouts, loss of sync or some other optic degradation or physical layer issue. Imagine the scenario, a Storage Administrator is measuring the performance of his infrastructure or troubleshooting a performance issue and is seeing lots of traffic via MB/s – unaware that many of the environment’s transactions are actually being cancelled across the Fabric!

This lack of visibility into transactions has also led to many storage architects being reluctant to aggressively use lower tiers of storage as poor I/O performance is often attributed to the storage arrays when often bottlenecks in the storage infrastructure are actually the root cause. Measuring performance via Exchange Completion Times enables measurement and monitoring of storage I/O performance, hence ensuring that applications can be correlated and assigned to their most cost- effective storage tier without sacrificing SLAs. With many Storage vendors adopting automated tiering within their arrays some would feel this challenge has now been met. The reality of automated tiering though is that LUNs or sub-LUNs are only dynamically relocated to different tiers based on the frequency of data access i.e. frequently accessed is more valuable so should reside on a higher tier and infrequently accessed data should be moved to lower tiers. So while using historical array performance and capacity data may seem a sufficient way to tier, it’s still too simplistic and lacks the insight for more optimized tiering decisions. Such an approach may have been sufficient to determine optimum data placement in the days of DAS when the I/O performance bottleneck was disk transfer rate but in the world of SANs and shared storage to look just at external transfer rates between SSD, Fibre Channel or SATA drives is a detached and inaccurate way to measure the effect of SAN performance on an application’s response time. For example congestion/problems in the SAN can result in severely degraded response times or cancelled transactions that fail to be acknowledged by the back end of the array. Furthermore incorrect HBA queue depths, the difference between sequential and random requests, link and physical layer errors all have an impact on response times and in turn application latency. By incorporating the Exchange Completion Time metric i.e. measuring I/O conversations across the SAN infrastructure into your tiering considerations, tiering can now accurately be based on comprehensive real time performance as opposed to device specific views.

Monitoring your FC SAN Storage environment in a comprehensive manner that incorporates the SAN fabric and provides metrics such as the Exchange Completion Time rapidly changes FC SAN troubleshooting from a reactive to proactive exercise. It also enables Server, Storage and Application administrators to have a common language of ‘response times’ thus eliminating any potential silos. With the knowledge of application I/O latency down to the millisecond, FC SAN Storage administrators can quickly be transformed from the initial point of blame to the initial point of resolution, while also ensuring optimum performance and availability of your mission critical data.

Demystifying the Cloud: IaaS, PaaS, SaaS, MaaS, CaaS & XaaS

2011-11-12T12:44:00.001-08:00

Generally IT folk, whether in Storage, Virtualization, Change Management or Project Management love the use of acronyms and synonyms to express key concepts amongst each other. What other industry would allow an individual to spurt a line such as "Have SOX seen the BCP and CAB approval for our VDC's DR SAN and will this then be added to the CMDB by CoB today?" without immediately flinching or bringing in a logopaedics specialist for help. More often than not, IT folk have also used these synonyms and acronyms as smokescreens to prevent outsiders from realizing "well this IT stuff is actually quite easy to understand and quite straightforward".

Hence no surprise that when the seemingly simple concept of Cloud Computing took off, so did the emergence of an abundance of acronyms and synonyms reaping a new breed of I.T. professionals who were the only ones that could correctly understand them, i.e., ‘The Cloud Specialist'. Despite this, the beauty of the Cloud (or as most people are starting to realise the synonym for the Internet) is that it not only encompasses the IT industry and their business demands but also the average end user who's only experience with IT is their iPhone and its App Store. So while EMC's extensive airport advertising may have initially confused a lot of tourists into thinking that the ‘Journey to the Cloud' was a slogan for an up and coming budget airline, the general public are certainly now becoming aware of ‘The Cloud'. End users are now bombarded with Clouds from Microsoft claiming that Windows 7 is your ‘Path to the Cloud', Pizza Restaurants offering free access to ‘the Cloud' and Apple iPhone owners having iCloud enforced upon them (no comment on the security issues of your email contacts and personal photos being uploaded to Apple's database). So while the idea of Public, Private and Hybrid Clouds become more familiar and understood even amongst the masses, it's with surprise that I often find people within the IT industry who are still unaware or unsure of Cloud Service acronyms such as IaaS, PaaS, SaaS, Maas, Caas or Xaas.

To understand why there are so many acronyms with the Cloud, it is important to appreciate that the Cloud has a number of services which each of these classify. The first of these, IaaS (Infrastructure as a Service) is when the consumer does not deal with the infrastructure, instead the responsibility of the equipment is outsourced to the Service Provider. The Service Provider not only owns the equipment but will also be responsible for its running and maintenance, where the consumer will be charged on a ‘pay as you use' basis. IaaS is often offered as a horizontally integrated service that includes not only the server and storage but also the connectivity domains. For example while the consumer may deploy and run their own applications and operating systems, the Iaas provider would typically provide the replication, backup and archiving (Storage), the powerful computing requirements (Server) or the network load balancing and firewalls (Connectivity domains).
PaaS provides the capability for consumers to have applications deployed without the burden and cost of buying and managing the hardware and software. In other words these are either consumer created or acquired web applications or services that are entirely accessible from the Internet. Usually created with programming languages and tools supported by the service provider these web applications enable the consumer to have control over the deployed applications and in some circumstances the application-hosting environment but without the complexity of the infrastructure i.e. the servers, operating systems or storage. Offering a quick time to market and services that can be provisioned as an integrated solution over the web, PaaS facilitates immediate business requirements such as application design, development and testing at a fraction of the normal cost.
Software as a service (SaaS) is the ability for a consumer to use on demand software that is provided by the service provider via a thin client device e.g. a web browser over the Internet. With SaaS the consumer has not only no management or control of the infrastructure such as the storage, servers, network, or operating systems, but also no control over the application's capabilities. Culled from what were originally referred to as (ASPs) Application Service Providers, SaaS is a quick and efficient delivery model for key business applications such as customer relationship management (CRM), enterprise resource planning (ERP), HR and payroll.

Monitoring as a Service (MaaS) is at present still an emerging piece of the Cloud jigsaw but an integral one for the future. In the same way that businesses realised that their infrastructure and key applications required monitoring tools that would ensure the proactive elimination of any downtime risks, Monitoring as a Service provides the option to offload a large majority of those costs by having it run as a service as opposed to a fully invested in house tool. So for example by logging onto a thin client or central web based dashboard which is hosted by the service provider, the consumer can monitor the status of their key applications regardless of location. Add the advantages of an easy set up and purchasing process and MaaS could be a key pay as you use model for the de-risking of applications that are initially being migrated to the Cloud.

Communication as a Service (CaaS), enables the consumer to utilize Enterprise level VoIP, VPNs, PBX and Unified Communications without the costly investment of purchasing, hosting and managing the infrastructure. With the service provider responsible for the management and running of these services also, the other advantage the consumer has is that they needn't require their own trained personnel, bringing significant OPEX as well as CAPEX costs.

Finally XaaS or ‘anything as a service' is the delivery of IT as a Service through hybrid Cloud computing and is a reference to either one or a combination of Software as a Service (SaaS), Infrastructure as a Service (IaaS), Platform as a Service (PaaS) Communications as a service (CaaS) or monitoring as a service (Maas). XaaS is quickly emerging as a term that is being readily recognized as services that were previously separated on either private or public Clouds are becoming transparent and integrated.

So as the term ‘The Cloud' finally breaks into the minds of the masses and takes meaning, the next phase will be to take the numerous services that are offered by the Cloud, mature them and enable consumers to fully understand their benefits. From Enterprise to SMB to end users, Cloud Services will inevitably bring immense benefits and cost savings. All that is now required is for consumers to know what all those unnecessarily complicated acronyms mean!

vSphere 5 & VAAI Demand Radical Changes to Storage Arrays

2011-09-05T12:29:00.000-07:00

The launch of vSphere 5 and its new storage related features will set the precedent for a complete rethink on how a new datacenter’s storage infrastructure should be designed and deployed. vSphere 5’s launch is not only an unabashed attempt at cornering every single aspect of the server market but is also a result for the growing need for methodical scalability that merges the I.T. silos and consequently combines the information of applications, servers, SANs and storage into a single comprehensive stack. In an almost ironic shift back towards the original principles of mainframe, VMware’s importance has already influenced vendors such as EMC with their VMAX and HDS with their VSP in adopting a scale out as opposed to scale up approach to Storage. With this direction being at the forefront of most storage vendors’ roadmaps for the foreseeable future it subsequently dictates a criterion far beyond the storage capacity requirement I.T. departments are traditionally used to. With such considerations the end of the traditional Storage Array could be sooner than we think as a transformation takes place on the makeup of future datacenters.

With the emergence of the Private Cloud, Storage Systems are already no longer being considered or accepted by a majority of end users as ‘data banks’ but rather ‘on demand global pools’ of processors and volumes. End users have finally matured into accepting dynamic tiering, thin / dynamic provisioning, wide striping, SSDs etc. as standard for their storage arrays as they demand optimum performance. What will dictate the next phase is that the enterprise storage architecture will no longer be accepted as a model for unused capacity but rather one that facilitates further consolidation and on demand scalability. Manoeuvres towards this direction have already taken place as some vendors have taken to adopting Sub-Lun tiering (in which LUNs are now merely containers) and 2.5 inch SAS disks. The next key shift towards this datacenter transformation is the new integration of the server and storage stack as brought about with vSphere 5’s initiatives, such as VAAI, VASA, SDRS, Storage vMotion etc.

Leading this revolution is Paul Maritz, who upon becoming the head of VMware, set his vision clear: rebrand VMware from a virtualization platform to the essential element of the Cloud. In doing that VMware needed to showcase their value beyond the Server Admin’s milieu. Introduced with vSphere 4.1, vSphere API Array integration (VAAI) was the first major shift towards the comprehensive Cloud vision that incorporated the Storage stack. Now with further enhancements in vSphere 5, VAAI compatibility will be an essential feature for any Storage Array.

VAAI has several features named primitives aimed at driving this change forward. There are several primitives associated with the vSphere API Array integration (VAAI) with the first being the Full Copy primitive. Essentially with VMware when copying of data occurs whether via VM cloning or Storage vMotion, the ESX server becomes responsible for first having to read every single block that has to be copied and then writing it back to the new location. Of course this adds a fundamental load and effect on the ESX server. So for example when deploying a 100GB VM from a template the entire 100 GB will have to be read by the vSphere host and then subsequently written requiring a total of 200 GB of I/O.

Instead of this host intensive process the Full Copy primitive operates by issuing a single SCSI command called the XCOPY. The XCOPY is sent for a contiguous set of blocks which in essence is a command to the storage to do the copying of each block from one logical block address to another. So hence a significant load is taken off the ESX server and instead put upon the Storage array that can more than easily deal with the copy operations resulting in very little I/O between the host and array.

The Second Primitive, Block Zeroing is also related to the virtual machine cloning process which in essence is merely a file copy process. When a VM is copied from one datastore to another it would copy all of the files that make up that VM. So for a 100 GB virtual disk file with only 5 GB of data, there would be blocks that are full as well as empty ones with free space i.e. where data is yet to be written. Any cloning process would entail not just IOPS for the data but also numerous repetitive SCSI commands to the array for each of the empty blocks that make up the virtual disk file.

Block zeroing instead removes the need for having to send these redundant SCSI commands from the host to the array. By simply informing the storage array of which blocks are zeros the host offloads the work to the array without having to send commands to zero out every block within the virtual disk.

The third VAAI primitive is named Hardware Lock Assist. The original implementation of VMFS used SCSI reservations to prevent data corruption when several servers shared a LUN. What typically occurs without VAAI in such situations though, are what are termed SCSI reservation conflicts. Being a normal part of the SCSI protocol, SCSI reservations occur to give exclusive access to a LUN so that competing devices do not cause data corruption. This especially used to occur during VMotion transfers and metadata updates of earlier versions of ESX and caused serious performance problems and poor response times as devices had to wait for access to the same LUN. Although these reservations are typically less than 1ms, many of them in rapid succession can cause a performance plateau with VMs on that datastore.

Hardware-Assisted Locking in essence is the elimination of this LUN-level locking based on SCSI reservations. How it works is that initially the ESX server will make a first read on the lock. If the lock is free, the server will then send a Compare and Swap command that is not only the lock data that the server wants to place into the lock but also the original free contents of the lock. The storage array will then read the lock again and compare the current data in the lock to what is in the Compare And Write command. If they are found to be the same, new data will be written into the lock. All of this is treated as a single atomic operation and is applied at the block level (not the LUN level) making parallel VMFS updates possible.

With the advent of vSphere 5, enhancements with VAAI have also come about for infrastructures deploying array-based thin-provisioning. Thin-provisioned LUNs have brought many benefits but also challenges such as the monitoring of space utilization and the reclamation of dead space.
To counter this the VAAI Thin Provisioning primitive has an Out-of-Space Condition which monitors the space usage on thin-provisioned LUNs. Via advanced warning users can prevent themselves from catastrophically running out of physical space.

Another aspect of this primitive is what is termed Dead Space Reclamation (originally coined as SCSI UNMAP). This provides the ability to reclaim blocks of thin-provisioned LUNs whenever virtual disks are deleted or migrated to different datastores.

Previously when a deletion of a snapshot, a VM or a Storage vMotion took place the VMFS would delete a file and sometimes leading to the filesystem reallocating pointers instead of issuing a SCSI WRITE ZERO command to zero out the blocks. This would lead to blocks previously used by the virtual machine being still reported as in use, resulting in the array providing incorrect information with regards to storage consumption.

With vSphere 5 this has subsequently changed as now instead of a SCSI WRITE ZERO command or reallocation of VMFS pointers, a SCSI UNMAP command is used. This in fact enables the array to release the specified Logical Block Address back to the pool, leading to better reporting and monitoring of disk space consumption as well as the reclamation of unused blocks.

Back in 2009, Paul Maritz boldly proclaimed the dawning of ‘a software mainframe’. With vSphere 5 and VAAI, part of that strategy is aiming at transforming the role of the Storage Array from a monolithic box of capacity to an off-loadable virtual resource of processors for ESX servers to perform better. Features such as SRM 5.0, SDRS, VASA (details on these to follow soon in upcoming blogs) are aimed at further enhancing this push. At VMworld 2011, Maritz proudly stated that a VM Is born every 6 seconds and that there are more than 20 million VMs in the world with more than 5.5 vmotions per second. With such serious figures, whether you’re deploying VMware or not, it’s a statement that would be foolish to ignore when budgeting and negotiating for your next Storage Array.

Silos will prevent Tier 1 Apps reaching the Cloud

2011-06-05T16:33:00.000-07:00

On a recent excursion to a tech event I had the pleasure of meeting a well-known ‘VM Guru’, (who shall remain nameless). Having read some of this individual’s material I was excited and intrigued to know his thoughts on how he was tackling the Storage challenges related to VMware especially with Fibre Channel SANs.

“Storage, that’s nothing to do with me, I’m a VirtGuy”, he proudly announced.

To which I retorted, “yes but if there are physical layer issues in your SAN fabric, or poorly configured Storage etc. it will affect the performance of your Virtual Machines and their applications, hence surely you also need some visibility and understanding beyond your Server’s HBAs?”

Seemingly annoyed with the question, he answered, “Why? I have SAN architects and a Storage team for that, it’s not my problem. I told you I’m a VirtGuy, I have my tools so I can check esxtop, vCenter etc…” as he then veered off into glorious delusions of grandeur of how he’d virtualized more servers than I’d had hot dinners. As fascinating as it was to hear him, it was at this point that my mind was side tracked into realizing that despite all the industry talk of ‘unified platforms’, ‘Apps, Servers & Storage as a Service’ i.e. the Cloud, the old challenge of bridging the gap between silos still had a long way to go.

Let’s face it Virtualization and the Cloud have brought unprecedented benefits but they’ve also brought challenges. One such challenge that is dangerously being

overlooked is that of the silos that exist within most IT infrastructures. Indeed it’s the silos that have led to the new phenomenon that is coined as, ‘The Virtual Stall’. The Virtual Stall was never an issue several years ago as Virtualization was happily adopted by Application owners to consolidate many of their ‘Crapplications’ that meant little or nothing to them and certainly didn’t carry the burden of a SLA. Storage teams were none the wiser as VM admins requested large capacities of storage for their VMFS and despite the odd performance problem no one was too bothered as these VMs rarely hosted Tier 1 Apps. With the advent of VDI, large VM backups and critical applications such as Exchange and SQL being virtualized, the ordeal of maintaining performance took root, resulting in the inevitable ‘blame game’ between silos. Fast forward to today and despite all the talk of Private Clouds, the fear factor of potential performance degradation resulting in the virtualization of mission critical applications has led to the ‘Virtual Stall’.

Business and Management have been convinced of the benefits of consolidation, reduction in data foot print, power/coolin

g etc. that they initially saw with the virtualization of their low tier applications. This has led them to want more of the same for higher end applications leading to what many organiz

ations are terming a ‘VMware First’ policy. Under pressure from them the silo of the application owners still don’t have a true understanding of server virtualization and hence are reluctant for their Tier 1 apps to be migrated from their physical platforms. At best they may accept two mission critical VMs on a physical server. Under pressure to prove the Application owners wrong and maintain the performance of virtualized applications, the silo of the VMware administrators will often over-provision from their pool of Memory, CPU and storage resources. Furthermore the VM Admin silo also lack a real understanding of Storage and at best will think in terms of capacity for their VMFS stores, while Storage Admin will think in terms of IOPS. As this lack of understanding and communication between the silos exists and grows so too do the challenges of making the most of the benefits of server virtualization.

One of the key mistakes is that it’s often over looked that whether on a virtualized or non-virtualized platform, application performance is heavily affected by its underlying storage infrastructure. The complexity of correctly configuring storage in accordance to application demands can range from deciding the right RAID level, number of disks per LUN, array cache sizes to the correct queue depth and fan-in / fan-out ratio. These and other variables can drastically influence how I/O loads are handled and ultimately how applications respond. With virtualized environments the situation is no different, with Storage related problems often being the cause of most VMware infrastructure mis-configurations that inadvertently affect performance.

Even with the option of Raw Device Mapping, the alternative for VMware storage configuration, VMFS is often the most preferred due to its immediate advantages in terms of provisioning and zoning. In this method several Virtual machines are able to access the same LUN or a pool of LUNs. This becomes far more simplistic as opposed to a one to one mapping ratio that is required for each LUN for each Virtual Machine with the RDM option. Additionally this makes backups far easier as the VMFS for the given Virtual Machines need only be dealt with instead of numerous individual LUNs that are mapped to many Virtual Machines. VMFS volumes can be as big as 2TB and with the concatenation of additional partitions which are termed VMFS extents, this can then be as large as 64TB i.e. 32 extents. With a Storage Admin unaware of such distinctions within VMware, it’s easy to also be unaware of the best practices with extents, such as creating these on new physical LUNs to facilitate additional LUN queues or throughput congestion. Coupled with this, if extents are not assigned the same RAID and disk type you quickly fall into a quagmire of horrendous performance problems. In fact it can be pointed out that the majority of VMware performance problems are in fact initiated at the beginning of the provisioning process or even earlier at the design phase and are a result of the distance between the silos.

As mentioned already application owners will pressure VM administrators to overprovision Memory and CPU to avoid any potential application slowdowns, while the VM administrator will falsely think along the lines of capacity for their VMFS in terms of Storage. At best a VM Admin may request the RAID level and the type of Storage e.g. 15K RPM FC disks but it is here that the discrepancy arises for the Storage administrator. The Storage Admin, used to provisioning LUNs on the basis of application requirements, will instead not be thinking of capacity but rather in terms of IOPS and RAID levels. Eventually though as there is no one to one mapping and the requested LUN is to be merely added to a VMFS, the storage administrator, not wishing to be the bottleneck of the process, will proceed to add the requested LUN to the pool. Herein is also the source of a lot of eventual performance problems as overtly busy LUNs begin to affect all of their aligned virtual machines as well as those that share the same datastore. Moreover if the LUN is part of a very busy RAID group on the backend of the storage array, such saturated I/O will impact all of the related physical spindles and hence all of the LUNS they share. What needs to be appreciated is that the workload of individual applications presented to individual volumes will be significantly different to that of multiple applications being consolidated onto a single VMFS volume. The numerous I/Os of multiple applications alone even if sequential, will push the Storage array to deal with these numerous requests as random, thus requiring different RAID level, LUN layout, cache capacity etc. considerations than those for individual applications.

Once these problems exist there is a customary troubleshooting procedure that VM and Storage administrators often follow which take from the metrics found in vCenter, esxtop, vscsiStats, IOMeter, Solaris IOSTAT, PerfMON and the Array management tool. This somewhat laborious process usually includes measuring the effective bandwidth and resource consumption between the VM and storage, moving and using other paths between the VMs and storage and even reconfiguring cache and RAID levels. To have even got to this point days if not weeks would have been spent in checking for excessive LUN and RAID group demands, understanding the VMFS LUN layout on the backend of the storage’s physical spindles, investigating the array’s front end, cache and processor utilization as well as bottlenecks on the ESX host ports. Some may even go to the lengths of playing around with the Queue Depth settings, which without an accurate insight is at best a guessing game based on rule of thumb. Despite all of these measures there is still no guarantee that this will identify or eliminate the performance issues, leaving VMware to be erroneously blamed as the cause or that the application is ‘unfit’ to be virtualized. Ironically so many of these problems could have been proactively avoided had there been a better understanding and communication between the silos in the design and provision phase.

While it could be argued that Application, Server / VM and Storage teams all have their own expertise and should stick to what they know, in today’s unified Cloud-driven climate remaining in a bat cave of ignorance justified by the knowledge that you’re an expert in your own field is nothing short of disastrous. Application owners, VMware and Storage Admin have to sit and communicate with each other and destroy the first barrier erected by silos i.e. knowledge sharing. This does not require that a Storage Admin set up a DRS cluster or a VM Admin start provisioning LUNs but what it does mean is that as projects roll out a common understanding of the requirements and the challenges be understood. As the technology brings everything into one stack with vStorage APIs, VAAI and terminology such as orchestration that describe single management panes which allow you to provision VMs and their Storage with a few clicks, the need for the ‘experts’ of their field to sit and share their knowledge has never been greater. Unless the challenge of breaking the silos is addressed we could be seeing Kate Bush’s premonition of Cloudbursting sooner than we think.

The True Optimum Queue Depth for VMware / vSphere

2011-06-03T03:44:00.000-07:00

An array’s Queue Depth in its most basic terms is the physical limit of exchanges that can be open on a storage port at any one time. The Queue Depth setting on the HBA will specify how many exchanges can be sent to a LUN at one time. Generally most VM Admins leave their Queue Depth settings at the manufacturer’s default with only the requirement to facilitate a small number of I/O intensive VMs/servers leading them to make an increase. The risk with changing or in fact not changing Queue Depths to their optimum can have severe detrimental effects on performance where any outstanding I/O queuing can cause bottlenecks. For example if Queue Depth settings are set too high the Storage ports will quickly become overrun or congested leading to poor application and VM performance or even worse data corruption or loss. Alternatively if Queue Depth settings are set too low, the Storage ports become underutilized thus leading to poor SAN efficiency. On the other hand should the Queue Depth be correctly optimized, performance of VMs and their corresponding LUNs can be vastly improved, hence the requirement for a methodology to accurately determine this is an imperative.

Generally VM Admins use esxtop to check for I/O Queue Depths and latency with the QUED column showing the queuing levels. With VirtualWisdom though, end users are now empowered with the only platform that can measure real-time aggregated queue depth regardless of storage vendor or device i.e. in a comprehensive manner that takes into consideration the whole process from Initiator to Target to LUN. VirtualWisdom’s unique ability to do this ensures accurately that storage ports are optimized for maximum application health, performance, and SAN efficiency.

The esxtop QUED column

So to begin with it is important to prevent the storage port from being over-run by considering both the number of servers that are connected to it as well as the number of LUNs it has available. By knowing the number of exchanges that are pending at any one time it is possible to manage the storage Queue Depths.

In order to properly manage the storage Queue Depths one must consider both the configuration settings at the host bus adapter (HBA) in a server and the physical limits on the storage arrays. It is important to determine what the Queue Depth limits are for each storage array. All of the HBAs that access a storage port must be configured with this limit in mind. Some HBA vendors allow setting HBA and LUN level Queue Depths, while some allow HBA level setting only.

The default value for the HBA can vary a great deal by manufacturer and version and are often set higher than what is optimal for most environments. If you set the queue depths too low on the HBA it could significantly impair the HBA’s performance and lead to under utilization of the capacity on the storage port (i.e. underutilizing storage resources). This occurs both because the network will be underutilized and the storage system will not be able to take advantage of its caching and serialization algorithms that greatly improve performance. Queue Depth settings on HBAs can also be used to throttle servers so that the most critical servers are allowed greater access to the necessary storage and network bandwidth.

To deal with this the initial step should be to baseline the Virtual environment to determine which servers already have their optimal settings and which ones are either set too high or too low. Using VirtualWisdom real time Queue Depth utilization can be reported for a given period. Such a report will show all of the initiators and the maximum queue depths that were recorded during the recording period. This table can be used as a method to compare the settings on the servers to the relative values of the applications that they support. The systems that are most critical should be set to higher Queue Depths than those that are less critical, however Queue Depth settings should still be within the vendor specified range. Unless Storage ports have been dedicated to a server, VirtualWisdom often shows that optimum Queue Depth settings should be between the ranges of 2-8, despite industry defaults tending to be between 32-256. To explain this further, VirtualWisdom can drill off a report that can show in descending order the Maximum Pending Exchanges and their corresponding initiators and server names. The Maximum Pending Exchanges are not only the maximum number of exchanges pending during the interval being recorded but also the exchanges that were opened in previous intervals that have not yet closed.

Report on Pending Exchanges i.e. HBA Queue Depth

So for example if a report such as this was produced for 100 ESX servers it’s important to consider whether your top initiators are hosting your highest priority applications and whether your initiators with low queue depth settings are hosting your lowest priority applications. Once the appropriate Queue Depth settings have been determined, an alarm can be created for any new HBAs that are added to the environment, especially any HBA that violates the assigned Queue Depth policy.

Once this is established the VirtualWisdom dashboard can be then be used to ensure that the combined Pending Exchanges from all of the HBAs are well balanced across the array and SAN fabric.

Making SAN Cheaper than NAS

2011-03-21T09:08:00.000-07:00

Are you making the most of your FC SAN?

There is a common myth that FC SAN is expensive, difficult to manage and troubleshoot. Coupled with this there are heavily marketed agendas to move customers away from FC SAN to new and allegedly more cost effective solutions.
But what if there was a way to…

• Reduce the amount of physical adapters, FC cables, SAN ports and Storage ports while concurrently improving your application response time, availability and SLAs?

• Simplify server FC I/O provision, enabling a more agile, scalable and dynamic deployment model?

• Gain the insight that on average most FC SAN and Storage ports are greatly underutilized averaging 5-10%, with only a minority of them needing their full bandwidth and therefore enabling you to drive higher levels of resource utilization and performance out of an already deployed solution?

• Know your FC SAN Storage related problems before they occur enabling you to transform your environment from a reactive to proactive one?

• Reduce the time it takes for troubleshooting your FC SAN Storage environment from days to minutes?

• Reduce the OPEX and CAPEX of your existent FC SAN Storage infrastructure such that it’s deemed not only the most reliable and secure but also the most cost effective solution of your environment?

I'm absolutely delighted to have the opportunity to discuss and answer some of these issues with ex-EMC vSpecialist and VM Guru Stephen Spellicy, in an upcoming webinar on the 30th of March. i welcome all to partake in the Q and A session and discussion as both Stephen and I showcase the way for a complete solution to optimize and consolidate your FC connectivity while ensuring the reliability and performance you have come to know and love with Fibre Channel. Find out how to also improve resource utilization, deployment and ongoing management of your FC connectivity while easily accessing and analyzing critical application-to-storage transaction data that allows you to pinpoint latency down to the millisecond.

Registration is free and available here:
http://info.virtualinstruments.com/webinar-fc-san-myths.html

In the meantime here's an insight into the exciting new technology that is IOV:

CRC Errors, Code Violation Errors, Class 3 Discards & Loss of Sync - Why Storage isn't Always to Blame!

2011-01-18T14:02:00.001-08:00

Storage is often
automatically pinpointed as the source of all problems. From System Admins, DBAs, Network guys to Application owners, all are quickly ready to point the figure at SAN Storage given the slightest hint of any performance degradation. Not really surprising though, considering it’s the common denominator amongst all silos. On the receiving end of this barrage of accusation is the SAN Storage team, who are then subjected to hours of troubleshooting only to prove that their Storage wasn’t responsible. On this circle goes until there reaches a point when the Storage team are faced with a problem that they can’t absolve themselves of blame, even though they know the Storage is working completely fine. With array-based management tools still severely lacking in their ability to pinpoint and solve storage network related problems and with server based tools doing exactly that i.e. looking at the server, there really is little if not nothing available to prove that the cause of latency is a slow draining device such as a flapping HBA, damaged cable or failing SFP. Herein lies the biggest paradox in that 99% of the time when unidentifiable SAN performance problems do occur, they are usually linked to trivial issues such as a failing SFP. In a 10,000 port environment, the million dollar question is ‘where do you begin to look for such a miniscule needle in such a gargantuan haystack?’

To solve this dilemma it’s imperative to know what to look for and have the right tools to find them, enabling your SAN storage environment to be a proactive and not a reactive fire-fighting / troubleshooting circus. So what are some of the metrics and signs that should be looked for when the Storage array, application team and servers all report everything as fine yet you still find yourself embroiled in performance problems?

Firstly to understand the context of these metrics / signs and the make up of FC transmissions, let’s use the analogy of a conversation. Firstly the Frames would be considered the words, the Sequences the sentences and an Exchange the conversation that they are all part of. With that premise it is important to first address the most basic of physical layer problems, namely Code Violation Errors. Code Violation Errors are the consequence of bit errors caused by corruption that occur in the sequence – i.e. any character corruption. A typical cause of this would be a failing HBA that would eventually start to suffer from optic degradation prior to its complete failure. I also recently experienced at one site Code Violation Errors when several SAN ports had been left enabled after their servers had been decommissioned. Some might think what’s the problem if they have nothing connected to them? In fact this scenario was creating millions of Code Violation Errors causing a CPU overhead on the SAN switch and subsequent degradation. With mission critical applications connected to the same SAN switch, performance problems became rife and without the identification of the Code Violation Errors could have led to weeks of troubleshooting with no success.

The build up of Code Violation Errors become even more troublesome as they eventually lead to what is referred to as a Loss of Sync. A Loss of Sync is usually indicative of incompatible speeds between points and again this is typical of optic degradation in the SAN infrastructure. For example if an SFP is failing, its optic signal will degrade and hence will not be at for example the 4Gbps it’s set at. Case point: a transmitting device such as a HBA is set at 4Gbps while the receiving end i.e. the SFP (unbeknownst to the end user) has degraded down to 1Gbps. Severe performance problems will occur as the two points constantly struggle with their incompatible speeds. Hence it’s an imperative to be alerted of any Loss of Sync as ultimately they are also an indication of an imminent Loss of Signal i.e. when the HBA or SFP are flapping and are about to fail. This leads to the nightmare scenario of an unplanned path failure in your SAN storage environment and worse still a possible outage if failover cannot occur.

One of the biggest culprits and a sure-fire hit to resolving performance problems is to look for what are termed CRC errors. CRC Errors usually indicate some kind of physical problem within the FC link and are indicative of code violation errors that have led to consequent corruption inside the FC data frame. Usually caused by a flapping SFP or a very old / bent / damaged cable, once CRC errors are acknowledged by the receiver, the receiver would reject the request leaving the Frame having to be resent. For example as an analogy imagine a newspaper delivery boy, who while cycling to his destination loses some of the pages of the paper prior to delivery. Upon delivery the receiver would request for the newspaper to be redelivered with the missing pages. This would entail the delivery boy having to cycle back to find the missing pages and bring back the newspaper as a whole. In the context of a CRC error a Frame that should typically take only a few milliseconds to deliver could take up to 60 seconds in being rejected and resent. Such response times can be catastrophic to a mission critical application and it’s underlying business. By gaining an insight into CRC errors and their root cause one can immediately pinpoint which bent cable or old SFP is responsible and proactively replace them long before they start to cause poor application response times or even worse a loss to your business.

The other FC SAN gremlin is what is termed a Class 3 discard. Of the various services of data transport defined by the Fibre Channel ANSI Standard, the most commonly used is Class 3. Ideal for high throughput, Class-3 is essentially a datagram service based on frame switching and is a connectionless service. Class 3’s main advantage comes from not giving an acknowledgement that a frame has been rejected or busied by a destination device or Fabric. The benefits of this are that it firstly significantly reduces the overhead on the transmitting device and secondly allows for more bandwidth availability for transmission which would otherwise be reduced. Furthermore the lack of acknowledgements removes the potential delays between devices caused by round-trips of information transfers. As for data integrity, Class 3 Flow control has this handled by higher-level protocols such as TCP due to Fibre Channel not checking the corrupted or missing frames. Hence any discovery of a corrupted packet by the higher-level protocol on the receiving device instantly initiates a retransmission of the sequence. All of this sounds great until the non-acknowledgement of rejected frames starts to also bring about Class 3’s disadvantage. This is that inevitably a Fabric will become busy with traffic and will consequently discard frames, hence the name Class 3 discards. Due to this the receiving device’s higher-level protocol’s subsequent request for retransmission of sequences will then degrade the device and fabric throughput.

Another indication of Class 3 discards are zoning conflicts where a frame has been transmitted and cannot reach a destination, hence concluding in the SAN initiating a Class 3 discard. This is caused by either legacy or zoning mistakes where for example a decommissioned Storage system was not unzoned from a server or vice versa leading to continuous frames being discarded and degraded throughput as sequences are retransmitted. This then results in performance problems, potential application degradation and automatic finger pointing at the Storage System for a problem that can’t automatically be identified. By resolving the zoning conflict and spreading the load of the SAN throughput across the right ports, the heavy traffic or zoning issues which cause the Class 3 discards can be quickly removed bringing immediate performance and throughput improvements. By gaining an insight into the occurrence and amount of Class 3 discards, huge performance problems can be quickly remediated before they occur and thus another reason as to why the Storage shouldn’t automatically be blamed.

These are just some of the metrics / signs to look for which can ultimately save you from weeks of troubleshooting and guessing. By first acknowledging these metrics, identifying when they occur and proactively eliminating them, the SAN storage environment will quickly evolve and transform into a healthy, proactive and optimized one. Furthermore by eliminating each of these issues you also empower yourself by eliminating their consequent problems such as application slowdown, poor response times, unplanned outages and long drawn out troubleshooting exercises which eventually lead to fingerpointing fights. Ideally what will occur is a paradigm shift where instead of application owners complaining to the Storage team, the Storage team will proactively identify problems prior to their existence. Here lies the key to making the ‘always blaming the Storage’ syndrome a thing of the past.

Undressing Victoria – Could Hitachi's New VSP Rock the EMC boat?

2010-11-21T10:30:00.000-08:00

Back in 2004 HDS launched the USP, which was then followed by the great but not so radically different USP-V in 2007. Within that same time frame, HDS’ main rival in the Enterprise Storage market EMC, busily went about launching the Symmetrix DMX-3, then the DMX-4 and most recently the VMAX. Launching so-called revolutionary features such as FAST, (which HDS had been doing previously for years i.e. Tiered Storage Manager) EMC’s marketing machine quickly created an atmosphere wherein the Storage World became obsessed with all things ‘V’ namely VSphere, VMAX and VPLEX. With marketing so powerful that it extended to international airport posters advertising EMC’s ability to ‘take you to the Private Cloud’, you could easily forgive Hitachi for possibly becoming complacent and content with being a company renowned by the masses for just making great vacuum cleaners. Well thank goodness, after three years in the making, a codename of Victoria, a semi-decent marketing campaign and a ‘V’ to its final name, HDS have at last launched the new VSP Enterprise array…and yes it’s been worth the wait.

Marketed as a 3D scaling storage system, it was pleasing to realize that it wasn’t a reference to the tinted glasses needed to look at its rather revolting vomit green cabinet. (So yes it certainly can’t compare to the Knight Rider looks of the VMAX and probably won’t be appearing in an episode of ‘24’). Aesthetics aside and more importantly though the 3D refers to the terms scale up, scale out and scale deep. What HDS mean by this is that you can scale up by adding more resources to the VSP system, you can scale out by adding more disk blocks, host connections and Virtual Storage Directors, and you can scale deep by virtualising external heterogenous arrays behind the VSP. From this premise it’s also evident that HDS are looking at the VSP to be the foundational block of the recently announced but yet to be released cloud platform, the UCP.

While the scale deep is an old tradition that HDS have mastered for years, it’s easy to note that the scale out and Virtual Storage Directors terms bear more than a passing resemblance to the concept introduced by EMC’s VMAX. With four Virtual Storage Directors in each system and with four cores within each Virtual Storage Director the VSP houses a total of 16 cores. Essentially the masterminds of the machine, Virtual Storage Directors are responsible for managing the VSPs internal operations such as mapping and partitioning. The VSP can then be expanded into a mammoth system of 32 Cores by combining two VSP systems using the PCIe Hitachi Data Switch, scaling up to 2048 drives with a Terabyte of cache. So while an EMC aficionado may immediately point out that the VMAX can offer 128 cores, which dwarfs the VSP’s 32 Cores, it’s worth remembering that with Storage Virtualization the number of cores that can potentially be housed behind the VSP are in the hundreds.

Another point, there is no equivalent to the USPVM - the mini-me USPV which couldn’t scale up to the size of its big brother. Instead the VSP starts as a single pair of Virtual Storage Directors with no internal storage that can act as a pure virtualization platform to homogenize externally attached multi-vendor arrays. With such a proposition, one can just imagine the quivering of DS8000s, VMAXs, Clariions and EVAs ‘confettied’ within datacenters now faced with the prospect of being marginalized as a portion of the potential 255PB of LUNs that can sit behind the VSP’s Directors.

Of course this is also a great sales pitch to eventually get the same VSP stacked up later with internal storage that can range from 256 SSDs in either STEC’s 200GB 2.5-inch or 400GB 3.5-inch format as well as up to 2.5PB of 3.5-inch SATA drives. Add to that HDS have taken the pioneering route of adopting the capability to house up to 1.2 PB of SAS 2.5-inch drives. Yes, that’s right the HDS VSP has a SAS backend and it’s ready to have a 6Gbps SAS interface. While I’m no fan of SSDs sitting on the backend of a Storage system behind a RAID controller, processors, SAN switches etc. (can’t wait for DRM to hit the mainstream market), nevertheless a full duplex SAS backend is a definite improvement in taking advantage of the IOPs and throughput capability of SSDs. With up to 128 paths out to the disks and solid state drives, HDS are calling this switching fabric the Grid Switch Layer. Of course when you add in the idea of 2.5 inch drives using less power, increase in IOPS due to a higher spindle count and a reduction of one less cabinet on your datacenter floor, you suddenly see a nice ROI figure being mustered up by your local HDS account manager. Expect EMC and co. to follow suit.

Also gone are those somewhat prehistoric battery backups that resided in the USPV and were legacy from the USP. Instead you will find between the aforementioned Grid Switch layer and the back end enclosures that the VSP hosts an extra layer of cache. This feature eliminates the need for the old battery backups. Instead the Virtual Storage Director’s data is stored in this cache and de-staged to solid state memory in the event of a power loss etc. hence ensuring data protection. It’s a simple idea but a welcome one for field engineers who can vouch for the pain of having to replace one of those battery packs. Indeed other legacy complications have been reduced due to the fact that the Control Memory (still responsible for all the metadata of the VSP’s operations) is now located on Virtual Storage Director boards and DIMMs, removing the requirement of separate dedicated Shared Memory and Control Memory boards.

Furthermore despite having borrowed the VMAX concept of coupling engines as well as using Intel processors for their Virtual Storage Directors, HDS have still retained a unique stamp by forsaking the Rapid IO interconnects chosen by EMC for their much more familiar Star Fabric architecture. So unlike EMC’s complete overhaul of their Direct Matrix architecture, HDS have maintained their non-blocking crossbar architecture switch to the back-end while having their global cache shared amongst multiple controllers. This familiar HDS method is the internal network of the VSP that manages its data via the Drives, Virtual Storage Directors, BEDs, FEDs and Cache.

So while HDS have inadvertently acknowledged EMC’s insight to go the Intel route they’ve also seemingly taken a leaf out of VMware’s DRS book by having custom I/O routing ASICs. Point being that on both the FEDs and BEDs of the VSP, data accelerator ASICs designed by Hitachi themselves, have now been built for managing the I/O traffic. Unlike the USPV where the ACP and CHP processors were tied to particular ports, the VSP instead makes a resource pool of CPU from which the ASICs can then assign to any front end or back end port that requires them at any given time. Personally I think this is a fantastic idea and step forward as it quickly eliminates a lot of the performance tuning that was previously required to get the same effect. With such a VMware-esque feature it’s somewhat ironic then that the VSP doesn’t yet support VAAI, although news is that it’s coming very soon.

Another ground-breaking step and one I’m most excited about is the VSP’s new Sub Lun Tiering feature. Using the now (thanks partly to Marc Farley’s terrific YouTube rant) infamous HDS 42MB page size, new policy based tiering will instead work on the page level instead of the LUN. Hence as a particular page becomes more or less active or “hot”, the VSP will automatically upgrade or downgrade the tier for that page only, regardless of whether it’s on external or internal storage. The objective here is pretty clear – an attempt to optimize your usage of SSDs so you can justify buying more of them. Also ironically what was once considered HDS’ Achilles heel with regards to storage efficiency, the 42MB page size now works out to be ideal. Imagine the nightmares of a smaller page size - valuable Storage Processors’ CPU utilized in the desperate search for numerous 50Kb page sizes that heat up and need to be moved up to tier 0; not a pretty thought. As this feature is sure to be emulated by other vendors it will be interesting to see what page sizes they’ll be coming up with.

Also speaking of other vendors, HP who recently achieved the takeover of the year with their purchase of 3PAR has also launched the VSP albeit with a much nicer cabinet and the OEM moniker of P9500. What is interesting here is that the P9500 (VSP) is clearly a higher range platform than the InServ arrays and if indications are correct HP have no intention of disbanding their EVA range (reports have already surfaced of an EVA now called P6000). So with the OEM deal still intact, HP currently has every intention of also marketing and pushing forward the VSP / P9500. Indeed while at a meeting at one of HP’s headquarters during the week of the P9500’s release I was delightfully told of the P9500 amazing APEX functionality. APEX sounded incredible as I was told of an application-level QOS control, which would give Pillar’s similar feature a run for their money. Strange then that I hadn’t heard of any such feature during the HDS launch. Upon further reading of APEX, it was explained that mission-critical data could be given bandwidth priority over less important data. It was then I suddenly realized something familiar. This was nothing but a remarketed version of HDS’ Server Priority Manager’s functionality which had been around for years (you’ve probably never heard of it because of HDS’ poor marketing but it’s actually very good). In fact the only uniqueness of APEX is that for HPUX platforms it does indeed allow the prioritization of CPU, cache and storage resources. So not really that significant a differentiator from the VSP especially if you don’t run HPUX (and to be honest I think they’d have more success pitching how much nicer their cabinet looks). Nonetheless, differentiators or not, the addition of the P9500 to HP’s storage portfolio will only add further credence to their growing status of a Storage powerhouse.

Another welcome addition / change is the replacement of the demonically slow Storage Navigator management GUI in place for a much faster and greener looking GUI. HDS have also announced a whole new refurbishment of their Command Suite software. As well as being quicker and more user friendly there’s also better integration with VMware allowing you to manage storage for Virtual machines. A welcome change for a SRM that often looked and performed in an outdated manner that was not befitting of the array (I still have nightmares of carving up LDEVs on the USP pre Quick Format days).

So with new features still to be released such as integration with VMware’s VAAI, support for FCOE and primary deduplication, the VSP has come a long way from its predecessor the USPV. Taking the best from their competitors and integrating it with their own way of doing things is not a new concept for HDS and with the VSP they certainly have done that. But HDS now have a genuinely new product which surpasses the minor gap filled in between the USP and USPV that successfully incorporates its characteristic tools such as dynamic provisioning and virtualization with bleeding edge technology such as Sub Lun Tiering. There will be inevitable criticisms from competitors. There will be inevitable squabbles between the vendors. There will be inevitable comparisons between arrays. One thing’s for sure though, expect a lot of the VSP’s new features to be incorporated in other upcoming arrays pretty soon, Hitachi or not. In the words of Simon Cowell, “Glad to see them back in the game!”

N.B. I received a great explanation and post about APEX from Calvin Zito - also known as HPStorageGuy. He clarifies the fact that there is more of a distinction than what was originally posted by me - or in his words "Bottom line, there is no HDS equivalent of APEX" (-:

Here's the link: http://h30507.www3.hp.com/t5/Around-the-Storage-Block-Blog/Application-Performance-Extender-setting-the-record-straight/ba-p/83533#feedback-success

'Well Ours Goes to 8' - Why Going 8Gbps From 4Gbps Doesn't Necessitate Double the Bandwidth

2010-10-09T18:31:00.000-07:00

A wise man once told me that if there were a major car crash further up the highway, having a faster car would only get me to the accident quicker. Obvious right? Not so it seems when the wisdom of these words is applied to the analogy of the growing number of SAN infrastructures currently upgrading from 4Gbps to 8Gps. ‘Faster means quicker, means better’ is the commonly heard sales pitch used to seduce vulnerable IT Directors who dream of ‘a guaranteed performance improvement that would solve the headache of their ever slowing applications’. Sadly though for many of those that bit the 8Gbps apple, the significant improvement never came and like a culprit with no shame the same voices returned claiming that this was the fault of the outdated servers, HBAs and storage systems which also now needed to be upgraded. So down the 8Gbps road they went which now extended from the fabric all the way to the server platform, but still no significant improvement and if so certainly not one that could justify such a heavy investment. Like any infrastructure, being unaware of the SAN inevitably means that any unseen problems caused by error statistics such as CRC errors, physical link errors, protocol errors, code violations, class 3 discards etc. (i.e. the car crash) would remain, regardless of whether you get there at 4Gbps or 8Gbps. So how could such a simple concept be lost amongst the numerous 4Gbps to 8Gbps upgrades that are now taking place across the SAN stratosphere?

The main reason is that there are clearly several seemingly instant advantages with the 8Gbps standard. Having one byte consisting of 8 bits, giving you a potential 800 MB per second gives you the immediate impression that you are able to potentially double the transmission of your data within the same single cable. Logic would then dictate that with both SAN switches and storage systems having 8Gbps ports, you also now have the freedom to double the number of hosts to a single storage port without the fear of any performance impact. Logic would also conclude that extra bandwidth would be a blessing in a virtual environment where dozens of VMs scramble for a limited number of ports while blade servers subsequently struggle to house the physical space for their growing HBA demands. Couple this with the ever-nearing cost equivalence to their 4Gbps component counterparts and such advantages become unavoidable choices for end users.

Indeed it’s the drive for ‘more throughput’ in this virtualisation era that has really kicked the 8Gbps juggernaut into top gear. Pre-virtualisation world, (which surprisingly wasn’t even that long ago yet already seems like an aeon) the relationship between server, application, SAN and storage were straightforward and one-dimensional. A single host with one application would connect to a dual redundant SAN fabric that in turn would be mapped to a single LUN. Today everything has multiplied, with a single physical server hosting numerous virtual servers and applications being connected to several storage interfaces and numerous LUNs.

Solutions such as N_Port ID Virtualization NPIV and N_Port Virtualization (NPV) have gone even further by enabling the virtualization of host and switch ports. Now via NPIV your single HBA can be termed an N_Port and consequently register multiple WWPNs and N_Port ID numbers. So now what was once just a single physical server can now house numerous virtual machines each with their own Port IDs, which in turn allows them to be independently zoned or mapped to LUNs. On the switch side, NPV presents the switch port as an NPIV host to the other switches. Hence expanding a SAN can be rapidly deployed without the burden of worrying about multiple domain IDs.

So while the case to upgrade to 8Gbps is on the offset quite compelling, further analysis would show that this isn’t necessarily the case. Reality and not logic shows that a lot of the aforementioned advantages have been related to ‘guess work’ and assumptions. Moreover and ironically the rush to 8Gbps is actually causing more problems than were previously existent within data centers unbeknownst to the majority of end users due to their inability to soundly monitor what’s happening in the SAN. To begin with if we revisit the concept of FC bit rates and their constant increase from 2Gbps to 4Gbps and now 8Gbps, one should be aware that the consequence is a proportionally decreasing bit period. Hence this now shrunken window of data requires an even more robust physical infrastructure than before and becomes even more susceptible to potential errors - think Michael Schumacher driving his Ferrari top speed on the same public road Morgan Freeman took Miss Daisy.

While you may not have had performance issues with 4Gbps, by upgrading to 8Gbps and its greater sensitivity to light budget you instantly expose yourself to more bit stream errors, bit-error rates and multiple retries i.e. delays, disruption and performance degradation of your mission critical applications. Of course this isn’t always the case but when FC cables are bent at 70 degrees or more, quality of optical transceivers / in-line connectors are not upgraded or small specks of dirt reside on the face of optical cable junctions, your environment suddenly becomes doubly susceptible to jitters and major errors on your SAN fabric. Factors which were previously transparent at 4Gbps become significant performance degraders in the highly sensitive mould of 8GBps.

So as organizations upgrade to 8Gbps without having taken these factors into consideration, we see countless troubleshooting and even HBA replacements as there is no real insight into these transmission errors from current SRM tools. Orange OM2 fiber-optic cables may get replaced for aqua OM3 fiber-optic cables and SFP transceivers swapped for SFP+ transceivers leaving administrators thinking they’ve solved the problem. Worst of all though such fire-fighting tactics often lead to a temporary elimination of performance problems, only to then without any explicable reason rear their ugly head like a persistent zombie from a horror flick that refuses to die.

Given the recent revelations in the industry that SAN fabrics are being over-provisioned on average by at least a factor of 5 times, there clearly is little reason for most companies to upgrade to 8Gbps. When all of your applications are receiving the bandwidth that only 5% of your applications actually need, going straight to 8Gbps leads to even poorer configuration and further waste. This scenario becomes even more complicated given the fact that server virtualization has led administrators to over- provision their SAN infrastructure in a fear that they can’t accommodate their bandwidth requirements. Also with an increase of SSDs being deployed in the majority of enterprise infrastructures, going up to 8Gbps seems a natural way of making the most of their expensive disk investment. Problem is that having SSDs running on upgraded yet over-provisioned links which already suffer from jitters may give some performance improvement over their mechanical disk counterparts but are hardly running at optimum levels.

To solve such a dilemma and gain the true benefits of an 8Gbps upgrade it’s important to have an instrument which captures both directions of every SCSI I/O transaction from start to finish on every link carrying your business-critical data. In a recent discussion with IBM’s DS8000 specialist Jens Wissenbach, it was agreed that the solution of deploying TAPs on all the key links within the data center is the only way to truly detect the number of light levels, signal quality, throughput metrics, latency and response times, as well as protocol violations. With such real-time visibility into your FC infrastructure the administrator can quickly determine if any of the applications are in actual need of an excess to 4Gbps or where in fact the performance problems are coming from whether that be a bent cable, a speck of dirt or an outdated SFP.

TAPs such as those provided by the company Virtual Instruments, will soon be the natural replacement for patch panels across all enterprise data centers. But their role could also be the tool that allows end users to provision their SAN links to properly accommodate their SSD and VMware requirements without over-provisioning and being blinded by performance degradation that is beyond the scope of their SRM tool. So as Fibre Channel vendors are planning to start rolling out 16Gbps products for next year and with the news that the standard for 32Gbps Fibre Channel is already being worked on, it’s imperative that such upgrades take place with the correct preparation so as to maximize the benefits of such an investment.

Cloud Wars: The 3PAR Strikes Back

2010-09-06T19:26:00.000-07:00

A long time ago (November 2009 to be precise), in a Cloud far far away, the Rebel Alliance of EMC, Cisco and VMware joined forces to form what are now dubbed Acadia and the VCE coalition. Soon after came the launches of VBlocks 0, 1 and 2 each respectively incorporating the EMC Celerra, Clariion or VMax with a stack of Cisco blades and switches and a layer of VMware virtualization to suit. Marketed as ‘best of breed’ and ‘ready configured to client specifications’ it was an immediate launch pad for any customer looking to deploy a private cloud with a single pane of management. Whether it was an already mature virtualization infrastructure looking to quickly expand or an organization wanting to enter the virtualization stratosphere with minimum fuss, in house training etc. the VBlock was quickly carving itself a significant market share. Hence no surprise that EMC’s VP of Global Marketing and CTO Chuck Hollis can hardly hide his glee in interviews at the current lack of direct competition to the VCE’s destiny to rule the Cloud’s Galactic Empire. But one should never underestimate the power of the HP side, now counterstriking in their salaciously exciting sweeping aside of Dell and consequent takeover of the brilliant storage platforms of 3PAR. The Cloud Wars begun they have.

At $33 a share, 3Par now belong to HP. So despite being the household name that they are and arguably the world’s biggest technology company in sales terms, HP will only now be entering the storage market with a product worthy of their clout. HP EVA administrators may beg to differ but any end user who’s had to go through the nightmare of doing a firmware upgrade on a HP EVA will certainly join me in the chorus of disapproval at the very mention of ‘Enterprise’ within the EVA name. A good modular storage system – yes, enterprise – certainly not and hence why HP have had an OEM deal with HDS to rebrand their USPV and USPVM range as XP24000 and XP12000 respectively albeit with some microcode changes. It was a nice arrangement but surely all of this will now change with the acquisition of 3PAR.

3PAR is one of those classic examples of a company with pioneering developments and products but an inability to turn in a profit. 3PAR have arguably led the way with their developments with their Utility Storage range; architectures that provide a multi-tenant platform on which service providers can deliver both virtualized and scalable enterprise IT as a utility service, in other words Software as a Service (SaaS) and Infrastructure as a Service (IaaS) models. Then there’s their InServ storage server range, which constitutes models such as the T400 and T800 which directly compete with other Enterprise arrays such as the EMC VMax and the HDS USPV, or should I say the HP XP24000. With an operating system software suite named InForm they have a management pane that scales across both their high end T-class and modular F-class range i.e. the F200 and F400. A much better transition model than the MSA, EVA to XP range currently being offered by HP.

Another pioneering achievement for 3PAR was their revolutionary thin provisioning, a mechanism now adopted by practically every storage vendor on the market. Therein though emerged the ironic twist of fate as the very technology which gave them a major edge over their competitors was the same one that companies such as EMC and HDS adopted, marketed and created the allusion of being their very own. So now with the increased sales channels which come from being under the HP umbrella, the 3PAR range can finally penetrate a customer base worthy of its quality product and showcase its technology firsthand.

Another issue is that the new technology that will come with 3PAR will also enable HP to offer something far greater and beneficial for their existent client base than their current storage offerings. I personally now fear greatly for the future of HDS having already suffered the blow of the end of the OEM agreement with SUN and now what inevitably will be the end of their relationship with HP. It makes no business sense for HP to continue with HDS and their XP range for several reasons. While as great as the USPV is, nothing has happened or changed for several years while HP’s competitors such as IBM, NetApp and EMC continue to develop and enhance their storage system range. Due to this HP’s high end range has also stagnated and in a market where customers are being told of the wonders of a VMax and VPlex, it’s a hard sale to keep plugging a product such as the XP/USPV which hasn’t changed for more than four years. Step in the 3PAR R&D team and you have gurus who are adapting and changing with the times bringing some of the latest and greatest developments in storage – are HP really going to wait around for the USPV2 while EMC corner the market left, right and centre?

It’s the new developments that will also give HP an added edge in the market and bring the 3PAR technology at the forefront while also allowing HP to bring a direct competitive model to the VBlock. Just recently 3PAR announced their latest integration of their InServwith VMware’s vSphere4 to quickly build cloud infrastructures for their shared, virtualized utility service offerings. 3PAR have also made no secret of their Utility Storage range being specifically designed for virtual datacenters and the delivery of IT as a service. With the addition of their Adaptive Optimization, their Autonomic Storage Tiering application and tight integration with VMware’s vStorage, 3PAR are already a storage company looking at embracing the Cloud phenomenon. 3Par's virtual storage solutions will not only position HP as a one-stop storage solution, but also carries the massive potential of an integrated HP blade / VMware virtualization / 3PAR Storage Cloud platform that directly targets the VCE coalition’s VBlock.

So while it may be considered by some that HP’s takeover of 3PAR is a satisfactory result for EMC, with no 3PAR product threatening their Dell deals, I would strongly beg to differ. The HP 3PAR takeover is a statement of intent. No need for HDS and the XP range. No more playing second fiddle in the storage market to EMC. No more allowing VPLEX to run free reign when they now have a virtualized and autonomically managed cloud storage system. And no more allowing the VBlock to corner the Cloud market. Arise HP, the Force is strong with you.

Monitoring the SAN shine with Virtual Instruments

2010-08-31T17:52:00.000-07:00

It was about three months ago that one of my friends had informed me he was leaving HDS to join a company named Virtual Instruments. ‘Virtual Instruments?’ I asked myself, trying to fathom if I’d have heard of them before only to realize that I had once seen a write up on their SAN monitoring solution, which was then termed NetWisdom. I was then inadvertently asked to mention Virtual Instruments in one of my blogs - nice try pal but I had made it clear several times before to vendors requesting the same that I didn’t want my blog to become an advertising platform. Despite this though I was still intrigued by what could have persuaded someone to leave a genuinely stable position at HDS to a company I hadn’t really had much exposure to myself. Fast forward a few months, several whitepapers and numerous discussions and I find myself writing a blog about the very said company.

Simple fact is it’s rare to find a solution or product in the storage and virtualization market that can truly be regarded as unique. More often than not most new developments fall victim to what I term the ‘six month catch up’ syndrome in which a vendor brings out a new feature only for its main competitor to initially bash it and then subsequently release a rebranded and supposedly better version six months later. The original proponents of thin provisioning, automated tiered storage, deduplication, SSD flash drives etc. can all pay testament to this. It is hence why I have taken great interest in a company that currently occupies a niche in the SAN monitoring market and as yet doesn’t seem to have worthy competitor, namely Virtual Instruments.

My own experience of Storage monitoring has always been a pain in the sense that nine times out of ten it was a defensive exercise in proving to the applications, database or server guys that the problem didn’t lie with the storage. Storage most of the time is fairly straightforward, wherein if there are any performance problems with the storage system they’ve usually stemmed from any immediate change that may have occurred. For example provision a write intensive LUN to an already busy RAID group and you only have to count the seconds before your IT director rings your phone on the verge of a heart attack at how significantly his reporting times have increased. But then there was always the other situation when a problem would occur with no apparent changes having been made. Such situations required the old hat method of troubleshooting supposed storage problems by pinpointing whether the problem was between the Storage and the SAN fabric or between the Server and the SAN but therein dwelled the Bermuda Triangle at the centre of it all i.e. the SAN. Try to get a deeper look into the central meeting point of your Storage Infrastructure and to see what real time changes have occurred on your SAN fabric and you’d subsequently enter a labyrinth of guesses and predictions.

Such a situation occurred to me when I was asked to analyze and fix an ever-slowing backup of an Oracle database. Having bought more LTO4 tapes, incorporating a destaging device, spending exorbitant amounts of money on man days for the vendor’s technical consultants, playing around with the switches buffer credits and even considering buying more FC disks, the client still hadn’t resolved the situation. Now enter yours truly into the labyrinth of guesses and predictions. Thankfully I was able to solve the issue by staying up all night and running a Solaris IOSTAT, while simultaneously having the storage system up on another screen. Eventually I was able to pinpoint (albeit with trial and error tactics) the problem to rather large block sizes and particular LUNs that were using the same BEDs and causing havoc on their respected RAID groups. With several more sleepless nights to verify the conclusion, the problem was finally resolved. Looking back surely there was a better, cost effective and more productive way to have solved this issue. There was but I just wasn’t aware of it.

Furthermore ask any Storage guy that’s familiar with SAN management/monitoring software such as HDS’ Tuning Manager, EMC’s ControlCenter, HP’s Storage Essentials and their like and they’ll know full well that despite all the SNIA SMI-S compliancy they still fail to provide metrics beyond the customary RAID group utilization, historic IOPS/sec, cache hit rate, disk response times etc. in other words from the perspective of the end-user there really is little to monitor and hence troubleshoot. Frustratingly such solutions still fail to provide performance metrics from an application to storage system view and thus also fail to allow the end user to verify if they are indeed meeting the SLAs for that application. Put this scenario in the ever growing virtual server environment and you are further blinded by not knowing the relation between the I/Os and the virtual machines from which they originated.

Moreover Storage vendors don’t seem to be in a rush to solve this problem either and the pessimist in me says this is understandable when such a solution would inevitably lead to a non-procurement of unnecessary hardware. With a precise analysis and pinpointing of performance problems/degradation and you have the consequent annulment of the haphazard ‘let’s throw some more storage at it’, ‘let’s buy SSDs’ or ‘let’s upgrade our Storage System’ solutions that are currently music to the ears of storage vendor sales guys. So amidst these partial viewing vendor provided monitoring tools, which lack that essential I/O transaction-level visibility, Virtual Instruments (VI) pushes forth it’s solution, which boldly claims to encompass the most comprehensive monitoring and management of end-to-end SAN traffic. From the intricacies of a virtual machine’s application to the Fibre Channel cable that’s plugged into your USPV, VMax etc. VI say they have an insight. So looking back had I had VI’s ability to instantly access trending data on metrics such as MB/sec, CRC errors, log ins and outs etc. I could have instantly pinpointed and resolved many of the labyrinth quests I had ventured through so many times in the past.

Looking even closer at VI, there are situations beyond the SAN troubleshooting syndrome in which it can benefit an organization. Like most datacenters if you have one of the Empire State Building-esque monolithic storage systems it is more than likely being under utilized with the majority of its residing applications not requiring the cost and performance of such a system. So while most organizations are aware of this and look to saving costs by tiering their infrastructure onto cheaper storage via the alignment of their data values to the underlying storage platform, it’s seldom a seen reality due to the headaches and lack of insight related to such operations. Tiering off an application onto a cheaper storage platform requires the justification from the Storage Manager that there will be no performance impact to the end users but due to the lack of precise monitoring information, many are not prepared to take that risk. In an indirect acknowledgement to this problem, several storage vendors have looked at introducing automated tiering software for their arrays which in essence merely looks at the LUN utilization before migrating them to either higher-performance drives or cheaper SATA drives. In reality this is still a rather crude way of tiering an infrastructure when you consider it ignores SAN fabric congestion or improper HBA queue depths. In such a situation a monitoring tool that tracks I/Os across the SAN infrastructure without being pigeonholed to a specific device is axiomatic in the enablement of performance optimization and the consequent delivery of Tier I SLAs with cheaper storage – cue VI and their VirtualWisdom 2.0 solution.

In the same way that server virtualisation exposed the under utilization of physical server CPU and Memory, the VirtualWisdom solution is doing the same for the SAN. While vendors are more than pleased to further sell more upgraded modules packed with ports for their enterprise directors, it is becoming increasingly apparent that most SAN fabrics are significantly over-provisioned with utilization rates often being less than 10%. While many SAN fabric architects seem to overlook fan in ratios and oversubscription rates in a rush to finish deployments within specified project deadlines, underutilized SAN ports are now an ever-increasing reality that in turn bring with them the additional costs of switch and storage ports, SFPs and cables.

Within the context of server virtualisation itself, which has undoubtedly brought many advantages with it, one irritating side affect has been the rapid expansion of FC traffic to accommodate the increased number of servers going through a single SAN switch port and the complexity now required to monitor it. Then there’s the virtual maze which starts with applications within the Virtual Machines that are in turn running on multi-socket and multi-core servers, which are then connected to a VSAN infrastructure only to finally end up on storage systems which also incorporate virtualization layers whether that be with externally attached storage systems or thinly-provisioned disks. Finding an end-to-end monitoring solution in such a cascade of complexities seems an almost impossibility. Not so it seems for the team at Virtual Instruments.
Advancing upon the original NetWisdom premise, VI’s updated Virtual Wisdom 2.0 has a virtual software probe named ProbeV. The ProbeV collects the necessary information from the SAN switches via SNMP and on a port to port basis metrics on information such as the number of frames and bytes are collated alongside potential faults such as CRC errors synchronization loss, packet discards or link resets /failures. Then via the installation of splitters (which VI name TAPs - Traffic Access Points) between the storage array ports and the rest of the SAN, a percentage of the light from the fibre cable is then copied to a data recorder for playback and analysis. VI’s Fibre-Channel probes (ProbeFCXs) then analyze every frame header, measuring every SCSI I/O transaction from beginning to end. This enables a view of traffic performance whether related to the LUN, HBA, read/write level, or application level, allowing the user to instantly detect application performance slowdowns or transmission errors. The concept seems straightforward enough but it’s a concept no one else has yet been able to put in practice, despite growing competition from products such as Akorri's BalancePoint, Aptare's StorageConsole or Emulex's OneCommand Vision.

Added to this VI’s capabilities can also provide a clear advantage in preparing for a potential virtualization deployment or dare I fall for the marketing terminology – a move to the private cloud. Lack of insight of performance metrics has evidently led to the stagnation of the majority of organizations virtualising their tier 1 applications. Server virtualization has reaped many benefits for many organizations, but ask those same organizations how many of them have migrated their IO intensive tier 1 applications from their SPAARC based physical platforms to an Intel based virtual one and you’re probably looking at a paltry figure. The simple reason is risk and fear of performance degradation, despite logic showing that a virtual platform with resources set up as a pool could potentially bring numerous advantages. Put this now in the context of a world where cloud computing is the new buzz as more and more organizations look to outsource many of their services and applications and you then have even fewer numbers willing to launch their mission critical applications from the supposed safety and assured performance of the in-house datacenter to the unknown territory of the clouds. It is here where VirtualWisdom 2.0 has the potential to be absolutely huge in the market and at the forefront of the inevitable shift of tier 1 applications to the cloud. While I admittedly I find it hard to currently envision a future where a bank launches it’s OLTP into the cloud based on security issues alone, I’d be blinkered to not realize that there is a future where some mission-critical applications will indeed take that route. With VirtualWisdom’s ability to pinpoint virtualized application performance bottlenecks in the SAN, it’s a given that the consequences will lead to an instantly significant higher virtual infrastructure utilization and subsequent ROI.

The VI strategy is simple in that by recognizing I/O as the largest cause of application latency, VirtualWisdom’s inclusion of baseline comparisons of I/O performance, bandwidth utilization and average I/O completions comfortably provide the necessary insight fundamental to any major virtualization or cloud considerations an organization may be planning for. With its ProbeVM, a virtual software probe that collects status from VMware servers via vCenter, the data flow from virtual machine through to the storage system can be comprehensively analyzed with historical and real-time performance dashboards leading to an enhanced as well as accurate understanding of resource utilization and performance requirements. With a predictive analysis feature based on real production data the tool also provides the user the ability to accurately understand the effects of any potential SAN configuration or deployment changes. With every transaction from Virtual Machine to LUN being monitored, latency sources can quickly be identified whether it’s from the SAN or the application itself, enabling a virtual environment to be easily diagnosed and remedied should any performance issues occur. With such metrics at their disposal and the resultant confidence given to the administrator, the worry of meeting SLAs could quickly become a thing of the past while also rapidly hastening the shift towards tier 1 applications being on virtualized platforms. So despite growing attention being given to other VM monitoring tools such as Xangati or Hyperic, they’re solutions still lack the comprehensive nature of VI.

The advantages to blue-chip, big corporate customers are obvious and as their SAN and virtual environments continue to grow, an investment into a VirtualWisdom solution should soon become compulsory for any end of year budget approval. In saying that though, the future of VI also quite clearly lies beyond the big corporates with benefits which include the enablement of an organization to have real- time proactive monitoring and alerting, consolidation, preemptive analysis of any changes within the SAN or Virtual environment and comprehensive trend analysis of application, host HBA, switches, virtualization appliances, storage ports and LUN performance. Any company therefore looking to either consolidate their costly over-provisioned SAN, accelerate troubleshooting, improve their VMware server utilization & capacity planning, implement a tiering infrastructure or migrate to a cloud would find the CAPEX improvements that come with VirtualWisdom a figure too hard to ignore. So while Storage vendors don’t seem to be in any rush to fill this gap, they too have an opportunity to undercut their competitors by working alongside VI by promoting its benefits as a complement to their latest hardware, something which EMC, HDS, IBM and most recently Dell have cottoned on to having signed an agreement to sell the VI range as part their portfolio. Despite certain pretenders claiming to take its throne, FC is certainly here to stay for the foreseeable future. If the market/customer base is allowed to fully understand and recognize its need, then there’s no preventing a future when just about every SAN fabric comes part and parcel with a VI solution ensuring its optimal use. Whether VI eventually get bought out by one of the large whales or continue to swim the shores independently, there is no denying that companies will need to seriously consider the VI option if they’re to avoid drowning in the apprehensive nature of virtual infrastructure growth or the ever increasing costs of under-utilized SAN fabrics.

VDI – A Vulnerably Dangerous Investment or A Virtual Dream Inclusion?

2010-08-21T09:52:00.000-07:00

PCs are part of everyday life in just about every organization. First there’s the purchase of the hardware and the necessary software followed by an inventory recorded and maintained by the IT department. Then normal procedure would dictate that the same IT department would then install all required applications before delivering them physically to the end user. Then over a period of time the laptop/PC would be maintained by the IT department with software updates, patches, troubleshooting etc. to ensure full utilization of employees. Once the PC/laptop becomes outdated, the IT department is then tasked with the monotonous task of removing the hardware, deleting sensitive data and removing any installed applications to free up licenses. All of this is done to enable the whole cycle to be repeated all over again. So in this vicious circle, there are obvious opportunities to better manage resources and save unnecessary OPEX & CAPEX costs, one such solution being virtual desktops.

Having witnessed the financial rewards of server virtualization, enterprises are now taking note of the benefits and usage of virtualization to support their desktop workloads. Consolidation, centralization are now no longer buzz words which were once used for marketing spin but are instead tangible realities for IT managers who initially took that unknown plunge into what was then the deep mystical waters of virtualization. Now they’re also realizing that by enabling thin clients the cost of their endpoint hardware is also significantly driven down by the consequent lifespan extension of existing PCs. Indeed the future of endpoint devices is one that could revolutionize their existent IT offices – a future of PC/laptop-less office desks replaced by thin client compatible portable iPads? Anything is now possible.

There’s also no doubting that VDI brings with it even further advantages one being improved security. With data always being administered via the datacenter rather than from the vulnerability of an end user’s desktop, risks of data loss or theft are instantly mitigated. No longer can sensitive data potentially walk out of the company’s front doors. Also with centralized administration, data can instantly be protected from scenarios where access needs to be limited or copying needs protection. For example a company that has numerous outsourcers / contractors on site can quickly set their data and application access to be specified or even turned off. Indeed there is nothing stopping an organization in setting up ‘a contractor’ desktop template which can be provisioned instantly and then decommissioned the moment the outsourced party’s contract expires.

By centralizing the infrastructure, fully compliant backup policies can also become significantly easier. With PCs and hard drives constantly crashing leading to potential data loss, the centralized virtual desktop has an underlying infrastructure which is continuously backed up. Additionally with the desktop instance not being bound to the PC’s local storage but instead stored in the server, recovery from potential outages are significantly quicker with even the option of reverting the virtual desktops back to their last known good states. Imagine the amount of work the customary employees that constantly bombard the IT helpdesk with countless “help I’ve accidentally deleted my hard drive” phone calls could actually get done now, not to mention the amount of time it will free up for your IT helpdesk team. In fact you might even end up with an IT helpdesk that gets to answer the phone instead of taking you straight to voicemail.

Additionally an IT helpdesk team would also be better utilized with the centralized, server-based approach allowing for both the maintenance of desktop images and specific user data all without having to visit the end user’s office. Hence with nothing needing to be installed on the endpoint, deployment becomes incredibly faster and easier with VDI than the traditional PC desktop deployment. This can also be extended to the laborious practice of having to individually visit each desktop to patch applications, provision and decommission users, as well as upgrade to newer operating systems. By removing such activities, the OPEX savings are more than substantial.

OPEX savings can also be seen with the added benefit of optimizing the productivity of highly paid non-technical end users by avoiding them having to needlessly maintain their desktop applications and data. Furthermore the productivity of employees can also be improved significantly by a centralized control of which applications are used by end users and a full monitoring of their usage, so long gone should be the days of employees downloading torrents or mindlessly chatting away on social networks during working hours. Even the infamously slow start up time of Windows which has consequently brought with it the traditional yet unofficial morning coffee/cigarette break can be eradicated with the faster Windows boot up times found with VDI. Even lack of access to an employee’s corporate PC can no longer be used as an excuse to not log in from home or elsewhere remotely when required – a manager’s dream and a slacker’s nightmare.

So with all these benefits, where lies the risk or obstacle to adopting a VDI infrastructure for your company? Well as with most technology there rarely exists a one solution fits all scenario and VDI is no different. Prior to any consideration for VDI, a company must first assess their infrastructure and whether VDI could indeed reap these benefits or alternatively possibly cause it more problems.

One of the first issues to look for is whether the organization has a high percentage of end users which manipulate complex or very large files. In other words if a high proportion of end users are constantly in need of using multimedia, 2D or 3D modeling applications, or VOIP, than VDI should possibly be reconsidered for a better managed desktop environment. The performance limitations that came about with server-based computing platforms such as Microsoft's Terminal Services with regards to bandwidth, latency and graphics capabilities are still fresh in the mind of many old school IT end users and without the correct pre-assessment those old monsters could rear their ugly head. For example an infrastructure that has many end users using high performance / real time applications should think carefully before going down the VDI route regardless of what the sales guys claim.

Despite this though if having taken all this into consideration and realizing your environment is suited to a VDI deployment the benefits and consequent savings are extensive despite the initial expenditure. As for which solution to take this leads to another careful consideration and one that needs to be investigated beyond the usual vendor marketing hype.

Firstly when it comes to server virtualization, there currently is no threatening competition (certainly not in the Enterprise infrastructure) to VMware’s VSphere 4. In the context of desktop virtualization though, the story has been somewhat different. Citrix’s XenDeskTop for those who’ve deployed it certainly know that it has better application compatibility than VMview 3. Add to the problems of multimedia freeze framing that would often occur with the VMview 3 solution and Citrix looked to have cornered a market in the virtual sphere which initially seemed destined to be monopolized by VMware. Since then VMware have hit back with VMview 4 which brought in the vastly improved PCOIP display protocol which dwarfs their original RDS protocol and simplified their integration with Active Directory and overall installation of the product, but in performance terms XenDeskTop still has an edge. So it comes as no surprise that rumours are rife that VMWorld 2010 which is soon to take place in a couple of weeks will be the launching pad for VMview 4.5 and a consequent onslaught on the Citrix VDI model. Subsequent retaliation is bound to follow from Citrix who seemed to have moved their focus away from the server virtualization realm in favour of the VDI milieu which can only be better for the clients that they are aiming for. Already features such as Offline Desktop, which allow end users to download and run their virtual desktops offline and then later resynchronize with the data center are being developed beyond the beta stage.

So the fact remains that quickly provisioning desktops from a master image and instantly administering policies, patches and updates without affecting user settings, data or preferences is an advantage many will find hard to ignore. So while VDI has still many areas for improvement, depending on your infrastructure it may already be an appropriate time to reap the rewards of its numerous benefits.

VSphere 4 still leaves Microsoft Hyper V-entilating

2010-08-03T08:45:00.000-07:00

When faced with a tirade of client consultations and disaster recovery proposals/assessments, you can’t help but be inundated with opportunities to showcase the benefits of server virtualization and more specifically VMware’s Site Recovery Manager. It’s a given that if an environment has a significant amount of applications running on X86 platforms, then virtualization is the way to go not just for all the consolidation and TCO savings but for the ease in which high availability, redundancy and business continuity can be deployed. Add to that the benefit of a virtualized disaster recovery solution that can easily be tested, failed over or failed back. With what was once a complex procedure, testing can now be done via a simple GUI based recovery plan. Thus one should consequently see the eradication of trepidation that often existed in testing out how full proof an existent DR procedure actually was. Long gone should be the days of the archaic approach of the 1000 page Doomsday Book-like disaster recovery plans which the network, server and storage guys had to rummage through during a recovery situation, often becoming a disaster within itself. Hence then there really is little argument to not go with a virtualized DR site and more specifically VMware’s Site Recovery Manager, but not so it seems if you’ve been cornered and inculcated by the Microsoft Hyper V Sales team.

Before I embark further, let’s be clear that I am not an employee or sales guy for VMware - I’m just a techie at heart who loves to showcase great technology. Furthermore let it go on record that I’ve never really had a bone of contention with Microsoft before – their Office products are great, Exchange still looks fab and I still run Windows on my laptop (albeit on VMware Fusion). I even didn’t take that much offense when I recently purchased Windows 7 only to realize that it was just a well marketed patch for the heir to the disastrous Windows ME throne i.e. Windows Vista. I also took it with a pinch of salt that Microsoft were falsely telling customers that Exchange would run better on local disks as opposed to the SAN in an attempt to safeguard themselves from the ongoing threat of Google Apps (a point well exposed and iterated on David Vellante’s Wikibon article, “Why Microsoft has it’s head up it’s DAS”). Additionally my purchase of Office 2010 in which I struggled to fathom the significant difference between Office 2007, still didn’t irk me that much. What has turned out to be the straw that broke the camel’s back though is the constant claims Microsoft are making that Hyper-V is somehow an equally good substitute to VMware and consequently pushing customers to avoid a Disaster Recovery Plan that includes Site Recovery Manager. So what exactly are the main differences between the two hypervisors and why is it that I so audaciously refuse to even consider Hyper-V as an alternative to VSphere 4?

Firstly one of the contentions often faced with virtualizing is the notion that some applications don’t perform well if at all when on a virtualized platform. This is true when put in the context of Hyper V, which currently limits the number of vCPUs to only 4. That’s pretty much a no go for CPU thirsty applications leading to an erroneous idea that a large set of applications should be excluded from virtualization. This is not the case when put in the VSphere 4 context where guests can have up to 8 cores of vCPUs. In an industry which is following a trend of CPUs scaling up by adding cores instead of increasing clock rates, the future of high-end x86 servers provides a vast potential for just about any CPU hungry application to run on a virtualized platform – something VSphere 4 is already taking the lead in.

Then there’s the management infrastructure in which Hyper V uses software named Systems Center (SC) and more specifically the Systems Center Virtual Machine Manager (SCVMM), whereas the VSphere4 equivalent is named vCenter Server. With Hyper-V being part of a complete Microsoft virtualization solution, System Center is generally used to manage Windows Server deployments. The System Center Virtual Machine Manager on the other hand not only manages Hyper-V-hosted guests but also Virtual Server, VMware Server and VMware ESX and GSX guests. Ironically this can then also be extended to managing vMotion operations between ESX hosts, (perhaps an inadvertent admission from Microsoft that vMotion wipes the floor off their equivalent Live Migration). Compared to vCenter Server which can either be a physical or virtual machine this comes across as somewhat paltry when VSphere 4 now offers the ability to allow multiple vCenter servers to be linked together and controlled from a single console, enabling a consolidated management of thousands of Virtual Machines and several Datacenters. Add to this the functionality that vCenter Server provides a search-based navigation tool that enables the finding of virtual machines, physical hosts and other inventory objects based on a user defined criteria and you have the ability to quickly find unused virtual machines or resources in the largest of environments all through a single management pane.

Taking the linked management capabilities of vCenter further, VSphere 4 also offers what they term the vNetwork Distributed Switch. Previously for an ESX server a virtual network switch was provisioned and managed and configured. With the vNetwork Distributed Switch, virtual switches can now span multiple ESX servers while also allowing the integration of third-party distributed switches. For example the Cisco Nexus 1000v is the gateway for the network gurus to enter the world of server virtualization and take the reins of the virtual network which were previously being run by VM system admins. Put this in the context of multiple vCenter Servers in the new linked mode and end users have the capability to not only manage numerous virtual machines but also the virtual network switches. In an Enterprise environment where there are hundreds of servers and thousands of virtual machines, what previously would have been a per-ESX switch configuration change can now be done centrally and in one go with the vNetwork Distributed Switch. Hyper V as of yet has no equivalent.

That broad approach has also pushed VMware to not only incorporate the network guys into their world, but also the security and backup gurus. With VSphere 4’s VMSafe, VMware have now enabled the use of 3rd party security products within their Virtual Machines. An avenue for the security guys to at last enter the virtual matrix they previously had little or no input in. Then there’s the doorway that VSphere 4 has opened for backup gurus such as Veeam to plug into virtual machines and take advantage of the latest developments such as Change Block Tracking and vStorage APIs bringing customers a more sophisticated and sound approach to VM backups. Hyper V still has no VMsafe equivalent and certainly no Change Block Tracking.

Furthermore as Microsoft flaunt Hyper V’s latest developments, scrutiny shows that they are merely features that have been available on VMware for several years and even then still don’t measure up in terms of performance. Point in case being Hyper V’s rather ironically titled ’Quick Motion’. For high availability and unplanned downtime protection Hyper-V clusters have a functionality that restarts Virtual Machines on other cluster nodes if a node fails. With ‘Quick Motion’ a Virtual Machine is then moved between cluster hosts. Where it fails though is in its inability to do the action instantly as is the case with VMware’s vMotion and HA features. This hardly exudes confidence in Hyper V when a potential move that can take several seconds leaves you exposed to the risk of a network connection failure which consequently results in further unplanned downtime. Subsequently Quick Motion’s inability to seamlessly move Virtual Machines across physical platforms results in downtime requirements for any potential server maintenance. This is certainly not the case with VMware and vMotion wherein server maintenance requiring downtime is a thing of the past.

Moreover so seamless is the vMotion process that the end user has no idea that his virtual machine has just crossed physical platforms while they were inputting new data. This leads us to Hyper V’s reaction and improved offering now termed Live Migration which Microsoft claim is now on a par with vMotion. Upon further inspection this still isn’t the case as the amount of vMotion operations that can be simultaneously done between physical servers is still far more limited with Hyper V. Additionally while Hyper V claims to be gaining ground, VMware in return have shot even further ahead with VSphere4’s Storage vMotion capabilities which allows ‘on the fly’ relocation of virtual disks between the storage resources within their given cluster. So as VMware advances and fine tunes its features such as Distributed Resource Scheduler, Distributed Power Management (DPM), Thin Provisioning, High Availability (HA) etc., Hyper V is only just announcing similar functions.

Another issue with Hyper-V is that it’s simply an add-on of Windows Server which relies on a Windows 2008 parent partition i.e. it’s not a bare metal hypervisor as virtual machines have to run on the physical system’s operating system, (something akin to VMware’s Workstation). Despite Microsoft’s claims that the device drivers have low latency access to the hardware, thus providing a hypervisor-like layer that runs alongside the full Windows Server software, in practical terms those that have deployed both Hyper V and VMware can testify the performance stats are still not comparable. One of the reasons for this is that VMware have optimized their drivers with the hardware vendors themselves unlike Hyper V which sadly is stuck in the ‘Windows’ world.

This leads to my next point that with VSphere 4 there is no reliance on a general operating system and the various operating systems that are now supported by VMware continues to grow. Microsoft on the other hand, being the potential sinking ship that she is in the Enterprise Datacenter have tried to counter this advantage with marketing Hyper V as being able to run on a larger variety of hardware configurations. One snag they don’t talk about so much is that it has to be a hardware configuration that is designed to support Windows. Ironic when one of the great things about virtualization is that Virtual Machines with just about any operating system can now be run together on the same physical server, sharing pools or resources – not so for Microsoft and Hyper V who desperately try to corner customers to remain on a made-for-PC operating system that somehow got drafted into datacenters. Question now is how many more inevitable reboots will it take on a Windows Enterprise Server before IT managers say enough is enough?

Then there are some of the new features that were introduced in VSphere 4 which still have failed to take similar shape in the Hyper V realm. For example VMDirectPath I/O which allows device drivers in virtual machines to bypass the virtualization layer and access the physical resources directly – a great feature for workloads that need constant and frequent access to I/O devices.

There’s also the Hot-Add features wherein a virtual machine running Windows 2000 or above can have its network cards, SCSI adaptors, sound cards, CD-ROMs added or removed while still powered on. They even go further by letting your Win 2003 or above VM hot add memory or CPU and even extend your VMDK files – all while the machine is still running. There’s still nothing ‘hot’ to add from the Hyper V front.

Also instead of the headache inducing complexities that come with Microsoft’s Cluster Service, VSphere 4 comes with Fault tolerance – a far easier alternative for mission critical applications that can’t tolerate downtime or data loss. By simply creating a duplicate virtual machine on a separate physical host and via vLockstep technology to ensure consistency of data, VSphere 4 offers a long awaited and straightforward alternative to complex clustering that further enhances the benefits of virtualization. No surprise then that currently the Microsoft Hyper V sales guys tend to belittle it as no great advantage.

Another VSphere 4 feature which also holds great benefits and is non-existent in Hyper V is that of Memory overcommitment. This feature allows the allocation of more RAM to virtual machines than is physically available on the physical host. Via techniques such as Transparent page sharing, virtual machines can share their common code thus leading to significant savings in the all too common situation of having to add more memory to an existent server which equates to more than the cost price of the server.

So while Hyper V has also recently caught up with a Site Recovery Manager equivalent with the Citrix Essentials for Hyper V package, it’s still doing just that i.e. playing catch up. One of the main arguments for Hyper V is that it’s free or nearly free but again that’s the marketing jargon that fails to elaborate that you have to buy a license for a Windows Server first and hence help maintain the dwindling lifespan of Microsoft within the Datacenter. Another selling point that Hyper V had was that they were better aimed for small to medium sized businesses due to their cheaper cost….the recent announcement of VSphere 4.1 may now also put bed to that claim. So like all great empires, collapses are imminent and while I don’t believe Microsoft are going to the I.T. Black Hole, they certainly don’t look like catching up with VMware in the ever emerging and growing market of virtualization.

NetApp Justifies Storage Efficiency Tag with Primary Deduplication

2010-06-04T09:51:00.000-07:00

The pendulum has shifted. We are in an era in which Storage Managers are in the ascendancy while vendors must shape up to meet customer demands in order to survive the current economic plight. Long gone are the days of disk happy vendors who could easily shift expensive boxes of FC disks or Account Managers who boasted their huge margins at the selling of skyscraper storage systems to clients who faced an uphill struggle to meet their

constantly growing storage demands. With responses such as thin/dynamic/virtual provisioning arrays and automated storage tiering, vendors have taken a step towards giving customers solutions that will enable them to use more of what they already have as well as utilise cheaper disks. Another such feature now starting to really prick the conscience of vendors as customers become more savvy is that of primary deduplication or the more aptly termed ‘data reduction’. So as this cost saving surge continues some vendors have cheekily tried to counteract it with sales pitches for exorbitantly priced Flash SSDs (which promise 10 times performance yet shamelessly sit on the back end of Stor

age systems dependent on the latency of their BEDs and RAID controllers) as a means to keep margins up. But not the WAFL kings NetApp….

Mention deduplication and you most likely think of backup environments where redundant data is eliminated, leaving only one copy of the data and an index of the duplicated data should it ever be required for restoration. With only the unique data stored the immediate benefits of deduplication are obvious from a reduction in backup storage capacity, power, space and cooling requirements to reduction in the amount of data sent across the WAN for remote backups, replication and disaster recovery. Not only that, deduplication savings has also shifted the backup paradigm from tape to disk allowing quicker restores and reduced media handling errors (and yes I have made it no secret of giving kudos to Data Domain in this respect). Shift this concept now to primary storage though and you have a different proposition with different challenges and advantages.

Primary storage is accessed or written to constantly therefore necessitating that any deduplication process must be fast enough to eliminate any potential overhead or delay to data access. Add to the equation that unlike backup data the amounts of duplicate data are not in the same proportion as that found in Primary storage and you also have a lesser yield in deduplication ratios. Despite this though, NetApp have taken Primary deduplication by the horns and are offering genuine data reduction that extends beyond the false marketing of archiving and tiering being data reduction techniques when in fact all they are is the shoving of data onto different platforms.

Most vendors on the ‘data reduction’ bandwagon have gone with file level deduplication which looks at the file system itself replacing identical files with one copy and links for the duplicate files. Hence there is no requirement for the file to be decompressed or reassembled upon end user request due to the same data merely having numerous links. Therefore the main advantage is that data access should be without any added latency. In real terms though this minimalist approach doesn’t produce data reduction ratios that yield anything significant for the user to be particularly excited about.

On the flip side what is referred to as sub file level deduplication has an approach familiar to those who already use deduplication for their backups. Using the hash based technology; files are first broken into chunks. Each chunk of data is then assigned a unique identification, whereupon duplicated identifications of chunks are replaced with a pointer to the original chunk. Such an approach brings the added advantage of discovering duplicate patterns in random places irregardless of how the data is saved. With the addition of compression end users can also significantly reduce the size of chunks. Of course this also adds the catch 22 situation of deduplication achieving better efficiency with smaller chunks, while compression is more effective with larger chunks. Hence why NetApp have yet to incorporate compression alongside their sub level deduplication. Despite this though NetApp are showing results that when put in a virtual context are more than impressive.

One of the first major vendors to incorporate primary data deduplication, NetApp is comfortably verifying their ‘storage efficiency’ selling tag when put in the context of server and desktop virtualisation. One of the many benefits of VMware (or other server virtualisation platforms) is their ability to rapidly deploy new virtual machines from stored templates. Each of these VM templates includes a configuration file and several virtual disk files. It is these virtual disk files that include the operating system, common applications and patch systems or updates and it is these that are constantly duplicated each time a cloned VM is deployed. Imagine now a deployment of 200 like for like VMs and then put NetApp’s primary deduplication process wherein multiple machines end up sharing the same physical blocks in a FAS system and you’ve got some serious reduction numbers and storage efficiency. With reduction results of 75% to 90%, NetApp’s advantage comes from their long established snapshot-magic producing WAFL (write anywhere file level) technology. With its in built CRC checksum for each block of data store, the WAFL already has block-based pointers. By running the deduplication at scheduled times all checksums are examined, with the filer doing a block-level comparison of blocks if any of the checksums match. If a match is identified, then one of the WAFL block-based pointers simply replaces the duplicated block. Due to the scheduled nature of the operation occurring during quiet periods, the performance impact is also not that intrusive giving the NetApp solution significant storage savings especially when similar operating systems and applications are grouped into the same datastores. Add to the mix that NetApp’s PAM (Performance Accelerator Module) is also dedupe-aware, common block reads are quickly satisfied from cache bringing even faster responses by not having to search through every virtual disk file (VMDK). NetApp also ‘go further, faster’ so to speak with the addition of their flex clone technology which rapidly deploys VM clones which are also prededuplicated.

So while arguments may be raised that NetApp’s sub level deduplication suffers from the physical layer constraints of WAFL’s 4KB block size or their lack of compression, the truth is that they have deliberately avoided such alternatives. If they’d have opted for using sliding block chunking where a window is passed along the file stream to seek out a more naturally occurring internal file or added compression algorithms, the overhead that would come with such additions would render most of the advantages of primary

dedupe worthless. Yes, Ocarina and Storwize have appliances that compress and uncompress data as it’s alternatively stored and read but what performance overhead do such technologies have when hundreds of end users concurrently access the same email attachment? As for Oracle’s Solaris ZFS file system sub level deduplication which is yet to see the light of day one wonders how much hot water it will get Oracle into should it turn out to be a direct rip off of the NetApp model.

Bottom line is as long as the primary deduplication model you employ gives you the reduction numbers worth the inevitable overhead then it's more than a beneficial cost saving feature. Furthermore while I’m the first to admit that NetApp certainly have their flaws but when it comes to primary deduplication and consequent data reduction they really are making your storage more efficient.

EMC's VPLEX & Planned Takeover of DataDirect Networks

2010-05-28T08:35:00.000-07:00

Talk to the average Storage Engineer who manages the growth of your datacenter’s modular system about Petaflops, Exabytes, Petabytes of Archives or 1TB of sustained bandwidth and you’ll probably find them scratching their heads in disbelief. This is the reality that does exist in the world of super computing and what is sometimes referred to as Extreme Storage. While some Storage Managers would feel they are suffering with their exponential data growth and decreasing budgets, their problems can’t be classified as ‘Extreme’ unless they’re dealing with ExaBytes (1018 bytes) of storage with trillions of data transactions per second, trillions of files and a data transfer rate from storage to application that exceeds a TB per second. Couple that with the conundrum that it’s for relatively few users and requires the data to be secure, both for short-term and long-term retention and then you have a real case for Extreme Storage……. well at least for now.

Such figures though are not the concern of the average datacenter manager or storage vendors with architectures that are catered and designed for IOPS centric database driven applications so much so that even SNIA has yet to give Extreme Storage the relevance of a definition in their Storage Dictionary. Not EMC though, where if my sources are correct, they deem Extreme Storage not only a key to their own future but also that of the Storage industry’s such that they are already concocting an audacious takeover plan of the company DataDirect Networks.

Before I embark upon my controversial claim, let’s rewind back a few weeks to EMC World, Boston where most of the buzz centred on the launch of the new VPLEX. A nifty idea that would take cloud computing enthusiasts to an ever approaching reality by creating heterogeneous pools of storage that can be accessed and shared over a distance. Couple that with VMware integration and you have the ability to VMotion your applications across datacenters that are miles apart - a great idea and one that strikes a double whammy at both Storage vendors HDS/IBM/HP/NetApp and the virtualisation ‘catch-up guys’ MS Hyper-V and Red Hat. As IT Directors yearn for a virtual infrastructure for their applications that goes beyond the physical limitations of the datacenter, EMC’s trump card of setting up a centrally managed pool of virtual resources spread across numerous datacenters via VPLEX is nothing short of a ‘virtual’ revolution. With Site Recovery Manager, VMware already had the edge over their competitors by in essence providing an extended version of their ‘high availability’ concept that could span across data centers. With VPLEX the VMotion concept of moving a virtual server across physical platforms ‘on the fly’ can now also be extended across datacenters. Moreover while EMC have currently failed to corner the market of virtualisation of heterogeneous storage dominated by HDS and IBM with their product Invista, the launch of the VPLEX now takes that battle head on with the added value of cross-site virtualisation. So how then does this link to my bold prediction that Extreme Storage is next on EMC’s radar with more significantly a proposed takeover of the company DDN?

The VPLEX model is poised to have four versions, two of which are already available namely VPLEX Local and VPLEX Metro with VPLEX Geo and VPLEX Global to follow suit. VPLEX Local is the straightforward virtualisation of heterogeneous storage systems behind one management pane within your datacenter, a solution that has successfully been offered by HDS for several years. VPLEX Metro though allows the concept to stretch up to 100km, hence enabling the virtualisaion of storage spanning datacenters across cities. Based on a combination of hardware and software which is placed between the traditional FC attached storage systems and the servers, the VPLEX rack virtualizes the heterogeneous mix of disk arrays into what EMC term ‘a federated pool of virtual storage’. As for the hardware itself, it contains a management server, FC switches, Ethernet switches, the standard redundant power supplies and the VPLEX engines. Within each engine rests a pair of quad core Intel CPUs and directors which each contain 32 FC ports with 8Gbps bandwidth. With an active-active cluster spread over one to four VPLEX engines the requirements to seamlessly VMotion applications across a 100km distance is more than easily met, hence being coined VPLEX Metro. The question that now stands is for the proposed VPLEX Geo and VPLEX Global i.e. would such hardware and performance stats add up for say data that needs to be VMotioned across continents as the name suggests? Indeed such distances and endeavours would not be the requirement of EMC’s regular customer base of industries that demand financial transaction processing but rather those that are facing a content nightmare and need the expertise and performance figures that are associated with Extreme Storage.

When you’re talking Extreme Storage you’re talking DDN i.e. DataDirect Networks. While relatively unknown, DDN still possess an impressive resume of HPC clients from NASA, Lawrence Livermore Laboratories to movie special effects users like Pacific Title & Art Studio. Thus as far as being a company that can act as a platform from which EMC can build out its ‘global’ and ‘geo’ cloud storage offerings, DDN already have credible references to do so quite easily.

Furthermore a potential acquisition of DDN will allow EMC to penetrate a HPC customer base that they’re currently unfamiliar to. Fields ranging from High Energy Physics companies such as Fermilab, Nuclear research organisations such as CERN, particle physics research companies such as DESY to National Security and Intelligence are all potential clients that EMC could take on with a new Extreme Storage Platform that incorporates VPLEX and deals with large data that is locally or globally distributed with long-term retention. It would clearly give EMC a major distinction from its current major competitors.

Ironically though it’s one of EMC’s current competitors, HP that have already made moves into Extreme Storage with their Ibrox based HP 9100 Extreme. The HP 9100 was marketed as an Extreme Storage system and was shamelessly targeting Web companies and their like who required multipetabytes of data storage. HP’s aim was to profit from an emerging market of heavy users such as ever growing and popular social networks with their online subscriber information and video content as well as users of video surveillance systems and research organizations. While this was a brave attempt even HP had to concede to DDN’s supremacy and expertise in the field when they only this week agreed an OEM relationship for DataDirect Networks (DDN)’s S2A9900 disk array to be bundled with the Lustre File System resold by the SCI group within HP. Indeed HP are now like every large HPC OEM vendor out there – reselling DDN. With partnerships already with IBM, Dell and SGI, the one big name missing from the list is EMC. Now with the VPLEX Global and Geo offering soon to be unveiled, a relationship with DDN whether it be an acquisition or OEM seems inevitable.

In fact DDN and EMC are certainly no strangers to each other, when last year the former launched a direct onslaught on EMC's Atmos cloud storage product with their Web-Optimised Scaler (WOS). Designed for geographically dispersed global storage clouds, the WOS is a geo-cluster of quasi-filers that store objects with an API-access to a global namespace. Boasting scalability that is currently growing beyond 200 billion files of storage and 1 million file reads per second for objects, such stats are effortlessly achieved through the simultaneous access of numerous WOS boxes. Hence not only flooring EMC's Atmos in terms of transaction rates and file retrieval rates, being a file-based product the WOS also hits the EMC NAS jugular namely the Celerra. As for how the WOS works on a global scale; the storing of objects, which are files or groups of files are each given a unique object number which identifies the datacenter containing the WOS system that stores the object and the object itself. Datacenters are linked via WOS nodes which form the WOS cloud while the WOS API is used to access servers to read or write objects to the WOS cloud. A straightforward concept but the question now is how much of this explanation will replace the word DDN with EMC and WOS with VPLEX Global come the launch of the latest EMC masterplan? Put this in the wider context of the upcoming VPLEX Geo and Global, and I have little doubt that EMC Execs (renowned for preferring to spend outrageously than OEM a potential competitor) are furiously sharpening their pencils, carefully concocting a takeover of the still relatively small yet growing company that is DDN.

After the rapidly swift takeover of Data Domain last year, nothing surprises me anymore with regards to the financial clout of EMC. So while a takeover of DDN will not only bring about the removal of the competitive edge that DDN currently poses and enable EMC’s vision of ‘VPLEXing’ across the globe to become an instant reality, the benefits of such a deal would bring even more so to EMC’s constantly growing portfolio. In the words of Bob Dylan,“Times are a changing” and the digital content explosion brought about by the rapid growth of online, nearline and backup data pools has left the traditional storage systems designed by EMC and their like defunct and inadequate to compete in such a vast growing market. Like a crumbling empire the domination of transactional data that factored so heavily in the design of storage systems has ended with an unscrupulous coup de etat of unstructured data requiring extreme performance, scalability and density becoming the mainstream. EMC have clocked on to this and are pushing their future be involved in this direction. Should a DDN deal go through, EMC will not only have advanced themselves into a new customer base but would also bring in vast technical expertise ranging from high-speed FPGA parity calculation accelerators instead of RAID systems, high speed Infiniband interconnects etc. that can only enhance their current Enterprise and Modular range. As for EMC’s direct competitors such as HP, IBM, HDS etc. who will they have to turn to for an OEM deal or expertise should they also decide to enter the fast growing market trend towards Extreme Storage……perhaps EMC themselves if these predicted developments are to bear fruit.

The Unified Storage Battlefield Could Decide the Future of Storage

2010-04-29T11:05:00.000-07:00

In the past week HDS finally revealed their response to the VMware-Cisco-EMC alliance with the launch of a unified computing platform including integrated storage, server, and networking technology. With the aid of Microsoft, HDS have stated that their centralized storage, server and networking platform will be launched early next year. In the tradition of my enemy’s enemy is my friend, HDS have also signed an OEM deal with Microsoft under which Microsoft System Center Operations Manager, System Center Virtual Machine Manager and Windows Server 2008 R2 will be tightly integrated with Hyper-V. Added to this is HDS Dynamic Provisioning and the HDS Storage Cluster for Microsoft Hyper-V. Moreover despite the secrecy, the networking brains behind the platform are most probably Brocade, the grandfathers of SAN who also now have a sound grip on IP networking since their acquisition of Foundry back in 2008.

Well, it’s no surprise that with the current turmoil brought upon by the disbandment of the SUN OEM deal, HDS are desperate to announce a new product despite it being more than six months away. But the trend towards Unified Storage is one that is being followed by many in an attempt to adhere to the economic climate and the rapid drive towards consolidation. While at one point it was NetApp’s domain of which no one seemed to be interested in, the Unified Storage demand has grown considerably with customers seeing the mass of potential and savings that come with running and managing files and applications from a single device. By consolidating file-based and block-based access in a single platform and hence supporting FC, iSCSI, and NAS, customers immediately reap the benefits of reduced hardware requirements, lower capital expenditures and simplified single pane management. So now the war of vendors has entered a new battlefield in which nearly all are making a bid to usurp the lion’s share of the spoils. But like in every battle there will ultimately be casualties…

Cisco, the IP kings have bravely entered the arena and are pushing forward plans with their combination of networking, blade servers and storage in a single architecture i.e the Unified Computing System (UCS) platform. Whether they can convince customers that an IP switch company can build servers remains to be seen, but Cisco already proved doubters wrong when they successfully entered the SAN market by drawing on the IP niche that they had established in just about every data center in the world.

HP's acquisition of 3Com on the other hand was instigated to provide the networking brains for their ‘converged computing’ model that binds server, storage, and networking resources. How the powerhouse of HP will fair is not at as difficult to predict given the success of their blade systems and credence amongst customers as a server platform provider. But are they entering the arena too late and how will this fair with their OEM relationship with HDS?

Within this battlefield of generals, there are also some charlatans who have cheekily tried to gain some market share just by coining the term ‘unified storage’. IBM and NEC for example, have brought out backup and recovery systems within a single architecture that lack any NAS support, yet still coin the term ‘unified storage’. Such pretenders may suffer an early death especially when smaller companies such as BlueArc go the whole nine yards with their Titan systems that not only support SAN and NAS but can also utilize WAN via Riverbed's Steelhead networking solution.

Then there’s the SUN 7000 series from Oracle’s Sun Microsystems. A great bargain for the amount of functionality that it provides from unlimited snapshots, integral data compression, iSCSI thin provisioning, virus scanning, remote replication as well as the expected support for CIFS, NFS, HTTP, FTP and iSCSI. Additionally the 7000 series supports RAID 5, RAID 6 arrays and ZFS Hybrid storage pools which can capitalize on the high performance of Flash memory devices and DRAM memory. Yet despite how great the 7000 is, it’s coming from a camp that has been mortally wounded with the messy Oracle takeover and the bureaucracy that surrounds it, to which customers are now suffering the effects of. Will customers purchase a great product that will immerse it into an eon of political wrangling when they need and rely on quick and timely support?

It’s evident that HDS or anyone else for that matter, which coins the term ‘Unified Storage’, is going to have a tough time dealing with EMC. The marketing machine which currently knows no bounds, made an unashamed onslaught on the small business market cornered by NetApp when they launched the Celerra. While in essence it was just a Clariion with a NAS gateway, it fully supported SAN and NAS as well as NFS 2, 3 and 4, and CIFS file sharing. Furthermore EMC’s entry into the market seems to be with a strategic plan that seems to span the company as a whole, which is minimizing its different hardware platforms.

When EMC released the V-Max platform, one of the most notable things was its usage of hardware components that were already available on other EMC hardware platforms. From the Clariion-esque disk drives, flash drives, DAE’s, LCC’s, Intel x64 CPU’s, fans to power supplies, the Celerra, like the V-Max is also made in the same mould. With the Clariion, CDL, EDL and Celerra platforms all sharing similar hardware components, it’s only a matter of time before the anomalous architecture of the archive platform, Centera is either changed to fit the mould or replaced completely in favour of a unified platform that seamlessly integrates with the Celerra or Clariion.

As Cisco had done before them when they added SAN to their IP portfolio and what NetApp have done to some extent with ONTAP, EMC’s common hardware direction could eventually lead to underlying software being the only thing which distinguishes different EMC platforms.

So while currently unified storage limits the level of control in file-based versus block-based I/O and hence does give lesser performance than its dedicated block-based counterpart, a strategic approach that takes a long term look at the term ‘unified’ could change the face of high end storage systems in the future. As storage systems move further towards consolidation, it is indeed the winner in the battlefield of unified storage that that will eventually draw others to a new beginning and approach and ultimately the end of the current trend of 7 feet tall high end enterprise systems that have housed data centers for so many years. A self tiering SATA / SSD Unified Platform without FC disks?….Let’s watch this space.

Data Domain's CPU Centric Deduplication Genius is no Dupe

2010-04-06T19:26:00.000-07:00

Last year EMC’s somewhat controversial acquisition of Data Domain right under the noses of NetApp raised several eyebrows to say the least. Considering the reported amount of $2.1 billion and their already deduplication packed portfolio which consisted of the source based Avamar, the file-level deduplication/compression of its Celerra filer and their Quantum dedupe integrated VTLs, some heads were left scratching as to what actually was the big deal with the target based deduplication solution of Data Domain. Almost a year on and with Data Domain’s DD880 being adopted by an ever growing customer base, the heads have stopped scratching and are paying close attention as to what is probably the most significant advancement in backup technology of the last decade.

With deduplication currently being all the rage, with possibly only ‘Cloud Computing’ overshadowing it, the benefits of deduplication are becoming an exigency for backup and storage architects. With most backup software producing copious amounts of duplicate data stored in multiple locations, deduplication offers the ability to eliminate those redundancies and hence use less storage, less bandwidth for backups and hence shrink backup windows. With source based and file level based deduplication offerings, it is Data Domain’s target based solution i.e. the big black box that is clearly taking the lead and producing the big percentages in terms of data reduction. So what exactly is so amazing about the Data Domain solution, when upon initial glance at for example the DD880 model, all one can see is just a big black box? Even installing one of the Data Domain boxes hardly requires much brainpower apart from the assignment of an IP address and a bit of cabling. And as for the GUI, one could easily forget about it as the point of the ‘big black box’ is that you just leave it there to do its thing and sure enough it does its thing.

And while the big black box sits there in your data center the figures start to jump out at you where an average backup environment can see a reduction of up to 20 times. For example a typical environment with a first full backup of 1TB with only 250GB of physical data will immediately see a quadrupled reduction. If such an environment was to take weekly backups with a logical growth rate of 1.4TB per week but with only a physical growth of 58GB per week, the approximate reduction could go up to more than 20 times within four months:

Reduction =
First Full + (Cumulative Logical Growth x Number of weeks) / Physical Full + (Cumulative Physical Growth x Number of weeks)

e.g. After 25 weeks
Reduction = 1TB + (1.4TB x 25) / 0.250TB + (0.058TB x 25)
= 35TB / 1.7TB
= 21 times less data is backed up

So how does Data Domain come up with such impressive results? Upon closer inspection, despite being considered the ‘latest technology’, Data Domain’s target based deduplication solution has actually been around since 2003, so in other words these guys have been doing this for years. Now in 2010 with the DD880, to term their latest ‘cutting edge’ would be somewhat misleading when a more suitable term would be ‘consistently advancing’. Those consistent advancements have come from the magic of the big black box being based on its CPU-centric architecture and hence not reliant upon adding more disk drives. So whenever Intel unveils a new processor, Data Domain does likewise with its incorporation into their big black box. Consequently the new DD880’s stunning results are the result of its incorporation of a quad-socket quad-core processor system. With such CPU power the DD880 can easily handle aggregate throughput to up to 5.4 TB per hour and single-stream throughput of up to 1.2 TB per hour while supporting up to 71 TB of usable capacity, leaving its competitors in its wake. Having adopted such an architecture, Data Domain have pretty much guaranteed a future of advancing their inline deduplication architecture by taking advantage of every inevitable advance on Intel's CPUs.

Unlike the source based offerings, Data Domain’s Target-based solution is controlled by a storage system rather than a host and thus takes the files or volumes from the disk and simply dumps them onto to the disk-based backup target. The result is a more robust and sounder solution to a high change-rate environment or one with large databases where RPOs can be met a lot easier than with a source-based dedupe solution.

Another conundrum that Data Domain’s solution brings up is the future of tape based backups. The cheap RAID 6 protected 1 TB / 500 GB 7.2k rpm SATA HDD disks used by the DD880 alongside the amount of data reduced via its deduplication also brings into question the whole cost advantage of backing up to tape. If there’s less data to back up and hence fewer disks than tape required, what argument remains for avoiding the more efficient disk to disk back up procedure? An elimination of redundant data with a factor of 20:1 brings the economics of disk backup closer than ever to those of tape backups. Couple that with the extra costs of tape backups often failing, the tricky recovery procedures of tape based backups as well as backup windows which are increasingly scrutinized; this could well be the beginning of the end of the Tape Run guys having to do their regular rounds to the safe.

Furthermore with compatibility already with CIFS, NFS, NDMP and the Symantec OpenStorage, word is already out that development work is being done to integrate closer with EMC’s other juggernauts VMware and Networker. So while deduplication and its many forms saturate the market and bring in major cost savings to backup architectures across the globe, it is Data Domain’s CPU based, target based inline solution which has the most promising foundation and future and currently unsurpassable results. $2.1 billion? Sounds like a bargain.

Consider this Before Jumping on the Cloud

2010-03-13T12:16:00.002-08:00

Thinking about jumping on the cloud? True, I’ve had no qualms in showing my skepticism towards the marketing of ‘cloud computing’ and it being the mere repackaging of solutions which have existed for years, but the fact is it still addresses a concept and reality that exists and one which holds numerous benefits. Indeed abandoningan existing on site IT infrastructure for a cloud provider that most often or not can’t offer the same level of security, control or performance is not an easy decision but one which CIOs and IT executives are seriously considering when weighing up the economic benefits. As with any change though, a move towards the cloud necessitates a sound and comprehensive assessment to avoid the trap of a short term benefit turning into a long term nightmare.

Integration Considerations:

Firstly your company should assess its business strategy and whether a cloud computing infrastructure could integrate with it. With all the convenience that cloud offers, the risk of data residing beyond your company’s vicinity still presents a significant security risk. Hence depending on your company, whether that be a startup, small, medium or enterprise an assessment of how much risk you are prepared to take in order to cut costs will initially be the first factor in deciding which data / applications to outsource to the cloud. This is what often leads enterprises to be content with using the cloud for archival while smaller business find it easier to go the whole nine yards. Once that’s decided your design for integration should first evaluate each process and system and determine the number of simultaneous requests that need to be handled. As always availability of information also needs to be paramount, regardless of whether the information resides internally or on the cloud. Thus the key for enterprises is to not think of the cloud as a substitute for their processes, policies and security but rather an extension of their existent architecture.

Choosing the Right Cloud:

If your company needs access to e-mail, e-commerce, gaming applications, etc. with cost reduction being the primary concern over security and availability then the public cloud provides the service to access such applications via a thin device with minimal financial overhead. Here the start up business can thrive but companies with an already substantial customer base and Tier 1 applications such as OLTP should think otherwise. In such an instance a hybrid cloud may be most suitable wherein the Tier1 application data remains in house with the cloud being utilized for backup, archiving and testing bringing in the consequent cost savings. Enterprises on the other hand with strict Sarbanes Oxley and security controls on customer data and service levels may outweigh the risk of information access control over cost and leave the cloud altogether, that is until security in the cloud becomes further enhanced.

Testing the Cloud with your Existent Virtualised Infrastructure:

The balance of scales will always alternate between the reliable and secure nature of data centers with the far cheaper ‘pay as you use’ option of cloud computing. Therefore a way to get a taste of the cloud and assess its suitability to your environment is to utilize the virtualization that is already taking place within your data center. If you are one of the few archaic infrastructures that has not already moved towards the virtualization of applications and infrastructure then you are already missing out on immediate cost benefits which could be existent within your own data center or what the marketing teams tag ‘the internal’ or ‘private’ cloud. With the independence of applications from their physical infrastructure, your in house virtualisation already provides the gateway and flexibility to cloud offerings which are based on external virtual infrastructures. By choosing a non-core application to experiment with such as your e-mail archiving you can quickly set up a test environment and get a feel of the bottlenecks, security, performance and billing procedures of the cloud. Upon sufficient testing and having gained an idea of the metrics you required, you will have a better idea and peace of mind on how best to deploy more mature applications to the cloud.

Choosing the Right Applications:

Choosing the right applications is also a serious precursor as not all applications are best suited for the cloud. If PaaS and IaaS are a consideration then a migration may be a necessity if the cloud may not support your application especially if there are incompatibility issues between the platform software and your application. Another issue is related to applications that require real time response that can’t be put at the risk of potential network latency, congestion or bandwidth clogging. Tight backup windows, graphics intensive applications or applications handling large I/O would need serious consideration and investigation of the SLAs provided by the cloud prior to any migration. On the flip side applications linked to your non core business operations may be ideal for deployment such as a marketing campaign website or a standalone application that doesn't require much interaction with other applications. Again applications which are used for testing purposes can easily be deployed to the cloud with minimal costs and risks to the business especially with many cloud providers offering free trial periods.

To conclude while network latency, data availability, security and application support are all valid concerns for customers thinking of utilizing the cloud, a sound approach and pre-analysis could easily alleviate them, bringing in major cost savings to your business. In other words jumping on the cloud bandwagon without the right assessment might see you falling straight through from the sky; but making sure the cloud you do jump on is substantial enough to support and sustain your company could also be the soundest business investment you’ll ever make.

Some think it's as easy as building bricks:

Are SUN to end Hitachi Data Systems OEM Relationship?

2010-02-26T14:08:00.000-08:00

On January 26th just prior to the official announcement of Oracle’s takeover of SUN Microsystems, I confidently predicted in my article “SUN’s Oracle Merger” with regards to SUN’s storage portfolio that “One certainty is that the OEM partnership with HDS’ enterprise arrays will continue.” Perhaps it’s time to eat some humble pie. If current indications are anything to go by, it’s more than likely that the SUN agreement of reselling HDS Enterprise Storage may be coming to an abrupt end.

Oracle clearly have a different business approach to their customers than SUN Microsystems did, and that includes dealing with Hitachi Data Systems. Admittedly I‘ve never been a great fan of SUN’s storage systems, often finding them to be the epitome of a server company that builds storage i.e. a box with lots of disks and sparse in-built intelligence. But with the recent launch of the 7000 series (which still may come under scrutiny by NetApp considering it’s more than coincidental similarities) and the intelligent storage systems built by Larry Ellison’s other plaything Pillar Data Systems, their modular market is now pretty well covered. How Oracle/SUN could plan to cover the gap that will be left from a potential removal of the USPV / USPVM (ST9990V /ST9985V) could lie with the approach shown with the recent Exadata V2. Oracle databases that are directly attached to boxes chocked full of flash drives may well be the answer Oracle/ SUN will be offering to get themselves free from the entanglement of Enterprise storage vendors.

While there seems to be a game plan of some sort in the Oracle/SUN camp, if this supposition were to come true it will have major implications on Hitachi Data Systems and their next steps forward. Personally I’m happy if this will happen as it may at last be the kick up the ‘Back End Director’ that HDS need to finally start marketing and addressing a customer base, certainly within the EMEA region that is still oblivious to its existence. I’ve often shown my frustration at HDS and their inability and lack of drive to push forward their brand and products to consumers who have settled for inferior products from other vendors that were merely marketed better. Resting on the laurels that HP and SUN were rebadging and reselling their Enterprise Systems and doing all the hard work for them, the downside was that HDS’ cross bar architecture storage systems and virtualization technology were firmly placed in thousands of datacenters, unbeknownst to the IT Directors that bought them. Another issue was that unlike the SUN relationship in which only the colour of the doors and the SUN badge changed, HP buy HDS Enterprise Storage Systems and actually change the microcode making them more ‘HP’ than ‘HDS’. So a true untainted HDS USPV could now potentially only be purchased from Hitachi Data Systems themselves. This could be the beginning of a HDS revolution or a slow withering death of sales.

But I’m confident if the leadership at HDS takes the right steps and investment, this could finally be the key to a future market share they have been lacking in. There is no doubting the quality of the HDS Enterprise range from the still reliant cross bar architecture and vIrtualisation through the array USPV systems. Hence maintaining those sales and support deals with existent SUN customers may not be such a great overhaul especially with an updated USPV on the horizon. Where the real challenge lies is drawing customers to the equally good modular AMS and WMS range which are rarely found in Datacenters yet alone virtualized behind their own Enterprise Storage Systems. Also the HNAS range made by BlueArc are also a range to be reckoned with but are hardly making NetApp sales guys break a sweat as potential customers are often unaware of their existence. Plus all the latest initiatives which HDS have taken such as High Availability Manager, IT Operations Analyzer, or the Hitachi Content Archive platform HCP, as excellent as they are, are still not making the waves and marketing noise their credentials deserve.

So in a twist of fate, should the continuation of the SUN OEM relationship fall through, Hitachi Data Systems may be forced into being the marketing machine it up to now has shied away from in order to maintain and advance its presence in the industry. The positive thing is that the products are and always have been good enough – now it’s time for the marketing guys to promote it.

The Best Back Up Solution for VSphere4?

2010-02-19T10:44:00.000-08:00

When VMware first introduced VCB as part of the ESX package, it never did seem more than a temporary / complimentary solution for customers who had a small environment of 100 VMs or less. With the launch of VSphere4 and the subsequent introduction of APIs which allowed external applications and scripts to communicate directly to the ESX, it was apparent that VMware was beginning the gradual move to offload the backup solution to the Backup experts. Now having run with VSphere4 for more than six months, it seems a good time to assess and evaluate who has taken advantage and the lead with incorporating all the latest features of ESX4.

To recap; with VMs being encapsulated in a single disk file the principal was that image-level backups instead of traditional file-level allowed backups to be much faster. With VSphere4, VMware introduced improved support for thin provisioning which not only had the potential to reduce the amount of actual storage allocated but also shorten backup windows. The idea was straightforward; by using thin provisioning the system admin is given the ability to over-commit disk space, especially handy as the majority of VMs don’t use all of their allocated disk space. Thus this eradicates the problem with the majority of disk-to-disk backup applications using image-level backups i.e. no more backing up of a complete virtual disk file of a VM when most of it wasn’t actually used.

Another new addition to VSphere4 and it’s vStorage APIs was the feature of CBT (Changed Block Tracking). With CBT the VMkernel could now track the changed blocks of a VM’s virtual disk. Just by querying the information using an API call to the VMkernel, CBT alleviates the burden from the backup applications having to scan or keep track of changed blocks. This results in much quicker incremental backups as the overhead of scanning the whole VM image for changes since the last backup could now be eradicated.

So in looking around for solutions which best incorporated these new features I eventually came across Veeam. Utilising the thin provisioning feature to remove the overhead of no longer seeking out empty disk blocks and the unnecessary backing up of those empty blocks, Veeam also incorporates compression algorithms on the target backup device. Hence Veeam have a solution that not only reduces the amount of space used on the source host datastores but also the target backup storage device.

Furthermore Veeam are currently the only third party API that offer support for CBT, although ESXpress are promising something similar with their upcoming version 4. Veeam have come up with several different modes to utilise the CBT feature namely, SAN mode, Virtual Appliance mode and Network mode. Each mode brings less I/O to each device depending on your setup and thus less resource consumption when performing backups consequently leading to reduced backup windows.

So while Veeam are currently leading the way, the time is certainly ripe for more third party APIs to be developed and incorporated making VM backup nightmares a thing of the past.