Sharepoint Virtualization - Numero Drei

Hopefully I won't seem like a person who waffles. Or someone who jumps to conclusions. I have tried to be objective and I tried to make the testing meaningful. I will share here an unsanitized version of what I published to our team:

Our tests were performed with a two server MOSS 2007 farm with the same databases running on a fairly robust (physical) SQL cluster. One server was an indexer only, and the other served the SSP, web front end, etc. No special Sharepoint caching was turned on for any sites. The physical servers tested were HP servers with 2GB RAM and two (older) 3.4 ghz dual core processors in each. The virtual servers ran with 2GB RAM and variously 2 or 4 processors on an ESX system that I believe had two dual quad core machines and LOTS of RAM.

Three categories of test were performed:

• Indexing performance
• Web page traffic
• Document library traffic/search queries

Results were monitored from various sources - HP OpenView, Loadrunner, WAPT, Windows native, and subjective.

1. Indexing
Indexing virtual to virtual with 2 processor machines seemingly outperformed physical in all categories except CPU usage, which was significantly higher on the virtuals (though acceptable). The time to complete the index was almost the same for physical and virtual farms.

We did not have any HPOV data on the indexer.

2. Web Page Traffic
An older script was run with Loadrunner, simulating 250 users for 15 minutes. The script was web page views only. In general, the virtuals outperformed the physicals in these tests. Neither was overloaded.

3. Document Library - File and Search
A short script was created to download two large files (30 MB) repetitively and do some search queries. This is a realistic document library scenario of Intranet content being downloaded simultaneously by multiple users. The script was run with WAPT via several different PC's, with anywhere from 20 to 200 sessions simulated. The tests were run from the same part of the building and presumably through a small number of ports on the same network switch.

None of the systems performed this test well. Heavy file activity can be the bane of SharePoint's existence, even though we often use it as a document management system that is easy to use and access from any office. Large files are common, and access to such files usually cannot be managed (if it were managed, we could easily cache such files and more easily serve up large files to many users).

The physical servers were slow but remained responsive during this test. CPU utilization was usually below 50%, with some higher spikes. Four processor virtualized servers remained responsive during this test, but were very slow (slower than the physicals) and CPU utilization was between 30 and 75%, with some higher spikes. Two processor virtualized was almost non-responsive. CPU utilization rose to 100% and stayed there as long as the test was permitted to run. This happened for as few as twenty connections.

Conclusions
I don't want to draw too many conclusions until we have had further discussion of the testing. There may have been other limitations to tests 1 and 3 from the SQL server. Microsoft spends more time discussing SQL than it does SharePoint servers in the latest tuning info http://go.microsoft.com/fwlink/?LinkID=105621&clcid=0x409. I previously distributed this Microsoft tuning document (published in late December).

Microsoft recommends using 4 or 5 web front ends (without much qualification). Serving up files over 15 MB or having content databases over 50 GB is taxing on SharePoint and may degrade performance significantly. This latest tuning information from Microsoft is very conservative.

Our tests indicate that multiple processors are a key to maintaining reasonable performance from web front ends. Indexing seems to work fine from a virtual server.


So am I objective or what? My tests did little to confirm any of my original line of thinking.

Virtualizing SharePoint Part Deux

Be sure to also read the next part - Numero Drei

We are still not finished with our virtualization tests, but I had to document this early stage of enlightenment (especially since the part one post seems to have been pretty far off the mark). Since virtualization very likely will be a part of my SharePoint experience from now on, I imagine there will be a part 3, 4, etc.

The state of the art of virtualization is a lot better than I thought. One of our IT groups has been building out the infrastructure for VM such that the virtuals will have a much better future than physical machines. Using multiple dual quad core servers with scads of memory, VMWare ESX is pretty amazing stuff.

The purpose of this study is to determine our server needs for a build-out of SharePoint to include new farms in Europe and Asia with up to a terabyte of data in each, and increasing North American capacity to up to two terabytes. There are around 5000 users in 30+ locations (heavily weighted to North America). The current farm is smaller (4 machines in our North America data center and less than 200GB of content). It is currently running on brand new IBM boxes (our SQL cluster is a little bit older). New 64 bit multiple quad core SQL machines will be used (VERY physical - brutes) and all the disk is on SANs.

The way it looks from the early virtualization testing, my SharePoint future is all virtual. 100% virtual. Indexer, WFEs, apps (SQL will have those big, new 64 bit physical machines). For SharePoint servers, tested performance of a VM is comparable (often better) than physical, with so many other advantages that I probably won't want ANY physical servers for the buildout, and I will probably give up the physicals that we already have.

When I started this process I was concerned that the risks of going virtual were high. I am almost convinced that it is the best way to go for high availability and performance. It does not seem to save a lot of money, but it saves some. There may be some hard to quantify costs savings (administration, power, cooling, etc.) and fortunately I will not be under much pressure to put numbers on such things (I'd make them up!).

The final conclusions will come in future posts, but this is encouraging.

Holiday Valley February 16, 2008

After my first skiing trip to the Rockies last December I had hoped to get to someplace closer to home a bit sooner so I could evaluate my old skis vs. the hot new stuff I rented at Vail. I also wanted to judge the lameness of the comparative experience.

Too much time has passed, though, so I couldn't judge the skis. My long, old things seemed OK. The experience is what it is. A hill a third the length of somplace like Vail (but it feels like a fifth) and terain that probably isn't a fifth in size (you can't compare them, really). The snow was perfect at Vail vs. your typical Eastern ice in New York (ice is damn fast, though, and a challenge). The lines were a bit long this weekend too, but bearable. They were sledom long at Vail.



My conclusion is that I can still have fun on day trips to Holiday Valley, and do them like I always have, with healthy trips to the Hearth (my usual restaurant) and the bar, which is so unlike my Vail experience, where I only hit the bar afterwards.



I have a story of an altercation with one of the bartenders at the Yodeler Lodge bar. I ordered an Ellicottville Brewing Co. IPA (or pale ale) for Bryce and an Ellicottville Brewing Co. Amber for myself. She brought me back what looked and tasted like Miller Genuine Draft. I told her it was the wrong beer and she said it wasn't. I said the kegs were switched when she insisted it was right. She even poured another to show me. I told her it was OBVIOUSLY the wrong beer - the color was all wrong (that EBC Amber is very pretty in the glass, it doesn't look anything like Miller). She dramatically threw the one beer in the sink, asked me what I wanted then and took the other away. She practically threw my change at me. I should have thrown the tip at her.

Virtualizing - Part 1

Be sure to read the other parts of this series (because it changes a lot!):
http://bobklass.blogspot.com/2008/02/virtualizing-sharepoint-part-deux.html
http://bobklass.blogspot.com/2008/02/sharepoint-virtualization-numero-drei.html


Hopefully it won't be a misnomer to call this part 1, but at this point I really don't have a lot of hard info, just my thoughts. Later, when we've tried it, I'll add more (we all hope, right?).

If you think about how the Sharepoint components work, it is easy to rule out SQL and indexers for virtualization (at least for a large farm, as we are deploying). That leaves web front ends, query servers and application servers (Excel, Infopath). We have run our farm with query server and web front end combined. The applications will be new to us very soon.

So for us, the low risk trial will be to use the (very beefy) virtuals for applications. This especially makes sense since these are processor intensive. With WFE's, there is too much risk of underperformance. What if the cache reads are too slow? We can't have it not work well.

So I am advocating that we proceed this way.

I've been lax with my posts as I try to rest my right arm - too many mouse clicks. Very painful. I'll be back!

Check out part deux.