Windows Home Server v2 'Vail' Beta: Drive Extender v2 Dissected
by Ryan Smith on April 27, 2010 10:39 PM EST- Posted in
- Operating Systems
- Ryan's Ramblings
- WHS
- Windows
Yesterday Microsoft released the first public beta of the next version of Windows Home Server, currently going under the codename of Vail (or as we like to call it, WHS v2). WHS v2 has been something of a poorly kept secret, as word leaked out about its development as early as 2008. In more recent times an internal beta leaked out late last year, confirming that WHS v2 existed and giving everyone an idea of what Microsoft has in store for the next iteration of their fledgling home server OS.
As was the case with WHS v1, WHS v2 is having what we expect to be a protracted open beta period to squish bugs and to solicit feedback. This is very much a beta release by Microsoft’s definition (as opposed to Google’s definition) so it’s by no means production-worthy, but it’s to the point where it’s safe to throw it on a spare box or virtual machine and poke at it. Being based on Windows Server 2008 R2 (WHS v1 was based on Server 2003), the underlying OS is already complete, so what’s in flux are WHS-specific features.
We’ll cover WHS v2 more in depth once it’s closer to shipping, but there’s one interesting thing that caught our eye with WHS v2 that we’d like to talk about: Drive Extender.
Drive Extender v2: What’s New
Drive Extender was the biggest component of the secret sauce that made WHS unique from any other Microsoft OS. It was Drive Extender that abstracted the individual hard drives from the user so that the OS could present a single storage pool, and it was Drive Extender that enabled RAID-1 like file duplication on WHS v1. Drive Extender was also the most problematic component of WHS v1 however: it had to be partially rewritten for WHS Power Pack 1 after it was discovered that Drive Extender was leading to file corruption under certain situations. The new Drive Extender solved the corruption issues, but it also slightly changed the mechanism of how it worked, doing away with the “landing zone” concept in favor of writing files to their final destination in the first place.
For WHS v2, Microsoft has once again rewritten Drive Extender, this time giving it a full overhaul. Microsoft hasn’t released the full details on how Drive Extender v2 works, but we do know a few things that are worth discussing.
At the highest level, Drive Extender v1 was file based, sitting on top of the file system with the network shares going through it to access files. Hard drives in turn were NTFS formatted and contained complete files, with Drive Extender directing the network sharing service to the right drive through the use of NTFS tombstones and reparse points.
Drive Extender v2 is turning this upside down. The new Drive Extender is implemented below the file system, putting it between NTFS and the hard drives. With this change also comes a change in how data is written to disk – no longer are the drives NTFS formatted with the files spread among them, rather now they’re using a custom format that only WHS v2 can currently read. With Drive Extender v2 sitting between NTFS and the hard drives, anything that hits NTFS direct now receives full abstraction without the need to go through shared folders. In a nutshell, while Drive Extender v1 was file based, v2 is block based.
With the move to block based storage and a custom file system also comes a change in how data is stored on hard drives. In place of storing files on a plain NTFS partition, WHS v2 now uses a custom file system that stores data in 1GB chunks, with files and parts of files residing in these chunks. In the case of folder duplication, the appropriate chunks are duplicated instead of files/folders as they were on WHS v1.
Microsoft also threw in a bit more for data integrity, even on single-drive systems. Drive Extender v2 now keeps an ECC hash of each 512 byte hard drive sector (not to be confused with a chunk) in order to find and correct sector errors that on-drive ECC can’t correct – or worse, silently passes on. MS’s ECC mechanism can correct 1-bit and 2-bit errors, though we don’t know if it can detect larger errors. MS pegs the overhead at this at roughly 12% capacity, so our best guess is that they’re using a 64bit hash of each sector.
The Good & The Bad
All of these changes to Drive Extender will bring some significant changes to how WHS will operate. In terms of benefits, the biggest benefit is that from a high-level view WHS now operates almost exactly like NTFS. The NTFS volumes that Drive Extender v2 generates can be treated like any other NTFS volumes, enabling support for NTFS features such as Encrypted File System and NTFS compression that WHS v1 couldn’t handle. This also goes for application compatibility in general – Drive Extender v2 means that application compatibility is greatly improved for applications that want to interact directly with the drive pool, with this being one of Microsoft’s big goals for WHS v2.
Meanwhile at the low-level, using block based storage means that Drive Extender v2 doesn’t behave as if it was tacked on. Open files are no longer a problem for Drive Extender, allowing it to duplicate and move those files while they’re open, something Drive Extender v1 couldn’t do and which would lead to file conflicts. Speaking of duplication, it’s now instantaneous instead of requiring Drive Extender to periodically reorganize and copy files. The move to block based storage also means that WHS v2 data drives and their data can be integrated in to the storage pool without the need to move the data in to the pool before wiping and adding the drive, which means it’s going to be a great deal easier to rebuild a system after a system drive failure. Finally, thanks to chunks file size is no longer limited by the largest drive - a single file can span multiple drives by residing in chunks on multiple drives.
However this comes with some very notable downsides, some of which MS will have to deal with before launching WHS v2. The biggest downside here is that only WHS v2 can read MS’s new data drive file system – you can no longer pull a drive from a WHS and pull off files by reading it as a plain NTFS volume. This was one of WHS’s best recovery features, as a failing drive could be hooked up to a more liberal OS (i.e. Linux) and have the data pulled off that way.
The other major downside to the new Drive Extender is that the chunking system means that a file isn’t necessarily stored on a single disk. We’ve already touched on the fact that excessively large files can be chopped up, but this theoretically applies to any file big enough that it can’t fit in to leftover space in an existing chunk. This in turn gives WHS v2 a very RAID-0 like weakness to drive failures, as a single drive failure could now take out many more files depending on how many files have their pieces on the failed drive. Our assumption is that Drive Extender v2 is designed to keep files whole and on the same disk if at all possible, but we won’t be able to confirm this until MS releases the full documentation on Drive Extender v2 closer to WHS v2’s launch.
First Thoughts
In discussing WHS v2 and Drive Extender v2, there’s one thing that keeps coming to mind: ZFS. ZFS is Sun/Oracle’s next-generation server file system that has been in continuing development now for a few years. ZFS is a block based file system for use in creating storage pools that features easy expandability and block hashing (among other features) making it not unlike Drive Extender v2. ZFS ultimately goes well beyond what Drive Extender v2 can do, but the fundamental feature set the two share is close enough that Drive Extender v2 can easily be classified as a subset of ZFS. Whether this is intentional or not, based on what we know thus far about Drive Extender v2, Microsoft has created what amounts to a ZFS-lite low-level file system with NTFS sitting on top of it. Make no mistake, this could prove to be a very, very cool feature, and is something we’re going to be keeping an eye on as WHS v2 approaches release.
In the meantime, Microsoft is still taking feature suggestions and bug reports on WHS v2 through Microsoft Connect. We don’t have any specific idea on how long WHS v2 will be in beta, but we’d bet on a Winter 2010 release similar to WHS v1 back in 2007, which means it would be in beta for at least a couple more months.
52 Comments
View All Comments
strikeback03 - Wednesday, April 28, 2010 - link
Speaking of wanting it to run on Atom processors, I read that for streaming video it is going to transcode the stream on the fly. I can't see an Atom processor doing much transcoding.QChronoD - Wednesday, April 28, 2010 - link
Considering how many formats most of the new settop boxes can handle natively, I don't see the system needing to do much transcoding anymore. (Unless you are using a 360 and can't use anything but MPEG2 and WMV.)strikeback03 - Thursday, April 29, 2010 - link
The review I saw said it was for remote viewing, so I would assume they mean so you aren't streaming a full-size DVD rip over the internet.strikeback03 - Wednesday, April 28, 2010 - link
My understanding from another review is that any computer can only be part of one WHS network at a time. So to properly test WHS v2 wouldn't you need not only a spare machine to run it on, but other systems which did not need to be part of the main WHS network?Roland00 - Wednesday, April 28, 2010 - link
With how cheap flash memory is, and the fact they are no longer using the system partition as a landing zone, I don't see why OEMS make the system partition on bundled flash drives. Make the system partition on a bundled flash drive, and then have a spot for a second flash drive you plug in. And schedule automatic backups of the system partition onto the second flash drive. If the system partition drive fails, well you move up the bundled second drive up a usb port, and then you buy a generic usb flash drive to plug into the second slot and use that as your backup for your system partition.4gbs of flash memory is going for $10 dollars on the open market. There is little reasons not to do this.
funkyd99 - Wednesday, April 28, 2010 - link
These were my thoughts exactly! A small, quick 40GB SSD (although it sounds like 60GB is written into the system requirements) containing the OS tucked away inside a Chenbro ES34069 mini-ITX case, leaving 4 hot swappable bays available for the data disks. That's my dream home server, anyway :) I'm not sure where some of you find room for 10+ drive cases!clex - Wednesday, April 28, 2010 - link
I put mine in the guest room. When people stay in there they usually say, "I can barely hear it." Then, at about 1 in the morning the backups begin and all those drives start buzzing. After that I shut it off at night and forgo backups while anyone is visiting.Ryan Smith - Wednesday, April 28, 2010 - link
The Flash used in an SSD is much more expensive than your cheap stuff that goes in to USB flash drives and such, since it needs to be far more reliable and faster. Right now for the price of a 64GB SSD, you can pick up a 1.5TB hard drive.For that reason I don't expect to see OEMs go the SSD route on WHS boxes any time soon. They'll simply stick in one large, cheap hard drive.
Roland00 - Wednesday, April 28, 2010 - link
I understand that SSD memory is the higher bin stuff, but is it really necessary for WHS system drive? If you aren't using the system drive as a landing drive, than you don't really need a big system drive, nor are you writing to it all the time like a traditional OS drive. You don't need a SSD. I don't see why you couldn't limit the system drive to 4 or 8 gbs of information, and just use cheap flash memory.And if the cheap flash memory fails, it doesn't matter for the OEMS have an automatic backup to another cheap flash drive which the user provides. If the flash drive fails, just plug in a new one.
It will be so much easier than using a big hard drive as a storage drive and a system drive. If this "combo" drive fails than it is a pain in the ass to rebuild the array for the non technically inclined (it is easy if it is just a storage drive). My mom can easily move a flash drive from slot B to slot A if the computer tells her a problem is occurring with slot A. My mom is also educated enough to go buy a new 4 or 8gb of flash drive from best buy or another retail store, if the OS tells her she needs a new storage device.
It just makes no sense to me from an HP or ACER standpoint.
AMv8(1day) - Thursday, April 29, 2010 - link
That would be perfect, I was planning on separating the OS and storage so that I could run up to 6 2TB HDDs exclusively for storage with a 32-64GB 2.5" SSD as the primary OS and a cheap USB stick as backup.By the way, anyone looking for a good small 5+ HDD capable box should look at the mini-ITX/DTX capable Lian Li PC-Q08 or the Fractal Design Array mini-ITX case. There are a few mini-ITX boards that have come out recently that support 6 SATA ports + 1 eSATA port.