A history of the DIVX DVD
In 1998 (almost 20 years ago), a new DVD rental format arrived named DIVX (aka Digital Video Express). It purported to be a DVD rental format that had no late fees and the media didn’t need to be returned… at least those were the benefits purported to the consumer. What they didn’t tell you was that you would need to buy a brand new expensive DVD player to play them. Let’s explore.
DIVX versus DivX
To get this confusion cleared up quickly, DIVX was a brand name assigned to a new DVD rental standard introduced by Circuit City and the entertainment law firm Ziffren, Brittenham, Branca and Fischer in 1998. However, it’s not entirely clear what problem Circuit City was trying to solve by introducing the DIVX rental format when DVD was already useful enough for rentals.
The DIVX brand name, introduced by Circuit City, bears no relationship to the DivX or Xvid video encoding standards. Even though there is no relationship by Circuit City to the DivX encoder, there is a slight reverse relationship from the DivX encoder to the DIVX brand. In fact, the original name of the DivX encoder was actually DivX ;-)
Yes, this encoder name included the winking smiley. This smiley was actually a nod (and sarcasm) towards Circuit City’s then soon-to-be-defunct DIVX rental standard. Here’s what the DivX Wikipedia article says of the early days of the video encoder named DivX ;-).
DivX ;-) (not DivX) 3.11 Alpha and later 3.xx versions refers to a hacked version of the Microsoft MPEG-4 Version 3 video codec (not to be confused with MPEG-4 Part 3) from Windows Media Tools 4 codecs. The video codec, which was actually not MPEG-4 compliant, was extracted around 1998 by French hacker Jerome Rota (also known as Gej) at Montpellier.
So then, what does DivX ;-) have to do with the DIVX DVD format? Not much other than DivX ;-) making a tongue-in-cheek poke at Circuit City’s DIVX rental format. Hopefully, this clears up any confusion around this convoluted naming.
DIVX as a rental standard
The primary impetus to build the new DIVX rental standard by Circuit City was probably brand recognition. At the time, Circuit City was considered the second largest electronics retailer behind Best Buy. The Circuit City management was obviously willing to do anything to become the number one electronics retailer, including dreaming up technology ideas that didn’t need to be built. Meaning that by 1998, Blockbuster had the rental market sewn up. However, Circuit City sought to disrupt that by trying to create a new standard that not only simultaneously upset Blockbuster’s cart, but introduced a new format that would bring more recognition to the Circuit City brand (and, of course, generate more hardware and rental sales). As a side note, Circuit City was also the second largest appliance retailer behind Sears at that time.
Here’s the Circuit City DIVX promo video (skip to 0:17 to begin an unrealistic family scenario, press 1 to skip the intro entirely or jump to 4:19 to begin use case demonstration). Don’t feel obligated to watch the whole thing.
Now, let’s watch this training video to better understand how then CEO Richard L. Sharp saw DIVX’s future within Circuit City. Pay close attention to his statements during the opening segments of this video. Again, don’t feel obligated to watch the whole thing.
Unfortunately, Circuit City’s management goals were way too ambitious and overconfident. They also dropped into a rabbit hole with this DIVX venture that took them away from their core retail business and caused them to spend millions to create and support a format that didn’t live up to the hype. In fact, it might even be considered that failure of the DIVX format ushered in the downward slide of Circuit City into oblivion. While Circuit City was a reasonable electronics retailer, they didn’t have any presence in the video rental market. When they introduced DIVX, the assumed strategy was to add $4-5 rentals and boost DVD player sales in the Circuit City and Good Guys stores. The management team thought that this rental business would somehow take them to the next level. If only they had adopted standard DVD rentals instead.
As shown in the demo, DIVX boasted a 48 hour rental period with no need to return the disc when the rental period ended. However, to use DIVX, you had to invest in a brand new type of DVD player that also supported the DIVX format. Tada! Here’s the catch. This is also where Circuit City comes into the picture. You’d have to run on over to your local Circuit City (or one of several other retailers owned by CC, like Good Guys) to buy one of these newfangled DIVX DVD player doodads. A doodad that might cost you $100-150 more than a DVD player without DIVX. At least, this is what the management at Circuit City hoped you would do.
This idea for introducing this new format was a huge misfire for Circuit City. In addition to the picture quality problems described just below, the DIVX player contained a modem that required the player to dial-up and register itself before you could play any DIVX discs. It also apparently dialed-up twice a day to register any new purchases and download advertisements. This modem required a physical telephone line plugged into the unit to dial home. This then authorized not only your player, but supplied the player with the necessary information to authorize playback of a rental disc you recently picked up. This concept all worked reasonably well, except for the fact that several privacy groups felt that this dial home feature meant that Circuit City (or whomever) could keep tabs on your viewing habits. Little did we know then exactly how much spying would become commonplace with sites like Google and Facebook. Anyway, that privacy concern didn’t help boost efforts to sell DIVX into main stream. Of course, it wasn’t the only problem.
Poor Movie Quality
The actual DIVX DVDs themselves failed to contain the more advanced features found on a standard DVD, such as 16:9 anamorphic widescreen needed to fill a large flat panel. DIVX movie discs also failed to contain alternate audio commentary and extra features commonly found on standard DVDs. Instead, these DIVX DVDs simply contained 4:3 cropped pan and scan versions of the film… a subpar version. This was a huge misfire for the format. When you can get a better looking film on standard DVD, why would you rent the crappier DIVX format for $5? Yet more consumer dilemma.
Consumer Misunderstanding
Because a DIVX DVD appears to be a standard DVD (it looks the same), some consumers didn’t understand that they needed to buy a new player to play the DIVX media. Instead, they bought the DIVX disc, took it home and inserted it into their regular DVD players only to find that it failed to work. They would then find that they couldn’t return the disc because it was open. The misunderstanding of this new format caused grief among would-be consumers and left a sour taste for this format. This problem only served to fracture the DVD market. Worse, who’s willing to buy a brand new and expensive DIVX player just to recover a $5 loss? Not many. This problem didn’t serve Circuit City well.
It seems that Circuit City’s commercials likely didn’t much help clear this misunderstanding. Let’s watch a Circuit City commercial from this era with that same guy from the Demo reel:
There were also a number of commercials released during 1998 and 1999 that failed to mention DIVX at all… like the following commercial from 1998. You’d think a company like Circuit City spending millions to try and force adoption of their new brainchild would advertise the DIVX format on every commercial, if even only a mention at the very end. Nope. You can’t sell something if you don’t market it.
Landfill Problems
One of the touted benefits to consumers would be throw-away discs. You’d spend $4-5 for each disc, but you didn’t need to return any discs after the 48 hour watching period was over. This also meant no late fees. You simply tossed the disc into the garbage can. This idea was to hit Blockbuster where it hurt. Blockbuster was the king of late fees at the time. A few months after introduction of this idea, Circuit City stores set up recycle containers to entice users to recycle used DIVX discs at Circuit City stores instead of throwing them into the trash. Not sure how well that worked. I don’t think this wasteful idea went over well with consumers, particularly after AOL’s constant barrage of wasted CDs ended up everywhere at the time.
Licensing Issues and Retailers
As a result of Circuit City’s involvement with DIVX (along with a legal team), for other retailers to sell DIVX compatible players required paying a licensing fee to Circuit City. As a result of the licensing fees, Best Buy and other retailers shunned the players choosing to avoid paying those fees. It’s no wonder, either. Why would you ever agree to pay another retailer money for the privilege of selling that retailer’s product in your store? You wouldn’t. This was a completely foreseeable miscalculation by the Circuit City management team.
This meant that Circuit City and other stores owned by Circuit City ended up the sole sellers of these players (and the DIVX format). Without wider support via other retailers, this format really had no hope of surviving. Circuit City should have dropped the idea for licensing fees quickly just to get better entrenchment for the format. It’s not like it wasn’t already costing Circuit City a mint to keep this format alive. Stupid is as stupid does.
Movie Studio Support
On the plus side for the format, because of what studios considered weak protection technologies associated with standard DVD (CSS), many studios jumped on board with the DIVX’s CSS + Triple DES protection standard. This boosted the initial ~20 titles when it arrived in summer of 1998 to well over 400 titles by early 1999. Some early studio adopters were 20th Century Fox, Disney and Dreamworks. Wikipedia says:
The initial trial of the DIVX format was run in the San Francisco and Richmond, California, areas starting on June 8, 1998. Initially only a single Zenith player was available, along with 19 titles. A nationwide rollout began three months later, on September 25, with players and 150 titles available in 190 stores. In total 87,000 players were sold during 1998, with 535,000 discs across 300 titles being sold.
The studios felt that the DIVX format offered a more solid encryption technique to protect their movie content. I’m sure they did. Due to the arcane structure needed to authorize the movie rental, it meant jumping through hoops just to get your movie to play. The movie studios love making consumers jump through hoops to play their content.
This quick studio adoption rate was a bone of contention because some studios began exclusively releasing their films onto the DIVX format instead of DVD. This issue caused further problems for the format and more consumer backlash erupted and threatened to fracture the industry into a new format war.
On the other hand, Sony and Warner Home Video, which at the time apparently comprised up to 40% of the movie rental market, refused to release their movies on DIVX. The primary reason for this refusal was that both of these companies had a stake in the success of standard DVD format. Supporting the DIVX standard would be a conflict of interest.
By spring of 1999, the number of titles had increased to over 410. Little did Circuit City or the consumers realize the end was near for DIVX. Due to mounting pressures and costs, Circuit City didn’t realize how much of an albatross that DIVX would become. There was just no way Circuit City could go this new format alone without wider industry investment and consumer acceptance.
Overconfidence
Circuit City’s biggest mistake was its heavily miscalculated financial ability to support this newly created format. After all, Circuit City is a retailer, not a tech innovator. Driving a new tech format through a retail company already has many hurdles and reputational issues to overcome. Circuit City was also too confident in its ability to entice other retailers to make this format succeed. Those retailers didn’t bite. Even in 1998 when this format came about, Internet RFCs were still a thing. Circuit City entirely avoided the RFC and Whitepaper approach that had become commonplace to announce new technologies. Instead, they launched this format without much fanfare hoping that the party train would show up. It didn’t.
Because of all of the above and including backlash from consumers and lack of retailer support, Circuit City way overestimated its ability to get this format adopted… and why would anyone want to adopt this format? With licensing fees, there was no incentive for non-affiliated retailers to adopt some other retailer’s idea as practical or realistic… especially when the standard DVD already provided a better rental and sales format.
Without the necessary support by consumers and other retailers alike, the format was doomed from the go. By the summer of 1999 (just 1 year after it launched), the format officially died on June 16th, 1999 (almost exactly year since it had launched). However, due to format commitments to existing consumers, it would limp this format along until 2001. Wikipedia writes of the DIVX demise:
The format was discontinued on June 16, 1999, because of the costs of introducing the format, as well as its very limited acceptance by the general public. It was shot down by Blockbuster Video stores not wanting to carry it. Also Circuit City announced a $114 million after-tax loss, and Variety estimated the total loss on the scheme was around $337 million. Over the next two years the DIVX system was phased out. Customers could still view all their DIVX discs and were given a $100 refund for every player that was purchased before June 16, 1999. All discs that were unsold at the end of the summer of 1999 were destroyed. The program officially cut off access to accounts on July 7, 2001.
Retail, DRM and Tech Innovation Don’t Mix
Due to the conflict of interest between the Circuit City chain, other retailers, licensing and this new format, there was ultimately no way this idea could survive. Circuit City made so many missteps along the way to adoption, the format was doomed from the outset. Even the management should have been able to foresee this event. If Circuit City had spun off the DIVX idea into a separate holding company that Circuit City had founded and enticed other retailers in (to avoid licensing requirements), the standard might have had a chance of surviving. While DRM was a relatively new thing in 1999, consumers could already begin to see how it could become a problem in the way they viewed content with DIVX (and other formats).
The Future of the Movies at Home
Even if DIVX had managed to make the slightest dent in the rental market (hint: it didn’t), the future of Internet streaming movies would have still seen to its demise. Even in 1997, a year before DIVX came to exist, Reed Hastings was in the process of setting up Netflix. By 2002, Redbox led the downfall of Blockbuster through it’s DVD rental kiosks. Some people blame Netflix on the death of Blockbuster, but it is firmly the self-service and low cost nature of the Redbox kiosk that ushered Blockbuster out the door. Yes, Netflix started Blockbuster’s problems, Redbox nailed Blockbuster’s doors shut. Blockbuster simply couldn’t compete with $1 DVD rentals at a time when Blockbuster was still charging sometimes $5 per disc. Netflix chose a per month plan fee with limits and was (and still is) charging well more than $1 per disc that Redbox adopted. In fact, Redbox is still the best deal going for both DVD and Blu-ray rentals, even though their prices have somewhat increased.
Getting back to DIVX, Netflix’s movie streaming, along side Redbox, Amazon, Hulu, Vudu, Crackle, YouTube, YouTube Red and others would have killed the DIVX DVD format anyway. Ultimately, DIVX didn’t have a place in the market or a problem to solve. It was already behind the times when it was introduced by a company that didn’t have the capital to invest in the longevity of such a format.
In short, Circuit City bit off well more than they could chew with DIVX. Today, these DIVX players are essentially worthless for playing DIVX format discs. Because the players could play standard DVD format discs also, this is their only redeeming point. There’s no way to authorize the players or discs as the service has been dismantled. If you have any DIVX discs in your collection, they can no longer be played as there’s no way to authorize the players or discs.
Even today, DVD is so behind the times when compared with UltraHD 4K, even that would have killed DIVX in short order. Ultimately, even if DIVX had managed to survive longer than 1 year on the market, it would have eventually died because of movie streaming services. There was just no way for DIVX to compete with that. However, it died long before that happened simply because of Circuit City.
Final Death of DIVX
The DIVX format supported limited viewings as well as unlimited viewings (DIVX Silver). Limited viewings of a disc were based on your rental period. Unlimited viewings cost more and was known as DIVX Silver. Why this is important is that the players still needed to dial home to verify the viewing of each play of the movie. After June 30, 2001, the DIVX service was shuttered including the dial home feature. For those who had purchased into DIVX Silver for some of their films, they could request a refund before the service was shuttered. This meant that any further viewings of DIVX movies after June 30, 2001 were impossible, rendering the DIVX format and the DIVX portion of the players useless.
The LA Times wrote of DVIX’s failure on June 17, 1999:
But the venture never connected with consumers and represented a major miscalculation of both the market and the video industry by the nation’s second-largest consumer electronics retailer.
The failure of Divx is an embarrassment for Richard Sharp, chief executive of Circuit City Stores. Sharp fought an uphill battle to promote the venture, which became a significant drag on Circuit City’s bottom line.
Sharp declined to comment Wednesday, but the market cheered the decision to junk Divx. Circuit City’s stock closed at $90.38, up $8.38 on the New York Stock Exchange.
A Cautionary Tale
This whole DIVX situation serves as a cautionary tale for early adopters of technology when produced by a company that’s never been in that business. This is particularly a problem considering the DIVX players required so much constant hand-holding with home base. If that home base connection was unavailable (i.e., Circuit City closed the service), the movies would stop working, which is exactly what happened in the end. Why would you, as a consumer, want to buy into a media format that’s so heavily dependent on a third party’s continued success? The other problem is that the players chose to use a phone line instead of phoning home over the Internet. Of course, had the format lived, it would have been relatively trivial to introduce new players that supported Internet always-on capabilities.
The real cautionary tale here is that consumers should never early adopt into entertainment content that relies on phoning home to authorize each viewing. One could argue that Netflix is a form of this, but I’d argue it isn’t. When you use Netflix, the movie is either there or it isn’t. There’s no pulling-the-rug-out-from-under tactics. Meaning, you leave your media sitting for a few months only to find that it will no longer play. Standard DVD movies have never required authorization per play. However, Blu-ray technology has instituted a somewhat similar phone home approach, but so far this hasn’t been an issue. However, should Sony die or the servers cease to exist that enables a specific Blu-ray to function, we could find that Blu-rays become coasters at some point in the future like the DIVX media.
If you happen to own a DIVX player and any DIVX media, know that it’s dead and it’s not coming back. There is really no way to revive it. The decryption keys and the authorization service that allowed each movie to work have long been dismantled. As far as I know, there has never been anyone willing to reverse engineer this phone home service to allow old DIVX media to play. Though, why bother? The movies were mostly of inferior quality. Other than as a novelty of showing a functional DIVX movie off on a YouTube electronics history channel or possibly for nostalgia, there’s no other legitimate reasons to want to watch DIVX movies today.
↩︎
Restore a Mac formatted 6th Gen iPod nano in Windows 7
I recently picked up a sixth generation iPod nano refurbished from Gamestop. When I got home and plugged it into iTunes for Windows 7, iTunes recognized it as a Macintosh formatted iPod and said that it needed to be restored. Here’s where the fun begins.. not. Several things happened after I plugged it in. First, Windows recognized it as drive O: and opened a requester wanting to format the iPod. This format panel stays open until cancelled. Second, when I tried to restore the iPod, iTunes kept showing me error 1436, which is a rather non-descript error that takes you to a mostly generic Apple help page that is only moderately helpful. I take that back, this help page wasn’t helpful at all.
Note, Macintosh formatted iPods cannot be used with Windows. However, Windows formatted iPods can be used on both Windows and Macs. So, this is simply a problem that exists because this iPod was originally formatted on a Mac. Such stupid issues that cause such time wasting problems.
How did the first restore go?
It didn’t. I realized the above mentioned Windows disk format panel had the iPod open and the 1436 error was due to this. However, that was just the beginning of the problems. When I cancelled that panel and I tried the restore again, I got a different issue. Basically, iTunes opens a progress bar that keeps moving without any progress. I wasn’t sure if this progress panel was normal or abnormal. Although, I suspected abnormal after 3 minutes without any changes. So, I began searching for how long an iPod restore should take. I found that restore should complete in only a few minutes (less actually). So, I knew something was wrong when it wasn’t making any progress.
Disk Mode
It was clear that iTunes wasn’t going to restore this iPod through its normal means. I began searching on the net for how to recover this iPod and ran into a site that led me to Apple’s How to put an iPod in Disk Mode help page. This page is actually very useful and where the 1436 error page should have led me but didn’t.
What is Disk Mode? Disk Mode puts the iPod into a state that allows it to be formatted as a disk. Well, you don’t really want to format it. Instead, in Disk Mode, it gets rid of all that pesky Macintosh formatting garbage and actually lets you restore it properly. For the sixth gen iPod nano, to put it in Disk Mode, press and hold the power and volume down buttons until the screen turns black and the Apple logo appears. When you see the Apple logo, press and hold both volume up and down buttons until the iPod shows a white screen. This is the Disk Mode screen.
Recovering
At this point, I plugged the iPod back in with iTunes running and iTunes saw that the iPod was ‘corrupted’ and asked to restore it. Well, the restoration this time went like a champ. No issues at all. However, after I restored it, I did have to close out of iTunes and restart iTunes. Until I did that, iTunes kept telling me that the iPod was in ‘Recovery Mode’ even though I knew that it wasn’t based on the screen of the iPod. After restarting iTunes, that stopped and it finally recognized the iPod as new and let me put music on it. Yay!
So, there you have it. Although, it should have been as simple as plug-in and restore. But, Apple had to make this a chore because of the PC vs Mac formatting thing. Seriously, is that even necessary?
Design
Let me take a moment to commend Apple on this design of this iPod nano. When the first long skinny nano was first released, I thought it was kind of cool, but not worth it. Then the smaller squatty nano arrived and I liked that design so much that I bought one. I got my use out of that and eventually bought an iPod touch. However, the iPod touch isn’t useful in all circumstances and I wanted something smaller and lighter. When this nano was released, I always thought it was a great idea and well executed save for the fact that it has no application support. So, here’s where Apple dropped the ball on this one.
The size and weight is awesome. The look is great, especially if you get a watch band. It just needed a refresh to add a few more features like Bluetooth, video (although, not really necessary in my book) and apps support. I loved the square display because this is the exact image ratio of CD covers. So, it was the perfect marriage between a music player and a user interface. Some people complained that the touch display was overkill. Perhaps, but I always liked it, but I have never needed one of these. I still don’t really need one. The reason I bought one is because Apple has discontinued this model in lieu of it’s bigger screen cousin.
The new nano, however is neither nano in size nor is it really that small. This nano was the perfect size and perfect shape. It truly deserved the name nano. However, the new nano is really not deserving of that name. The screen is too big and it’s really just a dumbed down iPod touch. Yes, the new nano has video capabilities, but so what? I don’t plan on ever loading video on it. Without WiFi or streaming mechanisms, there’s no point. I realize Apple wants to enrich their ecosystem (read, sell more videos to people), but this isn’t the device to do it. In fact. this latest nano design to ship late 2012 is really not that great looking. I feel that it’s stepping too far into the same territory as the iPod touch. So, why do this? It’s also bigger, bulkier and likely heavier. The battery life is probably shorter even. It’s no longer a small portable player.
The 6th generation iPod nano (this one I just bought) is truly small and light. It can go just about anywhere and has a built-in clip even! It lacks some features, yes, but for a music player I certainly don’t miss them. If you’re thinking of buying a 6th generation iPod nano, you should do it now while the Apple outlet still has them in stock. Yes, they are refurbished, but they’re still quite spectacular little music players. However, don’t go into the purchase expecting the feature-set of an iPhone or an iPod touch. It’s not here. If you go into the purchase thinking it’s an iPod shuffle with a display, then you won’t be disappointed with the purchase.
Apple’s ever changing product line
What I don’t get about Apple is removing a product from its product lineup that clearly has no competition in the marketplace at all, let alone having no competition even within its own product lineup. Yet, here we are. Apple is dropping the 6th generation design in lieu of the 7th generation design that’s bigger and bulkier (and likely heavier). In fact, it looks a lot like a smaller dumbed-down iPod touch.
In reality, the 7th gen nano is so close to becoming a tiny iPod touch clone that it clearly competes with the Touch. This is bad. The 6th generation nano (pictured above) in no way competes with the iPod touch, other than it has a tiny touch screen. The 6th generation nano design clearly still has a place in Apple’s lineup. I just don’t get why they dump products from their lineup and replace them with designs that aren’t likely to sell better (0ther than to those people who complained you couldn’t play video on the 6th gen nano). The 6th gen nano is great for the gym or while running. However, after this newest nano is introduced, if you want a square sized small music player, you have to get a shuffle with no display. The bigger bulkier 7th gen design just won’t work for most activity use cases. Apple, your design team needs to better understand how these devices are actually being used before you put pen to paper on new designs, let alone release them for public consumption. Why is it always just one device? Why can’t you have both in the product lineup?
Of course, if they had retained an updated 6th gen model along with adding the 7th gen model, then that would make a lot more sense. Removing the older model in lieu of this one, this is not a replacement design. You can’t wear this one like a watch. So, that whole functionality is gone. What I would like to have seen is two models. A 6th gen revamped to add more features like Bluetooth and perhaps a camera and, at the same time, introducing this new video capable model. The updated 6th gen doesn’t need to playback movies, the screen is too tiny for that. In fact, the screen on this new 7th gen model is too tiny for that. Even the iPod touch is too tiny for watching movies, in practicality. It’s not until you get to the iPad does watching a movie even become practical. In a pinch, yes you could watch a video or movie, but you’d be seriously straining your eyes. I’d rather do that (or rather, not strain my eyes) with a much bigger screen. No, an updated square-format touch screen iPod is still very much necessary in the lineup. I understand Apple’s need for change here, but not for the use case that’s now lost with this 7th generation iPod. Sometimes, Apple just doesn’t seem to get it. This is just one of a new series of cracks in the armor that is the new Jobs-less era Apple. Welcome to the new Apple folks.
↩︎
How to format NTFS on MacOS X
This article is designed to show you how to mount and manage NTFS partitions in MacOS X. Note the prerequisites below as it’s not quite as straightforward as one would hope. That is, there is no native MacOS X tool to accomplish this, but it can be done. First things first:
Disclaimer
This article discusses commands that will format, destroy or otherwise wipe data from hard drives. If you are uncomfortable working with commands like these, you shouldn’t attempt to follow this article. This information is provided as-is and all risk is incurred solely by the reader. If you wipe your data accidentally by the use of the information contained in this article, you solely accept all risk. This author accepts no liability for the use or misuse of the commands explored in this article.
Prerequisites
Right up front I’m going to say that to accomplish this task, you must have the following prerequisites set up:
- VirtualBox installed (free)
- Windows 7 (any flavor) installed in VirtualBox (you can probably use Windows XP, but the commands may be different) (Windows is not free)
For reading / writing to NTFS formatted partitions (optional), you will need one of the following:
- For writing to NTFS partitions on MacOS X:
- Tuxera NTFS (not free) or
- ntfs-3g (free)
- For reading from NTFS, MacOS X can natively mount and read from NTFS partitions in read-only mode. This is built into Mac OS X.
If you plan on writing to NTFS partitions, I highly recommend Tuxera over ntfs-3g. Tuxera is stable and I’ve had no troubles with it corrupting NTFS volumes which would require a ‘chkdsk’ operation to fix. On the other hand, ntfs-3g regularly corrupts volumes and will require chkdsk to clean up the volume periodically. Do not override MacOS X’s native NTFS mounter and have it write to volumes (even though it is possible). The MacOS X native NTFS mounter will corrupt disks in write mode. Use Tuxera or ntfs-3g instead.
Why NTFS on Mac OS X?
If you’re like me, I have a Mac at work and Windows at home. Because Mac can mount NTFS, but Windows has no hope of mounting MacOS Journaled filesystems, I opted to use NTFS as my disk carry standard. Note, I use large 1-2TB sized hard drives and NTFS is much more efficient with space allocation than FAT32 for these sized disks. So, this is why I use NTFS as my carry around standard for both Windows and Mac.
How to format a new hard drive with NTFS on Mac OS X
Once you have Windows 7 installed in VirtualBox and working, shut it down for the moment. Note, I will assume that you know how to install Windows 7 in VirtualBox. If not, let me know and I can write a separate article on how to do this.
Now, go to Mac OS X and open a command terminal (/Applications/Utilities/Terminal.app). Connect the disk to your Mac via USB or whatever method you wish the drive to connect. Once you have it connected, you will need to determine which /dev/diskX device it is using. There are several ways of doing this. However, the easiest way is with the ‘diskutil’ command:
$ diskutil list /dev/disk0 #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *500.1 GB disk0 1: EFI 209.7 MB disk0s1 2: Apple_HFS Macintosh HD 499.8 GB disk0s2 /dev/disk1 #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *2.0 TB disk1 /dev/disk2 #: TYPE NAME SIZE IDENTIFIER 0: Apple_partition_scheme *119.6 MB disk2 1: Apple_partition_map 32.3 KB disk2s1 2: Apple_HFS VirtualBox 119.5 MB disk2s2
Locate the drive that appears to be the size of your new hard drive. If the hard drive is blank (a brand new drive), it shouldn’t show any additional partitions. In my case, I’ve identified that I want to use /dev/disk1. Remember this device file path because you will need it for creating the raw disk vmdk file. Note the nomenclature above: The /dev/disk1 is the device to access the entire drive from sector 0 to the very end. The /dev/diskXsX files access individual partitions created on the device. Make sure you’ve noted the correct /dev/disk here or you could overwrite the wrong drive.
Don’t create any partitions with MacOS X in Disk Utility or in diskutil as these won’t be used (or useful) in Windows. In fact, if you create any partitions with Disk Utility, you will need to ‘clean’ the drive in Windows.
Creating a raw disk vmdk for VirtualBox
This next part will create a raw connector between VirtualBox and your physical drive. This will allow Windows to directly access the entire physical /dev/disk1 drive from within VirtualBox Windows. Giving Windows access to the entire drive will let you manage the entire drive from within Windows including creating partitions and formatting them.
To create the connector, you will use the following command in Mac OS X from a terminal shell:
$ vboxmanage internalcommands createrawvmdk \ -filename "/path/to/VirtualBox VMs/Windows/disk1.vmdk" -rawdisk /dev/disk1
It’s a good idea to create the disk1.vmdk where your Windows VirtualBox VM lives. Note, if vboxmanage isn’t in your PATH, you will need to add it to your PATH to execute this command or, alternatively, specify the exact path to the vboxmanage command. In my case, this is located in /usr/bin/vboxmanage. This command will create a file named disk1.vmdk that will be used inside your Windows VirtualBox machine to access the hard drive. Note that creating the vmdk doesn’t connect the drive to your VirtualBox Windows system. That’s the next step. Make note of the path to disk1.vmdk as you will also need this for the next step.
Additional notes, if the drive already has any partitions on it (NTFS or MacOS), you will need to unmount any mounted partitions before Windows can access it and before you can createrawvmdk with vboxmanage. Check ‘df’ to see if any partitions on drive are mounted. To unmount, either drop the partition(s) on the trashcan, use umount /path/to/partition or use diskutil unmount /path/to/partition. You will need to unmount all partitions on the drive in question before Windows or vboxmanage can access it. Even one mounted partition will prevent VirtualBox from gaining access to the disk.
Note, if this is a brand new drive, it should be blank and it won’t attempt to mount anything. MacOS may ask you to format it, but just click ‘ignore’. Don’t have MacOS X format the drive. However, if you are re-using a previously used drive and wanting to format over what’s on it, I would suggest you zero the drive (see ‘Zeroing a drive’ below) as the fastest way to clear the drive of partition information.
Hooking up the raw disk vmdk to VirtualBox
Open VirtualBox. In VirtualBox, highlight your Windows virtual machine and click the ‘Settings’ cog at the top.
- Click the Storage icon.
- Click the ‘SATA Controller’
- Click on the ‘Add Hard Disk’ icon (3 disks stacked).
- When the ? panel appears, click on ‘Choose existing disk’.
- Navigate to the folder where you created ‘disk1.vmdk’, select it and click ‘Open’.
- The disk1.vmdk connector will now appear under SATA Controller
You are ready to launch VirtualBox. Note, if /dev/disk1 isn’t owned by your user account, VirtualBox may fail to open this drive and show an error panel. If you see any error panels, check to make sure no partitions are mounted and then check the permissions of /dev/disk1 with ls -l /dev/disk1 and, if necessary, chown $LOGNAME /dev/disk1. The drive must not have any partitions actively mounted and /dev/disk1 must be owned by your user account on MacOS X. Also make sure that the vmdk file you created above is owned by your user account as you may need to become root to createrawvmdk.
Launching VirtualBox
Click the ‘Start’ button to start your Windows VirtualBox. Once you’re at the Windows login panel, log into Windows as you normally would. Note, if the hard drive goes to sleep, you may have to wait for it to wake up for Windows to finish loading.
Once inside Windows, do the following:
- Start->All Programs->Accessories->Command Prompt
- Type in ‘diskpart’
- At the DISKPART> prompt, type ‘list disk’ and look for the drive (based on the size of the drive).
- Note, if you have more than one drive that’s the same exact size, you’ll want to be extra careful when changing things as you could overwrite the wrong drive. If this is the case, follow these next steps at your own risk!
DISKPART> list disk
Disk ### Status Size Free Dyn Gpt -------- ------------- ------- ------- --- --- Disk 0 Online 40 GB 0 B Disk 1 Online 1863 GB 0 B *
- In my case, I am using Disk 1. So, type in ‘select disk 1’. It will say ‘Disk 1 is now the selected disk.’
- From here on down, use these commands at your own risk. They are destructive commands and will wipe the drive and data from the drive. If you are uncertain about what’s on the drive or you need to keep a copy, you should stop here and backup the data before proceeding. You have been warned.
- Note, ‘Disk 1’ is coincidentally named the same as /dev/disk1 on the Mac. It may not always follow the same naming scheme on all systems.
- To ensure the drive is fully blank type in ‘clean’ and press enter.
- The clean command will wipe all partitions and volumes from the drive and make the drive ‘blank’.
- From here, you can repartition the drive as necessary.
Creating a partition, formatting and mounting the drive in Windows
- Using diskpart, here are the commands to create one partition using the whole drive, format it NTFS and mount it as G: (see commands below):
DISKPART> select disk 1
Disk 1 is now the selected disk
DISKPART> clean
DiskPart succeeded in cleaning the disk.
DISKPART> create partition primary
DiskPart succeeded in creating the specified partition.
DISKPART> list partition
Partition ### Type Size Offset ------------- ---------------- ------- ------- * Partition 1 Primary 1863 GB 1024 KB
DISKPART> select partition 1
Partition 1 is now the selected partition.
DISKPART> format fs=ntfs label="Data" quick
100 percent completed
DiskPart successfully formatted the volume.
DISKPART> assign letter=g
DiskPart successfully assigned the drive letter or mount point.
DISKPART> exit
Leaving DiskPart...
- The drive is now formatted as NTFS and mounted as G:. You should see the drive in Windows Explorer.
- Note, unless you want to spend hours formatting a 1-2TB sized drive, you should format it as QUICK.
- If you want to validate the drive is good, then you may want to do a full format on the drive. New drives are generally good already, so QUICK is a much better option to get the drive formatted faster.
- If you want to review the drive in Disk Management Console, in the command shell type in diskmgmt.msc
- When the window opens, you should find your Data drive listed as ‘Disk 1’
Note, the reason to use ‘diskpart’ over Disk Management Console is that you can’t use ‘clean’ in Disk Management Console, this command is only available in the diskpart tool and it’s the only way to completely clean the drive of all partitions to make the drive blank again. This is especially handy if you happen to have previously formatted the drive with MacOS X Journaled FS and there’s an EFI partition on the drive. The only way to get rid of a Mac EFI partition is to ‘clean’ the drive as above.
Annoyances and Caveats
MacOS X always tries to mount recognizable removable (USB) partitions when they become available. So, as soon as you have formatted the drive and have shut down Windows, Mac will likely mount the NTFS drive under /Volumes/Data. You can check this with ‘df’ in Mac terminal or by opening Finder. If you find that it is mounted in Mac, you must unmount it before you can start VirtualBox to use the drive in Windows. If you try to start VirtualBox with a mounted partition in Mac OS X, you will see a red error panel in VirtualBox. Mac and Windows will not share a physical volume. So you must make sure MacOS X has unmounted the volume before you start VirtualBox with the disk1.vmdk physical drive.
Also, the raw vmdk drive is specific to that single hard drive. You will need to go through the steps of creating a new raw vmdk for each new hard drive you want to format in Windows unless you know for certain that each hard drive is truly identical. The reason is that vboxmanage discovers the geometry of the drive and writes it to the vmdk. So, each raw vmdk is tailored to each drive’s size and geometry. It is recommended that you not try to reuse an existing physical vmdk with another drive. Always create a new raw vmdk for each drive you wish to manage in Windows.
Zeroing a drive
While the clean command clears off all partition information in Windows, you can also clean off the drive in MacOS X. The way to do this is by using dd. Again, this command is destructive, so be sure you know which drive you are operating on before you press enter. Once you press enter, the drive will be wiped of data. Use this section at your own risk.
To clean the drive use the following:
$ dd if=/dev/zero of=/dev/disk1 bs=4096 count=10000
This command will write 10000 * 4096 byte blocks with all zeros. This should overwrite any partition information and clear the drive off. You may not need to do this as the diskpart ‘clean’ command may be sufficient.
Using chkdsk
If the drive has become corrupted or is acting in a way you think may be a problem, you can always go back into Windows with the data1.vmdk connector and run chkdsk on the volume. You can also use this on any NTFS or FAT32 volume you may have. You will just need to create a physical vmdk connector and attach it to your Windows SATA controller and make sure MacOS X doesn’t have it mounted. Then, launch VirtualBox and clean it up.
Tuxera
If you are using Tuxera to mount NTFS, once you exit out of Windows with your freshly formatted NTFS volume, Tuxera should immediately see the volume and mount it. This will show you that NTFS has been formatted properly on the drive. You can now read and write to this volume as necessary.
Note that this method to format a drive with NTFS is the safest way on Mac OS X. While there may be some native tools floating around out there, using Windows to format NTFS will ensure the volume is 100% compliant with NTFS and Windows. Using third party tools not written by Microsoft could lead to data corruption or improperly formatted volumes.
Of course, you could always connect the drive directly to a Windows system and format it that way. ;)
Random Thoughts - Randocity! 
2 comments