This article is more than 1 year old
How Windows NTFS finally made it into Linux
Microsoft's New Technology File System has been with us for decades and at long, long last it's going to be fully supported for penguins
Opinion Love it or hate it, Linux users in a Windows world must deal with Microsoft's New Technology File System (NTFS). This has always been a pain in the rump. Even after Microsoft finally gave up on its anti-Linux rhetoric and released its patents to the open-source community and expressively opened up its exFAT patents, we still couldn't get into NTFS.
Things have changed. Starting with the Linux 5.15 kernel, NTFS is finally being fully supported in Linux. This journey hasn't been easy.
Microsoft introduced NTFS, a proprietary – naturally – journaling file system in Windows NT 3.1 in 1993. From there, it replaced 1977's File Allocation Table (FAT) file system across Windows.
Now just because something is proprietary doesn't mean that Linux and open-source developers can't reverse-engineer it or finagle some way of using it. Just see what the Samba crew has done over the years with Server Message Block (SMB) and the Wine folks, who have got numerous Windows applications to run on the *nix operating system family. NTFS, however, proved to be much more difficult to port over.
Oh, there were ways to work with NTFS drives from Linux, but it wasn't pretty. There were two main earlier open-source Linux NTFS drivers: NTFS-3G and the Captive NTFS driver.
NTFS-3G, which was introduced in 2007 can work in Linux as a userspace program with the Filesystem in Userspace (FUSE). It has a reputation for being slow. As Linus Torvalds wrote on the Linux Kernel Mailing List (LKML) recently, the "user-space FUSE thing does indeed work reasonably well. It performs horribly badly".
NTFS-3G's creator and CTO of Tuxera, its parent company, Szabolcs Szakacsits, however, told Torvalds that a better review of NTFS-3G and the new Linux kernel NTFS driver will show the "user space ntfs-3g was about 21% faster overall than the kernel space ntfs3." That said, Szakacsits added that "Ntfs-3g always aimed for stability, features, interoperability, and portability, not for best possible performance." He also added "Userspace drivers can have major disadvantages for certain workloads" but then asksed "how relevant are those for NTFS users?"
You can find out for yourself soon by testing the userspace NTFS-3G against the new Linux kernel NTFS3 driver.
The Captive NTFS driver could read and write to NTFS. To pull this trick off, however, it used the original Windows ntfs.sys driver. Using a proprietary driver in open-source software is always troublesome, especially back in those bad old days. In the event, the project didn't last for long. The code itself is still around, but the project itself is long dead.
While the open-source community was working on these projects, the company Paragon was taking a different approach. Its Microsoft NTFS for Linux by Paragon Software used the proprietary Paragon File System Link, a cross-platform file system driver to read and write from NTFS drives. The program name says it all. Paragon produced its software in partnership with Microsoft.
The result was a fast, efficient, and, yes, proprietary NTFS Linux system. But, while Paragon did well with this as a business for years, Microsoft was no longer the proprietary powerhouse it had been under Bill Gates and Steve Ballmer. Today's Microsoft has realized it was on the wrong side of history with open source.
What was Paragon to do with a pro-open-source Microsoft? Well, at first, the company rapidly went into the first three stages of grief: Denial and isolation, anger, and bargaining. In May 2020, it published a piece entitled The hidden cost of "free" exFAT, where you can see all three stages on display.
By September 2020, Paragon had moved on to the final stage: Acceptance. The company "contributed the read-write NTFS kernel driver under the GPL to the Linux community, for hopeful inclusion in due time to the mainline kernel.
That was wonderful news… except for this one little thing. The NTFS3 code was in no way, shape, or form ready to be added to the kernel. That's the thing with proprietary code – and why I'm cynical about the idea that Microsoft could simply open-source, say, all of Windows 7 – it's often badly written. Unlike open-source, where everyone can see your work, proprietary software can hide its sins from watchers.
First, the code, with 27,000 lines, was much too big. Linux kernel developer Nikolay Borisov wrote: "So, how exactly do you expect someone to review this monstrosity?"
Good question. The answer: You can't.
Besides, as Linus Torvalds said, it would be nice if Paragon would, you know, "actually submit it."
Would it be too much to ask for "Paragon [to] just make a git pull request for it"?
Paragon, to no surprise, didn't know how to deal with the Linux kernel e-mail and Git-driven workflow. Mea culpa, Paragon agreed that it "will be maintaining this implementation," but that "we'll need several days to prepare a proper pull request before sending it to you."
Torvalds was glad to hear this, but he admonished the Paragon developers: "The one other thing I do want when there are big new pieces like this being added is to ask you to make sure that everything is signed-off properly, and that there is no internal confusion about the GPLv2 inside Paragon, and that any legal people, etc. are all aware of this all and are on board. The last thing we want to see is some 'Oops, we didn't mean to do this' brouhaha six months later."
Over a year later, they didn't have a GPL fight. Thank goodness for small miracles. But, after a long, messy process, NTFS3, and thus NTFS, is finally in the Linux 5.15 kernel release candidate. If it all goes well, you can expect to see this kernel arrive in time for Halloween. Feel free to make a Microsoft Halloween documents joke if you like, but with Microsoft's tacit blessing, NTFS will finally be fully and efficiently within Linux.
Who would have dreamed that – even five years ago after Microsoft claimed it loved Linux? We live in interesting times. ®
Updated at 2238 UTC on 14 October 2021 to add
The article has been updated with information from Linus Torvalds and NTFS-3G author Szabolcs Szakacsits.
Biting the hand that feeds IT
