I have a newly created NTFS partition (external drive, connected via FireWire 400). Copying files to it seems to work at a desired speed, but with high CPU load of up to 63% for the mount.ntfs process using the latsest 1.2712 version with kernel 2.6.24.19

Is this normal? Is there anything I can do to lower the load?

Hi,

If you experience this when copying a lot of small files sitting on a long path, you can make a try with the version available on

http://pagesperso-orange.fr/b.andre/security.html

This version caches directory attributes, which reduces the CPU usage significantly. You need not use the ownerships and permissions features.

Regards

Jean-Pierre

Actually, it happens even when I copy a single file that takes more than a few seconds to go through (>100MB)

Please see: http://www.ntfs-3g.org/support.html#cpu100

I am using Gentoo linux on x86 Intel P4 2.8Ghz CPU with 2G memory and kernel version 2.6.25

I have noticed, since I started fiddling with a disk image and VirtualBox, that large files (40G) on NTFS partitions, under linux use insane amounts of CPU when creating the large files.

For example, I have a 40G virtual disk image - it works how I want, so I make a backup by copying it. I use command line 'cp' to perform the copy.

During the copy, mount.ntfs-3g soaks up over 60% of my Intel P4 2.8Ghz Hyperthreaded CPU.

I have 2 NTFS partitions and the behavior is the same, it doesn't matter which I write to, the usage is the same.

I am running ntfs3g version 1.2812 (I am using ~x86) and the disks on which the NTFS systems exist, are SATA.

Writing large amounts to my other XFS filesystems, even the ones that are software raided, do not use so much CPU.

The NTFS partitions are large and have only been created in the last few months and do not contain very much data.
They are virtually unused by windows - I formatted them as NTFS so I had the option of using them in windows.

I have checked the performance FAQ and none seems to apply to me.

Is there anything I can do to obtain logs/performance statistics that can be helpful?

...Lyall

Explained here: http://forum.ntfs-3g.org/viewtopic.php?p=3824#3824

The post you pointed me to would appear to be with regard to sparse files.

The files I am copying are not sparse, they are 40GB pre-allocated (actually, they are dd images of physical disks).

The NTFS partition I am am duplicating the files on (making a copy in the same dir as the source) contains only 487 files, in total with 56% free space (327G used out of 580G), most of that taken up by about 7 40G disk images plus a few smaller ones.

I see the same CPU behavior when I copy the 40G from the mostly empty filesystem into an NTFS which has 376G of 580G available but contains over 35,000 files (music collection of about 10,000 spread over many directories plus about 24,000 small <100k images - webcam time lapse pics, all in a single directory).

I am ok if this high cpu behavior is expected. I am just grateful that we even have an NTFS driver for linux, that works!

If there are any linux tools that provide output which would help confirm this, I am happy to execute and provide the output.

...Lyall

cp creates sparse files (autodetection). Solution is to use the --sparse=never option.

dd uses the most inefficient block size possible: http://ntfs-3g.org/support.html#dd
The solution is to use the 'bs=1M' option.

If it's still slow then please wait for the stable, high-performance big block write support.

The recursive directory copy case is also known and will be addressed.

Thanks!

I have the exact same problem.

9-10 MB/s write speed (USB 2.0)
Uses as much cpu as it can get
Didn't do this with my old harddrive
Sparse/non-sparse doesn't matter
No fragmentation whatsoever (thanks to paragon total defrag)
blksize=4096

This was fixed by building the latest stable release from source

Nope

I am using 2009.3.8 ntfs-3g on my Gentoo using "Linux version 2.6.29-gentoo-r5 (root@lyalls-pc) (gcc version 4.3.2 (Gentoo 4.3.2-r3 p1.6, pie-10.1.5)"

It has taken over an HOUR (so far) to copy from my xfs filesystem to my ntfs-3g filesystem (which was checked by windows only the other day), located on a SATA 1TB drive.
I am copying a 6G file using 'cp --sparse=always'.
cp is not using any CPU.
ntfs-3g is chewing an entire core of my quad core Intel CPU and has only copied 2.2G after an hour - 4G to go!

I copied the file FROM ntfs to my xfs filesystem in minutes, using exactly the same copy command.

I am not shy on memory (4G), the disk is new, modern CPU, basically, ntfs-3g sucks big time when writing big files.

...Lyall

Hi,


You might want to use the Advanced ntfs-3g. Creating sparse files has been improved significantly (http://pagesperso-orange.fr/b.andre/adv ... fs-3g.html)

Using a quad-core will not help in your situation, ntfs-3g is single-threaded.

How full is your target partition ?

Regards

Jean-Pierre

My target partition is 48% full, after the copy completes. It's a 580G partition (one of 4 primary partitions on a 1TB disk).

Using Gentoo, I don't have an option for using ntfs-3g any newer than 2009.3.8 - unless I create the the ebuild myself - something I have limited experience with.

FYI: Copying the 6G file from ntfs-3g to XFS takes 5 minutes, 41 seconds. From XFS to ntfs-3g, it takes hours.

The following tests have been done with cp, default parameters, on my Gentoo 32 bit system using Intel quad core.

Copying a file of 2.1G of random data takes 1 minute 26 seconds from XFS to ntfs-3g.
Copying the same 2.1G of random data from ntfs-3g to XFS takes 1 minute 8 seconds.

Copying a file of 2.8G of NULLS trailed by the output of 'date' from XFS to ntfs-3g takes 1 minute 32 seconds
Copying the same 2.8G of NULLS and trailing date from ntfs-3g to XFS takes 1 minute 28 seconds.

For a 1.8G file which contains about 48K of nulls, then the output of 'date', repeated until reaching the 1.8G length, copying from XFS to ntfs-3g takes 1 minutes 2 seconds.
Copying that same file from ntfs-3g to XFS takes 56 seconds.

All this sounds really good, but when I try a real life example - in this case, copying a Virtual Machine disk image - things go to hell in a hand basket.

I copy a 6G vdi file and it gets to about 500MB then slows to a crawl. Watching it, it's in the order of just over 1MB per second, whilst using 100% of one core - constantly. No I/O Wait, 75% system Idle - nothing else happening on either the source nor ntfs target. I note that the CPU started out at about 33% (typical for the other tests) until the file hit the 500 odd MB mark, then CPU sky rocketed.

If I can help by supplying any data or running some sort of diagnostic, I am happy to do so, all you have to do is tell me what to do - so long as it does not involve me having to trash my system :)

...Lyall

This issue is well-known, documented and solved. For a start please see: http://ntfs-3g.org/support.html#slowwrite

What could you do?

1. Use a "proper" NTFS defragmenter (unfortunately I have no hint). Defrag + copy must be much faster than waiting to end the copy.
2. Use the Advanced NTFS-3G driver. The next stable NTFS-3G will have the relevant performance improvements.
3. The commercial NTFS-3G driver has huge additional performance improvements (e.g. the maximum write speed is 1.8 GB/sec) and hopefully it can be released very soon.

Regards, Szaka

I am using 2009.3.8, which, according to the link, means that should solve my problems.
I can't use the advanced version without a fair bit of effort to create an ebuild, gentoo does not have an ebuild for the advanced version.
I am copying a VM Image, but the reason I am copying it is because I have just gone through a whole 'zero unused blocks' in an attempt to reduce the storage requirements.

As noted in my previous post, I have no problems with files of random data and I created, what I thought, would be a very sparse file, which worked fine.

As an additional note, I am copying my 6G Virtual Machine image to a USB Key, which is also formatted in NTFS.
It exhibited the fast initial startup and at about 420MB, CPU shot up and the whole thing slowed down to around half a megabyte per second.
I note that it's activity light flashes about every 5 seconds then 15 seconds then another 5 seconds then another 15 seconds - pretty much consistently.
I would expect a pretty much constant flickering as data is written - it does flash constantly with smaller files and pretty much flashed constantly up till the 420MB point.
I guess I won't be turning off my computer tonight, as I try to complete these two copies - both concurrently, chewing 2 CPU cores at 100%.
It's interesting to note that the 16G USB Key seems to be gaining, in terms of total copy throughput, when compared to the 580GB NTFS partition. It's as though the driver is building up some sort of list and doing a sequential search, rather than an optimised search and doing this search *very* often.

Something is really quite wrong here.

Everything works as expected. You use 'cp --sparse=always' when it was suggested above to use 'cp --sparse=never'. You're just making your situation much worse doing the opposite which was recommended.

You also don't need ebuild, only

./configure
make unstall

Removal:

make uninstall

You are correct in that cp --sparse=never makes a huge difference. The problem is that the default options for cp is 'sparse=auto' - which will create sparse files - which causes ntfs-3g to, in my mind, fail.

I think it's a bit unreasonable to expect everyone to use --sparse=never when the built in default is auto.

Still, after having tried it, I have a work around that gives me good throughput.

Thank you.

Nobody should use --sparse=never. The default is perfectly fine. Your case is special (copying huge sparse files) and it was only your choice not to use our fix/improvement.

I did not make the choice to not use sparse=never, sparse=auto is the default. The choice I made was to use the default, which caused ntfs-3g to fail.

In my opinion, this is a defect in ntfs-3g - a great product - but a defect nevertheless.

Yes, it's a defect. This is why it was fixed/improved in the Advanced NTFS-3G. This is what I have meant as a choice, not the --sparse=never workaround ;-)