I am getting an error code "-13" on files which may well be encrypted on an ntfs partition. Is it likely that this is the reason?

Christopher

Correct. EFS is not supported at the moment.

I'm trying to use ntfs-3g (on lvm snapshot of a vmware disk image) for backup; as stated above, backup for EFS encrypted files fails with "access denied" at the moment.

Actual decryption of an encrypted file is neither required nor actually wanted behaviour. Instead it would be optimal to backup encrypted data + encryption metadata so a restore of encrypted data + attributes would allow decryption for authorized users in windows.

would the "the backing up ntfs file attributes" thread also cover the encryption metadata? and if so: what's still missing for this usecase be at the moment?

Thanks for your time,

Martin

Hi,


AFAIK most encryption metadata are appended to the file itself : signatures and symmetric encryptions keys encrypted with the public keys of all users allowed to decrypt. The secret keys needed to decrypt the symmetric key is withheld by their owners and are not associated to the file. The only thing missing is probably a flag mentioning this a an EFS encrypted file.

Not exactly, though it would cover the encryption flags mentioned above.

The ability to make a raw copy of data and to restore it, keeping track of the encryption status. For instance in a "tar" format there is nothing to record the encryption format.

Regards

Jean-Pierre

Hi


Just wanting to fix my mislocated plurals which may cause confusion (there is one signature, one symmetric encryption key and several encrypted symmetric encryption keys) :

"AFAIK most encryption metadata are appended to the file itself : the signature and the symmetric encryption key encrypted with the public keys of all users allowed to decrypt. The secret keys needed to decrypt the symmetric key are withheld by their owners and are not associated to the file."

Regards

Jean-Pierre

doesn't seem to be the case - filesize (both in windows and viewed via NTFS-3G) stays the same as for the unencrypted version.
could the encryption header also be an attribute?

I'll see if I can convince the driver to give me access to the raw data first; keeping track of the attributes using a seperate metadata extraction process (using getfattr/setfattr for proof of concept) works just fine for me.

Thanks, Martin

Hi,


It should. The file size is related to bytes returned to an application. You should check the space on disk (with du, for a file of 4095 bytes, are you getting 4 ? - mind the storage alignment)

Regards

Jean-Pierre

Hi,

Yes, I'm getting 4 in this case. both from NTFS-3G and from windows properties (size on disk)

Reading of raw data works after disabling the checks for NAttrEncrypted; mapping of NTFS attributes related to Encryption doesn't work yet, have to spend some time on that

Hi again,


Strange, and complex to backup.
There is a view of the structure of an encrypted file on
http://technet.microsoft.com/en-us/libr ... 57116.aspx
but I can have wrongly interpreted where the "file header" is located.

Regards

Jean-Pierre

Hi,

there's quite a lot of info regarding the fields & data structures of EFS files already present in layout.h

let's see what I manage to break trying to make this visible.

Thanks, Martin

Hi,
OK, update:
* reading system.ntfs_attrib to determine if a file is encrypted works just fine:
400000 is FILE_ATTR_ENCRYPTED

* I've added a new system attribute ntfs_efsinfo that returns the contents of AT_LOGGED_UTILITY_STREAM where NTFS is storing its EFS metadata; it's almost identical to the ntfs_get_ntfs_reparse_data function used for mapping reparse data to an extended attribute.


Looks promising so far - haven't done the 2nd part (adding EFS info to a file) - if this actually works, it's one more thing that clonig a volume via ntfs-3g can handle. I'll be back:-)

Hi,

Sigh - that would have been too easy; first effort doesn't work. Comparing the information from ntfsinfo for the original and the copied inode shows a few obvious differences: FILE_ATTR_ENCRYPTED is set not just for the inode itself but also for a couple of other attributes:



New file:


Any pointers on how to change the missing encryption flags would be apreciated :-)

Thanks, Martin

Hi,
Looks like that was the major problem left.

After adding the AT_ENCRYPTED bit to the flags for the unnamed AT_DATA attribute windows can successfully decrypt the copied file, so backing up/cloning EFS encrypted files actually works.

one difference remains: In windows, the files created on linux don't get get the color coding for encrypted files; instead they show up just like "normal" files - once I select them in file explorer, the color changes and they show up as encrypted.

Currently, the xattr containing the efs info is in the system namespace and does not get returned by listxattr; saving encryped data without the corresponding xattr is however not useful, i.e I wonder if the efs xattr should be listed so rsync/tar w/ xattr support can be used on efs encrypted files?

One known problem so far: handling of encrypted files with resident AT_DATA doesn't seem to work - I end up seeing encrypted data in windows :-(

I've attached a patch with the changes I've made up to now.

Hi,


Congratulations !

So there must be some residual difference. Can you check whether this is related to the flags in parent directory :
a) clone an encrypted file into a directory which was already marked by Windows as encrypted applied to contained objects,
b) move by Windows an encrypted file into another directory not marked for encryption.

It would be reasonable to list the efs info in a listxattr. The ACLs and the reparse point are not because this would lead to contradictions in the saved data, which should not happen with efs.
You have however to check the sequence of actions when restoring by tar or rsync. I suppose you first have to restore the data as if the file was not encrypted, then you set the efs info, which implies the flags.

Have you checked this to be possible on Windows : create a small file (eg less than 100 bytes) and check whether Windows stores the data as resident and encrypted.

Regards

Jean-Pierre

Hi,

Tried that - no difference; the files created by linux show up as normal files and change color once I select them in windows. Tried comparing ntfsinfo data before/after windows access: we get a new attribute (+ a new entry in the attribute list)
I don't really see why existence of an object ID would change how a file is displayed but I'll try to see what happens if I add one..


Good point, have to see how tar/rsync handle this. I tried adding an entry to list in ntfs_fuse_listxattr (src/ntfs3g.c) - looks like I'm trying something stupid here, I get a segfault when using getfattr --dump.

It doesn't. Looks like encrypted data MUST be non-resident. Created a small file, data is resident. encrypt it ==> data is no longer resident. looks like I need to make the data atrribute non-resident when adding efs data as well - should be easy using ntfs_attr_make_non_resident, right?

Thanks for your feedback

Martin

Hi Martin,


Hum, this is an unknown area to me. The object ID could be related to some other data.

Should be easy to fix. But I remember fuse filters non-user extended-attributes, so you might be in a dead end !

Yes, that is the way to do it.

Regards

Jean-Pierre

Hi Jean-Pierre,
I know now that the object ID isn't relevant. The Info used for choosing the directory color in windows comes from the index entry in the directory inode:So far, I haven't set the ENCRYPTED Flag in the Index entry, and I'm not sure yet how to go about that. renaming an encrypted file results in an index entry with correctly set flags btw, so copies made by rsync turn out OK (rsync writes a temp file, restores xattrs to the temp file and finaly renames it). Display color in windows is correct.

You're quite right there. I haven't actually seen where this filtering takes place yet, but "system." attributes definitely don't show up in listxattr while "user." attributes do. Currently I've just changed the attribute name to user.ntfs_efsinfo which does not match the other special attributes. On the other hand, I've just seen that rsync also filters xattrs: system xattrs are never copied, non-user xatrrs are copied only for user root. So I guess puting the attribute into user space might be best after all?

With the attribute in user namesapce and the bug in ntfs_fuse_listxattr fixed, I can now copy efs encrypted files/directories using rsync.

Found that directory inodes can also have an LOGGED_UTILITY_STREAM attribute, even though there's no encrypted data at this level; this seems to tell windows how to treat new files in the directory. Not a problem, just have to make sure I don't try to change a non-existing unnamed AT_DATA attribute for a directory inode.

Making resident $DATA attributes non-resident doesn't seem to be as easy as I thought: "na = ntfs_attr_open(ni, AT_DATA, AT_UNNAMED, 0);" returns an error.
For the call ntfs_attr_make_non_resident I need both a search context and an ntfs_attr. Is the sequence (minus all checks)
reasonable or am I trying something stupid here?

Thanks, Martin

-

Hi Martin,


Which field ? Could it be the attribute flag in the $INDEX_ROOT of parent directory ?

The exact criterion must still be understood. You would not want your files be made unusable by some process (eg chkdisk).

user namespace is meant for user applications and not supposed to be file system dependent. But you might view encryption as an user application service also meaningful on ext3. Anyway you may have no choice.

Which error and what is your sequence of actions ? Has your data been entered before you make the attribute non-resident ? Is this before you set the encryption flags ? You may violate some constraint not applicable to encrypted files.

This is roughly what is done in ntfs_resident_attr_resize(), you probably need to search deeper.

Regards

Jean-Pierre

Hi Jean-Pierre,
I'm talking about the directory entries for the actual files (found in $INDEX_ALLOCATION in my case). The dump in my previous message is one of the index entries from the index block. Each directory entry has a copy of the referenced inodes File attributes. Turns out to be quite easy to fix: I just added a call to NInoFileNameSetDirty, which results in these flags being updated.
Definitely. I'd say we can tag this as understood. Windows currently seems to be quite tolerant about mismatched info in the directory entry and the inode itself but we sure don't want to depend on this.
Looks like I've got no choice. The efs info needs to be preserved across different filesystems (i.e: backup efs encrypted files to ext3 or to xfs). This + fuse filtering of system attributes in listxattr + rsync not replicating system attributes means I'm stuck with a user attribute.
sequence of actions turns out to have been the culprit. I ran into a sanity check in attrib.c (Inode 138 has corrupt attribute flags (0x0 <> 0x4000): Input/output error) because I had set the encrypted flag on the inode too early.
Current code now handles:
- EFS Info at directory level (display color, automaticaly encrypt new files added to the directory)
- add EFS info to files with resident AT_DATA Attribute (make non-resident)
- list efs attribute so tar/rsync can transparently handle it
I'd say this has now reached "works for me" status, i.e I can use ntfs-3g for backups and restore the backed up data even if it's been encrypted using efs.
I've attached a patch with my efs related changes.

Thanks again for your help & input!

Hi again,


Do you mean this field (in red) ?

File attributes: ARCHIVE ENCRYPTED (0x00000000)
Namespace: Win32
Filename: 'encrypted_testfile.txt'


This is a very nice achievement, congratulations.

I would like to make my own tests in order to decide whether I include this in the advanced version together with data compression which shares parts of code (adaptations obviously needed). This is planned for June, so you have time to check the pre-release versions.
Can you post (or deposit on a server) a tar file with sample encrypted files for me to test ? Also a copyright notice to include into the modified source files). You can use the "pm" for confidential matters.

Regards

Jean-Pierre

Hi,

Guess what: it wasn't that easy. I found problems with the last few bytes of files created in linux being corrupted when reading/decrypting them in windows. A new patch should now fix this issue. Also, alternate data streams are now supported. To enable reading of encrypted files, you must specify the mount option "efs_read_raw".

Hi Martin,

Very interesting.


Encryption schemes generally operate on fixed size blocks, so the last block has to be padded to appropriate size. As a consequence, after encryption, part of original information is located beyond the apparent file size.

Sure it was not. Congratulations.

Regards

Jean-Pierre

Hi,

OK; another patch:
* efs_info xattr is just a simple dump of the logged_data_stream attribute
* info about number of padding bytes gets appended to the AT_DATA streams
* couple of bugfixes

Hi Martin,

Good job !

I will test your patch, and let you know what the tests show.

Regards

Jean-Pierre