Learning FUSE and Python's C interface

For some time now, I wanted to have a look at Python's interfaces to C libraries. And yesterday I wondered how to implement a filesystem with FUSE. So, in this article we will explore implementing a user-space file-system with Python.

WORK IN PROGRESS

Installing FUSE

I'm using XUbuntu, so the installation instructions will be Debian/Ubuntu specific. And unspectacular:

    ~/fuse-test$ sudo apt-get install fuse

Now, for some mounts.

Using sshfs

What is sshfs? sshfs is a popular FUSE filesystem that allows you to mount a directory from another host, using ssh as transport.

First, let's prepare a directory to mount.

    ~/fuse-test$ ssh remotehost
    murat@remotehost:~$ mkdir test
    murat@remotehost:~$ touch test/testfile
    murat@remotehost:~$ exit

For sshfs there's a package readily available in the Ubuntu apt mirrors. Also, we need a directory to mount to.

    ~/fuse-test$ sudo apt-get install sshfs
    ~/fuse-test$ mkdir mountpoint

Now, we can mount the remote directory test from remotehost.

    ~/fuse-test$ sshfs remotehost:test mountpoint
    ~/fuse-test$ cat /etc/mtab | grep remotehost
    remotehost:test /home/murat/fuse-test/mountpoint fuse.sshfs rw,nosuid,nodev,user=murat 0 0

mtab is looking good. So, what's inside our mount?

    ~/fuse-test$ ls mountpoint
    testfile

Cake for everyone! And unmount, using fusermount.

    ~/fuse-test$ fusermount -u temp-fuse/

Hello World FS

Looking for an easy intermediate step to writing my own file system, I stumbled over the examples that are shipped with FUSE. In particular, there is the Hello World filesystem. Let's make that one work.

The examples are contained in the development package for libfuse, so we install that.

    ~/fuse-test$ sudo apt-get install libfuse-dev
    ~/fuse-test$ ls /usr/share/doc/libfuse-dev/examples/
    cusexmp.c  fioclient.c  fselclient.c  fusexmp_fh.c  hello_ll.c  null.c
    fioc.c     fsel.c       fusexmp.c     hello.c       Makefile
    ~/fuse-test$ cp -r  /usr/share/doc/libfuse-dev/examples ./
    ~/fuse-test$ cd examples/

Compile it.

    ~/fuse-test/examples $ make
  cc -Wall -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse   fusexmp.c -lfuse   -o fusexmp
  # more make output here

We can now go ahead and mount the filesystem.

    ~/examples $ ./hello ../mountpoint/
    cd ../mountpoint/
    ~/fuse-test/mountpoint$ ls
    hello
    ~/fuse-test/mountpoint$ cat hello 
    Hello World!
    ~/fuse-test/mountpoint$ cd ..
    ~/fuse-test$ fusermount -u mountpoint

Let's have a look at the available Python bindings.

by the way: undefined reference to `fuse_main_real'

Here you might encounter the following error. If you don't, skip ahead.

    cc -Wall -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse    -lfuse    fusexmp.c   -o fusexmp
    /tmp/ccUDCZVj.o: In function `main':
    fusexmp.c:(.text+0x677): undefined reference to `fuse_main_real'
    collect2: error: ld returned 1 exit status
    make: *** [fusexmp] Error 1

The issue has been documented on the FUSE mailing list. In Ubuntu 10.10, the linking behaviour of GCC has been changed, now enabling the option --as-needed by default.

For the compiling/linking step above, this is important because libfuse is referenced before the C file. When the symbols of the library, most notably fuse_main_real are checked for references, they are dismissed, because fusexmp.c has not been considered yet.

When you have a look at the Makefile, you'll notice that there are no explicit rules for compiling and linking the C files. The implicit rules to compile and link C files are the culprit here. Of course, they can be overriden, for example like so:

    %: %.c
    $(CC) $(CFLAGS) $@.c $(LDFLAGS) -o $@

Append this to the makefile and you're good to go. Run make again and your example files should now be compiled successfully.

fuse-python

For starters, I'll have a look at the official Python bindings.

Installation

Probably a piece of work: it's ancient. Installation is trivial, though, because it's available as a package in Ubuntu: apt-get install python-fuse. (Yes, the name is correct. The package name follows Debian's naming conventions for python packages.)

After installing, its examples can be found in /usr/share/doc/python-fuse/examples: namely hello.py and xmp.py. They work out of the box:

    ~/fuse-test$ python /usr/share/doc/python-fuse/examples/hello.py mountpoint/
    ~/fuse-test$ cat mountpoint/hello 
    Hello World!

NopFS: The Bare Minimum

The most minimal Filesystem I could come up with, is the following NopFS.

import fuse

fuse.fuse_python_api = (0, 2)


class NopFS(fuse.Fuse):
    pass


def main():
    server = NopFS()
    server.parse()
    server.main()

if __name__ == '__main__':
    main()

It shows nicely, how the API works:

3: Declare the targeted version of the FUSE API,
6: Subclass fuse.Fuse,
13: Call the classes main() function when the module is executed.

It can already be mounted like the previous examples:

~/fuse-test$ python nopfs.py mountpoint

When doing an ls, FUSE notices that this functionality has not been implemented yet:

~/fuse-test$ cd mountpoint && ls
ls: reading directory .: Function not implemented

LsFS: Directory Contents

So, let's make ls work. For that, the following functions are necessary at least: getattr(path) and readdir(path, offset). readdir provides the list of files in the root directory, and getattr gives information on each file, so ls knows whether a path identifies a directory or a file, who owns it, and so on.

By the way, how do you know what methods to implement? Find out in this note about system calls.

Let us add one file some_file and one directory some_dir to the root directory. The following LsFS filesystem does just that.

class LsFS(fuse.Fuse):
    def getattr(self, path):
        st = fuse.Stat()
        st.st_nlink = 1
        if path[1:] == "some_dir" or path == '/':
            st.st_mode = stat.S_IFDIR | 0755
        elif path[1:] == "some_file":
            st.st_mode = stat.S_IFREG | 0644
        else:
            return -errno.ENOENT
        return st

    def readdir(self, path, offset):
        if path == "/":
            for name in [".", "..", "some_file", "some_dir"]:
                yield fuse.Direntry(name)

3: getattr's job is to return a fuse.Stat object filled with information about path. It contains information about type, access rights and times, and so on.
4: fuse.Stat is initialized with “undefined” values. Usually, that boils down to a zero, which is harmless in most cases. Two fields need to be set: st_nlink (number of hardlinks to that inode) and st_mode (type, access rights, …). Here, the hardlinks are initialized wrongly, as for example directories should have at least two hardlinks. But I just want to make the code work.
5: FUSE always provides the absolute path relative to the mount root. That is why the first character, usually a '/' on UNIX, is removed.
6-8: The Stat structure is filled according to the path, making some_dir globally readable and searchable directory, and so on.
10: For anything other, the error code for “unknown file” is returned.
13: readdir should return a generator producing an DirEntry object for each inode in the directory. Note there is another, more low-level mode of operation which uses the offset parameter. More about it in the API docs.

After mounting, an ls reveals:

~/fuse-test$ ls -l mountpoint/
total 0
drwxr-xr-x 2 root root 0 Jan  1  1970 some_dir
-rw-r--r-- 1 root root 0 Jan  1  1970 some_file

Yay!

A “few” things were omitted. At least: Making some_dir's contents actually readable. Setting other stat-related fields. Checking access rights.

By the way, ever run into an Invalid argument error? Weirdly enough, this need not have anything to do with invalid arguments. Read the note on this error to learn more.

For a fuse-python implementation, there are two design choices to be considered.

PyFS: Reading and Writing Files

The methods above we can use to implement the Python filesystem. I will be using the OO approach. Since it's just too much details code, I'll only show the most important snippets here. Refer to the source code of PyFS for the complete implementation. By default, I am importing and making available three modules under /lib:

~/fuse-test$ ls mountpoint/lib/
json  os  sys

The next obvious feature is reading: For that we implement the read method on the file access class.

class FileMapping(object):
    def read(self, size, offset):
        return self._read_from_string(
            self.get_text(),
            size,
            offset,
        )

get_text() retrieves the string representation of the path. For /lib/os/pathsep, it resolves the name and returns :. _read_from_string returns the proper range of the text, taking size and offset into account. (Obviously, this is inefficient. Don't care, though.)

Something very nice that we can do already, is making the files representing the functions executable. For that to work, getattr needs to set the executable bit in the st_mode field and read needs to return an executable file that will call the function and print it's result. Implemented, this looks like so:

~/fuse-test$ mountpoint/lib/os/path/join /this/is/a "path/to" be/joined
/this/is/a/path/to/be/joined

The next thing, we can do is add modules to /lib by appending to the pseudo-file /run/modules. We could use an executable to be called like so: /bin/import re. That is nicely readable, but means we're only doing reading again, (reading the script that will parse the input and add it to the filesystem), which is boring. Also, since this executable would be run as a new process, we'd need inter-process communication, which is annoying. So, what about this: echo "re" >> /run/modules? For this to work, we need getattr to make this file writable, and the write() method needs to process the module name that will be written to it:

    def write(self, buf, offset):
        if not self.append and offset != 0:
            raise IOError(errno.EPERM)
        addlib(buf.strip())
        return len(buf)

First, we make sure that a module is either added to the existing ones, or replaces all of them. Then, the addlib() function adds the module with the given name to the list of modules accessible under /lib. Put together:

~/fuse-test$ ls mountpoint/lib/
json  os  sys
~/fuse-test$ echo re >> mountpoint/run/modules 
~/fuse-test$ ls mountpoint/lib/
json  os  re  sys
~/fuse-test$ mountpoint/lib/re/match ".*(l+).*" hello
<_sre.SRE_Match object at 0x7ff2a94b44e0>

By the way, you might run into “Invalid Argument” trouble here. The good news: You might be innocent; it's a bug! Have a look at this note regarding write errors.

What about echo "re" > /run/modules? This would clear the list of modules first, only leaving the re module mounted afterwards. To support truncating a file, we would need to implement the truncate() on the filesystem object, and ftruncate() on the file class. Since the wanted echo command will use the former, here it is:

    def truncate(self, path, len):
        if path != "/run/modules":
            raise IOError(-errno.EPERM)
        if len != 0:
            raise IOError(-errno.EPERM)
        clear_modules_list()

The first check makes sure, we only clear the module list when called upon the special file. The second check ensure we only do complete truncations. Put together:

~/fuse-test$ ls mountpoint/lib/
json  os  sys
~/fuse-test$ echo re > mountpoint/run/modules 
~/fuse-test$ ls mountpoint/lib/
re

Concluding thoughts

Not maintained.
Code-wise, it's a mess.
API-wise, too. (Either make it pythonic, or don't. Whatever you do, do it right.)
Documentation: None. Examples only.
Buggy in release.

So, that's it for fuse-python. The complete source code of all examples is available on Github.

by the way: Design Choices

Class structure: monolithic vs. composite

Using the monolithic approach, all functionality is provided by overriding methods directly in the subclass. This results in a more direct mapping of the FUSE API.

Using the other approach, functionality to access files and directories is encapsulated in dedicated classes. For each directory or file to be accessed, an object of this class is created. Aside from grouping semantically related code pieces, the object oriented approach allows to store Since this also uses the constructor for “opening”, this

For example, the opendir() function will be implemented directly. Access to files and directories can

Error handling: errno vs. exceptions

The second choice is about how errors are propagated to FUSE. In principle, this is possible by either errno codes or by raising exceptions that contain the codes. In the latter case, the fuse.ErrnoWrapper, a facade which is always added to API functions, will catch the exception and return the errno code.

When choosing the composite approach for the class design, the exception approach should be preferred, because in the constructor of the access classes you cannot return an errno code.

by the way: fuse_python_api

fuse.fuse_python_api = (0, 2) http://sourceforge.net/p/fuse/fuse-python/ci/master/tree/README.new_fusepy_api

RuntimeError: fuse.fuse_python_api not defined.

! Please define fuse.fuse_python_api internally (eg.
! 
! (1)  fuse.fuse_python_api = (0, 2)
! 
! ) or in the enviroment (eg. 
! 
! (2)  FUSE_PYTHON_API=0.1
! 
! ).
!
! If you are actually developing a filesystem, probably (1) is the way to go.
! If you are using a filesystem written before 2007 Q2, probably (2) is what
! you want."

by the way: Tracing System Calls

Apparently, FUSE is aware that the nop filesystem does not support anything yet. What is needed to make ls work? Tracing the system calls of an ls on an empty directory, brings us a step further:

~/fuse-test$ mkdir testdir
~/fuse-test$ strace -v -e file -e \!process ls testdir 2>&1 1>/dev/null
...
stat("testdir", {st_dev=makedev(252, 1), st_ino=5643452, st_mode=S_IFDIR|0775, st_nlink=2, st_uid=1000, st_gid=1000, st_blksize=4096, st_blocks=8, st_size=4096, st_atime=2013/10/31-22:42:45, st_mtime=2013/10/31-22:42:45, st_ctime=2013/10/31-22:42:45}) = 0
openat(AT_FDCWD, "testdir", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
getdents(3, {{d_ino=5643452, d_off=6816274797489594284, d_reclen=24, d_name="."} {d_ino=5636714, d_off=9223372036854775807, d_reclen=24, d_name=".." }}, 32768) = 48
getdents(3, {}, 32768)                  = 0
close(3)                                = 0
...

First, the root dir is analyzed with stat. Then, a file handle to the directory is opened with openat, it's two entries . and .. are retrieved with getdents. Finally, the handle is released with close.

API documentation gives some hints to what these system calls will be mapped to: stat requires a getattr,

class DirectoryMapping(object):
    def __init__(self, path):
        self.path = path

    def readdir(self, offset):
        if self.path == "/":
            for name in [".", "..", "hello"]:
                yield fuse.Direntry(name)
        else:
            raise OSError(errno.ENOENT, os.strerror(errno.ENOENT))

~/fuse-test$ ls mountpoint
ls: cannot access mountpoint: Function not implemented
~/fuse-test$ (cd mountpoint && ls)
ls: cannot access hello: Function not implemented
hello
~/fuse-test$ (cd mountpoint && ls -l)
ls: cannot access hello: Function not implemented
total 0
?????????? ? ? ? ?            ? hello

by the way: Invalid argument

In respect to misspelled names, fuse-python does a bad job at exposing the root cause of the problem. As an example, let us assume a name in LsFS had been mistyped: S_IFREF instead of S_IFREG:

st.st_mode = stat.S_IFREF | 0644

Then, when running ls, we'd get this:

~/fuse-test$ ls -l mountpoint/
ls: cannot access mountpoint/some_file: Invalid argument
total 0
drwxr-xr-x 2 root root 0 Jan  1  1970 some_dir
?????????? ? ?    ?    ?            ? some_file

Contrary to what the reader of that error message might think, the API method's signature was correctly handling the given arguments. In this case, you need good logging to understand the actual error.

by the way: Error on opening a file for writing

When implementing the module import mechanism, I stumbled over the following behavior in the xmp:

~/fuse-test$ echo "world" >> mountpoint/hello
~/fuse-test$ echo "world" > mountpoint/hello
bash: mountpoint/hello: Invalid argument

So, I could successfully write a file, but not truncate it. In the PyFS, I could do neither.

When stracing that, it turns out that opening the file (with truncate flag) did not work.

open("mountpoint/hello", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EINVAL (Invalid argument)

This rang a bell. open() is one of the functions getting special treatment in Fuse.lowwrap(). There are three kinds of return values possible: an errno integer, a FuseFileInfo object and an instance of the file-handling class. The latter being my case.

In the C function open_func(), I found that the attributes keep_cache and direct_io are accessed on the return value.

This, in turn, reminded me about a bugfix, I've seen: Turns out that Môshe van der Sterre fixed this one, too. (Thanks, there!) Commit 7e29c2 makes sure that checking for those attributes does not leave an exception state open (when the attributes don't exist).

Long story short: To fix this, either get the HEAD version of fuse-python, or make sure that the attributes can be resolved on your filehandler object. Like so:

class FileWrapper(object):
    def __init__(self, path, flags, *mode):
        # ...
        self.keep_cache = False
        self.direct_io = False

Obviously, you should set the attributes depending on what your filesystem supports.

fusepy

fusepy provides ctypes based Python bindings to FUSE.

Installation

Can't be installed in a virtualenv. That means: Must install system-wide and can't tst fuse-python and fusepy at the same time, since both use "fuse" as module name. Meh. This is what you get for unintentionally importing the wrong one:

Traceback (most recent call last):
  File "/home/murat/ws/fuse-and-python/phuse/fusepython/pyfs.py", line 238, in 
    class PyFS(fuse.Fuse):
AttributeError: 'module' object has no attribute 'Fuse'

NopFS: The bare minimum

Again, here's the minimal filesystem: doing nothing, but mountable.

import fuse


class NopFS(fuse.Operations):
    pass

if __name__ == '__main__':
    fuse.FUSE(NopFS(), sys.argv[1])

The code is self-explanatory. The differences to fuse-python are worth mentioning, though.

4: We subclass fuse.Operations. As can be seen below, this is not the class we hand control to. fusepy prefers composition over inheritance for separating the filesystem logic from the generic code. Makes things much more understandable for me.
8: We have to retrieve the mountpoint argument ourselves and then pass it to fusepy.

Now, this can be mounted as before (without the -s, though). A ls on the directory reveals something nice about fusepy.

~/fuse-test$ ls mountpoint/
~/fuse-test$ ls -hal mountpoint/
total 4,0K
drwxr-xr-x  2 root  root     0 Jan  1  1970 .
drwxrwxr-x 10 murat murat 4,0K Dez 26 16:28 ..

fusepy comes along with some minimal, but nice defaults. In this case, readdir always returns ['.', '..'] and that's why we're not running into errors immediately.

LsFS: Directory Contents

How do we add a file and a directory with fusepy? It's fairly simple:

class LsFS(fuse.Operations):
    def getattr(self, path, fh=None):
        if path[1:] == "some_dir" or path == '/':
            return dict(
                st_mode=(stat.S_IFDIR | 0755),
                st_nlink=2,
            )
        elif path[1:] == "some_file":
            return dict(
                st_mode=(stat.S_IFREG | 0644),
                st_nlink=1,
            )
        else:
            raise fuse.FuseOSError(errno.ENOENT)

    def readdir(self, path, fh):
        if path == "/":
            return [".", "..", "some_file", "some_dir"]

4: Subclass fuse.Operations

PyFS: Reading and Writing Files

Concluding thoughts

The general command-line options -f, -d and -s cannot be used from the command-line when using fusepy. Instead, they must be passed as boolean keyword arguments to the FUSE constructor: foreground (-f), debug (-d) and nothreads (-s).

if __name__ == '__main__':
    fuse.FUSE(LsFS(), sys.argv[1], foreground=True, debug=True, nothreads=True)

by the way: Common Error Messages

fusermount: failed to open /etc/fuse.conf: Permission denied

The fuse group is missing. Add it and log out and back in.

getattr returns string instead of dict: ~/fuse-test$ ls mountpoint ls: cannot access mountpoint: Bad address you did something wrong (import error, name error, ...) ~/fuse-test$ ls mountpoint ls: cannot access mountpoint: Input/output error