{"id":38214,"date":"2014-06-19T14:16:23","date_gmt":"2014-06-19T12:16:23","guid":{"rendered":"https:\/\/www.mugaict.com\/synetosviluppo\/dive-filesystems-zfs-part-1\/"},"modified":"2014-06-19T14:16:23","modified_gmt":"2014-06-19T12:16:23","slug":"dive-filesystems-zfs-part-1","status":"publish","type":"post","link":"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/","title":{"rendered":"A dive into filesystems \u2013 ZFS \u2013 part 1"},"content":{"rendered":"<p><span style=\"padding-left:2em;\">I&#8217;m starting a series of posts about ZFS. I will try to provide as much as possible architectural diagrams and code examples. The series is for people wanting to learn more about general filesystem architecture, especially about ZFS (OpenZFS).<\/span><\/p>\n<p><span style=\"padding-left:2em;\">I will restrict myself for now to the implementation and architecture on <a href=\"http:\/\/wiki.illumos.org\/display\/illumos\/About+illumos\">Illumos,<\/a> the open source fork of OpenSolaris.<\/span><\/p>\n<p><span style=\"padding-left:2em;\">Let&#8217;s start with the higher level abstraction, the Virtual File System. Like the most of the Unix systems Illumos uses a framework called Virtual File System (VFS) which represent an abstraction layer under which multiple concrete filesystems can be implemented. One of the first versions of VFS was introduced in SunOS 2.0 with the introduction of NFS (Network file system). It&#8217;s good to know that the implementation on Illumos was actually the first one ever made regarding virtual file systems. The SunOS implementation was the one used in the first commercial version of a Unix operating system <a href=\"http:\/\/en.wikipedia.org\/wiki\/System_V_Release_4\">Unix System V Release 4<\/a>.<\/span><\/p>\n<p>VFS allows almost any objects to be abstracted as files and filesystems. I will mention a few of the file system categories that are in use today:<\/p>\n<ul>\n<li>Storage based filesystems like ZFS, UFS, ext4<\/li>\n<li>Network based filesystems like NFS (Network file system), CIFS (Common Internet File System)<\/li>\n<li>Pseudo filesystems like procfs (\/proc) which gives the ability to map the address space of a process as a series of files or devfs for managing devices<\/li>\n<\/ul>\n<p><span style=\"padding-left:2em;\">Within VFS there are 2 key concepts. The first one is the virtual file which is abstracted as a vnode object and the second one is the virtual file system objects or vfs objects. A vnode provides file related functions and vfs provides file system related functions. In Illumos the vnode operations are defined in usr\/src\/uts\/common\/sys\/vnode.h and contained within the structure vnodeops_t.<\/span><\/p>\n<pre class=\"language-clike line-numbers\">\n<code class=\"language-clike line-numbers\">\n\/*\n * Operations on vnodes.  Note: File systems must never operate directly\n * on a 'vnodeops' structure -- it WILL change in future releases!  They\n * must use vn_make_ops() to create the structure.\n *\/\ntypedef struct vnodeops {\n\tconst char *vnop_name;\n\tVNODE_OPS;\t\/* Signatures of all vnode operations (vops) *\/\n} vnodeops_t;\n<\/code>\n<\/pre>\n<p>To mention some of the vnode operations available: open, close, read, write, seek, sync. All the function signatures are in the VNODE_OPS.<\/p>\n<p><span style=\"padding-left:2em;\">The vnode and vfs functions delegate to appropriate concrete filesystem implementations. All file related functions reach the vfs\/vnode layer through a system call and from there they are directed to the appropriate filesystem implementation. The diagram below show the higher level architecture of the VFS.<br \/>\n<a href=\"\/wp-content\/uploads\/2014\/06\/vfs1.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"\/wp-content\/uploads\/2014\/06\/vfs1.jpg\" alt=\"vfs1\" width=\"1508\" height=\"1067\" class=\"alignnone size-full wp-image-8501\" \/><\/a><\/p>\n<p><span style=\"padding-left:2em;\"><br \/>\nFiles are referenced in process space using file descriptors. File descriptors are integer numbers, specifically a C type int. For example, the 3 standard POSIX file descriptors corresponding to the 3 streams: STDIN &#8211; 0, STDOUT &#8211; 1 and STDERR &#8211; 2. The file descriptors are assigned when the file is opened and then freed when the file is closed.<br \/>\n<\/span><\/p>\n<p><span style=\"padding-left:2em;\">Each process has it&#8217;s own file list. Per process file information in Illumos is kept in a structure called uf_info_t. This can be found in usr\/src\/uts\/common\/sys\/user.h.<\/span><\/p>\n<pre class=\"language-clike line-numbers\">\n<code class=\"language-clike line-numbers\">\n\/*\n * Per-process file information.\n *\/\ntypedef struct uf_info {\n\tkmutex_t\tfi_lock;\t\/* see below *\/\n\tint\t\tfi_badfd;\t\/* bad file descriptor # *\/\n\tint\t\tfi_action;\t\/* action to take on bad fd use *\/\n\tint\t\tfi_nfiles;\t\/* number of entries in fi_list[] *\/\n\tuf_entry_t *volatile fi_list;\t\/* current file list *\/\n\tuf_rlist_t\t*fi_rlist;\t\/* retired file lists *\/\n} uf_info_t;\n<\/code>\n<\/pre>\n<p>The file lists are indexed by the integer file descriptor. Now let&#8217;s see how we can normally get to this information, the current file list. In usr\/src\/uts\/common\/sys\/user.h we have the user_t structure defined which contains all the per process data related to a user. Through user_t, by accessing it&#8217;s field u_finfo (this is of type uf_info_t) we get to the per process file information and to the current file list. A good example is in the Dtrace code (dtrace.c): uf_info_t *finfo = &amp;curthread-&gt;t_procp-&gt;p_user.u_finfo. There are tools that will allow you to see the fi_list based on the process ID, like pfiles for example.<\/p>\n<p><span style=\"padding-left:2em;\">The file list contains elements of type uf_entry_t. The definition for uf_entry_t is again in the same file usr\/src\/uts\/common\/sys\/user.h.<\/span><\/p>\n<pre class=\"language-clike line-numbers\">\n<code class=\"language-clike line-numbers\">\n\/*\n * Entry in the per-process list of open files.\n * Note: only certain fields are copied in flist_grow() and flist_fork().\n * This is indicated in brackets in the structure member comments.\n *\/\ntypedef struct uf_entry {\n\tkmutex_t\tuf_lock;\t\/* per-fd lock [never copied] *\/\n\tstruct file\t*uf_file;\t\/* file pointer [grow, fork] *\/\n\tstruct fpollinfo *uf_fpollinfo;\t\/* poll state [grow] *\/\n\tint\t\tuf_refcnt;\t\/* LWPs accessing this file [grow] *\/\n\tint\t\tuf_alloc;\t\/* right subtree allocs [grow, fork] *\/\n\tshort\t\tuf_flag;\t\/* fcntl F_GETFD flags [grow, fork] *\/\n\tshort\t\tuf_busy;\t\/* file is allocated [grow, fork] *\/\n\tkcondvar_t\tuf_wanted_cv;\t\/* waiting for setf() [never copied] *\/\n\tkcondvar_t\tuf_closing_cv;\t\/* waiting for close() [never copied] *\/\n\tstruct portfd \t*uf_portfd;\t\/* associated with port [grow] *\/\n\t\/* Avoid false sharing - pad to coherency granularity (64 bytes) *\/\n\tchar\t\tuf_pad[64 - sizeof (kmutex_t) - 2 * sizeof (void*) -\n\t\t2 * sizeof (int) - 2 * sizeof (short) -\n\t\t2 * sizeof (kcondvar_t) - sizeof (struct portfd *)];\n} uf_entry_t;\n<\/code>\n<\/pre>\n<p>As we can see the uf_entry_t contains the pointer to the file in it&#8217;s field: uf_file. Let&#8217;s take a look further on down the path, to see reference between the file pointer and it&#8217;s attached vnode. In Illumos the definition of the struct file resides in usr\/src\/uts\/common\/sys\/file.h.<\/p>\n<pre class=\"language-clike line-numbers\">\n<code class=\"language-clike line-numbers\">\n\/*\n * One file structure is allocated for each open\/creat\/pipe call.\n * Main use is to hold the read\/write pointer associated with\n * each open file.\n *\/\ntypedef struct file {\n\tkmutex_t\tf_tlock;\t\/* short term lock *\/\n\tushort_t\tf_flag;\n\tushort_t\tf_flag2;\t\/* extra flags (FSEARCH, FEXEC) *\/\n\tstruct vnode\t*f_vnode;\t\/* pointer to vnode structure *\/\n\toffset_t\tf_offset;\t\/* read\/write character pointer *\/\n\tstruct cred\t*f_cred;\t\/* credentials of user who opened it *\/\n\tstruct f_audit_data\t*f_audit_data;\t\/* file audit data *\/\n\tint\t\tf_count;\t\/* reference count *\/\n} file_t;\n<\/code>\n<\/pre>\n<p>From the file pointer we can now reach the system wide attached vnode through the f_vnode field. Multiple processes can hold references to the same vnode. We are almost at the end of the virtual filesystem level here, as we move towards the concrete filesystem implementation. The last barrier is basically the vnode. The vnode definition resides in usr\/src\/uts\/common\/sys\/vnode.h.<\/p>\n<pre class=\"language-clike line-numbers\">\n<code class=\"language-clike line-numbers\">\ntypedef struct vnode {\n\tkmutex_t\tv_lock;\t\t\/* protects vnode fields *\/\n\tuint_t\t\tv_flag;\t\t\/* vnode flags (see below) *\/\n\tuint_t\t\tv_count;\t\/* reference count *\/\n\tvoid\t\t*v_data;\t\/* private data for fs *\/\n\tstruct vfs\t*v_vfsp;\t\/* ptr to containing VFS *\/\n\tstruct stdata\t*v_stream;\t\/* associated stream *\/\n\tenum vtype\tv_type;\t\t\/* vnode type *\/\n\tdev_t\t\tv_rdev;\t\t\/* device (VCHR, VBLK) *\/\n\t...\n} vnode_t;\n<\/code>\n<\/pre>\n<p>The entry point into filesystem specific implementation is the v_data field within the vnode_t structure. Having ZFS underneath, v_data will point to a znode. In ZFS the znode is the equivalent of the the UFS inode. The v_data is casted to a znode_t structure through the macro: #define  VTOZ(VP)  ((znode_t *)(VP)-&gt;v_data). More about the znode and the ZFS posix operations implementation in the following posts.<\/p>\n<p><span style=\"padding-left:2em;\">Let&#8217;s make a short recap through the code path of arriving from process level user data to the ZFS Posix Layer implementation.<\/span><\/p>\n<pre class=\"language-clike line-numbers\">\n<code class=\"language-clike line-numbers\">\nproc_t process;\nuser_t p_user = process->p_user; \/\/ the user structure\nuf_info_t file_info = p_user->u_finfo; \/\/ per process file information\nuf_entry_t file_entry = file_info->fi_list[0]; \/\/ This is in case we have fd 0\nfile_t p_file = file_entry->uf_file; \/\/ pointer to file struct\nvnode_t file_vnode = p_file->f_vnode; \/\/ the vnode\n(znode_t *)file_vnode->v_data \/\/ reached the ZFS znode\n<\/code>\n<\/pre>\n<p>The next post will dive into the ZFS posix layer and into the code path of certain sys calls.<\/p>\n<p>Happy coding.<\/p>\n<blockquote class=\"blockquote-reverse\">\n<footer>written by Vadim Comanescu.<\/footer>\n<\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;m starting a series of posts about ZFS. I will try to provide as much as possible architectural diagrams and code examples. The series is for<span class=\"excerpt-hellip\"> [\u2026]<\/span><\/p>\n","protected":false},"author":5,"featured_media":38215,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[502,468,494,501,503,449],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.9 (Yoast SEO v21.9.1) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>A dive into filesystems \u2013 ZFS \u2013 part 1 - Syneto<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"A dive into filesystems \u2013 ZFS \u2013 part 1\" \/>\n<meta property=\"og:description\" content=\"I&#8217;m starting a series of posts about ZFS. I will try to provide as much as possible architectural diagrams and code examples. The series is for [\u2026]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/\" \/>\n<meta property=\"og:site_name\" content=\"Syneto\" \/>\n<meta property=\"article:published_time\" content=\"2014-06-19T12:16:23+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/syneto.eu\/wp-content\/uploads\/2021\/08\/misc-technical-article-cover-3.png\" \/>\n\t<meta property=\"og:image:width\" content=\"761\" \/>\n\t<meta property=\"og:image:height\" content=\"311\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Vadim Com\u0103nescu\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Vadim Com\u0103nescu\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/\",\"url\":\"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/\",\"name\":\"A dive into filesystems \u2013 ZFS \u2013 part 1 - Syneto\",\"isPartOf\":{\"@id\":\"https:\/\/syneto.eu\/#website\"},\"datePublished\":\"2014-06-19T12:16:23+00:00\",\"dateModified\":\"2014-06-19T12:16:23+00:00\",\"author\":{\"@id\":\"https:\/\/syneto.eu\/#\/schema\/person\/28486cf754a19c4c231b82ff24605c5d\"},\"breadcrumb\":{\"@id\":\"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/syneto.eu\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"A dive into filesystems \u2013 ZFS \u2013 part 1\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/syneto.eu\/#website\",\"url\":\"https:\/\/syneto.eu\/\",\"name\":\"Syneto\",\"description\":\"Data Management, Resiliency and Security\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/syneto.eu\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/syneto.eu\/#\/schema\/person\/28486cf754a19c4c231b82ff24605c5d\",\"name\":\"Vadim Com\u0103nescu\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/syneto.eu\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/52e6cc2537fc62c0d71d63c1c2ad8407?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/52e6cc2537fc62c0d71d63c1c2ad8407?s=96&d=mm&r=g\",\"caption\":\"Vadim Com\u0103nescu\"},\"url\":\"https:\/\/syneto.eu\/author\/vadim-comanescu\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"A dive into filesystems \u2013 ZFS \u2013 part 1 - Syneto","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/","og_locale":"en_US","og_type":"article","og_title":"A dive into filesystems \u2013 ZFS \u2013 part 1","og_description":"I&#8217;m starting a series of posts about ZFS. I will try to provide as much as possible architectural diagrams and code examples. The series is for [\u2026]","og_url":"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/","og_site_name":"Syneto","article_published_time":"2014-06-19T12:16:23+00:00","og_image":[{"width":761,"height":311,"url":"https:\/\/syneto.eu\/wp-content\/uploads\/2021\/08\/misc-technical-article-cover-3.png","type":"image\/png"}],"author":"Vadim Com\u0103nescu","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Vadim Com\u0103nescu","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/","url":"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/","name":"A dive into filesystems \u2013 ZFS \u2013 part 1 - Syneto","isPartOf":{"@id":"https:\/\/syneto.eu\/#website"},"datePublished":"2014-06-19T12:16:23+00:00","dateModified":"2014-06-19T12:16:23+00:00","author":{"@id":"https:\/\/syneto.eu\/#\/schema\/person\/28486cf754a19c4c231b82ff24605c5d"},"breadcrumb":{"@id":"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/syneto.eu\/dive-filesystems-zfs-part-1\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/syneto.eu\/"},{"@type":"ListItem","position":2,"name":"A dive into filesystems \u2013 ZFS \u2013 part 1"}]},{"@type":"WebSite","@id":"https:\/\/syneto.eu\/#website","url":"https:\/\/syneto.eu\/","name":"Syneto","description":"Data Management, Resiliency and Security","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/syneto.eu\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/syneto.eu\/#\/schema\/person\/28486cf754a19c4c231b82ff24605c5d","name":"Vadim Com\u0103nescu","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/syneto.eu\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/52e6cc2537fc62c0d71d63c1c2ad8407?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/52e6cc2537fc62c0d71d63c1c2ad8407?s=96&d=mm&r=g","caption":"Vadim Com\u0103nescu"},"url":"https:\/\/syneto.eu\/author\/vadim-comanescu\/"}]}},"_links":{"self":[{"href":"https:\/\/syneto.eu\/wp-json\/wp\/v2\/posts\/38214"}],"collection":[{"href":"https:\/\/syneto.eu\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/syneto.eu\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/syneto.eu\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/syneto.eu\/wp-json\/wp\/v2\/comments?post=38214"}],"version-history":[{"count":0,"href":"https:\/\/syneto.eu\/wp-json\/wp\/v2\/posts\/38214\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/syneto.eu\/wp-json\/wp\/v2\/media\/38215"}],"wp:attachment":[{"href":"https:\/\/syneto.eu\/wp-json\/wp\/v2\/media?parent=38214"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/syneto.eu\/wp-json\/wp\/v2\/categories?post=38214"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/syneto.eu\/wp-json\/wp\/v2\/tags?post=38214"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}