xfs
[Top] [All Lists]

[PATCH TARBALL, DEV ONLY] xfsprogs: metadata CRC support V2

To: xfs@xxxxxxxxxxx
Subject: [PATCH TARBALL, DEV ONLY] xfsprogs: metadata CRC support V2
From: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Sat, 13 Apr 2013 00:55:55 +1000
Delivered-to: xfs@xxxxxxxxxxx
User-agent: Mutt/1.5.21 (2010-09-15)
Hi folks,

I've attached a tarball containing the patch series that adds
metadata CRC support to xfsprogs. It is still very much a work in
progress, but is sufficient to start running xfstests on CRC enable
filesystems.

THere is much missing functionality, especially with respect to
directly/attribute modifications. Only mkfs and xfs_repair have been
updated to explicitly support CRC on directory/sttribute blocks,
though all the other tools will stil run the result might be a
little unpredictable.

The patch series is as follows:

Dave Chinner (26):
      xfsprogs: sync code to current kernel code
      xfsprogs: update libxfs to 3.9-rc1 + xfsdev
      xfsprogs: add CRC32c infrastructure
      xfsprogs: updata libxlog to current kernel code
      libxfs: add crc format changes to generic btrees
      xfsprogs: add crc format chagnes to ag headers
      libxfs: change quota buffer formats
      libxfs: add version 3 inode support
      libxfs: add support for crc headers on remote symlinks
      xfs: add CRC checks to block format directory blocks
      xfs: add CRC checking to dir2 free blocks
      xfs: add CRC checking to dir2 data blocks
      xfs: add CRC checking to dir2 leaf blocks
      xfs: shortform directory offsets change for dir3 format
      xfs: add CRCs to dir2/da node blocks
      xfs: add CRCs to attr leaf blocks
      xfs: split remote attribute code out
      xfs: add CRC protection to remote attributes
      xfs: add buffer types to directory and attribute buffers
      xfs: buffer type overruns blf_flags field
      xfs: add CRC checks to the superblock
      xfs: implement extended feature masks
      xfsprogs: introduce CRC support into mkfs.xfs
      xfsprogs: add crc format support to repair
      xfsprogs: add crc format support to db
      xfs_repair: update for dir/attr crc format changes.

Diffstat is below [1].

The changes are roughly:

        - sync to current kernel code
        - apply userspace converted version of kernel patches
        - update xfsprogs utilities to be aware of CRCs

Much of the last 4 patches needs to be split up and redirected back
to the patches that introduce the bugs that are fixed or
conversions that are missed.

Repair needs support for remote attribute and symlink block CRC
support, as well as verifying the new feature masks.  mkfs needs to
handle thenew feature masks correctly, too.  xfs_db needs to be able
to display all crc enabled formats, and needs to ensure all
read-only functions work properly on CRC enabled filesystems.  I
think that metadump will also need some attention. So there's plenty
of work yet to do, though it is already usable.

For non-CRC filesystems, the utilities appear to work without
regressions in xfstests. Most of the testing I have done is on
non-CRC filesytsems.

For CRC enabled filesystems, xfstests will run for a short while
before the kernel bugs out with an attribute corruption. Over the
next week I hope to get the kernel code to the point where it will
run most of xfstests with CRCs enabled:

$ sudo MKFS_OPTIONS="-m crc=1" ./check -g auto
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 test-4 3.9.0-rc4-dgc+
MKFS_OPTIONS  -- -f -m crc=1 /dev/vdb
MOUNT_OPTIONS -- /dev/vdb /mnt/scr

generic/001      4s
generic/002      0s
generic/005      0s
generic/006      1s
generic/007      1s
generic/010      [not run] dbtest was not built for this platform
generic/011      1s
generic/013      12s
generic/014      3s
generic/015      0s
generic/020
Message from syslogd@test-4 at Apr 13 00:35:45 ...
 kernel:[  451.268820] XFS: Assertion failed: ichdr->firstused >= ichdr->count 
* sizeof(xfs_attr_leaf_entry_t) + xfs_attr3_leaf_hdr_size(leaf), file: 
fs/xfs/xfs_attr_leaf.c, line: 1427
$

Finally, I'm sending this update out as a quilt series style tarball
rather than email patches as some of the patches are too large for
the list and will get blocked. Hence you get patches and a series
file that I've just exported from my current guilt tree.

Obligatory warning: If you enable CRCs, the code will eat your
data. Use this only only for testing with data you don't care about.

Comments, flames and testing results all welcome.

Cheers,

Dave.

[1] diffstat:

 copy/xfs_copy.c            |    5 +-
 db/agf.c                   |    3 +
 db/agfl.c                  |   16 +
 db/agfl.h                  |    2 +
 db/agi.c                   |    3 +
 db/attr.c                  |   20 +-
 db/btblock.c               |  145 +
 db/btblock.h               |   10 +
 db/check.c                 |   55 +-
 db/dir.c                   |    8 +-
 db/dir.h                   |    6 +
 db/dir2.c                  |   16 +-
 db/dir2.h                  |   32 +
 db/dir2sf.c                |   24 +-
 db/dirshort.c              |    1 +
 db/field.c                 |   16 +
 db/field.h                 |    8 +
 db/freesp.c                |   11 +-
 db/init.c                  |    4 +
 db/inode.c                 |   27 +
 db/inode.h                 |    3 +
 db/metadump.c              |   27 +-
 db/sb.c                    |   40 +-
 db/type.c                  |   35 +-
 db/type.h                  |    3 +-
 estimate/xfs_estimate.c    |    3 +
 fsr/xfs_fsr.c              |    2 +-
 include/Makefile           |   13 +-
 include/libxfs.h           |  222 +-
 include/libxlog.h          |   34 +-
 include/linux.h            |    1 +
 include/platform_defs.h.in |    5 +
 include/swab.h             |   38 +
 include/xfs.h              |    1 +
 include/xfs_ag.h           |   81 +-
 include/xfs_alloc.h        |   57 +-
 include/xfs_alloc_btree.h  |   29 +-
 include/xfs_arch.h         |   16 +
 include/xfs_attr_leaf.h    |  135 +-
 include/xfs_attr_remote.h  |   52 +
 include/xfs_bmap.h         |  325 +-
 include/xfs_bmap_btree.h   |   21 +-
 include/xfs_btree.h        |   87 +-
 include/xfs_buf_item.h     |  117 +-
 include/xfs_cksum.h        |   63 +
 include/xfs_da_btree.h     |  185 +-
 include/xfs_dinode.h       |   47 +-
 include/xfs_dir1.h         |   13 +
 include/xfs_dir2.h         |   95 +-
 include/xfs_dir2_block.h   |   92 -
 include/xfs_dir2_data.h    |  184 -
 include/xfs_dir2_format.h  |  836 +++++
 include/xfs_dir2_leaf.h    |  253 --
 include/xfs_dir2_node.h    |  100 -
 include/xfs_dir2_sf.h      |  171 -
 include/xfs_fs.h           |   38 +-
 include/xfs_ialloc.h       |   17 +-
 include/xfs_ialloc_btree.h |   12 +-
 include/xfs_inode.h        |  263 +-
 include/xfs_inode_item.h   |   22 +-
 include/xfs_inum.h         |   16 -
 include/xfs_log.h          |   31 +-
 include/xfs_log_priv.h     |  101 +-
 include/xfs_mount.h        |   48 +-
 include/xfs_quota.h        |   58 +-
 include/xfs_rtalloc.h      |    4 +-
 include/xfs_sb.h           |  172 +-
 include/xfs_symlink.h      |   45 +
 include/xfs_trace.h        |   69 +-
 include/xfs_trans.h        |   88 +-
 include/xfs_types.h        |   25 +-
 libxfs/Makefile            |   26 +-
 libxfs/crc32.c             | 1036 ++++++
 libxfs/crc32defs.h         |   72 +
 libxfs/gen_crc32table.c    |  144 +
 libxfs/init.c              |   26 +-
 libxfs/logitem.c           |   20 +-
 libxfs/rdwr.c              |  522 ++-
 libxfs/trans.c             |   55 +-
 libxfs/util.c              |  230 +-
 libxfs/xfs.h               |  147 +-
 libxfs/xfs_alloc.c         |  694 ++--
 libxfs/xfs_alloc_btree.c   |  140 +-
 libxfs/xfs_attr.c          |  546 +--
 libxfs/xfs_attr_leaf.c     | 1858 ++++++----
 libxfs/xfs_attr_remote.c   |  527 +++
 libxfs/xfs_bmap.c          | 8950 
++++++++++++++++++++++++----------------------
 libxfs/xfs_bmap_btree.c    |  124 +-
 libxfs/xfs_btree.c         |  415 ++-
 libxfs/xfs_da_btree.c      | 2598 ++++++++------
 libxfs/xfs_dir2.c          |  149 +-
 libxfs/xfs_dir2_block.c    |  806 +++--
 libxfs/xfs_dir2_data.c     |  604 ++--
 libxfs/xfs_dir2_leaf.c     | 1342 ++++---
 libxfs/xfs_dir2_node.c     | 1455 ++++----
 libxfs/xfs_dir2_priv.h     |  160 +
 libxfs/xfs_dir2_sf.c       |  326 +-
 libxfs/xfs_ialloc.c        |  632 ++--
 libxfs/xfs_ialloc_btree.c  |   87 +-
 libxfs/xfs_inode.c         |  505 ++-
 libxfs/xfs_mount.c         |  259 +-
 libxfs/xfs_rtalloc.c       |   77 +-
 libxfs/xfs_symlink.c       |  154 +
 libxfs/xfs_trans.c         |    9 +-
 libxlog/xfs_log_recover.c  |  299 +-
 logprint/log_copy.c        |    2 +-
 logprint/log_dump.c        |    2 +-
 logprint/log_misc.c        |    9 +-
 logprint/log_print_all.c   |    7 +-
 logprint/log_print_trans.c |   22 +-
 logprint/logprint.c        |    8 +-
 logprint/logprint.h        |   10 +-
 mdrestore/Makefile         |    2 +-
 mkfs/maxtrres.c            |    4 +-
 mkfs/proto.c               |   14 +-
 mkfs/xfs_mkfs.c            |  162 +-
 mkfs/xfs_mkfs.h            |   12 +-
 repair/agheader.c          |   36 +-
 repair/attr_repair.c       |   63 +-
 repair/dino_chunks.c       |   13 +-
 repair/dinode.c            |  208 +-
 repair/dir.c               |   96 +-
 repair/dir2.c              |  497 +--
 repair/dir2.h              |   28 +-
 repair/phase2.c            |   13 +-
 repair/phase3.c            |    2 +-
 repair/phase5.c            |  157 +-
 repair/phase6.c            |  327 +-
 repair/prefetch.c          |   11 +-
 repair/rt.c                |    4 +-
 repair/scan.c              |  166 +-
 repair/scan.h              |   12 +-
 repair/versions.c          |    2 +-
 repair/xfs_repair.c        |    9 +-
 134 files changed, 19036 insertions(+), 12663 deletions(-)
 create mode 100644 include/xfs_attr_remote.h
 create mode 100644 include/xfs_cksum.h
 create mode 100644 include/xfs_dir1.h
 delete mode 100644 include/xfs_dir2_block.h
 delete mode 100644 include/xfs_dir2_data.h
 create mode 100644 include/xfs_dir2_format.h
 delete mode 100644 include/xfs_dir2_leaf.h
 delete mode 100644 include/xfs_dir2_node.h
 delete mode 100644 include/xfs_dir2_sf.h
 create mode 100644 include/xfs_symlink.h
 create mode 100644 libxfs/crc32.c
 create mode 100644 libxfs/crc32defs.h
 create mode 100644 libxfs/gen_crc32table.c
 create mode 100644 libxfs/xfs_attr_remote.c
 create mode 100644 libxfs/xfs_dir2_priv.h
 create mode 100644 libxfs/xfs_symlink.c

-- 
Dave Chinner
david@xxxxxxxxxxxxx

Attachment: xfsprogs-crc-patchset-v2.tar.gz
Description: Text document

<Prev in Thread] Current Thread [Next in Thread>