A recent paper at the FAST08 conference highlighted a large number
of unchecked error paths in Linux filesystems and I/O layers. As a
subsystem, XFS had the highest aggregate numbers of bad error
propagation. A tarball which contains a quilt patch series of 32
patches aimed at improving this situation can be found here:
http://oss.sgi.com/~dgc/xfs/error-check/xfs-error-checking.tar.gz
The paper "EIO: Error Handling is Occasionally Correct" can be found
here:
http://www.cs.wisc.edu/adsl/Publications/eio-fast08.html
And the in depth results here:
http://www.cs.wisc.edu/adsl/Publications/eio-fast08/readme.html
http://www.cs.wisc.edu/adsl/Publications/eio-fast08/
The XFS results I've been working from are here:
http://www.cs.wisc.edu/adsl/Publications/eio-fast08/fullfs-xfs-without-false-positives.txt
and included below is an annotated version of this file as I've
worked through it. The graph of the XFS error paths is a good
visual representation of how the bad error paths tend to cluster
together:
http://www.cs.wisc.edu/adsl/Publications/eio-fast08/singlefs-xfs.pdf
(you'll need at least 800% zoom to be able to read it at all)
The paper analysed a 2.6.15 kernel, but I've been working against an
xfs-dev tree (~2.6.24). Of the 101 reported problems for the 2.6.15
kernel that was analysed:
- 7 did not exist anymore (bhv layer, dirv1, write path changes)
- 11 were false positives that were not modified
- 24 were false positives that have been patched to remove
(e.g. int xfs_foo() to void xfs_foo())
- 37 real problems where an error needed to be returned and are
fixed in the patch series.
- 3 where there is no error path to return an error and no
point in even warning about it (ENOSPC flushing)
- 10 where there is no error path to return an error, but
patched to warn to the syslog about potential data loss
or metadata I/O errors
- 4 were already fixed in the xfs-dev tree
- 2 where the error is ignored because we must continue anyway
(patched to warn to syslog)
- 4 that I haven't yet fixed (xfs_buf_iostrategy and
xfs_buf_iostart) because I need to think about them more.
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
---------------------------------------- fs/xfs/ ----
d 1 xfs_write -> _xfs_log_force
fs/xfs/linux-2.6/xfs_lrw.c 881
d 2 xfs_write -> _xfs_log_force
fs/xfs/linux-2.6/xfs_lrw.c 884
F 3 xfs_flush_device -> _xfs_log_force
fs/xfs/linux-2.6/xfs_super.c 547
F 4 xfs_qm_dqflush -> _xfs_log_force
fs/xfs/quota/xfs_dquot.c 1294
F 5 xfs_qm_dqflock_pushbuf_wait -> _xfs_log_force
fs/xfs/quota/xfs_dquot.c 1591
F 6 xfs_qm_dqunpin_wait -> _xfs_log_force
fs/xfs/quota/xfs_dquot_item.c 204
F 7 xfs_qm_dquot_logitem_pushbuf -> _xfs_log_force
fs/xfs/quota/xfs_dquot_item.c 267
F 8 xfs_alloc_search_busy -> _xfs_log_force
fs/xfs/xfs_alloc.c 2593
F 9 xfs_iunpin_wait -> _xfs_log_force
fs/xfs/xfs_inode.c 2847
F 10 xfs_iflush -> _xfs_log_force
fs/xfs/xfs_inode.c 3243
F 11 xfs_inode_item_pushbuf -> _xfs_log_force
fs/xfs/xfs_inode_item.c 819
P 12 xfs_log_unmount_write -> _xfs_log_force
fs/xfs/xfs_log.c 529
F 13 xlog_recover_finish -> _xfs_log_force
fs/xfs/xfs_log_recover.c 3961
F 14 xfs_unmountfs -> _xfs_log_force
fs/xfs/xfs_mount.c 1088
F 15 xfs_trans_push_ail -> _xfs_log_force
fs/xfs/xfs_trans_ail.c 198
F 16 xfs_syncsub -> _xfs_log_force
fs/xfs/xfs_vfsops.c 1440
F 17 xfs_syncsub -> _xfs_log_force
fs/xfs/xfs_vfsops.c 1455
F 18 xfs_syncsub -> _xfs_log_force
fs/xfs/xfs_vfsops.c 1491
F 19 xfs_syncsub -> _xfs_log_force
fs/xfs/xfs_vfsops.c 1543
P 20 xfs_fsync -> _xfs_log_force
fs/xfs/xfs_vnodeops.c 1129
P 21 xfs_qm_write_sb_changes -> _xfs_trans_commit
fs/xfs/quota/xfs_qm.c 2414
P 22 xfs_qm_scall_setqlim -> _xfs_trans_commit
fs/xfs/quota/xfs_qm_syscalls.c 739
P 23 xfs_itruncate_finish -> _xfs_trans_commit
fs/xfs/xfs_inode.c 1718
P 24 xlog_recover_process_efi -> _xfs_trans_commit
fs/xfs/xfs_log_recover.c 3047
P 25 xlog_recover_clear_agi_bucket -> _xfs_trans_commit
fs/xfs/xfs_log_recover.c 3174
PB 26 xfs_mount_log_sbunit -> _xfs_trans_commit
fs/xfs/xfs_mount.c 1579
P 27 xfs_growfs_rt_alloc -> _xfs_trans_commit
fs/xfs/xfs_rtalloc.c 154
P 28 xfs_growfs_rt_alloc -> _xfs_trans_commit
fs/xfs/xfs_rtalloc.c 191
P 29 xfs_growfs_rt -> _xfs_trans_commit
fs/xfs/xfs_rtalloc.c 2103
P 30 xfs_inactive_attrs -> _xfs_trans_commit
fs/xfs/xfs_vnodeops.c 1505
C 31 xfs_inactive -> _xfs_trans_commit
fs/xfs/xfs_vnodeops.c 1790
d 32 xfs_initialize_vnode -> bhv_insert
fs/xfs/linux-2.6/xfs_super.c 220
d 33 vfs_insertops -> bhv_insert
fs/xfs/linux-2.6/xfs_vfs.c 259
N 34 linvfs_truncate -> block_truncate_page
fs/xfs/linux-2.6/xfs_iops.c 651
G 35 fs_flushinval_pages -> filemap_fdatawait
fs/xfs/linux-2.6/xfs_fs_subr.c 83
G 36 fs_flush_pages -> filemap_fdatawait
fs/xfs/linux-2.6/xfs_fs_subr.c 108
G 37 fs_flushinval_pages -> filemap_fdatawrite
fs/xfs/linux-2.6/xfs_fs_subr.c 82
G 38 fs_flush_pages -> filemap_fdatawrite
fs/xfs/linux-2.6/xfs_fs_subr.c 105
n 39 xfs_flush_inode_work -> filemap_flush
fs/xfs/linux-2.6/xfs_super.c 508
PM 40 xlog_sync -> pagebuf_associate_memory
fs/xfs/xfs_log.c 1358
PM 41 xlog_sync -> pagebuf_associate_memory
fs/xfs/xfs_log.c 1395
PM 42 xlog_write_log_records -> pagebuf_associate_memory
fs/xfs/xfs_log_recover.c 1156
PM 43 xlog_write_log_records -> pagebuf_associate_memory
fs/xfs/xfs_log_recover.c 1159
PM 44 xlog_do_recovery_pass -> pagebuf_associate_memory
fs/xfs/xfs_log_recover.c 3646
PM 45 xlog_do_recovery_pass -> pagebuf_associate_memory
fs/xfs/xfs_log_recover.c 3653
PM 46 xlog_do_recovery_pass -> pagebuf_associate_memory
fs/xfs/xfs_log_recover.c 3705
PM 47 xlog_do_recovery_pass -> pagebuf_associate_memory
fs/xfs/xfs_log_recover.c 3711
M 48 xfs_buf_read_flags -> pagebuf_iostart
fs/xfs/linux-2.6/xfs_buf.c 636
M 49 xfsbufd -> pagebuf_iostrategy
fs/xfs/linux-2.6/xfs_buf.c 1755
M 50 xfs_flush_buftarg -> pagebuf_iostrategy
fs/xfs/linux-2.6/xfs_buf.c 1816
M 51 XFS_bwrite -> pagebuf_iostrategy
fs/xfs/linux-2.6/xfs_buf.h 503
n 52 xfs_flush_device_work -> sync_blockdev
fs/xfs/linux-2.6/xfs_super.c 533
f 53 exit_xfs_fs -> unregister_filesystem
fs/xfs/linux-2.6/xfs_super.c 999
PM 54 xfs_acl_vset -> xfs_acl_vremove
fs/xfs/xfs_acl.c 326
f 55 xfs_ialloc_ag_select -> xfs_alloc_pagf_init
fs/xfs/xfs_ialloc.c 411
P 56 xfs_qm_dqflush -> xfs_bawrite
fs/xfs/quota/xfs_dquot.c 1300
N 57 xfs_qm_dqflock_pushbuf_wait -> xfs_bawrite
fs/xfs/quota/xfs_dquot.c 1595
N 58 xfs_qm_dquot_logitem_pushbuf -> xfs_bawrite
fs/xfs/quota/xfs_dquot_item.c 275
N 59 xfs_buf_item_push -> xfs_bawrite
fs/xfs/xfs_buf_item.c 669
P 60 xfs_iflush -> xfs_bawrite
fs/xfs/xfs_inode.c 3249
N 61 xfs_inode_item_pushbuf -> xfs_bawrite
fs/xfs/xfs_inode_item.c 823
F 62 xfs_qm_dqflush -> xfs_bdwrite
fs/xfs/quota/xfs_dquot.c 1298
F 63 xfs_qm_dqiter_bufs -> xfs_bdwrite
fs/xfs/quota/xfs_qm.c 1551
F 64 xfs_iflush -> xfs_bdwrite
fs/xfs/xfs_inode.c 3247
F 65 xlog_recover_do_buffer_trans -> xfs_bdwrite
fs/xfs/xfs_log_recover.c 2271
F 66 xlog_recover_do_inode_trans -> xfs_bdwrite
fs/xfs/xfs_log_recover.c 2535
F 67 xlog_recover_do_dquot_trans -> xfs_bdwrite
fs/xfs/xfs_log_recover.c 2664
C 68 xfs_inactive -> xfs_bmap_finish
fs/xfs/xfs_vnodeops.c 1788
P 69 xfs_iomap_write_allocate -> xfs_bmap_last_offset
fs/xfs/xfs_iomap.c 787
d 70 xfs_dir_leaf_rebalance -> xfs_dir_leaf_compact
fs/xfs/xfs_dir_leaf.c 1146
d 71 xfs_dir_leaf_rebalance -> xfs_dir_leaf_compact
fs/xfs/xfs_dir_leaf.c 1176
d 72 xfs_dir_leaf_to_shortform -> xfs_dir_shortform_addname
fs/xfs/xfs_dir_leaf.c 693
P 73 xlog_recover_process_efi -> xfs_free_extent
fs/xfs/xfs_log_recover.c 3041
n 74 xfs_inode_item_push -> xfs_iflush
fs/xfs/xfs_inode_item.c 879
P 75 xlog_recover_do_inode_trans -> xfs_imap
fs/xfs/xfs_log_recover.c 2320
f 76 xfs_bmap_add_extent -> xfs_mod_incore_sb
fs/xfs/xfs_bmap.c 689
f 77 xfs_bmap_add_extent_hole_delay -> xfs_mod_incore_sb
fs/xfs/xfs_bmap.c 1918
f 78 xfs_bmap_del_extent -> xfs_mod_incore_sb
fs/xfs/xfs_bmap.c 3117
f 79 xfs_bmapi -> xfs_mod_incore_sb
fs/xfs/xfs_bmap.c 4801
f 80 xfs_bmapi -> xfs_mod_incore_sb
fs/xfs/xfs_bmap.c 4805
f 81 xfs_bunmapi -> xfs_mod_incore_sb
fs/xfs/xfs_bmap.c 5452
f 82 xfs_bunmapi -> xfs_mod_incore_sb
fs/xfs/xfs_bmap.c 5458
f 83 xfs_trans_reserve -> xfs_mod_incore_sb
fs/xfs/xfs_trans.c 305
P 84 xfs_qm_quotacheck -> xfs_mount_reset_sbqflags
fs/xfs/quota/xfs_qm.c 1962
N 85 xfs_qm_dqpurge -> xfs_qm_dqflush
fs/xfs/quota/xfs_dquot.c 1505
NB 86 xfs_qm_dquot_logitem_push -> xfs_qm_dqflush
fs/xfs/quota/xfs_dquot_item.c 168
N 87 xfs_qm_shake_freelist -> xfs_qm_dqflush
fs/xfs/quota/xfs_qm.c 2134
N 88 xfs_qm_dqreclaim_one -> xfs_qm_dqflush
fs/xfs/quota/xfs_qm.c 2306
PM 89 xfs_qm_quotacheck -> xfs_qm_dqflush_all
fs/xfs/quota/xfs_qm.c 1930
P 90 xfs_qm_scall_quotaoff -> xfs_qm_log_quotaoff
fs/xfs/quota/xfs_qm_syscalls.c 291
P 91 xfs_qm_scall_quotaoff -> xfs_qm_log_quotaoff_end
fs/xfs/quota/xfs_qm_syscalls.c 347
F 92 xfs_qm_newmount -> xfs_qm_mount_quotas
fs/xfs/quota/xfs_qm_bhv.c 273
F 93 xfs_qm_endmount -> xfs_qm_mount_quotas
fs/xfs/quota/xfs_qm_bhv.c 301
PM 94 xfs_quiesce_fs -> xfs_syncsub
fs/xfs/xfs_vfsops.c 632
P 95 xlog_recover_process_efi -> xfs_trans_reserve
fs/xfs/xfs_log_recover.c 3036
P 96 xlog_recover_clear_agi_bucket -> xfs_trans_reserve
fs/xfs/xfs_log_recover.c 3152
P 97 xfs_qm_scall_trunc_qfiles -> xfs_truncate_file
fs/xfs/quota/xfs_qm_syscalls.c 395
P 98 xfs_qm_scall_trunc_qfiles -> xfs_truncate_file
fs/xfs/quota/xfs_qm_syscalls.c 404
P 99 xfs_log_unmount_write -> xlog_state_release_iclog
fs/xfs/xfs_log.c 570
P 100 xfs_log_unmount_write -> xlog_state_release_iclog
fs/xfs/xfs_log.c 606
f 101 xfs_log_force_umount -> xlog_state_sync_all
fs/xfs/xfs_log.c 3586
f = false positive
F = false positive + patch to remove condition
G = patch in mainline git tree already
M = __must_check annotations found this as well
P = real, patch to fix
n = no error path to return error
N = no error path to return error, patch to warn about error added
d = does not exist anymore.
B = some other bug found and fixed at same time
C = error ignored, must continue anyway. If silent, made noisy
Notes:
- all the xfs_mod_incore_sb() are false positive because they are freeing
blocks or extents which means there can never be an error returned. The only
error that can be returned is ENOSPC when trying to allocate blocks....
- none of the callers of xfs_mount_log_sb() check the return value.
- new function xfs_log_sbcount failed to check return of xfs_trans_commit.
Callers are failing to check return value.
- most of the callers to xfs_log_force() are not interested in errors - they'll
get them through other means (i.e. log error implies filesystem shutdown).
Only a handful of callers really should return errors, such as fsync(),
sync writes or synchronous transaction commits.
|