<div dir="ltr">Hi xfsers:<div><br></div><div>I got some troubles on the performance of  xfs. <div>The environment is , </div><div>     xfs version is 3.2.1, </div><div>     centos 7.1, </div><div>     kernel version:3.10.0-229.el7.x86_64.</div><div>     pcie-ssd card, </div><div>     mkfs: mkfs.xfs /dev/hioa2 -f -n size=64k -i size=512 -d agcount=40 -l size=1024m.</div></div><div>     mount: mount /dev/hioa2 /mnt/  -t xfs -o rw,noexec,nodev,noatime,nodiratime,nobarrier,discard,inode64,logbsize=256k,delaylog</div><div>I use the following command to test iops: fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randwrite -size=50G -filename=/mnt/test -name="EBS 4KB randwrite test" -iodepth=64 -runtime=60</div><div>The results is normal at the beginning which is about 210k±，but some seconds later, the results down to 19k±.</div><div><br></div><div>I did a senond test , </div><div>     umount the /dev/hioa2, </div><div>     fio -ioengine=libaio -bs=4k -direct=1  -thread -rw=randwrite -filename=/dev/hioa2  -name="EBS 8KB randwrite test" -iodepth=64 -runtime=60</div><div>The results was normal, the iops is about 210k± all the time.</div><div><br></div><div>Then I use perf-tools to trace xfs:</div><div>     # ./funccount -i 1 'xfs*'</div><div><br></div><div>In the first test case , when the iops seems normal , the results is as follows:</div><div><div><div><div>FUNC                              COUNT</div><div>xfs_free_eofblocks                    1</div><div>xfs_attr_get                          5</div><div>xfs_attr_get_int                      5</div><div>xfs_attr_name_to_xname                5</div><div>xfs_attr_shortform_getvalue           5</div><div>xfs_da_hashname                       5</div><div>xfs_ilock_attr_map_shared             5</div><div>xfs_xattr_get                         5</div><div>xfs_fs_statfs                         7</div><div>xfs_icsb_count                        7</div><div>xfs_icsb_sync_counters                7</div><div>xfs_icsb_sync_counters_locked         7</div><div>xfs_can_free_eofblocks                9</div><div>xfs_file_release                      9</div><div>xfs_release                           9</div><div>xfs_icsb_counter_disabled            21</div><div>xfs_file_open                        30</div><div>xfs_file_aio_read                    33</div><div>xfs_readlink                         37</div><div>xfs_vn_follow_link                   37</div><div>xfs_vn_put_link                      37</div><div>xfs_buf_delwri_submit_nowait         41</div><div>xfs_trans_ail_cursor_first           41</div><div>xfs_trans_ail_cursor_init            41</div><div>xfs_vn_getattr                       44</div><div>xfs_file_mmap                        48</div><div>xfs_cil_prepare_item.isra.1        1004</div><div>xfs_extent_busy_clear              1004</div><div>xfs_inode_item_committing          1004</div><div>xfs_inode_item_data_fork_size.     1004</div><div>xfs_inode_item_format              1004</div><div>xfs_inode_item_format_data_for     1004</div><div>xfs_inode_item_size                1004</div><div>xfs_inode_item_unlock              1004</div><div>xfs_log_calc_unit_res              1004</div><div>xfs_log_commit_cil                 1004</div><div>xfs_log_done                       1004</div><div>xfs_log_reserve                    1004</div><div>xfs_log_space_wake                 1004</div><div>xfs_log_ticket_put                 1004</div><div>xfs_trans_add_item                 1004</div><div>xfs_trans_alloc                    1004</div><div>xfs_trans_apply_dquot_deltas       1004</div><div>xfs_trans_commit                   1004</div><div>xfs_trans_free                     1004</div><div>xfs_trans_free_dqinfo              1004</div><div>xfs_trans_free_item_desc           1004</div><div>xfs_trans_free_items               1004</div><div>xfs_trans_ijoin                    1004</div><div>xfs_trans_log_inode                1004</div><div>xfs_trans_reserve                  1004</div><div>xfs_trans_unreserve_and_mod_sb     1004</div><div>xfs_vn_update_time                 1004</div><div>xfs_destroy_ioend                 84845</div><div>xfs_end_io_direct_write           84845</div><div>xfs_finish_ioend                  84845</div><div>xfs_file_aio_write                84880</div><div>xfs_file_dio_aio_write            84882</div><div>xfs_vm_direct_IO                  84882</div><div>xfs_get_blocks_direct             84883</div><div>xfs_bmapi_read                    84885</div><div>xfs_bmap_search_extents           84886</div><div>xfs_bmap_search_multi_extents     84886</div><div>xfs_iext_bno_to_ext               84887</div><div>xfs_iext_bno_to_irec              84887</div><div>xfs_iext_get_ext                  84890</div><div>xfs_iext_idx_to_irec              84890</div><div>xfs_ilock_data_map_shared         84891</div><div>xfs_map_buffer.isra.9             84892</div><div>xfs_file_aio_write_checks         84893</div><div>xfs_alloc_ioend                   84895</div><div>xfs_bmapi_trim_map.isra.11        84900</div><div>xfs_fsb_to_db                     84902</div><div>xfs_bmapi_update_map              84903</div><div>xfs_bmbt_get_all                 169793</div><div>xfs_find_bdev_for_inode          169794</div><div>xfs_ilock                        170868</div><div>xfs_iunlock                      170872</div><div>xfs_bmbt_get_blockcount          555016</div><div>xfs_bmbt_get_startoff           2277490</div></div></div></div><div><br></div><div>when the results down to 19k±， the trace results are:</div><div><div>FUNC                              COUNT</div><div>xfs_bitmap_empty                      1</div><div>xfs_free_eofblocks                    1</div><div>xfs_attr_get                          5</div><div>xfs_attr_get_int                      5</div><div>xfs_attr_name_to_xname                5</div><div>xfs_attr_shortform_getvalue           5</div><div>xfs_da_hashname                       5</div><div>xfs_ilock_attr_map_shared             5</div><div>xfs_xattr_get                         5</div><div>xfs_fs_statfs                         7</div><div>xfs_icsb_count                        7</div><div>xfs_icsb_sync_counters                7</div><div>xfs_icsb_sync_counters_locked         7</div><div>xfs_can_free_eofblocks                9</div><div>xfs_file_release                      9</div><div>xfs_release                           9</div><div>xfs_file_open                        30</div><div>xfs_file_aio_read                    33</div><div>xfs_readlink                         37</div><div>xfs_vn_follow_link                   37</div><div>xfs_vn_put_link                      37</div><div>xfs_buf_delwri_submit_nowait         41</div><div>xfs_trans_ail_cursor_first           41</div><div>xfs_trans_ail_cursor_init            41</div><div>xfs_alloc_ag_vextent                 43</div><div>xfs_alloc_ag_vextent_near            43</div><div>xfs_alloc_find_best_extent           43</div><div>xfs_alloc_fix_freelist               43</div><div>xfs_alloc_fixup_trees                43</div><div>xfs_alloc_read_agf                   43</div><div>xfs_alloc_read_agfl                  43</div><div>xfs_alloc_update                     43</div><div>xfs_alloc_update_counters.isra       43</div><div>xfs_alloc_vextent                    43</div><div>xfs_bmbt_alloc_block                 43</div><div>xfs_btree_get_buf_block.constp       43</div><div>xfs_btree_reada_bufs                 43</div><div>xfs_btree_readahead_sblock.isr       43</div><div>xfs_btree_split                      43</div><div>xfs_btree_split_worker               43</div><div>xfs_read_agf                         43</div><div>xfs_trans_get_buf_map                43</div><div>xfs_allocbt_dup_cursor               44</div><div>xfs_allocbt_get_minrecs              44</div><div>xfs_allocbt_init_rec_from_cur        44</div><div>xfs_allocbt_update_lastrec           44</div><div>xfs_alloc_fix_minleft                44</div><div>xfs_alloc_log_agf                    44</div><div>xfs_alloc_longest_free_extent        44</div><div>xfs_bmbt_init_rec_from_key           44</div><div>xfs_bmbt_update_cursor               44</div><div>xfs_btree_buf_to_ptr.isra.17         44</div><div>xfs_btree_check_sblock               44</div><div>xfs_btree_init_block_cur.isra.       44</div><div>xfs_btree_init_block_int             44</div><div>xfs_btree_islastblock                44</div><div>xfs_trans_mod_dquot_byino            44</div><div>xfs_trans_mod_sb                     44</div><div>xfs_vn_getattr                       44</div><div>xfs_file_mmap                        48</div><div>xfs_iext_irec_new                    57</div><div>xfs_iext_realloc_indirect            57</div><div>xfs_btree_copy_ptrs                  66</div><div>xfs_btree_shift_keys.isra.23         66</div><div>xfs_btree_shift_ptrs                 66</div><div>xfs_btree_log_ptrs                   72</div><div>xfs_alloc_get_rec                    86</div><div>xfs_alloc_lookup_eq                  86</div><div>xfs_alloc_compute_aligned            88</div><div>xfs_btree_get_rec                    88</div><div>xfs_extent_busy_trim                 88</div><div>xfs_iext_add_indirect_multi         108</div><div>xfs_alloc_compute_diff              110</div><div>xfs_alloc_fix_len                   110</div><div>xfs_allocbt_init_cursor             132</div><div>xfs_iext_irec_compact               163</div><div>xfs_iext_remove                     163</div><div>xfs_iext_remove_indirect            163</div><div>xfs_buf_item_relse                  173</div><div>xfs_allocbt_init_ptr_from_cur       176</div><div>xfs_btree_set_sibling.isra.12       176</div><div>xfs_buf_item_free                   177</div><div>xfs_buf_item_free_format            177</div><div>xfs_buf_hold                        248</div><div>xfs_buf_item_get_format             248</div><div>xfs_log_item_init                   248</div><div>xfs_allocbt_get_maxrecs             264</div><div>xfs_bmbt_get_minrecs                326</div><div>xfs_btree_dec_cursor                369</div><div>xfs_btree_delete                    369</div><div>xfs_btree_delrec                    369</div><div>xfs_bmbt_set_allf                   633</div><div>xfs_bmbt_set_startblock            1007</div><div>xfs_bmbt_set_startoff              1007</div><div>xfs_trans_alloc                    1029</div><div>xfs_vn_update_time                 1029</div><div>xfs_btree_lshift                   1048</div><div>xfs_btree_decrement                1332</div><div>xfs_allocbt_init_key_from_rec      1481</div><div>xfs_allocbt_key_diff               2427</div><div>xfs_btree_check_lblock             3653</div><div>xfs_btree_lastrec                  3653</div><div>xfs_buf_free                       3660</div><div>xfs_bmbt_dup_cursor                3696</div><div>xfs_btree_check_block              3696</div><div>xfs_buf_allocate_memory            3703</div><div>xfs_btree_dup_cursor               3738</div><div>xfs_btree_increment                4324</div><div>xfs_btree_make_block_unfull        4699</div><div>xfs_btree_rshift                   4699</div><div>xfs_btree_updkey                   4738</div><div>xfs_btree_copy_keys.isra.22        4824</div><div>xfs_btree_log_keys                 4832</div><div>xfs_iext_add                       4876</div><div>xfs_iext_insert                    4876</div><div>xfs_btree_reada_bufl               5465</div><div>xfs_btree_readahead_lblock.isr     5465</div><div>xfs_buf_readahead_map              5508</div><div>xfs_btree_readahead                5670</div><div>xfs_inode_item_format              6071</div><div>xfs_inode_item_format_data_for     6072</div><div>xfs_inode_item_size                6072</div><div>xfs_inode_item_data_fork_size.     6073</div><div>xfs_trans_log_inode                6115</div><div>xfs_buf_item_pin                   6315</div><div>xfs_bmapi_write                    6674</div><div>xfs_end_io                         6674</div><div>xfs_iomap_write_unwritten          6674</div><div>xfs_bmap_add_extent_unwritten_     6675</div><div>xfs_bmapi_convert_unwritten        6675</div><div>xfs_destroy_ioend                  6675</div><div>xfs_end_io_direct_write            6676</div><div>xfs_finish_ioend                   6677</div><div>xfs_bmap_finish                    6683</div><div>xfs_file_aio_write                 6784</div><div>xfs_bmapi_read                     6785</div><div>xfs_file_aio_write_checks          6785</div><div>xfs_file_dio_aio_write             6785</div><div>xfs_get_blocks_direct              6785</div><div>xfs_ilock_data_map_shared          6785</div><div>xfs_map_buffer.isra.9              6785</div><div>xfs_vm_direct_IO                   6785</div><div>xfs_alloc_ioend                    6786</div><div>xfs_fsb_to_db                      6786</div><div>xfs_bmbt_set_blockcount            7691</div><div>xfs_log_commit_cil                 7704</div><div>xfs_trans_commit                   7704</div><div>xfs_trans_free_items               7704</div><div>xfs_trans_reserve                  7705</div><div>xfs_trans_unreserve_and_mod_sb     7705</div><div>xfs_log_done                       7706</div><div>xfs_trans_free                     7706</div><div>xfs_log_reserve                    7708</div><div>xfs_trans_ijoin                    7708</div><div>xfs_extent_busy_clear              7712</div><div>xfs_inode_item_unlock              7712</div><div>xfs_log_space_wake                 7712</div><div>xfs_log_ticket_put                 7712</div><div>xfs_trans_apply_dquot_deltas       7712</div><div>xfs_inode_item_committing          7713</div><div>xfs_log_calc_unit_res              7713</div><div>xfs_trans_free_dqinfo              7713</div><div>xfs_bmbt_update                    8311</div><div>xfs_btree_update                   8354</div><div>xfs_bmbt_init_rec_from_cur         9758</div><div>xfs_bmbt_disk_set_all              9760</div><div>xfs_bmbt_set_all                   9760</div><div>xfs_btree_insert                   9795</div><div>xfs_btree_insrec                   9838</div><div>xfs_bmbt_init_cursor              10379</div><div>xfs_btree_del_cursor              10498</div><div>xfs_trans_del_item                11101</div><div>xfs_buf_item_dirty                11109</div><div>xfs_bmbt_lookup_eq                11551</div><div>xfs_bmbt_init_ptr_from_cur        11563</div><div>xfs_btree_get_iroot.isra.8        11563</div><div>xfs_buf_item_format               11690</div><div>xfs_buf_item_size                 11690</div><div>xfs_buf_item_size_segment.isra    11690</div><div>xfs_btree_lookup                  11723</div><div>xfs_btree_get_sibling.isra.11     11974</div><div>xfs_buf_item_unlock               12229</div><div>xfs_buf_item_committing           12237</div><div>xfs_icsb_modify_counters          13351</div><div>xfs_icsb_lock_cntr                13365</div><div>xfs_icsb_unlock_cntr              13365</div><div>xfs_iext_bno_to_irec              13458</div><div>xfs_bmap_search_extents           13459</div><div>xfs_bmap_search_multi_extents     13459</div><div>xfs_iext_bno_to_ext               13459</div><div>xfs_bmapi_trim_map.isra.11        13469</div><div>xfs_bmapi_update_map              13469</div><div>xfs_find_bdev_for_inode           13573</div><div>xfs_btree_shift_recs.isra.24      14798</div><div>xfs_cil_prepare_item.isra.1       17769</div><div>xfs_bmbt_disk_set_allf            18126</div><div>xfs_btree_is_lastrec              18581</div><div>xfs_btree_log_block               19648</div><div>xfs_btree_set_ptr_null.isra.10    19649</div><div>xfs_btree_offsets                 19703</div><div>xfs_iunlock                       21348</div><div>xfs_ilock                         21349</div><div>xfs_btree_copy_recs.isra.21       22849</div><div>xfs_buf_item_init                 23331</div><div>xfs_buf_unlock                    23345</div><div>xfs_btree_setbuf                  23948</div><div>xfs_btree_log_recs                24190</div><div>xfs_icsb_counter_disabled         26751</div><div>xfs_buf_read_map                  28786</div><div>xfs_buf_get_map                   28829</div><div>xfs_btree_ptr_is_null.isra.9      29274</div><div>xfs_btree_read_buf_block.isra.    29684</div><div>xfs_btree_set_refs.isra.13        29712</div><div>xfs_trans_add_item                31037</div><div>xfs_trans_free_item_desc          31052</div><div>xfs_buf_trylock                   32463</div><div>xfs_perag_put                     32606</div><div>xfs_perag_get                     32633</div><div>xfs_buf_rele                      32693</div><div>xfs_trans_brelse                  35148</div><div>xfs_btree_ptr_addr                38591</div><div>xfs_btree_ptr_offset              38738</div><div>xfs_trans_read_buf_map            40938</div><div>xfs_trans_buf_item_match          41000</div><div>xfs_btree_ptr_to_daddr            44562</div><div>xfs_btree_lookup_get_block        46548</div><div>xfs_trans_log_buf                 48783</div><div>xfs_buf_item_log                  48806</div><div>xfs_trans_buf_set_type            48815</div><div>xfs_iext_get_ext                  51712</div><div>xfs_bmbt_get_all                  53669</div><div>xfs_btree_get_block               54424</div><div>xfs_iext_idx_to_irec              56768</div><div>xfs_bmbt_get_maxrecs              57774</div><div>xfs_bmbt_get_blockcount           89650</div><div>xfs_bmbt_disk_get_startoff        98285</div><div>xfs_bmbt_init_key_from_rec        98285</div><div>xfs_btree_rec_addr               115199</div><div>xfs_btree_rec_offset             163635</div><div>xfs_btree_key_addr               176837</div><div>xfs_btree_key_offset             186525</div><div>xfs_bmbt_key_diff                254763</div><div>xfs_lookup_get_search_key        257190</div><div>xfs_bmbt_get_startoff            363158</div><div>xfs_next_bit                     631334</div><div>xfs_buf_offset                  1166477</div></div><div><br></div><div><br></div><div>Compare with the two results above, I found  when the iops down to 19k±, lots of functions were traced, such as xfs_buf_trylock, xfs_iext_insert, xfs_btree_insert, etc.</div><div><br></div><div>I cannot find what cause the iops down to 19k±, any suggestion ?<br></div><div><br></div><div><br></div></div>