lockmeter
[Top] [All Lists]

Re: [PATCH] spinlock metering (2.3.42)

To: Linus Torvalds <torvalds@xxxxxxxxxxxxx>
Subject: Re: [PATCH] spinlock metering (2.3.42)
From: Peter Rival <frival@xxxxxxxxxxx>
Date: Wed, 09 Feb 2000 15:44:00 -0500
Cc: John Hawkes <hawkes@xxxxxxx>, linux-kernel@xxxxxxxxxxxxxxxx, linux-smp@xxxxxxxxxxxxxxxx, lockmeter@xxxxxxxxxxx, ezolt@xxxxxxxxxxxxxxxx
Organization: Tru64 QMG Performance Engineering
References: <Pine.LNX.4.10.10002091128230.1345-100000@xxxxxxxxxxxxxxxxxxxxx>
Sender: owner-lockmeter@xxxxxxxxxxx
Linus Torvalds wrote:

> On Wed, 9 Feb 2000, Peter Rival wrote:
> >
> > Not to ask a stupid question, but what type of "real-world usage" would you
> > like to see?  I did most of the Alpha work (yes John I still owe you an
> > update - in your inbox by end of the day) so I have scads of example outputs
> > under various loads, but almost all of them are benchmark-driven.  If there
> > is anything of that ilk that would be remotely interesting to you, let me
> > know and I'll send you what I have (or generate a new one).
>
> Benchmark-driven is fine, if the benchmark is meaningful. Specweb or
> similar would be fascinating, as would lmbench or the SAP or Oracle load
> benchmarks.
>

I don't have Specweb specifically, but I know that Phil Ezolt did some work with
the Alpha lockmeter patches on 2.2 (Phil?).  I'm currently trying to get time on
our Specweb testbed as it stands now, so I'll accelerate that request if
possible.  I'm attaching a copy of the output from a lockstat trace of an AIM 
VII
(don't shoot - it's all I can find right now! ;) shared system benchmark at the
(I believe) 2000 user datapoint (4CPU EV67 ES40 w/ the max 2GB of memory).

I'll get lmbench numbers if they're more interesting to you; Oracle/SAP are a 
bit
tougher as there's no Oracle/SAP for AlphaLinux.  As soon as one is available, I
can get results.  Personally, I like the output of this one better than the
lockinfo in Tru64, but that's just me. :) HTH.

 - Pete
___________________________________________________________________________________________
System: Linux schooner.zk3.dec.com 2.3.34 #2 SMP Sat Jan 11 01:08:08 EST 2048 
alpha
Command: ./multitask 
All (4) CPUs
Selecting locks: threshhold: >0/sec  contention: >0%

Start time: Sat Jan 11 13:48:25 2048
End   time: Sat Jan 11 14:01:11 2048
Delta Time: 7843.86 sec, slots in use: 398, 

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
- - - - - -
SPINLOCKS            HOLD              WAIT
  TOT/SEC  CON   MEAN (   MAX  )   MEAN (   MAX  )       TOTAL      NOWAIT     
SPIN   REJECT  NAME

 28198.67   3%   2.6us(129502us)    59us(129497us)   221186423   216056912  
5129511        0  *TOTAL*

    65.38   1%   0.4us(   8.7us)   1.1us(   2.9us)      512838      512636      
202        0  backlog+0x14
    32.69   1%   0.4us(   4.8us)   1.0us(   2.6us)      256419      256293      
126        0    net_bh+0xa0
    32.69   1%   0.4us(   8.7us)   1.1us(   2.9us)      256419      256343      
 76        0    netif_rx+0x10c

  2342.76   3%   0.8us(    25us)   2.3us(   151us)    18376302    18006816   
369486        0  contig_page_data
  1171.43   3%   1.0us(    25us)   2.2us(   151us)     9188544     8967242   
221302        0    __alloc_pages+0xb4
  1171.33   2%   0.6us(    22us)   2.5us(   150us)     9187758     9039574   
148184        0    __free_pages_ok+0xe4

    32.42   1%   0.5us(    16us)   1.0us(   2.2us)      254326      254202      
124        0  device_request_lock
    24.30   1%   0.5us(    16us)   1.0us(   1.7us)      190571      190505      
 66        0    scsi_allocate_device+0x7c
     8.13   1%   0.5us(    10us)   1.0us(   2.2us)       63755       63697      
 58        0    scsi_release_command+0x40

   813.58   1%   0.4us(   743us)   1.7us(    65us)     6381626     6377221     
4405        0  files_lock
   201.94   1%   0.4us(   244us)   1.6us(    65us)     1583970     1582669     
1301        0    _fput+0x98
   196.95   1%   0.4us(   397us)   2.0us(    17us)     1544861     1544599      
262        0    file_move+0x38
   205.54   1%   0.5us(   743us)   1.8us(    32us)     1612255     1610226     
2029        0    get_empty_filp+0x2c
   205.54   1%   0.4us(   349us)   1.8us(   7.7us)     1612255     1611450      
805        0    get_empty_filp+0xd0
     3.61   1%   0.4us(    31us)   1.4us(   2.8us)       28285       28277      
  8        0    put_filp+0x48

   174.07   1%   0.5us(   342us)   1.6us(   9.7us)     1365355     1365194      
161        0  inode_lock
    25.17   1%   0.4us(    99us)   1.5us(   2.8us)      197441      197422      
 19        0    __mark_inode_dirty+0x38
    27.42   1%   0.5us(   342us)   1.4us(   9.7us)      215053      214986      
 67        0    get_empty_inode+0x4c
     1.27   1%   0.6us(    19us)   1.3us(   1.7us)       10000        9998      
  2        0    get_new_inode+0x74
     1.27   0%   0.7us(    37us)     0us                 10000       10000      
  0        0    iget4+0x74
    18.73   1%   0.5us(   252us)   1.8us(   3.2us)      146945      146935      
 10        0    insert_inode_hash+0x60
    43.77   1%   0.4us(    81us)   1.9us(   3.4us)      343325      343283      
 42        0    iput+0x68
    23.83   1%   0.4us(    88us)   1.7us(   3.1us)      186945      186929      
 16        0    iput+0x130
    28.69   1%   0.5us(    71us)   1.7us(   1.7us)      225053      225052      
  1        0    iput+0x198
     0.02   0%   2.0us(    12us)     0us                   154         154      
  0        0    sync_inodes+0x44
     3.88   1%   1.8us(   119us)   1.8us(   2.5us)       30439       30435      
  4        0    sync_inodes+0x154

   177.84   1%   1.6us(  1423us)   5.6us(   683us)     1394913     1387689     
7224        0  io_request_lock
     3.81   6%   1.3us(    19us)   7.3us(   655us)       29918       28157     
1761        0    __get_request_wait+0x268
    60.49   1%   0.9us(   733us)   5.9us(   683us)      474446      472148     
2298        0    __make_request+0x2d8
    11.33   1%   0.6us(    15us)   4.4us(    24us)       88899       88013      
886        0    add_request+0x130
     7.86   1%   6.3us(    25us)   9.4us(   348us)       61623       61253      
370        0    do_isp1020_intr_handler+0x4c
    60.49   1%   0.5us(   677us)   5.2us(    13us)      474446      474388      
 58        0    ll_rw_block+0x1f0
     8.13   2%   2.7us(    20us)   3.1us(   340us)       63755       62493     
1262        0    scsi_dispatch_cmd+0x1ec
     8.13   1%   0.5us(   5.9us)   3.3us(    14us)       63755       63713      
 42        0    scsi_old_done+0x620
     8.13   1%    12us(  1423us)   2.8us(   283us)       63755       63295      
460        0    scsi_queue_next_request+0x4c
     8.13   1%   1.5us(    15us)   3.3us(    17us)       63755       63741      
 14        0    scsi_request_fn+0x314
     0.15   3%   3.0us(   8.3us)    20us(   306us)        1185        1161      
 24        0    sym53c8xx_intr+0xe0
     1.20   1%   8.9us(   674us)   3.8us(    21us)        9376        9327      
 49        0    unplug_device+0x3c

  2063.67  10%    15us(129502us)   179us(129497us)    16187121    14671047  
1516074        0  kernel_flag
    11.54  21%   1.1us(   661us)   274us(  6642us)       90534       72350    
18184        0    _fput+0x3c
   195.27  12%    16us(  2640us)    66us( 10587us)     1531645     1361895   
169750        0    do_close+0x13c
    15.96  63%   958us(  3325us)   957us(126797us)      125180       47456    
77724        0    do_exit+0xf8
    15.96  65%   322us(129502us)  1154us( 22483us)      125180       44681    
80499        0    do_fork+0x130
     0.02   8%   6.7us(    23us)   532us(  5756us)         153         142      
 11        0    do_select+0x108
     0.34   6%   2.4us(    55us)    51us(  2138us)        2635        2492      
143        0    ext2_get_block+0x98
   447.20   9%   1.9us(   773us)   118us(129497us)     3507747     3193027   
314720        0    ext2_get_block+0x38c
     0.00   0%    34us(    45us)     0us                     3           3      
  0        0    fsync_dev+0x40
    65.89  15%   9.5us(   831us)    95us(  8513us)      516794      439284    
77510        0    osf_mmap+0xd0
     0.13  20%    15us(    79us)   271us(  2091us)        1000         808      
192        0    osf_shmat+0x48
     0.71  21%   544us(  1827us)   502us(  7188us)        5551        4409     
1142        0    schedule+0x69c
     4.73  22%   3.6us(   331us)   317us(124260us)       37107       29158     
7949        0    sock_map_fd+0x94
     0.38  17%   7.2us(    58us)   456us(  5496us)        3016        2517      
499        0    sock_poll+0x94
     0.02  33%  1049us(  5074us)   663us(  4224us)         151         102      
 49        0    sync_old_buffers+0x3c
     0.98  11%    13us(    62us)   239us(  4589us)        7680        6850      
830        0    sys_chdir+0x3c
     0.12   0%   5.1us(    36us)     0us                   910         910      
  0        0    sys_chmod+0x44
     0.12   0%   5.2us(    39us)     0us                   910         910      
  0        0    sys_chown+0x4c
     3.09  27%   482us(  1734us)   339us(  4524us)       24215       17814     
6401        0    sys_execve+0x4c
     0.00   0%   1.5us(    10us)     0us                    35          35      
  0        0    sys_fchdir+0xe8
    20.26   9%   0.6us(   101us)    59us(  8878us)      158894      144798    
14096        0    sys_fcntl+0xd0
    39.85   8%    11us(   725us)    73us(  9953us)      312555      288122    
24433        0    sys_getdents+0x58
     3.71  11%   1.3us(   714us)    84us(  4626us)       29072       25990     
3082        0    sys_ioctl+0xd4
    10.35   1%    15us(   291us)    63us(  2400us)       81165       80519      
646        0    sys_link+0x54
   877.50   6%   0.4us(   175us)    36us(  7658us)     6883004     6488031   
394973        0    sys_lseek+0xf4
     0.11   0%    40us(    92us)     0us                   840         840      
  0        0    sys_mkdir+0x50
    43.01  16%   1.6us(   194us)    55us(  7565us)      337380      284984    
52396        0    sys_newfstat+0x50
    37.11  14%   7.1us(   377us)    43us(  7045us)      291110      251155    
39955        0    sys_newlstat+0x44
    37.45  17%    10us(   405us)    72us( 11041us)      293732      245291    
48441        0    sys_newstat+0x44
   194.38  12%    11us(  1566us)    68us( 67063us)     1524716     1355861   
168855        0    sys_open+0x80
     3.09  15%   0.8us(    48us)   116us(  6871us)       24215       20760     
3455        0    sys_personality+0x3c
     0.13  19%    25us(    82us)   387us(  3225us)        1001         820      
181        0    sys_pipe+0x34
     0.19  14%   5.0us(    27us)   460us(  5743us)        1508        1307      
201        0    sys_poll+0x60
     1.27   8%   139us(  1145us)    28us(  7013us)       10000        9289      
711        0    sys_readlink+0x5c
     0.11   0%   455us(  2438us)     0us                   840         840      
  0        0    sys_rmdir+0x50
    28.97   3%    16us(  1681us)    77us(  6222us)      227270      222429     
4841        0    sys_unlink+0x48
     0.12   0%   5.3us(    42us)     0us                   910         910      
  0        0    sys_utimes+0x44
     0.02  18%   3.9us(    35us)   9.2us(  3913us)         149         123      
 26        0    tty_read+0xec
     0.03  48%   7.8us(   956us)   6.2us(  4742us)         207         109      
 98        0    tty_write+0x2b0
     3.58  15%   9.0us(   120us)   117us(  4606us)       28107       24026     
4081        0    unix_find_other+0x5c

    27.83   1%   4.9us(  1091us)    11us(   395us)      218289      217653      
636        0  loopback_dev+0x118
    27.83   1%   4.9us(  1091us)    11us(   395us)      218289      217653      
636        0    dev_queue_xmit+0x18c

    27.83   1%   0.4us(   316us)   1.6us(   7.7us)      218289      218228      
 61        0  loopback_dev+0x120
    27.83   1%   0.4us(   316us)   1.6us(   7.7us)      218289      218228      
 61        0    dev_queue_xmit+0x5c

  1244.21   1%   1.3us(  5670us)   1.7us(  3479us)     9759376     9729763    
29613        0  lru_list_lock
     1.92   1%   1.3us(    74us)   1.8us(   2.5us)       15035       15032      
  3        0    __bforget+0x34
     8.92   3%   0.6us(    69us)   3.4us(  3479us)       69979       68324     
1655        0    flush_dirty_buffers+0x48
     1.92   1%   1.5us(    55us)   1.3us(    11us)       15035       14953      
 82        0    insert_into_queues+0x9c
  1005.75   1%   0.5us(   690us)   1.6us(    67us)     7888931     7861151    
27780        0    refile_buffer+0x34
     0.00   0%  1377us(  2480us)     0us                    19          19      
  0        0    sync_buffers+0x15c
     1.37   1%   2.5us(  5670us)   1.0us(   1.5us)       10757       10756      
  1        0    sync_buffers+0x1c8
   224.33   1%   4.8us(   822us)   1.4us(    15us)     1759620     1759528      
 92        0    try_to_free_buffers+0x64

  4862.69   5%   0.6us(   970us)   1.4us(   820us)    38142240    36446576  
1695664        0  pagecache_lock
   497.85   8%   0.8us(   639us)   1.4us(   714us)     3905070     3630946   
274124        0    __find_get_page+0x44
  2258.15   5%   0.5us(   970us)   1.4us(   820us)    17712576    16882375   
830201        0    __find_lock_page+0x44
   224.37   7%   1.9us(   367us)   1.2us(    78us)     1759908     1651618   
108290        0    add_to_page_cache_unique+0x48
  1304.38   5%   0.5us(   727us)   1.4us(   316us)    10231376     9793515   
437861        0    do_generic_file_read+0x21c
   224.33   1%   0.5us(   360us)   1.4us(    72us)     1759620     1756404     
3216        0    remove_inode_page+0x38
   353.61   2%   0.4us(   405us)   1.5us(   343us)     2773690     2731718    
41972        0    truncate_inode_pages+0x54

   448.70   1%   0.4us(   358us)   2.0us(    33us)     3519528     3519327      
201        0  pagemap_lru_lock
   224.37   1%   0.5us(    93us)   1.7us(    33us)     1759908     1759857      
 51        0    add_to_page_cache_unique+0x128
   224.33   1%   0.4us(   358us)   2.1us(    12us)     1759620     1759470      
150        0    truncate_inode_pages+0xec

   256.13   8%    58us(   586us)   176us(  1369us)     2009033     1864156   
144877        0  runqueue_lock
    22.53  15%   2.7us(    28us)   286us(  1201us)      176742      150453    
26289        0    __wake_up+0x17c
     0.35  10%   2.2us(    10us)   181us(   676us)        2726        2464      
262        0    process_timeout+0x40
    35.61   3%   0.4us(    13us)   283us(  1146us)      279356      271310     
8046        0    release+0x44
     1.67  27%   2.6us(    11us)   136us(   876us)       13100        9682     
3418        0    schedule_tail+0x74
    46.07   6%   320us(   586us)   248us(  1369us)      361353      343131    
18222        0    schedule+0xfc
     6.16  20%   2.4us(    16us)   159us(  1283us)       48327       39111     
9216        0    schedule+0x3c8
     0.03   4%    33us(   426us)   169us(   651us)         232         225      
  7        0    schedule+0x744
   127.62   1%   0.3us(   4.1us)   232us(  1285us)     1001017      996692     
4325        0    send_sig_info+0x2f0
    16.09  60%   3.1us(    21us)   110us(  1090us)      126180       51088    
75092        0    wake_up_process+0x48

    51.25   1%   0.6us(    76us)    11us(   130us)      402000      401917      
 83        0  sem_ids+0x48
     0.13   1%   1.1us(    16us)   1.1us(   1.1us)        1000         999      
  1        0    semctl_down+0xb4
     0.13   1%   2.1us(    22us)    26us(    41us)        1000         999      
  1        0    semctl_main+0x80
    51.00   1%   0.6us(    76us)    11us(   130us)      400000      399919      
 81        0    sys_semop+0x13c

   162.55   1%   0.5us(    12us)   1.5us(   7.4us)     1275026     1274396      
630        0  timerlist_lock
    19.49   1%   0.5us(   8.8us)   1.7us(   4.7us)      152842      152605      
237        0    add_timer+0x44
    37.18   1%   0.6us(    12us)   1.5us(   7.4us)      291628      291414      
214        0    del_timer+0x40
     3.84   1%   0.7us(   3.8us)   1.4us(   3.2us)       30091       30023      
 68        0    mod_timer+0x44
    98.95   1%   0.5us(    11us)   1.3us(   2.7us)      776126      776020      
106        0    timer_bh+0x20c
     3.10   1%   0.4us(   3.4us)   1.3us(   2.1us)       24339       24334      
  5        0    timer_bh+0x42c

    15.29   1%   0.4us(   4.7us)   1.1us(   2.2us)      119924      119899      
 25        0  tqueue_lock
     0.43   1%   0.3us(   1.2us)   1.1us(   1.3us)        3357        3350      
  7        0    __get_request_wait+0x1d4
     1.20   1%   0.3us(   0.8us)   1.1us(   2.2us)        9376        9365      
 11        0    __make_request+0x35c
     0.36   1%   0.4us(   3.0us)   1.1us(   1.1us)        2833        2831      
  2        0    __wait_on_buffer+0x1e0
     6.63   1%   0.4us(   2.3us)   0.9us(   1.4us)       51993       51989      
  4        0    batch_entropy_store+0xe4
     0.01   0%   0.3us(   1.5us)     0us                   110         110      
  0        0    do_serial_bh+0x40
     0.01   0%   0.4us(   2.5us)     0us                   110         110      
  0        0    rs_sched_event+0x7c
     0.01   0%   0.4us(   0.5us)     0us                    70          70      
  0        0    sync_old_buffers+0xc0
     6.63   1%   0.3us(   4.7us)   1.5us(   1.5us)       52034       52033      
  1        0    tqueue_bh+0x40
     0.01   0%   0.4us(   1.0us)     0us                    41          41      
  0        0    tty_flip_buffer_push+0x80

   673.07   1%   1.0us(   429us)   1.0us(    63us)     5279436     5271869     
7567        0  unused_list_lock
   448.74   1%   0.4us(   409us)   1.0us(    63us)     3519816     3512338     
7478        0    get_unused_buffer_head+0x38
   224.33   1%   2.2us(   429us)   1.0us(   3.3us)     1759620     1759531      
 89        0    try_to_free_buffers+0xd0

     3.58   1%   0.5us(   2.3us)   0.9us(   5.9us)       28082       28081      
  1        0  wait_for_request
     1.79   0%   0.4us(   1.8us)     0us                 14041       14041      
  0        0    __get_request_wait+0x94
     1.79   1%   0.6us(   2.3us)   0.9us(   0.9us)       14041       14040      
  1        0    __get_request_wait+0x360

     3.78   1%   0.4us(    13us)   5.0us(   5.0us)       29638       29637      
  1        0  __down+0x1f0
     3.78   1%   0.7us(   4.9us)    14us(   250us)       29638       29607      
 31        0  __down+0x88
  3518.42   1%   0.7us(  1420us)   1.2us(    38us)    27597965    27487596   
110369        0  __wake_up+0x54
    34.24   1%   0.4us(    79us)   7.2us(   136us)      268604      268573      
 31        0  do_munmap+0x258
     0.13   1%   0.4us(    19us)    20us(    39us)        1000         999      
  1        0  inet_wait_for_connect+0x2a0
    97.36   1%   0.4us(   302us)   0.9us(   4.9us)      763711      763691      
 20        0  insert_vm_struct+0x184
     0.25   1%   0.7us(    29us)   1.1us(   1.3us)        2000        1999      
  1        0  ipc_addid+0xc8
   119.99   1%   0.8us(    20us)   1.4us(   3.8us)      941178      940751      
427        0  kfree+0xa4
   119.99   1%   0.8us(    17us)   1.3us(   4.1us)      941178      940658      
520        0  kmalloc+0x6c
  2576.84   3%   0.6us(    20us)   1.5us(    26us)    20212388    19769826   
442562        0  kmem_cache_alloc+0x4c
  2575.71   4%   0.5us(    25us)   1.5us(    26us)    20203540    19414071   
789469        0  kmem_cache_free+0x50
     0.05   1%   0.4us(   2.2us)   0.7us(   2.1us)         373         372      
  1        0  kmem_cache_grow+0x42c
    25.66   1%   0.4us(   5.6us)   0.9us(   1.0us)      201312      201308      
  4        0  skb_recv_datagram+0xe8
     4.73   1%   0.4us(    50us)    38us(   113us)       37107       36822      
285        0  sock_fasync+0xb4
     0.13   1%   0.4us(   0.9us)    12us(    23us)        1000         996      
  4        0  tcp_accept+0x6c
     0.38   1%   0.4us(    14us)    53us(   124us)        3000        2982      
 18        0  tcp_close+0x68
     0.38   1%   4.8us(    43us)    20us(    20us)        3000        2999      
  1        0  tcp_close+0x6e8
     2.92   5%   4.7us(   680us)   1.2us(    18us)       22913       21960      
953        0  tcp_data_wait+0x1e0
     2.92   1%   0.4us(    61us)    19us(   397us)       22913       22752      
161        0  tcp_data_wait+0x2a8
     2.16   1%    34us(  1105us)   7.3us(   275us)       16938       16818      
120        0  tcp_delack_timer+0x2c
    11.35   1%   1.4us(   715us)   3.3us(    20us)       89000       88893      
107        0  tcp_recvmsg+0x4e0
    11.35   2%   0.4us(   610us)    10us(    63us)       89000       87510     
1490        0  tcp_recvmsg+0x88
    15.21   5%    60us(  1365us)   1.4us(    61us)      119289      114403     
4886        0  tcp_v4_rcv+0x330
    11.35   1%   1.4us(    82us)   1.2us(   1.7us)       89000       88989      
 11        0  tcp_v4_sendmsg+0x184
    11.35   2%   0.4us(   125us)    23us(    86us)       89000       88039      
961        0  tcp_v4_sendmsg+0x60
    12.62   1%   0.3us(   1.6us)   1.0us(   1.2us)       99000       98999      
  1        0  udp_queue_rcv_skb+0x94
     0.42   1%   0.7us(   2.1us)   1.6us(   2.2us)        3312        3309      
  3        0  wait_for_packet+0x24c
     0.58   1%   0.4us(    17us)    36us(   426us)        4529        4490      
 39        0  wait_for_tcp_memory+0x2e8

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
- - - - - -
RWLOCK SPINLOCK      WAIT
  TOT/SEC  CON   MEAN (   MAX  )       TOTAL      NOWAIT     SPIN   REJECT  NAME

  7124.46   1%    81us(125881us)    55883246    55847811    35435        0  
*TOTAL*

    27.83   1%   0.5us(   1.2us)      218289      218255       34        0  
[0xfffffc005fc2f480]
    27.83   1%   0.5us(   1.2us)      218289      218255       34        0    
ip_output+0x78

     0.81   1%   0.1us(   0.2us)        6368        6367        1        0  
arp_tbl+0x134
     0.81   1%   0.1us(   0.2us)        6342        6341        1        0    
neigh_lookup+0x80
     0.00   0%     0us                    26          26        0        0    
neigh_periodic_timer+0x3c

     5.42   1%   0.3us(   0.7us)       42537       42534        3        0  
fib_hash_lock
     5.42   1%   0.3us(   0.7us)       42537       42534        3        0    
fn_hash_lookup+0x44

   796.44   1%   0.8us(    19us)     6247150     6232023    15127        0  
hash_table_lock
     1.92   0%     0us                 15035       15035        0        0    
__bforget+0x48
   568.27   1%   0.8us(    19us)     4457460     4442363    15097        0    
get_hash_table+0x98
     1.92   1%   0.0us(   0.4us)       15035       15034        1        0    
insert_into_queues+0xb0
   224.33   1%   0.2us(   1.2us)     1759620     1759591       29        0    
try_to_free_buffers+0x78

    28.04   1%   0.2us(   0.3us)      219968      219948       20        0  
inet_protocol_lock
    28.04   1%   0.2us(   0.3us)      219968      219948       20        0    
ip_local_deliver+0xec

     3.33   1%   0.2us(   0.7us)       26147       26140        7        0  
inetdev_lock
     0.81   1%   0.3us(   0.7us)        6342        6337        5        0    
arp_rcv+0x48
     0.00   0%     0us                    28          28        0        0    
fib_validate_source+0x60
     0.00   0%     0us                    15          15        0        0    
inet_select_addr+0x50
     2.52   1%   0.2us(   0.6us)       19762       19760        2        0    
ip_route_input_slow+0x4c

    32.69   1%   0.6us(   1.8us)      256419      256227      192        0  
ptype_lock
    32.69   1%   0.6us(   1.8us)      256419      256227      192        0    
net_bh+0x1a8

     3.46   1%   0.2us(   0.3us)       27104       27091       13        0  
rt_hash_lock
     2.73   1%   0.2us(   0.3us)       21413       21401       12        0    
ip_route_input+0xd4
     0.64   1%   0.2us(   0.2us)        5000        4999        1        0    
ip_route_output+0xdc
     0.08   0%     0us                   663         663        0        0    
rt_check_expire+0x7c
     0.00   0%     0us                    28          28        0        0    
rt_intern_hash+0xa0

   239.74   2%   151us(125881us)     1880507     1861435    19072        0  
tasklist_lock
     0.02   0%     0us                   154         154        0        0    
count_active_tasks+0x2c
    15.96   1%    82us(   755us)      125180      124604      576        0    
do_fork+0x8ec
    15.96   1%    14us(   644us)      125180      124536      644        0    
exit_notify+0x40
    15.96   1%    11us(  1187us)      125180      124266      914        0    
exit_notify+0x244
     0.00   6%   1.0us(   1.0us)          18          17        1        0    
get_pid+0xb0
     0.00   0%     0us                     1           1        0        0    
kill_pg_info+0x60
   127.49   1%   0.8us(    30us)     1000000      996102     3898        0    
kill_something_info+0x120
     1.27   1%   1.0us(   3.9us)       10000        9976       24        0    
proc_pid_lookup+0x90
     0.17   1%   1.2us(   960us)        1358        1352        6        0    
proc_root_lookup+0x6c
    15.96   6%   404us(125881us)      125180      118256     6924        0    
release+0xa4
     0.03   0%     0us                   232         232        0        0    
schedule+0x6d4
     0.00   0%     0us                     1           1        0        0    
sys_setsid+0x2c
    46.92   2%   1.1us(   987us)      368023      361938     6085        0    
sys_wait4+0x1f8

    13.60   1%   0.1us(   0.2us)      106679      106676        3        0  
udp_hash_lock
    12.62   1%   0.2us(   0.2us)       99000       98998        2        0    
udp_rcv+0x178
     0.25   1%   0.0us(   0.0us)        2000        1999        1        0    
udp_v4_get_port+0x58
     0.25   0%     0us                  2000        2000        0        0    
udp_v4_hash+0x64
     0.21   0%     0us                  1679        1679        0        0    
udp_v4_mcast_deliver+0x48
     0.25   0%     0us                  2000        2000        0        0    
udp_v4_unhash+0x50

    15.35   1%   1.4us(    24us)      120428      120353       75        0  
unix_table_lock
     7.68   1%   0.8us(   1.7us)       60214       60191       23        0    
unix_create1+0x17c
     7.68   1%   1.6us(    24us)       60214       60162       52        0    
unix_release_sock+0x40

   202.83   1%   7.9us(   730us)     1590980     1590646      334        0  
xtime_lock
     3.86   1%    30us(   730us)       30299       30235       64        0    
do_gettimeofday+0x3c
    98.95   1%   2.4us(    65us)      776126      775943      183        0    
timer_bh+0x40
   100.02   1%   3.5us(   272us)      784555      784468       87        0    
timer_interrupt+0x60

    15.96   1%   0.1us(   0.1us)      125180      125178        2        0  
copy_files+0x118
   201.44   1%   0.0us(   0.1us)     1580075     1580057       18        0  
do_close+0x4c
   200.67   1%   0.0us(   0.0us)     1574005     1573996        9        0  
do_close+0xcc
   205.54   1%   0.0us(   0.1us)     1612255     1612214       41        0  
get_unused_fd+0x30
     6.17   1%   0.0us(   0.0us)       48430       48429        1        0  
open_dentry+0x108
    45.39   1%   0.1us(   0.2us)      356032      356025        7        0  
osf_mmap+0xf4
    38.25   1%   0.1us(   0.2us)      300000      299996        4        0  
sock_def_readable+0x24
    22.95   1%   0.1us(   0.4us)      180000      179997        3        0  
sys_fcntl+0x54
    39.85   1%   0.1us(   0.2us)      312555      312548        7        0  
sys_getdents+0x7c
   877.50   1%   0.1us(   0.2us)     6883004     6882931       73        0  
sys_lseek+0x4c
    43.01   1%   0.1us(   0.1us)      337380      337371        9        0  
sys_newfstat+0x70
   190.78   1%   0.0us(   0.0us)     1496431     1496416       15        0  
sys_open+0xe4
  1354.29   1%   0.1us(   0.3us)    10622897    10622797      100        0  
sys_read+0x4c
  2320.47   1%   0.1us(   0.3us)    18201471    18201261      210        0  
sys_write+0x4c
    15.21   1%   0.2us(   0.3us)      119289      119263       26        0  
tcp_v4_rcv+0x1b8
    26.81   1%   0.3us(   0.7us)      210289      210277       12        0  
tcp_write_space+0x24
    12.62   1%   0.1us(   0.1us)       99000       98996        4        0  
udp_sendmsg+0x24c
    12.62   1%   0.1us(   0.2us)       99000       98999        1        0  
unix_dgram_sendmsg+0xb8
     7.68   1%   0.0us(   0.0us)       60214       60213        1        0  
unix_release_sock+0x68
     7.68   1%   0.1us(   0.1us)       60214       60213        1        0  
unix_release_sock+0x7c
    12.62   1%   0.1us(   0.2us)       99000       98998        2        0  
unix_stream_sendmsg+0x264
    12.62   1%   0.1us(   0.2us)       99000       98996        4        0  
unix_stream_sendmsg+0xa8
    28.83   1%   0.1us(   0.2us)      226107      226103        4        0  
unix_write_space+0x24
___________________________________________________________________________________________

<Prev in Thread] Current Thread [Next in Thread>