pcp
[Top] [All Lists]

Re: [pcp] pminfo -t aborts pmcd (#30)

To: performancecopilot/pcp <pcp@xxxxxxxxxxxxxxxxxx>
Subject: Re: [pcp] pminfo -t aborts pmcd (#30)
From: ABC <notifications@xxxxxxxxxx>
Date: Wed, 01 Jul 2015 00:49:25 -0700
Delivered-to: pcp@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha1; c=relaxed; d=github.com; h=from:reply-to:to:in-reply-to:references:subject:mime-version:content-type:content-transfer-encoding:list-id:list-archive:list-post:list-unsubscribe; s=s20150108; bh=LnXpGwTgCxtkwX9PZZ/OuYRVW30=; b=QPbMWe/XLZ6ZldCB HwHswTT1n5O7VlSn1wDkHLKSLYmXTkkqpWOnqYfvEtqotcc3jjvByxVKf5SvZtUI e4Qub9cNQKItG+TC12DAPViaP74hkoLlzq5l4k4vtKxNE65j5CbnwQA5PKdNy3tr Wbn+PGzv6xbfpga2yAf5A1hL1PM=
In-reply-to: <performancecopilot/pcp/issues/30@xxxxxxxxxx>
List-archive: https://github.com/performancecopilot/pcp
List-id: performancecopilot/pcp <pcp.performancecopilot.github.com>
List-post: <mailto:reply+00bd08b61a9375b8894dad4bcfa9ad7cb77e9a56c212072992cf0000000111ab5d8592a169ce057c1750@reply.github.com>
List-unsubscribe: <mailto:unsub+00bd08b61a9375b8894dad4bcfa9ad7cb77e9a56c212072992cf0000000111ab5d8592a169ce057c1750@reply.github.com>, <https://github.com/notifications/unsubscribe/AL0ItgNpQZCuSnxYku4zEwq6y4Ae6k15ks5oY5MFgaJpZM4FOwu4>
References: <performancecopilot/pcp/issues/30@xxxxxxxxxx>
Reply-to: performancecopilot/pcp <reply+00bd08b61a9375b8894dad4bcfa9ad7cb77e9a56c212072992cf0000000111ab5d8592a169ce057c1750@xxxxxxxxxxxxxxxx>

I didn't notice at first that pminfo -t output changes to IPC protocol failure from ipc.shm.max_segproc.

ipc.msg.num_smsghdr [number of system message headers (from msgctl(..,IPC_INFO,..))]
ipc.msg.max_seg [maximum number of message segments (from msgctl(..,IPC_INFO,..))]
ipc.shm.max_segsz [maximum shared segment size in bytes (from shmctl(..,IPC_INFO,..))]
ipc.shm.min_segsz [minimum shared segment size in bytes (from shmctl(..,IPC_INFO,..))]
ipc.shm.max_seg [maximum number of shared segments in system (from shmctl(..,IPC_INFO,..))]
ipc.shm.max_segproc: pmLookupName: IPC protocol failure
ipc.shm.max_shmsys: pmLookupName: IPC protocol failure
vfs.files.count: pmLookupName: IPC protocol failure
vfs.files.free: pmLookupName: IPC protocol failure
vfs.files.max: pmLookupName: IPC protocol failure
vfs.inodes.count: pmLookupName: IPC protocol failure

pminfo -T behaves similar to pminfo -t

ipc.shm.min_segsz
Help:
minimum shared segment size in bytes (from shmctl(..,IPC_INFO,..))

ipc.shm.max_seg
Help:
maximum number of shared segments in system (from shmctl(..,IPC_INFO,..))
ipc.shm.max_segproc: pmLookupName: IPC protocol failure
ipc.shm.max_shmsys: pmLookupName: IPC protocol failure
vfs.files.count: pmLookupName: IPC protocol failure
vfs.files.free: pmLookupName: IPC protocol failure

But, pminfo -T ipc does not abort pmcd.

Unfortunately, there is no debuginfo/dbg package for pcp for Ubuntu. gdb stack trace output when attached to pmcd process and run pminfo -t:

Program received signal SIGSEGV, Segmentation fault.
0xb728d43c in pmdaTreePMID () from /usr/lib/libpcp_pmda.so.3
(gdb) bt
#0  0xb728d43c in pmdaTreePMID () from /usr/lib/libpcp_pmda.so.3
#1  0xb725da14 in ?? () from /var/lib/pcp/pmdas/mmv/pmda_mmv.so
#2  0xb77b2393 in DoPMNSNames ()
#3  0xb77a969c in HandleClientInput ()
#4  0xb77a99de in ?? ()
#5  0xb77a8348 in main ()

(gdb) x/11i 0xb728d43c - 11
   0xb728d431 <pmdaTreePMID+17>:    inc    %esp
   0xb728d432 <pmdaTreePMID+18>:    and    $0x20,%al
   0xb728d434 <pmdaTreePMID+20>:    mov    0x24(%esp),%edx
   0xb728d438 <pmdaTreePMID+24>:    mov    0x28(%esp),%esi
=> 0xb728d43c <pmdaTreePMID+28>:    mov    (%eax),%eax
   0xb728d43e <pmdaTreePMID+30>:    mov    0x8(%eax),%eax
   0xb728d441 <pmdaTreePMID+33>:    call   0xb728d060
   0xb728d446 <pmdaTreePMID+38>:    test   %eax,%eax
   0xb728d448 <pmdaTreePMID+40>:    je     0xb728d468 <pmdaTreePMID+72>
   0xb728d44a <pmdaTreePMID+42>:    mov    0x14(%eax),%eax
   0xb728d44d <pmdaTreePMID+45>:    cmp    $0xffffffff,%eax
(gdb) i reg
eax            0x0  0
ecx            0xb72ad700   -1221929216
edx            0xb8c83bb0   -1194837072
ebx            0xb7260000   -1222246400
esp            0xbfbb5280   0xbfbb5280
ebp            0x44 0x44
esi            0xb8c83c1c   -1194836964
edi            0xb8c83c1c   -1194836964
eip            0xb728d43c   0xb728d43c <pmdaTreePMID+28>
eflags         0x210246 [ PF ZF IF RF ID ]
cs             0x73 115
ss             0x7b 123
ds             0x7b 123
es             0x7b 123
fs             0x0  0
gs             0x33 51

Looks like NULL pointer read.

Content of /var/log/pcp/pmcd/pmcd.log

Log for pmcd on n started Wed Jul  1 10:37:08 2015

[Wed Jul  1 10:37:08] pmcd(18026) Error: Permission clash for unix: with earlier statement for unix:

active agent dom   pid  in out ver protocol parameters
============ === ===== === === === ======== ==========
pmcd           2                 2 dso i:5  lib=/var/lib/pcp/pmdas/pmcd/pmda_pmcd.so entry=pmcd_init [0xb776b2e0]
linux         60                 2 dso i:4  lib=/var/lib/pcp/pmdas/linux/pmda_linux.so entry=linux_init [0xb726c780]
proc           3 18036  10  11   2 bin pipe cmd=/var/lib/pcp/pmdas/proc/pmdaproc -d 3
mmv           70                 2 dso i:4  lib=/var/lib/pcp/pmdas/mmv/pmda_mmv.so entry=mmv_init [0xb725dea0]
xfs           11 18042  12  13   2 bin pipe cmd=/var/lib/pcp/pmdas/xfs/pmdaxfs -d 11
jbd2         122                 2 dso i:4  lib=/var/lib/pcp/pmdas/jbd2/pmda_jbd2.so entry=jbd2_init [0xb7258860]

Host access list:
00 01 Cur/MaxCons host-spec                               host-mask                               lvl host-name
== == =========== ======================================= ======================================= === ==============
 y  y     0     0 127.0.1.1                               255.255.255.255                           0 localhost
 y  y     0     0 /                                       /                                         1 unix:
 n  n     0     0 0.0.0.0                                 0.0.0.0                                   4 .*
 n  n     0     0 ::                                      ::                                        8 :*
User access list empty: user-based access control turned off
Group access list empty: group-based access control turned off


pmcd: PID = 18026, PDU version = 2
pmcd request port(s):
  sts fd   port  family address
  === ==== ===== ====== =======
  ok  1026       unix   /var/run/pcp/pmcd.socket
  ok  1024 44321 inet   INADDR_ANY
  ok  1025 44321 ipv6   INADDR_ANY
[Wed Jul  1 10:43:28] pmcd(18026) Error: Unexpected signal 11 ...

Dumping to core ...

pcp output:

# pcp
Performance Co-Pilot configuration on rin:

 platform: Linux n 3.16.0-38-generic #52~14.04.1-Ubuntu SMP Fri May 8 09:44:48 UTC 2015 i686
 hardware: 2 cpus, 2 disks, 1995MB RAM
 timezone: MSK-3
     pmcd: Version 3.8.12-1, 6 agents, 1 client
     pmda: pmcd proc xfs linux mmv jbd2
 pmlogger: primary logger: n/20150701.10.46
     pmie: n: /var/log/pcp/pmie/n/pmie.log

I will try newer version of pcp later.


Reply to this email directly or view it on GitHub.

<Prev in Thread] Current Thread [Next in Thread>