Re: [pcp] Difference between: kernel.percpu.cpu.user and kernel.pernode.cpu.user

[Shirshendu Chakrabarti <shirshendu@xxxxxxx>]

Hi Mark,

I had a talk with my manager and he has agreed to give you access to one of our machines :)

I will setup a t1.micro machine which will replicate our environment.

Please let me know if this sounds good.

Thanks,

Shirshendu Chakrabarti

On Thu, Nov 27, 2014 at 5:23 PM, Shirshendu Chakrabarti <shirshendu@xxxxxxx> wrote:

Hi Mark,

Please see the responses below:

Please notice the cpu0 in the string above (bold)

Yes, I see the symlinks below forÂ

On Thu, Nov 27, 2014 at 4:17 PM, Mark Goodwin <mgoodwin@xxxxxxxxxx> wrote:

On 11/27/2014 07:34 PM, Shirshendu Chakrabarti wrote:
[...]

I am running on Amazon Linux - 3.4.71-63.amzn1.x86_64

is that an SMP kernel?

Yes it is a SMP kernel:

[root@pcp-test-shir3 pmlogger]# uname -a

Linux pcp-test-shir3.ops.use1b.aws.talk.to 3.4.71-63.98.amzn1.x86_64 #1ÂSMPÂTue Dec 3 01:59:29 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

please see the SMP in the string above (bold).

Also, top command, after pressing 1

top - 11:22:33 up 10 days, 16:54, Â1 user, Âload average: 0.00, 0.01, 0.05

Tasks: Â65 total, Â 1 running, Â64 sleeping, Â 0 stopped, Â 0 zombie

Cpu0ÂÂ: Â0.3%us, Â0.0%sy, Â0.0%ni, 99.7%id, Â0.0%wa, Â0.0%hi, Â0.0%si, Â0.0%st

Mem: Â Â608468k total, Â 192808k used, Â 415660k free, Â Â99576k buffers

Swap: Â Â Â Â0k total, Â Â Â Â0k used, Â Â Â Â0k free, Â Â34620k cached

 PID USER   ÂPR ÂNI ÂVIRT ÂRES ÂSHR S %CPU %MEM  ÂTIME+ ÂCOMMAND                                 ÂÂ

  1 root   Â20  0 19356 1352 1040 S Â0.0 Â0.2  0:00.83 init                                   Â

  2 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 kthreadd                                 Â

  3 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:00.07 ksoftirqd/0                               ÂÂ

  4 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 kworker/0:0                               ÂÂ

  5 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:00.01 kworker/u:0                               ÂÂ

  6 root   ÂRT  0   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 migration/0                               ÂÂ

  7 root    0 -20   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 cpuset                                  Â

  8 root    0 -20   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 khelper                                 ÂÂ

  9 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 kdevtmpfs                                ÂÂ

 Â10 root    0 -20   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 netns                                  ÂÂ

 Â11 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 kworker/u:1                               ÂÂ

 Â15 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:00.51 xenwatch                                 Â

 Â16 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 xenbus                                  Â

 Â83 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:02.49 sync_supers                               ÂÂ

 Â85 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:00.08 bdi-default                               ÂÂ

 Â86 root    0 -20   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 kintegrityd                               ÂÂ

 Â88 root    0 -20   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 kblockd                                 ÂÂ

 Â97 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:36.86 kworker/0:1                               ÂÂ

 102 root    0 -20   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 md                                    Â

 200 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:00.24 khungtaskd                                Â

 205 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:00.21 kswapd0                                 ÂÂ

 206 root   Â25  5   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 ksmd                                   Â

 276 root   Â20  0   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 fsnotify_mark                              ÂÂ

 281 root    0 -20   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 crypto                                  Â

 288 root    0 -20   0  Â0  Â0 S Â0.0 Â0.0  0:00.00 kthrotld                      Â

Â

I am running, PCP-3-10.0-1.

Please see its a t1.micro instance.

are there any 'cpu*' symlinks below /sys/devices/system/node/node0 ?
and does /sys/devices/system/node/node0/cpumap exist? This is where
the PCP "linux" PMDA gets it's node-cpu map from.

[root@pcp-test-shir3 node0]# pwd

/sys/devices/system/node/node0

[root@pcp-test-shir3 node0]# ls -l

total 0

--w------- 1 root root 4096 Nov 26 07:29 compact

lrwxrwxrwx 1 root root  Â0 Nov 26 07:29 cpu0 -> ../../cpu/cpu0

-r--r--r-- 1 root root 4096 Nov 26 07:29 cpulist

-r--r--r-- 1 root root 4096 Nov 16 18:28 cpumap

-r--r--r-- 1 root root 4096 Nov 26 07:29 distance

drwxr-xr-x 3 root root  Â0 Nov 26 07:29 hugepages

-r--r--r-- 1 root root 4096 Nov 16 18:28 meminfo

lrwxrwxrwx 1 root root  Â0 Nov 26 07:29 memory0 -> ../../memory/memory0

lrwxrwxrwx 1 root root  Â0 Nov 26 07:29 memory1 -> ../../memory/memory1

lrwxrwxrwx 1 root root  Â0 Nov 26 07:29 memory2 -> ../../memory/memory2

lrwxrwxrwx 1 root root  Â0 Nov 26 07:29 memory3 -> ../../memory/memory3

lrwxrwxrwx 1 root root  Â0 Nov 26 07:29 memory4 -> ../../memory/memory4

-r--r--r-- 1 root root 4096 Nov 16 18:32 numastat

drwxr-xr-x 2 root root  Â0 Nov 26 07:29 power

-rw-r--r-- 1 root root 4096 Nov 26 07:29 scan_unevictable_pages

lrwxrwxrwx 1 root root  Â0 Nov 16 18:28 subsystem -> ../../../../bus/node

-rw-r--r-- 1 root root 4096 Nov 16 18:28 uevent

-r--r--r-- 1 root root 4096 Nov 26 07:29 vmstat

I see a cpumap file, as can be seen above.

[root@pcp-test-shir3 node0]# less cpumap

00000001

cpumap (END)Â

For more details:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts_micro_instances.html

I have a few other questions:

1. Does PCP have any different requirements for running on the cloud.
2. Do the metrics need to be interpreted differently when running on cloud.

I don't know enough about the Amazon environment to answer that. Is it
possible for me to login and have a poke around? Or can we download an
instance and run it to investigate?

I am not sure if I can get you access to one of our machines. I will have to consult with our manager.

Thanks,

Shirshendu Chakrabarti

Â

Regards
-- Mark

Thanks,

Shirshendu Chakrabarti

On Thu, Nov 27, 2014 at 1:32 PM, Mark Goodwin <mgoodwin@xxxxxxxxxx
<mailto:mgoodwin@xxxxxxxxxx>> wrote:

  On 11/27/2014 05:17 PM, Shirshendu Chakrabarti wrote:

    Hi PCP team,

    Could anyone please respond the question above :)


  The difference is the instance domains and counter aggregation.

  kernel.pernode is aggregated over all CPUs on each numa node.
  On my laptop with 4 CPU cores on one (fake) numa node, we should
  expect kernel.all.cpu.user to equal kernel.pernode.cpu.user, and
  the sum of the per-cpu counters to equal the same, e.g. :

  # pminfo -f kernel.{all,pernode,percpu}.__cpu.user

  kernel.all.cpu.user
    Âvalue 254548080

  kernel.pernode.cpu.user
    Âinst [0 or "node0"] value 254548060

  kernel.percpu.cpu.user
    Âinst [0 or "cpu0"] value 66377110
    Âinst [1 or "cpu1"] value 61187360
    Âinst [2 or "cpu2"] value 64051110
    Âinst [3 or "cpu3"] value 62932480

  The mapping between CPUs and nodes is in hinv.map.cpu_node,
  where each CPU (instance) is mapped to a node number, e.g.

  # pminfo -f hinv.map.cpu_node

  hinv.map.cpu_node
    Âinst [0 or "cpu0"] value 0
    Âinst [1 or "cpu1"] value 0
    Âinst [2 or "cpu2"] value 0
    Âinst [3 or "cpu3"] value 0

  In your case, kernel.pernode.cpu.user is zero, which isn't correct.
  What platform and PCP version are you running?

  Regards
  -- Mark


    Thanks,

    Shirshendu Chakrabarti

    On Wed, Nov 26, 2014 at 3:47 PM, Shirshendu Chakrabarti
    <shirshendu@xxxxxxx <mailto:shirshendu@xxxxxxx>

    <mailto:shirshendu@xxxxxxx <mailto:shirshendu@xxxxxxx>>> wrote:

      ÂHi PCP team,

      ÂCould anyone in the team point me to any literature which explains the
      Âdifference between a per node metric and percpu metric. The
    one-liner and
      Âextended description are absent in pernode metric case and terse in
    percpu case.

      ÂFor example:

      Â[root@pcp-test-shir3 pmlogger]# pminfo -f kernel.percpu.cpu.user

      Âkernel.percpu.cpu.user
         inst [0 or "cpu0"] value 993240
      Â[root@pcp-test-shir3 pmlogger]# pminfo -f kernel.pernode.cpu.user

      Âkernel.pernode.cpu.user
         inst [0 or "node0"] value 0
      ÂI was under the assumption that, kernel.pernode.* and kernel.all.*
    metrics
      Âare the same. I am clearly missing something important.

      ÂThanks,

      ÂShirshendu Chakrabarti




    _________________________________________________
    pcp mailing list
    pcp@xxxxxxxxxxx <mailto:pcp@xxxxxxxxxxx>
    http://oss.sgi.com/mailman/__listinfo/pcp
    <http://oss.sgi.com/mailman/listinfo/pcp>