Simon Hayhurst (simon++at++zermatt.engr.sgi.com)
Tue, 3 Mar 1998 08:48:24 -0800 (PST)
I'm attaching here two pieces of information from Javier's encounter with this
problem, a piece of helpful source code and a quick plan of action to make
optimising the space a quicker process until autoplacement of DSOs can be made
more memory friendly (no date to offer on this).
Simon
On Mar 3, 1:26am, Mike Weiblen wrote:
> Subject: Re: Increasing shared arena.
> An arena requires a contiguous block of virutal memory space, but VM is
> fragmented by the placement of DSOs.
>
> There seem to be two effects at work:
> 1) VM is split (approx 1GB and 512MB) by default DSO address
> assignments.
> 2) arena is being placed in the smaller of the two largest VM blocks.
>
> Workarounds:
> A) use PFSHAREDBASE to place the arena in the larger (~1GB) block.
> B) move the DSOs to the bottom of VM using rqs, yielding one block of
> ~1.5GB.
> C) use 64-bit addressing
>
> It would be real nice if the OS would do the DSO packing automatically,
> perhaps as part of swmgr/inst.
>
> -- mew
>
> PS Many thanks to everyone involved in figuring this out
>
>
> Svend Tang-Petersen wrote:
> >
> > Hi pfAll.
> >
> > I'm trying to increase the shared arena to 1GB. I did reconfigure the
> > kernel and reboot, so when
> > I type 'limit' it actually shows 1GB.
> >
> > Secondly I set
> >
> > pfSharedArenaSize(512000000);
> > pfSemaArenaSize( 512000 );
> > pfInitArenas();
> > pfInit();
> >
> > but when I start the application, it exits with a message saying that it
> > was only able to
> > allocate 512MB.
> >
> > Any ideas ?
> >
> > (And yes, I have 1.28 GB)
> >
> > --
> >
> > Svend Tang-Petersen, MSc
> >
> > LEGO Silicon Graphics
> > Kloevermarken 120 Stationsparken 25
> > 7190 Billund 2600 Glostrup
> > Denmark Denmark
> >
> > e-mail: svend++at++digi.lego.com , svend++at++copen.sgi.com
>
> --
> Mike Weiblen talkto:972-960-2301 x292
> PARADIGM Simulation, Inc. faxto:972-960-9049
> 14900 Landmark, Suite 400 mailto:mew++at++paradigmsim.com
> Dallas TX 75240 http://www.paradigmsim.com
> =======================================================================
> List Archives, FAQ, FTP: http://www.sgi.com/Technology/Performer/
> Submissions: info-performer++at++sgi.com
> Admin. requests: info-performer-request++at++sgi.com
>-- End of excerpt from Mike Weiblen
--
--------------------------------------------------------------------------------
Simon Hayhurst | Phone (1)-650-933 6258
Applied Engineering | Fax (1)-650-964 8671
"Opening Minds, Closing Deals" | V-mail 933-6258
"Less Code, More Speed, Better Quality" | E-mail simon ++at++ sgi.com
--------------------------------------------------------------------------------
/* The program below will display the virtual
address map of a given process.
Available via anonymous FTP on ftp.sgi.com in the
directory ~ftp/support/Pipeline.
Compile and run it as follows:
cc dpa.c -o dpa; ./dpa pid
( where pid is the process id of a process
which can obtained from the ps(1) command. )
*/
#include "stdio.h"
#include "stdlib.h"
#include "sys/procfs.h"
#include "fcntl.h"
#include "errno.h"
static void doname(char *pname);
prmap_sgi_t *pmaps;
#define MAXMAP 2000
int maxmap = MAXMAP;
/* set of page state options */
char *stateopts[] = { "READ", "WRITE", "EXEC",
"SHARED", "BREAK", "STACK", "PHYS", "PRIMARY",
"SREGION", "COW", "NOTCACHED", "SHMEM" };
/* MA_* flags that correspond to above options */
int maopts[] = { MA_READ, MA_WRITE, MA_EXEC,
MA_SHARED, MA_BREAK, MA_STACK, MA_PHYS,
MA_PRIMARY, MA_SREGION, MA_COW, MA_NOTCACHED,
MA_SHMEM };
void main(int argc, char **argv) {
pid_t pid = -1;
char pname[20];
if(argc!=2) {
fprintf(stderr, "Usage: %s pid\n", \
argv[0]);
exit(1);
}
pid = (pid_t)atoi(argv[1]);
pmaps = malloc((maxmap + 1) *
sizeof(prmap_t));
sprintf(pname, "/proc/%05d", pid);
doname(pname);
exit(0);
}
static void doname(char *pname) {
int pfd;
struct prpsinfo psi;
struct prmap_sgi_arg pma;
int nmaps;
prmap_sgi_t *pmapp;
if ((pfd = open(pname, O_RDWR)) < 0) {
if (errno == EISDIR || errno == ESRCH || \
errno == ENOENT)
return;
if (errno != EACCES)
fprintf(stderr, "ERROR:Cannot open \
%s:%s\n", pname, strerror(errno));
return;
}
if (ioctl(pfd, PIOCPSINFO, &psi) != 0) {
if (errno != ESRCH)
fprintf(stderr, "ERROR:Cannot PSINFO \
%s:%s\n", pname, strerror(errno));
close(pfd);
return;
}
printf("pid %d, ",psi.pr_pid);
if (psi.pr_zomb == 0) {
pma.pr_vaddr = (caddr_t)pmaps;
pma.pr_size = sizeof(prmap_sgi_t) * maxmap;
if ((nmaps = ioctl(pfd, PIOCMAP_SGI, \
&pma)) < 0) {
fprintf(stderr, "ERROR:Cannot NMAP_SGI \
%s:%s\n", pname, strerror(errno));
close(pfd);
return;
}
printf("process has %d regions\n", nmaps);
}
if (psi.pr_zomb == 0) {
printf("addr\t\tsize\t\t off\t\tflags\n");
printf("----\t\t----\t\t ---\t\t-----\n");
for (pmapp = pmaps; pmapp->pr_mflags; \
pmapp++) {
printf("0x%x\t0x%x\t\t0x%x\t\t", \
pmapp->pr_vaddr, pmapp->pr_size, \
pmapp->pr_off);
printf("%s%s%s%s%s%s%s%s%s%s%s%s\n",
pmapp->pr_mflags & MA_READ ? " READ" : "",
pmapp->pr_mflags & MA_WRITE ? \
" WRITE" : "",
pmapp->pr_mflags & MA_EXEC ? " EXEC" : "",
pmapp->pr_mflags & MA_SHARED ? \
" SHARED" : "",
pmapp->pr_mflags & MA_BREAK ? \
" BREAK" : "",
pmapp->pr_mflags & MA_STACK ? \
" STACK" : "",
pmapp->pr_mflags & MA_PHYS ? " PHYS" : "",
pmapp->pr_mflags & MA_PRIMARY ? \
" PRIMARY" : "",
pmapp->pr_mflags & MA_SREGION ? \
" SREGION" : "",
pmapp->pr_mflags & MA_COW ? " COW" : "",
pmapp->pr_mflags & MA_NOTCACHED? \
" NOTCACHED":"",
pmapp->pr_mflags & MA_SHMEM ? \
" SHMEM" : "");
}
} else
printf(" Process is a zombie\n");
close(pfd);
}
Since you need a quick solution, please use the following steps:
a) Run your program first, asking for 512MB of arena size
(be sure that either the rlimits on your machine are set to use
all the memory or use my small routine vtUseAllMemory
before the arena initialization in the APP)
The code was:
struct rlimit limits;
/* increase the systune limit for the text size */
getrlimit( RLIMIT_VMEM, &limits );
limits.rlim_cur = limits.rlim_max;
setrlimit( RLIMIT_VMEM, &limits );
/* Resident at the same time */
getrlimit( RLIMIT_RSS, &limits );
limits.rlim_cur = limits.rlim_max;
setrlimit( RLIMIT_RSS, &limits );
b) While running, check the APP pid (process id)
(performer use to print out the pid for all threads)
c) While running, run ./dpa <APP_pid>
The source and instructions for dpa.c are provided at the end
of this email.
This will printout all the addresses chunks and sizes for
the APP.
d) Reading the pda table you should be able to find the big hole:
Find first the biggest jump in addresses.
Add to the last address before the jump the size of this
last allocation.
This address will be your PFSHAREDBASE for the next run.
e) Exit the performer program.
f) Run the program again but now with the desired PFSHAREDSIZE
and withe the above PFSHAREDBASE. In this case the program will map
the arena at the 1.2 GB hole.
This sounds complicated but it is not. Following please find an step by step
example, including calculations to achieve a big arena in perfly.
a) *********************************************************************
(sets the arena size to 512MB for initial testing: 512*1024*1024)
# setenv PFSHAREDSIZE 536870912
# ./perfly or whatever is the name of your program
b) *********************************************************************
In my case perfly prints out the pid list for each thread:
----------- Peformer Process State --------
Proc: APP pid:2350 <----------------------------
Proc: ISECT pid:2354
Proc: DBASE pid:2355
Proc: CLOCK pid:2351
... etc
If your application cannot provide this information you can figure it out using
# ps | grep <your_application_name>
Use to be the lowest pid for the APP.
In this example pid for the APP is 2350
c) ************************************************************************
# ./dpa 2350
pid 2350, process has 53 regions
addr size off flags
---- ---- --- -----
0x0 0x4000 0x0 READ WRITE COW
0x4000 0x4000 0x0 READ WRITE SHARED PHYS
0x8000 0x4000 0x4000 READ WRITE SHARED PHYS NOTCACHED
0xc000 0x4000 0x8000 READ WRITE SHARED PHYS
0x10000 0x4000 0x0 READ SHARED PHYS NOTCACHED
0x200000 0x4000 0x0 READ WRITE COW
0x4000000 0x44000 0x0 READ WRITE SHARED
0x4044000 0x4000 0x0 READ SHARED PHYS NOTCACHED
0x4048000 0x80000 0x0 READ WRITE SHARED
0x40c8000 0x80000 0x0 READ WRITE SHARED
0x4148000 0x800000 0x0 READ WRITE
0x9ef0000 0x18000 0x0 READ EXEC SHARED
0x9f14000 0x8000 0x14000 READ WRITE COW
0xad80000 0x28000 0x0 READ EXEC SHARED
0xadb4000 0x8000 0x24000 READ WRITE COW
0xd920000 0x78000 0x0 READ EXEC SHARED
0xd9a4000 0xc000 0x74000 READ WRITE COW
0xd9b0000 0x3c000 0x0 READ EXEC SHARED
0xd9f8000 0x4000 0x38000 READ WRITE COW
0xda20000 0x16c000 0x0 READ EXEC SHARED
0xdbd8000 0x14000 0x168000 READ WRITE COW
0xf610000 0x14000 0x0 READ EXEC SHARED
0xf634000 0x4000 0x14000 READ WRITE COW
0xf640000 0x70000 0x0 READ EXEC SHARED
0xf6c0000 0xc000 0x70000 READ WRITE COW
0xf6d0000 0x18000 0x0 READ EXEC SHARED
0xf6f4000 0x4000 0x14000 READ WRITE COW
0xf700000 0xd8000 0x0 READ EXEC SHARED
0xf7e4000 0x10000 0xd4000 READ WRITE COW
0xf840000 0x30000 0x0 READ EXEC SHARED
0xf880000 0x18000 0x30000 READ WRITE COW
0xf920000 0xc000 0x0 READ EXEC SHARED
0xf938000 0x4000 0x8000 READ WRITE COW
0xfa00000 0x100000 0x0 READ EXEC SHARED
0xfb4c000 0x14000 0xfc000 READ WRITE COW
0xfb60000 0x3c000 0x0 READ EXEC SHARED
0xfb9c000 0xc000 0x3c000 READ WRITE COW
0xfba8000 0x10000 0x0 READ WRITE COW
0x10000000 0x24000 0x0 READ EXEC SHARED PRIMARY
0x10030000 0xc000 0x20000 READ WRITE PRIMARY COW
0x1003c000 0xbc000 0x0 READ WRITE BREAK PRIMARY COW
0x5abe0000 0x310000 0x0 READ EXEC SHARED
0x5aefc000 0x484000 0x30c000 READ WRITE COW
0x5c1d0000 0x54000 0x0 READ EXEC SHARED
0x5c260000 0x10000 0x50000 READ WRITE COW
0x5c3d0000 0x4c000 0x0 READ EXEC SHARED
0x5c428000 0x1c000 0x48000 READ WRITE COW
0x5c4b0000 0x1c000 0x0 READ EXEC SHARED
0x5c4d8000 0xc000 0x18000 READ WRITE COW
0x5c4e4000 0x20000000 0x0 READ WRITE
0x7c4e4000 0x10000 0x0 READ WRITE SHARED
0x7c4f8000 0x2c000 0x0 READ WRITE SHARED
0x7ffe8000 0x10000 0x0 READ WRITE STACK COW
d) ********************************************************************
Looking into the pda table you will see:
That the 512MB arena (0x20000000 in hex) is mapped too close to the
end of the address space (0x5c4e4000), and that there is a big jump of
addresses:
0x1003c000 0xbc000 0x0 READ WRITE BREAK PRIMARY COW
0x5abe0000 0x310000 0x0 READ EXEC SHARED
>From 0x1003c000 to 0x5abe0000 there is a big hole, ~1.2 GB. This is where we
would like the arena to be placed, since there is more space that from
0x5c4e4000 to the end (just space for 512MB).
Let's calculate the address:
0x1003c000 + 0xbc000 = 0x100f8000
Just to be sure, let's select 0x10100000 as our base for the arena.
In this example, our maximum arena can be:
0x5abe0000-0x10100000 = 0x4aae0000 = 1,252,917,248 bytes (nearly 1.2GB)
and our base for the arena will be 0x10100000
Hence PFSHAREDBASE = 0x10100000, and just to be sure let's increase it
even more 0x11100000
(leave some extra room if it does not work with the initial estimation )
e) ******************************************************************
Exit the performer app.
f) ******************************************************************
Then if we wanted 1GB of arena (1024*1024*1024 = 1073741824 )
# setenv PFSHAREDSIZE 1073741824
# setenv PFSHAREDBASE 0x11100000
And then run the application and problem solved.
./perfly ... (now running with 1GB of arena size)
**********************************************************************
ONLY FOR YOUR INFORMATION:
If I run dpa again against the new APP pid it looks like:
#./dpa 2588
pid 2588, process has 55 regions
addr size off flags
---- ---- --- -----
0x0 0x4000 0x0 READ WRITE COW
0x4000 0x4000 0x0 READ WRITE SHARED PHYS
0x8000 0x4000 0x4000 READ WRITE SHARED PHYS NOTCACHED
0xc000 0x4000 0x8000 READ WRITE SHARED PHYS
0x10000 0x4000 0x0 READ SHARED PHYS NOTCACHED
0x200000 0x4000 0x0 READ WRITE COW
0x4000000 0x44000 0x0 READ WRITE SHARED
0x4044000 0x4000 0x0 READ SHARED PHYS NOTCACHED
0x4048000 0x80000 0x0 READ WRITE SHARED
0x40c8000 0x80000 0x0 READ WRITE SHARED
0x4148000 0x800000 0x0 READ WRITE
0x9ef0000 0x18000 0x0 READ EXEC SHARED
0x9f14000 0x8000 0x14000 READ WRITE COW
0xad80000 0x28000 0x0 READ EXEC SHARED
0xadb4000 0x8000 0x24000 READ WRITE COW
0xd920000 0x78000 0x0 READ EXEC SHARED
0xd9a4000 0xc000 0x74000 READ WRITE COW
0xd9b0000 0x3c000 0x0 READ EXEC SHARED
0xd9f8000 0x4000 0x38000 READ WRITE COW
0xda20000 0x16c000 0x0 READ EXEC SHARED
0xdbd8000 0x14000 0x168000 READ WRITE COW
0xf610000 0x14000 0x0 READ EXEC SHARED
0xf634000 0x4000 0x14000 READ WRITE COW
0xf640000 0x70000 0x0 READ EXEC SHARED
0xf6c0000 0xc000 0x70000 READ WRITE COW
0xf6d0000 0x18000 0x0 READ EXEC SHARED
0xf6f4000 0x4000 0x14000 READ WRITE COW
0xf700000 0xd8000 0x0 READ EXEC SHARED
0xf7e4000 0x10000 0xd4000 READ WRITE COW
0xf840000 0x30000 0x0 READ EXEC SHARED
0xf880000 0x18000 0x30000 READ WRITE COW
0xf920000 0xc000 0x0 READ EXEC SHARED
0xf938000 0x4000 0x8000 READ WRITE COW
0xfa00000 0x100000 0x0 READ EXEC SHARED
0xfb4c000 0x14000 0xfc000 READ WRITE COW
0xfb60000 0x3c000 0x0 READ EXEC SHARED
0xfb9c000 0xc000 0x3c000 READ WRITE COW
0xfba8000 0x20000 0x0 READ WRITE COW
0x10000000 0x24000 0x0 READ EXEC SHARED PRIMARY
0x10030000 0xc000 0x20000 READ WRITE PRIMARY COW
0x1003c000 0x114000 0x0 READ WRITE BREAK
PRIMARY COW
0x11100000 0x40000000 0x0 READ WRITE
0x51100000 0x10000 0x0 READ WRITE SHARED
0x51114000 0x2c000 0x0 READ WRITE SHARED
0x5abe0000 0x310000 0x0 READ EXEC SHARED
0x5aefc000 0x484000 0x30c000 READ WRITE COW
0x5c1d0000 0x54000 0x0 READ EXEC SHARED
0x5c260000 0x10000 0x50000 READ WRITE COW
0x5c3d0000 0x4c000 0x0 READ EXEC SHARED
0x5c428000 0x1c000 0x48000 READ WRITE COW
0x5c4b0000 0x1c000 0x0 READ EXEC SHARED
0x5c4d8000 0xc000 0x18000 READ WRITE COW
0x5c7a0000 0x4000 0x0 READ EXEC SHARED
0x5c7c0000 0x4000 0x0 READ WRITE COW
0x7ffe8000 0x10000 0x0 READ WRITE STACK COW
now you can see that
0x11100000 0x40000000 0x0 READ WRITE
is the arena and is mapped where we wanted (in the hole).
*************************************************************************
SOURCE CODE FOR dpa.c
/* The program below will display the virtual
address map of a given process.
Available via anonymous FTP on ftp.sgi.com in the
directory ~ftp/support/Pipeline.
Compile and run it as follows:
cc dpa.c -o dpa; ./dpa pid
( where pid is the process id of a process
which can obtained from the ps(1) command. )
*/
#include "stdio.h"
#include "stdlib.h"
#include "sys/procfs.h"
#include "fcntl.h"
#include "errno.h"
static void doname(char *pname);
prmap_sgi_t *pmaps;
#define MAXMAP 2000
int maxmap = MAXMAP;
/* set of page state options */
char *stateopts[] = { "READ", "WRITE", "EXEC",
"SHARED", "BREAK", "STACK", "PHYS", "PRIMARY",
"SREGION", "COW", "NOTCACHED", "SHMEM" };
/* MA_* flags that correspond to above options */
int maopts[] = { MA_READ, MA_WRITE, MA_EXEC,
MA_SHARED, MA_BREAK, MA_STACK, MA_PHYS,
MA_PRIMARY, MA_SREGION, MA_COW, MA_NOTCACHED,
MA_SHMEM };
void main(int argc, char **argv) {
pid_t pid = -1;
char pname[20];
if(argc!=2) {
fprintf(stderr, "Usage: %s pid\n", \
argv[0]);
exit(1);
}
pid = (pid_t)atoi(argv[1]);
pmaps = malloc((maxmap + 1) *
sizeof(prmap_t));
sprintf(pname, "/proc/%05d", pid);
doname(pname);
exit(0);
}
static void doname(char *pname) {
int pfd;
struct prpsinfo psi;
struct prmap_sgi_arg pma;
int nmaps;
prmap_sgi_t *pmapp;
if ((pfd = open(pname, O_RDWR)) < 0) {
if (errno == EISDIR || errno == ESRCH || \
errno == ENOENT)
return;
if (errno != EACCES)
fprintf(stderr, "ERROR:Cannot open \
%s:%s\n", pname, strerror(errno));
return;
}
if (ioctl(pfd, PIOCPSINFO, &psi) != 0) {
if (errno != ESRCH)
fprintf(stderr, "ERROR:Cannot PSINFO \
%s:%s\n", pname, strerror(errno));
close(pfd);
return;
}
printf("pid %d, ",psi.pr_pid);
if (psi.pr_zomb == 0) {
pma.pr_vaddr = (caddr_t)pmaps;
pma.pr_size = sizeof(prmap_sgi_t) * maxmap;
if ((nmaps = ioctl(pfd, PIOCMAP_SGI, \
&pma)) < 0) {
fprintf(stderr, "ERROR:Cannot NMAP_SGI \
%s:%s\n", pname, strerror(errno));
close(pfd);
return;
}
printf("process has %d regions\n", nmaps);
}
if (psi.pr_zomb == 0) {
printf("addr\t\tsize\t\t off\t\tflags\n");
printf("----\t\t----\t\t ---\t\t-----\n");
for (pmapp = pmaps; pmapp->pr_mflags; \
pmapp++) {
printf("0x%x\t0x%x\t\t0x%x\t\t", \
pmapp->pr_vaddr, pmapp->pr_size, \
pmapp->pr_off);
printf("%s%s%s%s%s%s%s%s%s%s%s%s\n",
pmapp->pr_mflags & MA_READ ? " READ" : "",
pmapp->pr_mflags & MA_WRITE ? \
" WRITE" : "",
pmapp->pr_mflags & MA_EXEC ? " EXEC" : "",
pmapp->pr_mflags & MA_SHARED ? \
" SHARED" : "",
pmapp->pr_mflags & MA_BREAK ? \
" BREAK" : "",
pmapp->pr_mflags & MA_STACK ? \
" STACK" : "",
pmapp->pr_mflags & MA_PHYS ? " PHYS" : "",
pmapp->pr_mflags & MA_PRIMARY ? \
" PRIMARY" : "",
pmapp->pr_mflags & MA_SREGION ? \
" SREGION" : "",
pmapp->pr_mflags & MA_COW ? " COW" : "",
pmapp->pr_mflags & MA_NOTCACHED? \
" NOTCACHED":"",
pmapp->pr_mflags & MA_SHMEM ? \
" SHMEM" : "");
}
} else
printf(" Process is a zombie\n");
close(pfd);
}
" NOTCACHED":"",
pmapp->pr_mflags & MA_SHMEM ? \
" SHMEM" : "");
}
} else
printf(" Process is a zombie\n");
close(pfd);
}
Hope this helps. We are working to make all the above unnecessary. It should be
fixed either in pf2.2 MR or in the default DSO placement in the OS.
-Javier
--
*****************************************************************
* Javier Castellar Arribas * Email: javier++at++sgi.com *
* * Vmail: 933-1589 *
* Member of Technical Staff * Phone: 415-933-1589 *
* Core Design - Applied Engineering * Fax: 415-964-8671 *
* Advanced Graphics Division * MailStop: 8L-525 *
*****************************************************************
* Silicon Graphics Inc. *
* 2011 N. Shoreline Boulevard, *
* Mountain View, California 94043-1386, USA *
*****************************************************************
"Violence is the last refuge of the incompetent"
Hardin Seldon
=======================================================================
List Archives, FAQ, FTP: http://www.sgi.com/Technology/Performer/
Submissions: info-performer++at++sgi.com
Admin. requests: info-performer-request++at++sgi.com
This archive was generated by hypermail 2.0b2 on Mon Aug 10 1998 - 17:56:58 PDT