Hi,
Since this is my first post to this list, let me first introduce myself.
My name is Dave Brolley and I am a software engineer at Red Hat in
Toronto, Canada. I've been at Red Hat for almost 15 years with the past
5 years or so spent working on the RHEL/Fedora tools team with my main
focus being on Systemtap.
I have been given the assignment of adding SSL/TLS capability to the
various components of PCP. This would give clients and agents the option
of using a secure connection with the pmcd if desired. Having started
the work, I wanted to present my approach for feedback, to make sure
that the approach is sound and that the work will stand some chance of
being accepted into the project.
All of the work described below can be found on the brolley/nss branch
in our pcp repository at git://sourceware.org/git/pcpfans.git
The Approach:
----------------------
The general approach is to provide the option of using NSS/NSPR for
socket I/O between PCP components. We (the tools team at Red Hat) chose
NSS/NSPR because it is available and used by applications on a variety
of platforms. Also because we have used it successfully in the
implementation of the systemtap compile-server and are, therefore,
familiar with the API. Using NSS/NSPR will also make it easy to provide
IPv6 support.
We recognize that PCP is supported on a wide variety of platforms, some
of which are not supported by NSS/NSPR and so the design allows for use
of the existing POSIX-based I/O based on the availability of NSS/NSPR
(automatic detection during configuration) or a specific choice made at
configuration time (--without-nss option).
The use of NSS/NSPR on its own does not provide SSL/TLS connections. The
client and server must both agree to and expect the use of a secure
connection. The idea would be to have an insecure connection made as
normal and then to provide some sort of protocol by which the client
and/or server can request an upgrade to a secure connection (think
STARTTLS). In this way existing pmcd clients and agents can still
connect to pmcd without change and future clients and agents can still
use insecure connections if they choose.
The Design:
----------------------
The idea is to provide an abstraction layer for I/O operations which
will shield mainline code from the choice of whether to use NSS/NSPR for
the actual I/O or not. This requires the abstraction of the data types
and function calls which make up the I/O API. In general I am attempting
to make the abstracted API look and feel like the POSIX API, which is
familiar to most developers.
The Implementation:
-----------------------------
So far, the data types which I have identified for abstraction are:
typedef struct sockaddr __pmSockAddr;
typedef struct sockaddr_in __pmSockAddrIn;
typedef struct in_addr __pmInAddr;
typedef unsigned int __pmIPAddr;
typedef int __pmFD;
typedef struct hostent __pmHostEnt;
typedef fd_set __pmFdSet;
When NSS/NSPR is used, these types are defined according to the NSS/NSPR
API:
typedef PRNetAddr __pmSockAddr;
typedef PRNetAddr __pmSockAddrIn;
typedef PRNetAddr __pmInAddr;
typedef unsigned long __pmIPAddr;
typedef PRFileDesc *__pmFD;
typedef PRHostEnt __pmHostEnt;
typedef struct __pmFdSet
{
size_t size;
PRPollDesc *elements;
} __pmFdSet;
These types are currently defined in src/include/pcp/pmio.h
There are many I/O functions which require abstraction. Some of them map
directly to POSIX API calls. Just to give you an idea, here are some
examples:
extern __pmFD __pmSocket(int domain, int type, int protocol);
extern int __pmSetSockOpt(__pmFD socket, int level, int option_name,
const void *option_value,
mysocklen_t option_len);
extern int __pmGetSockOpt(__pmFD socket, int level, int option_name,
void *option_value,
mysocklen_t *option_len);
extern int __pmConnect(__pmFD, __pmSockAddr *, mysocklen_t);
extern int __pmBind(__pmFD, __pmSockAddr *, mysocklen_t);
extern int __pmListen(__pmFD fd, int backlog);
extern __pmFD __pmAccept(__pmFD, __pmSockAddr *, mysocklen_t *);
extern ssize_t __pmRead(__pmFD fildes, void *buf, size_t nbyte);
extern ssize_t __pmWrite(__pmFD fildes, const void *buf, size_t nbyte);
extern ssize_t __pmSend(__pmFD socket, const void *buffer, size_t
length, int flags);
extern ssize_t __pmRecv(__pmFD socket, void *buffer, size_t length, int
flags);
extern void __pmCloseSocket(__pmFD);
Other abstractions are required in order to encapsulate differences in
the POSIX vs the NSS/NSPR APIs. For example, accessing information about
a host:
extern char *__pmAllocHostEntBuffer (void);
extern __pmHostEnt *__pmGetHostByName(const char *, __pmHostEnt
*hostEntry, char *buffer);
extern __pmHostEnt *__pmGetHostByAddr(__pmSockAddrIn *, __pmHostEnt
*hostEntry, char *buffer);
extern __pmHostEnt *__pmGetHostByInAddr(__pmInAddr *, __pmHostEnt
*hostEntry, char *buffer);
extern int __pmHostEntNumAddrs(const __pmHostEnt *he);
extern __pmInAddr *__pmHostEntGetInAddr(const __pmHostEnt *he, int ix);
extern __pmIPAddr *__pmHostEntGetIPAddr(const __pmHostEnt *he, int ix);
extern void __pmFreeHostEntBuffer (char *buffer);
The abstracted function prototypes are all currently in
src/include/pcp/impl.h and the implementations are in
src/libpcp/src/auxconnect.c
Issues:
-------------
In general, the abstraction process has gone fairly smoothly, so far,
and, as far as I can tell, I have kept the POSIX side of the abstraction
working throughout. The biggest issue I have run across is the
assumption that a file descriptor is an integer. This assumption is
pervasive throughout PCP and is manifested in the following ways:
- returning integer return codes on error and a file descriptor upon
success from many functions
- indexing arrays using file descriptors
- assuming that file numbers 0, 1 and 2 are stdin, stdout and stderr
respectively.
- printing file descriptors as integers
- keeping track of the "max" file descriptor
- for the purpose of calls to 'select'
- for the purpose of iteration
This has resulted in some abstractions designed to remove this
assumption from the mainline code. For example __pmUpdateMaxFD
A second issue is that many components use sockets, files and pipes
seamlessly and that file descriptors representing all of these are
passed to some functions. As a result, these functions must use the
abstracted data types and I have had to extend the POSIX-NSS/NSPR
abstraction to normal file and pipe I/O in many cases. I have, so far,
left as much code as I can unabstracted, however, for the sake of
consistency, it may make sense to convert all I/O (at least within the
pmcd) to use the abstraction layer.
Status:
-------------
As I mentioned above, I believe that I have kept the POSIX side of the
abstraction in working order. That is, if you configure using the
--without-nss option, or if your platform does not support NSS/NSPR,
then building my branch should result in a working PCP system.
The NSS/NSPR side of the abstraction is not fully implemented and
currently does not compile cleanly. It has not been tested at all.
The abstraction itself is still in flux and I have no doubt that there
will be more abstraction points added.
Testing:
--------------
Once I have both sides of the abstraction implemented, testing will be
the next area of focus. I understand that there is a test harness at
http://oss.sgi.com/cgi-bin/gitweb.cgi?p=pcp/pcpqa.git;a=summary
I will likely need help in getting things set up on my available
platforms and will certainly need assistance in testing on platforms
that are not available to me.
Summary:
------------------
My work (ongoing) can be found on the brolley/nss branch at
git://sourceware.org/git/pcpfans.git
I would appreciate feedback on the approach and on the abstractions that
have been made so far. Have I placed the abstraction at the correct
level and place within the system? Is the work headed in the right
direction for acceptance?
Comments, concerns and suggestions on any of this would be greatly
appreciated.
Dave
|