<TITLE>RPC Internals ..ce (Formerly, "RPC Implementation Guide") <AUTHOR>T.J. Berners&hyphen.Lee CERN DD/OC <DATE>Version 2.3.2, Last Revised February 1989 <COPYRIGHT>CERN 1986, 87, 88, 89 ..fo on This document provides internal details of the RPC system which are not covered by the RPC User Manual, and which a user should not need to know. In particular, it defines internal interfaces within the RPC system and message formats. ..fo off ..ce on <HP1>(Details about how to acquire a copy of this document are given in an appendix). DOCFIND RPC INTERNALS RPCIMP </HP1> </TITLEP> <TOC> </FRONTM> <BODY> <H1>Introduction ..tb 9 17 25 33 41 49 57 65 73 ..tb set ..hy set endpt 3 ..hy set minpt 3 ..ti set This document was written for those who intend to maintain, modify or port the RPC system, or (to a lesser extent) those who have an academic interest in its internal design. It assumes a prior knowledge of RPC principles, and of the User Manual <BIBREF REFID=RPCUSER> for this system. The amount of overlap with that manual has been reduced as far as possible, but has been unavoidable in some places. This document covers the internal functioning of the RPC system itself. In particular, it defines internal interfaces within the RPC system and message formats. It does not cover code which was developed in order to make some of the target environments more friendly, such as the AST system or the Event Handling system under MoniCa. There is some discussion of design criteria and choices, with comparison with other systems. The level of detail covered has had to be a compromise, in an effort to answer a reasonable number of likely questions in a reasonable space and time. <H1>Design Choices <i1 ix=1>Design choices This section introduces this particular RPC system to those who are familiar with other RPC systems, explaining some of the eccentricities of this system. These have followed from the requirements of the system, in particular the environments in which it has had to run. <h2>Requirements and Constraints <i1 ix=1>Constraints, on design <h3>Target environments RPC is used for a variety of applications in control and data acquisition systems for physics experiments and related test equipment. These systems must be distributed for three reasons: the severe geographical constraints, the system topology required to handle very large data rates, and the fact that the design teams are distributed between collaborating departments. In our case, processors involved are VAX ..bf FOOT; &S'TM. ..pf minicomputers, and 32 bit (M68000 series) and 16 bit (M6809) microprocessors, and personal computers (IBM and Macintosh). The operating environments are varied, and Unix is (at the time of writing) conspicuous by its absence from the online systems. Some of the systems are monotask systems, or (in VMS) systems in which process creation entails a large overhead. This lead to te use in many cases of "AST" (see Glossary) procedures, rather than the frequent forking of processes. <h3>Example applications Examples of the use of RPC include facilities such as the following, which have been provided remotely using RPC by various groups at CERN at elsewhere: <ul> <li>Remote error logging <li>File access <li>Terminal access <li>Graphics packages <li>Menu packages <li>Gas system control and monitoring <li>Control of processor farms <li>Remote program load and control <li>Remote database access <li>Remote FASTBUS access <li>Remote operating system commands <li>etc. </ul> <h3>Communications protocols <i1 ix=1>Protocols, requirements Whereas many commercial and academic RPC systems have been based on the Internet protocols, these we not generally available in most of the online environments. <ul> <li>DECNET ..bf FOOT; &S'TM., ..pf <li>raw Ethernet <li>ISO Class 4 Transport, <li>direct communication over 32 bit parallel cables, <li>RS232 links </ul> are currently used, with the adition of others possible in the future. The system had therefore to be designed to cope with both unreliable datagram and reliable connection oriented services, and to allow a migration path for users as the availability of facilities changed. It is general required to be able to reconfigure an application to run over a different protocol at run time, by changing tables or by control by the program itself. The use of unreliable datagram services was considered to be a temporary phase, and was covered by the invention of an ad hoc protocol summarised later in this document. <h3>Application Languages As FORTRAN has been a favorite language for physicists for a long time, there was a requirement to provide RPC stubs to interface to FORTRAN code. The first RPC application (The Valet-Plus) involved a lot of Pascal code, so Pascal compatibility was also required. Later, the OS9 system became important for users, and only supported C, and the Valet plus environment produced a requirement for stubs in PILS. In some cases (when using the CERN cross software, on the PCs, and originally on VAX/VMS) the support for Pascal was, when the system was developed, much stronger for the support for C. Support for FORTRAN was patchy in the microprocessor and PC areas, especially if it to be mixed with other languages. <h2>Design choices <h3>Implementation Language For the RPC system itself, FORTRAN was rejected as having insufficient power (no pointers or structures) and being insufficiently available. Because of its availablity at the time, Pascal was chosen as the initial implementation language for the system, whereas C would have been more usual. Since that time, support for C oriented environments has been provided, using the help of pascal to C conversion programs. FORTRAN callable stubs were provided for some time by producing "FORTRAN callable" pascal, but eventually it proved simpler to provide stubs in FORTRAN itself, and provide FORTRAN-callable entry points in the run time support. <i1 ix=1>Languages, implemention (The support for N languages on M machines tends to produce N*M mappings between them, e.g. between FORTRAN and C. It is easier to support the mapping in the limited case of run-time library entry points, rether than in the general case of a stub procedure produced by the RPC compiler. One then has only to support N compiler options and M runtime library flavours). The same became true of C and PILS <i1 ix=1>PILS <REFER ID=PILS>, so the RPC compiler now produces four different languages. <h3>Description Language ..cc 6 In our case, we chose an Ada ..bf FOOT; &S'TM. ..pf -like language to describe the procedure interfaces, as it was the only real time language in use locally which provides sufficient information about each parameter, namely its type and direction. For none of the other programming languages used was this true. <h3>Message format Standardisation of the format used in RPC messages is still an open question, despite its obvious advantages for interworking. One of the reasons for this is that different RPC systems have been optimised according to different, conflicting, criteria. There is a trade&hyphen.off, for example, in the coding scheme between generality, economy of message length, and encoding speed. In our case, we chose the Xerox Courier format, in which data sizes are defined at compilation time, and data is marshalled into bytes most significant byte first. The overhead of encoding data according to the X.409 standard was felt to be too high, both in time and space. In particular, transmission over RS232 lines put stringent constraints on the efficiency of encoding. Some systems deliberately check whether the communicating machines happen to be of the same type, and in this case skip the conversion of data into standard form. The time, space and complexity of this was not felt to be justified by the relatively small amount of time spent converting data formats. (This is especially so when the conversion routines called by the stub code are expanded in&hyphen.line by a good optimising compiler such as the VAX/Pascal compiler) <h3>Data Types The primary requirement was for FORTRAN style data types, but the type offered were extended on demand to the full set of types used by Pascal and C, with the exception of variant records (unions). We had to extend the Courier data types to include a real number type (sent in IEEE format). We also found that user program portability is greatly enhanced by the addition of a type which is represented as the native integer on any machine, but is constrained to lie in a 16 bit range, and is passed as 16 bits. The string type proved to have the greatest variation in its local representation, each version of pascal having its own representation, and a semantics slightly different from that of FORTRAN. In general, it was found to be important that the logical type of the data transferred be specified quite independently of the actual local format. The first determines the format of the messages, while the second determines the method by which the data is passed on the stack. The mapping between these two is a function of options supplied to the rpc compiler when the stubs are generated. It is not, for example, so useful to have a data type "zero terminated string": the data type is "string", which is mapped onto a zero&hyphen.terminated string when stubs are generated in C. The same procedure, written in FORTRAN, will use a string descriptor, while in pascal it would use one of the more or less exotic devices built into various compilers, or, for standard pascal, a pair of separate variables for the characters and the length. <h3>Addressing and Identifiers Unlike in some systems, no numbers or identifiers need to be globally unique. <i1>Unique identifiers The procedure number within the package is only known by the client and server stub modules, and is allocated at compilation time. The other three layers are allocated at load time. The package number is generated by, and only relevant in the context of, the server on which it it provided. It could be used as a capability in a capability&hyphen.based authorisation system. The client stub module knows only a logical name for its server counterpart: this is used to look up the full address of the remote package. Although this process currently uses local tables which are set up when the system is configured, provision is made for the search to be extended to a heirarchical sequence of name servers. We find that a distributed system is rarely built as a monolithic whole, but grows by the progressive amalgamation of independently designed sub systems. It is therefore important that a system may function independently as much as possible, while also being part of a larger system. The principle of information hiding in modular software then applies equally to distributed software, and requires that one does not rely on globally unique identifiers or centralised administration. <h3>Call Time vs. Load Time <i1>Call time <i1>Load time Trade&hyphen.offs exist when designing the run&hyphen.time support software for RPC. There is a distinction between systems designed for an occasional isolated call, and those designed to handle large amounts of traffic between particular partners. In our case, the goal was to optimise run&hyphen.time performance by doing as much work as possible at compilation time (type checking, for instance), and at program load time. At load time, any necessary connections are established between tasks. The media to be used, and the remote addresses, are looked up at this time. This allows a system to be reconfigured without recompilation or even relinking the executable images. Load time is also the appropriate time for starting server tasks, and checking client authorisation. Under the VAX/VMS ..bf FOOT; &S'TM. ..pf operating system, PILS, and TurboPascal, facilities exist for the automatic intialisation of software modules when they are loaded. In this case, any stub modules declare themselves to the run&hyphen.time support at load time. The remote binding is then carried out quite invisibly to the programmer, who does not have to write any code <i1>Initialisation, automatic to do it himself. Although it is possible for a program to expressly relink itself to different modules while it is running, in general any remote call is made over an established communication path to an established server module, and can therefore execute with no additional delay. This method of working also allows us to use connection&hyphen.oriented communications services (which happen to be available locally and are at a more advanced stage of standardisation than connectionless services) without a prohibitive overhead. The call&hyphen.time performance is then determined by parameter copying time and by operating system overheads. Without any assembly coding in these areas, a dummy call with n bytes of user data takes, across Ethernet, for example, about (5+0.08n) ms between VAX and an M68000 based system. <h2>Extra Facilities Although the primary goal was simply transparent procedure call between machines, experience has lead to a number of extra features, beyond the basic RPC function. These include <ul> <li>detailed trace outputs about remote calls and their parameters, to help in the debugging of distributed applications. <li> When the confidence in the remote machine, the remote application software, or the communication protocol is low, users have asked for optional application level timeouts on remote operations. This turns out to be important in a real time system for preventing some unforseen fault causing an unduly large amount of the system to hang. In this case, a maximum time allowed for a remote operation may be specified as a compiler option, or in the definition file. See PRAGMA TIMEOUT in the user manual. <i1>Pragmas <i1>Timeouts <li>Some users (especially FORTRAN programmers) have variables whose actual data type vairies at run time. There is provision for users to include their own marshalling algorithms to cope with this (on the assumption that <hp2>some</hp2> algorithm exists for interpreting the data). This also allows related implicit data such as FORTRAN common blocks to be transferred if necessary. <i1>COMMON blocks, transfer of See PRAGMA EXTERNAL_MARSHALLING in the user manual. <li> Also selectable at compilation time is the option of remote operations, once started, continuing concurrently with the calling program. This only applies to procedures which return no data. By adding a little pipelining, this option increases the performance of some control applications. See PRAGMA concurrent in the user manual. <li>Multiple servers: Dynamic Binding. In some cases, a client may have the choice, at run&hyphen.time, of many different, though congruent, server modules. For example, a control program may have access to identical servers on different parts of the apparatus. For these applications, facilities are provided for a client program to reconfigure itself to use different server modules, and to switch the binding rapidly between them. Obviously, in this case, the application program does become aware of the addresses of server modules, so the transparency is reduced. <li>Asynchronous servers. The lack of multitasking in many environments has produced a requirements for a server which will run application code, but then be interrupted at AST (see Glossary) level to service a remotely called procedure. <li>Two forms of reverse call proved to be required. In one ("callback"), a call is made from the server process back to the original client process during the original call. In the other ("CALLER addressing"), the server merely establishes a handle on the caller in order to call him back later asynchronously, outside the bounds of the original call. </ul> <h2>Retrospective comments In considering the design in the light of experience, there are a few areas in which things might usefully have been done differently. <ol> <li>The use of Courier format has been useful only in that it answered some basically arbitrary questions, and was stable. It has not, in practice, been used to interwork with any other systems to date. If one were to change the format, then from the technical point of view, the version number system <fn> The ideal version numbering would involve the specification of three numbers each particular version of the definition file. One would be the version number (V) of the file; one would be the lowest previous version number of server with which a client of version V would be compatible, and the other the lowest (previous) version number of client with which a server of version V would be compatible. The checking of version numbers would involve sending two of these from one side to the other. </fn> could have been more sophisticated, and checked at initialisation time, rather than every call. Also, the TID field of the message belongs to a lower layer of protocol, and so should have been included in an outer layer of message format. <li>The limitation of fixed size buffers has been awkward in some cases. It would have been useful to be able to send arbitrary large messages. There is a question as to whether this would compromise the speed. This is discussed later in more detail, with some workarounds. <li>If we had realised how long the ad hoc datagram protocol would be used, we would perhaps have made it more sophisticated. The deficiencies of this protocol are also discussed later. </ol> A list of possible extensions and improvements to the current system is kept elsewhere, as it changes (in both directions) from time to time. <H1>Operational Overview There follow a few notes on the way a remote call works. Some of this information is duplicated in the user manual. For more detail, see the flowcharts of the procedures in question, and comments in the source code. ..cc on ..fo off ..bx 5 35 / 45 75 ..tb set $ ..tb 20c 40c 60c $Client application$User$Server application $module$package$module $$calls ..bx 20 / 60 ..bx new 40 ..bx off ..bx off $User package$defined by$User package $calls$- definition file -$calls ..bx 20 / 60 ..bx new 10 30 / 50 70 $Client stub$Courier$Server stub $$Level 1 ..bx off ..bx can $RPC_xxx$$RPC_xxx $calls$$calls ..bx on 20 / 60 ..bx new 5 35 / 45 75 $RunTime Support$Courier$RunTime Support $(portable)$$(portable) ..bx off ..sk ..bx can $TS_xxx$$TS_xxx $calls$calls ..bx on 20 / 60 ..sk ..bx new 5 35 / 45 75 ..sk $General Communications$$General Communications $Support$$Support ..bx off ..sk ..bx 5 35 / 45 75 ..br $Specific underlying$(whatever$$Specific underlying $medium$protocol)$medium ..bx off ..fo on ..ce on ..us on ..tb set Software interfaces in an RPC link. <i1 ix=1>Interfaces, diagram of software Vertical interfaces are procedure calls between layers of software; horizontal interfaces are protocols at each level. ..us off ..ce off ..cc off ..tb 9 17 25 33 41 49 57 65 73 <H2>RPC_Open: setting up an RPC link <i1>RPC_Open The addressing and routing decisions which are necessary in order to connect two modules are made, as far as is possible, at initialisation time, in order to allow remote calls to run with as little overhead as possible. This has been referred to elsewhere <BIBREF refid=ecma127> as "semi&hyphen.lazy evaluation of binding". <i1>Binding A procedure is provided to set up an RPC service. It is called by, or on behalf of, the client stub module in order to establish contact with its partner server stub. The package required is identified by a name (a string of characters). The procedure, RPC_Open, is part of the RPC run&hyphen.time support. RPC_Open then returns to the caller a <i1>Handle <hp2>handle</hp2> (an integer) to be used when referencing the package. When RPC_Open is called, it in fact sets up a table within the local RPC RTS which gives, for that "handle", the network address of the relevant remote <i1>handle server, and the package number which will identify the package on the remote machine. The caller is unaware of these values. Under VMS, the logical name table is currently used for name translation, and under MoniCa, the MoniCa symbol table. Both allow interactive setting up using <i1>DEFINE command a DEFINE command. See the user manual for details for other systems. (The translation, or mapping, function is related to the verification of the user's right to access the remote package. (authorisation). Some work has been done on this, including a definition of requirements and a prototype database access implementation. This work has not yet been integrated with the RPC system.) ..bx off The client stub module xxx contains a procedure open_xxx which, when called, will call RPC_Open, with its own logical name, and hence acquire a handle which it can use for future calls. Under VMS, this procedure is flagged as an "initialise" procedure, and called automatically on image load. It is also called automatically under TurboPascal on the IBM PC or the Macintosh, or in PILS. <H2>Using the link: Stub Calls Run&hyphen.Time Support Having packed the buffer, the stub calls the run&hyphen.time support procedure RPC_CALL. It passes across the package handle, and the buffer. <H2>RTS calls Communication Service <i1>RPC_Call The RPC_CALL procedure performs a little more formatting of the message, inserting the reference number of the package on the remote machine. The run&hyphen.time support package uses the package handle to extract this from its tables, and also to get details of the how to contact the remote server. This includes which network to use, if there is more than one (eg RS232 and Ethernet), and any addresses which it will need to use on that network. If the communication service it uses is connection&hyphen.oritented, then it will use a connection which has already been set up by RPC_Open. Standard calling sequences in this area have now been defined by the CATS committee. The TS_xxx routines which it proposed are similar to the TS_xxx routines currently used by this system, and documented in a later section. <H2>Adaptation of communications calling sequence <i1>Transport service, adapter layer A layer of software exists which adapts the standard calling sequence and functionality required by the RPCRTS to the local facilities available is written specially as part of the port to each system. This layer provides four functions: <ul> <li>Selection of the apropriate medium <li> Calling the local routines, OS entry points, etc, to communicate over that medium <li> Handling of multiple simultaneous connections, for connection oriented services. <li>Protocols for ensuring reliability, over unrealiable datagram communication services. </ul> See "Transport Service" index entries. <H2>Server RTS system calls communication service. On the remote ("server") machine, the Run&hyphen.Time Support system is waiting for a message. The calling seqeunce here is again dependent on the communication service used. Depending on the type of server, blocking or asynchronous receive calls are used. <i1>spawning of servers The question of whether the RTS process forks (spawns) off a separate process to handle the call is up to the RTS in question. This is not currently an option. Under VMS, the overhead of process creation is very great. Under monotask systems (PC etc) it is impossible. Under the TCS system (DD/OC MODEL project VMS subtasks), a fork is made for each request. (This has been built up using the RPC_Queue_Server" and RPC_Service" primitives.) <H2>Server RTS system calls Server Stub When a message is received, the server RTS calls the server stub. <i1>Server stub, overview There is one stub procedure for each "package". Inside it is a big CASE statement, ("computed GOTO") to select, on the procedure number, the relevant unpacking/packing code and user procedure. <H2>Server stub calls User routine The User procedure is called in just the same way as it would have been called, were the calling program on the same machine. <H2>The Return of OUT Parameters Before the server stub calls the user routine, in the case of a "concurrent" procedure (not function) with only "IN" parameters, the stub calls RPC_early_return. This sends a reply back to the caller, who can then continue to execute while the called routine continues in paralel. If an error occurs, the stub may set the m_status field in the message to a bad (even) value, and the call will be rejected, with the given as rejection reason. In the normal case, when the stub returns to the RTS, the return parameters, which have all been packed back into the message buffer, are returned to the caller. <H2>Remote Errors <i1>Error codes Errors may occur on the called machine, either in the RTS or in the called routine. In either case, the errors will, where possible, be trapped, and a "reject" response given. In this context it would be very convenient to have a global format for error codes, so that the reason code can be interpreted on the remote machine. The portable RTS always generates VMS&hyphen.style error codes, with a special facility number. The Courier "reject" message type is used for this feature, with the VMS error as a parameter. <H2>Asynchronous Events, Interrupts and Break&hyphen.in In order to allow asynchonous events to be handled remotely, the RTS must allow a client stub so be called as a result of an unsolicited call message arriving. This requires that the communications service be capable of calling the RTS assynchonously, and that a priority channel be allocated to such messages. This is required, for example, in order to implement remote break&hyphen.in in a PILS or MoniCa system. <H2>Closing a service The party who calls RPC_Open is responsible for calling RPC_close when the service is no longer required. <h2>RPC buffers <i1>Message pool <i1>Message format <i1 see="Message">Buffer The RPC system uses a pool of fixed length buffers. The buffer length is defined (by the constant MAXLENGTH in the RPCCONST header file) when the library is built. Currently, the standard version has a buffer length if 1500 bytes, and the "large" version a length of 8000 bytes. <h3>Buffer Format <i1>Message, format in memory <fig place=float frame=none> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip16 <PICTURE NAME=FLOW16$S> ...skip16 <figcap><hp1>The format of an RPC_Message structure in memory. Alternative uses of the same space are separated horizointally. The maximum size of the message depends on the version of the run time system used. </hp1> </fig> The buffer format in memory consists of three parts. The first part is reserved for the internal use of the RPC RTS module and TS. It contains a link (for threading onto lists), a status longword for the general status of the transaction, which may be overwritten at any stage. There is also an index into the third part of the buffer which is used as a read and write pointer by the marshalling routines, and there is the number of the socket over which the call is being made. The second part is reserved for internal use by the transport service. It may be used for formatting a header to the message, where the communications interface cannot gather data from dispersed buffers: this saves the overhead of copying the data into an assembly area. The third part of the message structure is that which is actually transmitted between the parties. It's format is basically Courier standard, and is described later. <h3>Buffer passing and pointer swapping Messages are passed around by pointers, and in general the TS layer reserves the right to switch pointers. That is, the pointer itself is passed VAR, so that, by overwriting the pointer, the TS system can return a different buffer without having to copy the data. Therefore, for instance, in an M68020 system with a LANCE interface chip, the packet is read into a buffer by the LANCE DMA. The buffer is swapped with the empty one provided by the RPC system's READ request, so that the unmarshalling routines take it straight out of that buffer into the user memory area, with no extra copies at all. <h3>Buffer allocation strategy Some systems have more useable heap systems than others. <i1>Heap, use of In cases where the heap does not exist, or where unrealistic constraints are put on its use, the buffers are taken from a static pool of messages (array ..bf MONO static_messages ..pf ). This is controlled by the precompilation option NOHEAP. <i1>NOHEAP precompilation option At initialisation time, the messages in the array are put onto the free list. The size of the initial pool is selected at a compile&hyphen.time as a function of the environment. One must remember, in this context, that the ethernet and RS232 protocols require an extra buffer per socket to store the last call or the last reply sent (or both, if callback occurs). Under MoniCa, the LANCE chip requires (currently 4) buffers as a receive packet pool. In the case where the heap is usable, buffers are allocated from that. Even in this case, once a buffer has been acquired by the RPC system, it is kept permanently. When not in use, it is kept on a free list rather than being returned to the heap. This means that during normal running, buffers can be allocated and deallocated very rapidly, which is essential to the efficiency of the communications code. (Under VMS, Eileen Berman/FNAL found that a large proportion of the time when running over RS232 was actually spent in pascal 'new' when this was not done) <h3>Buffer size limitation When an application requires larger data transfers than the buffer size, it is necessary to get around the buffer size limitation. I note here a few techniques. <ol> <li>One possibility is always to write a loop in the application, to transfer the data in chunks. This was done (See note by Jorgen Petersen) for the HMINI histogram retrieval routines. <li>In the long term, it would be possible to build in segmentation algorithms into the RPC system. Unfortunately, checking explicitly for buffer overflow would slow down the marshalling. In principle, though, the compiler could detect when the parameters could be crossing buffer boundaries, and could (only in those cases) insert code which checks and allows segmentation. This would allow the much more common short RPC to remain optimal. A possible future project. <i1>Message size limitation <li>As buffer sizes become significantly large, the marshalling/unmarshalling time starts to dominate the communication overhead for a fast medium. In the subset of cases in which one is communicating between like machines, it then becomes very much more efficient to transfer the data directly between user memory and user memory, with no intermediate buffering. The CERN Host Interface (CHI), because it can byte swap, in fact allows direct transfer even between machines with different byte ordering. In this case, the (hand modified) stub does an explicit transfer of the data. This is used in the CHI for the remote FASTBUS block operations, when the block size excedes a 1kByte. The buffer size at which one resorts to this technique is a question of trade&hyphen.off between the reduced marshalling time and the increased overhead for one extra DMA. The same technique could be used over DECNET between two VAXes, if required. </ol> <h1>Compiler Overview <i1>Compiler overview <i1 see="Compiler">RPCC The RPCC compiler <fn> This section is in part derived from comments by Antonio Pastore in the source code. </fn> is organized in two parts: <ol> <li>The PARSER <li>The CODE GENERATOR </ol> <h2>Parser <i1>Parser The PARSER checks the input for SYNTACTICAL and LEXICAL errors and produces as output two trees, the TYPE and BLOCK trees. It uses a lexical analyser (GETOKEN) which returns tokens (identifiers, numeric values etc) and checks for reserved keywords of the langage. It can backtrack by one token: this is used for error recovery. The user defined data types are descibed by a list of named types, each of which has a name, and a pointer to the type descriptor tree. A type descriptor tree has as elements a structure which can define a simple type, or a complex type in terms of pointers to the descriptors of its subtypes. Field for each type also exist to define the maximum and minimum sizes of the type, in standard representation, and flags for any special options (external marshalling, etc.). <i1>Block, in RPCC A BLOCK in this context means either a procedure or a function. The BLOCKS tree (pointed to by BLOCKPTR) is a linked list of record each describing a block. Each BLOCK has a NAME, (as given by the user), and a list of parameters. Each record of this list has a NAME, the parameter's name (as given by the user), and a TYPE associated to it. This is a pointer to a type descriptor. The same routine as used to build a TYPE definition is used to analize a parameter declaration inside a BLOCK. <h2>Code generators The CODE GENERATOR reads the BLOCK tree and produces as output two files, the CLIENT and the SERVER stubs. It also optionally produces a declaration file in Pascal or C, with or without a definition of the user's types. Three code generator modules currently exist. One output generator (CODEGEN) will produce Pascal or C stubs; there is also one (CODEFOR) to produce FORTRAN, and one (CODEPILS) for PILS. <i1>Code generators They tend to use similar techniques, although in places the generation of Pascal types proves a simpler task to perform recursively than the generation of C or FORTRAN types. <h2>Source of the compiler The compiler is written with the (rare) use of the INCLUDE preprocessor. This is used to select machine&hyphen.dependent parts, maily in the OPTIONS module. Versions for Bsd 4.2 Unix/Ultrix, for VAXVMS and for the IBM PC can be generated in this way. The source is divided into modules: <dl> <dt>MAIN <dd>The main compiler file, which INCLUDEs each of the following modules in the order given: <dt>DECGLOB <dd>The declarations of constants, data types, and variables for the compiler as a whole. <dt>ERROR <dd>The error reporting and handling code. <dt>DEBUG <dd>The code to list the completed trees created by the compiler. This code is invoked by the debug options. (See user manual). <dt>INIT <dd>General initialisation code. This includes setting up data areas which are effectively constant, as strict Pascal does not provide for constant data declarations. <dt>OPTIONS <dd>This module handles the command line options. It contains implementations of unix style arc and argv functions for every system under which the compiler runs. Also, included are the (machine dependent) file open and file close procedures. <dt>GETOKEN <dd>The lexical analyser. GETOKEN is also the name of the principle procedure of this module. <dt>PARSER <dd>The parser. <dt>CODEFOR <dd>The code generator for FORTRAN stubs. <dt>CODEPILS <dd>The code generator for stubs in PILS. Derived from CODEFOR. PILS has more limited data types than fortran, but better string handling. <dt>CODEGEN <dd>A combined code generator for pascal and C. This module contains a certain amount of special code for different versions of Pascal, especially for the TurboPascal versions. TurboPascal requires UNIT declarations, and IMPLEMENTATION and INTERFACE sections producing separately. </dl> <H1>Transport Service <H2>General <i1>Transport service overview The Transport module (TS) is responsible for all the communications required by the RPC system. As normally most of the communications code is provided by existing operating system facilities, the TS module acts as an adapter layer, to connect the functions required by RPC to the form in which they are available from the operating system. They also perform functions which depend on whether the communication medium is basically connection oriented or connectionless. Where the underlying service is a datagram service, the TS layer must provide such functions as: <ul> <li>Reliability <li>Segmentation </ul> For connection oriented communication, those functions are normally already covered, but the TS layer must now handle: <ul> <li>Listening for connections (multiclient server) <li>Acceptance of connections <li>Actions when peer disconnects. <li>Message boundary marking (if stream oriented) </ul> The connectionless case is in principle more complicated, but the more complicated operating system interface for the connection oriented cases means that neither form of the TS layer is trivial. <h2>Connectionless communication protocol <i1>Protocol, datagram <i1 ix=1>Protocol, datagram The RPC system was originally designed to run using standard protocols, all of which were connection oriented. The built&hyphen.in protocol which was supposed to bridge the gap until these services were available was designed to be as simpple as possible. It does not handle segmentation at all. This means that in any one call over ethernet, the total data in the call or the reply cannot exceed one packet size (1500 bytes). For most applications, this was not considered a problem. The TS layer does use a simple protocol to give reliability in the case of lost packets. The assumption is made that the medium is only rarely unreliable <fn> This is in practice the case with ethernet, or local RS232 connections. In our experience to date, packets over ethernet had only been lost due to receiver overrun (a software problem), but not due to network saturation or packet corruption. For several months, RPC systems ran at CERN assuming complete reliability of the ethernet, without practical problems, before the protocol was introduced. </fn> so no effort was made to make the recovery fast. Different forms of reliability have been considered for RPC systems, different "call semantics". These are: <i1>Call semantics <ul> <li>"At most once": This is basically unreliable remote procedure call, in which one call is sent out, and if it or the reply is lost, no action is taken. <li>"At least once": In this case, one reply is required, and retransmissions are made, but no check is made that a call is not executed twice due to a retransmission. This is fine for idempotent <fn> <i1>Idempotent, definition <i1 ix=1>Idempotent, definition An idempotent opertation is one which may be executed any number (>0) of times, with the same result. Functions without side effects are idempotent, as are procedures which set a global state to a specified absolute value. </fn>operations. <li>"Exactly once": This is the correct semantics for a remote call which is to emulate a local call, idempotent or not. </ul> The TS layer keeps a notional "connection" between the server and any particular client, for the purposes of removing duplicate calls. The protocol used provides "exactly once" semantics except for the first call any connection, for which "at least once" semantics hold. The first call is normally an internal idempotent call (to look up the package number), so this is not a problem. A summary of the protocol is as follows: <ul> <li>Calls sent over a connection are numbered, starting at zero, then cycling between 1 and some maximum number. A call numbered zero is always acceptable - this restarts the sequence (starts a new "connection"). If ever such a packet is received, it is assumed that it is a new instantiation of a client which has sent it. <li>If the client does not receive a reply, it retransmits the call (with the same sequence number). The retransmission is done after a certain time, which doubles each retransmission ("exponential backoff"). <li>If the client receives a reply (return or reject) with the wrong sequence number, it ignores it. <li> If it receives a call with (nonzero) number matching that of the last rely sent, the server retransmits the last reply. Any other call is executed and replied to. <li>A multi&hyphen.client server keeps connections (i.e. a record of the last reply sent) indefinitely. If there were a very large number of possible clients, this would obviously eventully cause the server to run out of memory. In any case, if TS_Close is called, all messages are returned to the pool. </ul> <i1>Retransmission of packets This is not a perfect protocol, but is was felt that it was not worth the effort investing in a more complex one unless a standard existed. Aspects of its simplicity which allow easier implementation include: <ul> <li>The client is the only side which retransmits on timeout. The server is basically passive. <li>The server is not required to receive messages while executing the remote procedure. This allows implementaions on strictly single&hyphen.task systems. </ul> The simplicity of the protocol gives rise to the following restrictions: <ul> <li>The first call must be idempotent. <li>Large numbers of clients could make a server run out of memory <li>There is no way for a client to distinguish between a call which is taking a long time to execute, and a crashed server. </ul> (It is important that the protocol should run on a single task system. In practice, an inefficiency can arrise on such systems, in that ethernet drivers, when there are insufficient buffers, tend to throw away the most recent, rather than the latest, packets. This means that while a procedure is running (and therefore no servicing of incoming packets is possible, that the buffers will fill up with retransmitted calls. In some cases, if the client is faster than the server, then when the server responds, it loses the next call before it can clear out the backlog of retransmitted packets from the last call. This slows the system down by one retransmission on each call following a call which took a long time.) <H1>Software Interfaces and Flow Charts <i1>Interfaces, software, details <i1>Flow charts This chapter outlines the two interfaces: that between the stub and the run&hyphen.time support package, and that between the RTS and the transport service. Note that strings are all passed as 40 character arrays, by address. This means <i1>Strings passed to RTS that the caller must provide a 40 character string, blank filled if necessary. When using FORTRAN on the VAX, the string parameter must be passed using the built&hyphen.in dummy function %REF(thestring) in order to pass the string by address rather than by descriptor. The string length (currently 40), for pascal programmers, is defined in the standard include files as RPC_name_length. The definition of the calling conventions for the routines below must be taken as being the include files which are part of the implementation. These define the calls in terms of the relevant Pascal compiler (VAR or not VAR, etc). Those who program in other languages should ensure that the calling method they use corresponds. The "message" type is defined in the standard RPCTYPES include file issued with the RPC library. This is a Pascal include file, or C include file, depending on the system. Anyone writing code which needs to know the format of this message should either write in Pascal or C, or automate the generation of their include file from the pascal one. <hp1>In the flow charts, boxes drawn with heavy lines indicate procedures expanded elsewhere. </hp1> <H2>Run&hyphen.Time Support Calling Sequences <i1 see="user manual">Calling sequence, for RTS The calling sequences for these routines are given in the User Manual, which should be used for reference. <H3>RPC_Open <i1>RPC_Open Called by client stub, at initialisation. The service_name describes the remote service required. It may be a formatted physical address (program number, network address, etc) or it may be a logical name which is converted by the run&hyphen time system into an appropriate address. The handle returned is used in all future reference to the service by the client. May be called by the user application if he manages his own handles. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip1 <PICTURE NAME=FLOW1$$S> ...skip1 <FIGCAP><HP1>Flow chart of RPC_Open. </HP1> </FIG> RPC_Open, if successful, creates a new record on the client_list. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip29 <PICTURE NAME=FLOW29$S> ...skip29 <FIGCAP><HP1>Format of client_list entries </HP1> </FIG> The work of setting up a client handle is done by the procedure "Setup" which is also used by RPC_Switch. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip2A <PICTURE NAME=FLOW2A$S> ...skip2A <FIGCAP><HP1>Flow chart of Setup (part 1) <i1>Setup, flow chart </HP1> </FIG> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip2A <PICTURE NAME=FLOW2B$S> ...skip2B <FIGCAP><HP1>Flow chart of Setup (part 2) <i1>Setup, flow chart </HP1> </FIG> Find_Remote is a routine which interrogates the remote node to find out the package number of a package for which it only has the name. The call to Find_Remote is in fact a remote call to Find_Local, and is implemented by a standard stub. The stub is written into the RPCRTS module explicitly, as if it were not it would have mutual references with RPCRTS, which would complicate the implementation under strongly hierarchical systems such as TurboPascal. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip21 <PICTURE NAME=FLOW21$S> ...skip21 <FIGCAP><HP1> Find_Local simply looks up the package name and returns the package number </HP1> </FIG> <H3>RPC_Close <i1>RPC_Close Called by client set&hyphen.up when the service is no longer required. The handle value becomes invalid for future use. May be called by the user application if he manages his own handles. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip3 <PICTURE NAME=FLOW3$$S> ...skip3 <FIGCAP><HP1>Flow chart of RPC_Close </HP1> </FIG> <H3>RPC_Call <i1>RPC_Call Called by client stub. On entry, the buffer consists of a header (in which the procedure number has been written) followed the procedure arguments. The count tx_bytes includes only the number of bytes in the procedure argument array: it does not include the header. The timeout parameter specifies (nominally in 10us units) a maximum time allowed for the call. A negative value disabled the timeout (allows an infinite time). The buffer_size parameter indicates the total size in the buffer available for returned arguments (again, exclusive of the header). There are two versions of this routine. RPC_Call itself handles errors, whereas RPC_Call_Status returned error status to te caller. A possibility here is for the RTS to deallocate the buffer, and pass back a pointer to a different one used for recieving the data. This saves time in certain cases, especially if the communications service can accept a user buffer and deallocate it itself after the data has left. Therefore, the caller of RPC_Call must not keep any other record of the address of his buffer. ..us off <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip4A <PICTURE NAME=FLOW4A$S> ...skip4A <FIGCAP><HP1>Flow chart of RPC_Call and RPC_Call_Status. In the case of a local call, the routine Handle_Call is used. RPC_Call_Status returns the status value at the end of this part, wheras RPC_Call continues to treat the error. </HP1> </FIG> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip4B <PICTURE NAME=FLOW4B$S> ...skip4B <FIGCAP><HP1>Flow chart of RPC_Call (continued). </HP1> </FIG> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip5 <PICTURE NAME=FLOW5$$S> ...skip5 <FIGCAP><HP1>Flow chart of Handle Call. This routine is used for local calls as well as by a server of remote calls. </HP1> </FIG> <H3>RPC_Attach_Stub <i1>RPC_Attach_Stub Called by server stub on initialisation. The entry_point parameter passes the actual stub procedure to be attached. The service_name parameter gives the logical name of the server. This may be used by the run&hyphen.time support to update name tables. The service name will be the same for different instances of the same server. The program number returned may be used to disconnect the stub later. Internally, this calls RPC_Specify_Stub. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip6 <PICTURE NAME=FLOW6$$S> ...skip6 <FIGCAP><HP1>Flow chart of RPC_Attach_Stub </HP1> </FIG> This procedure sets up a new server stub record on the list "program_list". <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip28 <PICTURE NAME=FLOW28$S> ...skip28 <FIGCAP><HP1>Format of program_list entries </HP1> </FIG> <H3>RPC_Specify_Stub <i1>RPC_Specify_Stub This is identical to RPC_Attach_Stub, except the program number is generated by the user. This allows him to allocate "well known numbers" to represent particular packages. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip7 <PICTURE NAME=FLOW7$$S> ...skip7 <FIGCAP><HP1>Flow chart of RPC_Specify_Stub </HP1> </FIG> <H3>RPC_Detach_Stub <i1>RPC_Detach_Stub Called by server set&hyphen.up when the server is no longer required. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip8 <PICTURE NAME=FLOW8$$S> ...skip8 <FIGCAP><HP1>Flow chart of RPC_Detach_Stub </HP1> </FIG> <H3>A Server Stub <i1>Server stub, interface IN call_buffer: pointer to message ..bx off Called by the RTS. The length of the stub buffer should be obvious to the stub code, which is aware of its format. In case of error, the stub writes a bad status into the m_status field of the message. <H3>RPC_Early_Return <i1>RPC_Early_Return Called by the server stub, only in the case of a "concurrent" procedure. The message consists of a header (which is left intact by the stub) and a results array, which is filled. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip9 <PICTURE NAME=FLOW9$$S> ...skip9 <FIGCAP><HP1>Flow chart of RPC_Early_Return </HP1> </FIG> <H3>RPC_Loop_Server <i1>RPC_Loop_Server Called by: User server program. The set of clients to be serviced is identified by a string, which may in some cases include wildcards. This procedure is the simplest form of server, as it loops forever, servicing incoming requests. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip10 <PICTURE NAME=FLOW10$S> ...skip10 <FIGCAP><HP1>Flow chart of RPC_Loop_Server </HP1> </FIG> <H3>RPC_Start_Server <i1>RPC_Start_Server Called by: User server program. The set of clients to be serviced is identified by a string, which may in some cases include wildcards. This procedure queues a read request, and has the next message and successive messages serviced at AST level. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip22 <PICTURE NAME=FLOW22$S> ...skip22 <FIGCAP><HP1>Flow chart of RPC_Start_Server </HP1> </FIG> RPC_Start_Server specifies, as the AST procedure to run, RPC_Server_AST. This routine first handles the call, and then queues the next read on the channel. Originally, it used to queue the read in advance, which would speed up the system in some cases, but this causes problems if sophisticated stubs or user routines use the same communication channel while the call is being handled: messages would be received in the wrong order. (See the PIPELINE precompilation option). <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip30 <PICTURE NAME=FLOW30$S> ...skip30 <FIGCAP><HP1>Flow chart of RPC_Server_AST </HP1> </FIG> <H3>RPC_Create_Server <i1>RPC_Create_Server This routine sets up the communication channel for a server, but does not let it run. The socket returned may be used with the routines below to service requests as required. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip20 <PICTURE NAME=FLOW20$S> ...skip20 <FIGCAP><HP1>Flow chart of RPC_Create_Server </HP1> </FIG> <H3>RPC_Accept <i1>RPC_Accept This routine waits for a request on the given socket, and services it according to the currently attached stub modules. The timeout is in units of 10ms. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip11 <PICTURE NAME=FLOW11$S> ...skip11 <FIGCAP><HP1>Flow chart of RPC_Accept </HP1> </FIG> <H3>RPC_Queue_Server <i1>RPC_Queue_Server This routine queues an asynchonous user routine (AST) to be called when the next request arrives on the given socket (see RPC_create_server). It allocates space for the incoming message, and passes the address of this to the AST routine. The user AST routine has entry point astadr and is passed by address one parameter (info). This parameter is suitable to be passed to RPC_Service (see below). Its format is private to the RPC system. This parameter is all that is required by the service routine below: the user program may, however, want to perform some pre&hyphen. or post&hyphen. processing, such as forking off other tasks to handle the requests, or application functions. (For the format of the message buffer, see the rpc$types include file, as this contains several fields used internally by the RPC RTS, which may be changed between versions. It is not recommended that user code uses this buffer directly, but calls RPC_service to have it serviced.) This routine, with RPC_Service, was used to make a multitask server under the MODEL TCS scheduler. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip12 <PICTURE NAME=FLOW12$S> ...skip12 <FIGCAP><HP1>Flow chart of RPC_Queue_Server </HP1> </FIG> <H3>RPC_Service <i1>RPC_Service This is a procedure taking one record parameter as described in the preceding paragraph. It will handle one incoming call message, deallocate the message buffer passed to it, and return. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip13 <PICTURE NAME=FLOW13$S> ...skip13 <FIGCAP><HP1>Flow chart of RPC_Service </HP1> </FIG> <H3>RPC_Configure <i1>RPC_Configure This call allows one client to use more than one identical server in turn. The effect is for the connections to all open servers to be closed and reopened with the translation of their logical names being perfomed according to new values which may have been set up. A client process which wishes to use several identical servers may therefore repeatedly redefine the logicalnames which give the addresses of those servers, and call RPC_configure to cause future calls to be directed to the new servers. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip14 <PICTURE NAME=FLOW14$S> ...skip14 <FIGCAP><HP1>Flow chart of RPC_Configure </HP1> </FIG> <H3>RPC_Switch <i1>RPC_Switch This performs the reconfiguration function on a single package, referenced by a handle. The package must be already open (for the handle to be valid). The value of the handle is unchanged. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip15 <PICTURE NAME=FLOW15$S> ...skip15 <FIGCAP><HP1>Flow chart of RPC_Switch </HP1> </FIG> <H3>RPC_Establish <i1>RPC_Establish This procedure may be used in a client program to handle errors occuring in remote procedure calls. By default, if an unrecoverable error occurs in the communication with the server, the client program is stopped. ($EXIT occues under VAX/VMS, a pascal halt on other systems.) If the user wishes to handle the error himself, he passes the address of his error handler to rpc_establish. If this has been done, his handler will be called <hp1>instead</hp1> of the default handler. In order to revert to the default handler, RPC_establish is called with a value of zero. The user error handler is a routine taking one parameter, which is the message in question. If the user is programming in pascal, he will find the offending error status in the m_status field of the message. RPC_Establish simply places the given value into a global variable. <h3>RPC_Init This is called automatically where possible (e.g. VAX/VMS, TurboPascal, MoniCa). In other cases, it must be called explicitly by the user program. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip17 <PICTURE NAME=FLOW17$S> ...skip17 <FIGCAP><HP1>Flow chart for RPC_Init </HP1> </FIG> <h2>Internal Routines These routines are not normally called from outside the run time support. <h3>RPC_New <i1>RPC_New RPC_New is the routine which produces a new message buffer. It will allocate a message from the pool if the pool is not empty. Depending on whether the heap is in use, it may use the heap if the pool is empty. RPC_New may be called from the transport service as well as from stubs and internally. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip18 <PICTURE NAME=FLOW18$S> ...skip18 <FIGCAP><HP1>Flow chart for RPC_New </HP1> </FIG> <h3>RPC_Dispose <i1>RPC_Dispose, flow chart RPC_Dispose performs the reverse operation to RPC_New, except that it never returns a message to the heap, it keeps it for efficiency. ..go zz <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip19 <PICTURE NAME=FLOW19$S> ...skip19 <FIGCAP><HP1>Flow chart for RPC_Dispose </HP1> </FIG> ...zz <h2>TS layer TS_XXXX procedures <i1>Transport Service, calling sequence The calls used internally by the run&hyphen.time system are currently of the form given below. The status code is returned as a function value in each case. Outline flow charts are given below for some parts of a typical example of a TS adapter layer (TSPM68k) which supports RS232 and Ethernet communication. In some cases, the connection oriented communication handling is also shown. The reader is refered to the source for the system in question for the exact parameter passing conventions, data types, etc. <H3>TS_Open(socket, name) <i1>TS_Open <xmp> OUT status: integer (function value) OUT socket: integer IN name: string </xmp> All the information about the medium and about the protocol, service type and peer address where applicable are coded into the name string. A wildcard syntax in the string signifies connection to a class of peers. In this case the address for transmission is always to be taken from the source of the last received message. A socket number for reference is returned. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip23A <PICTURE NAME=FLOW23AS> ...skip23A <FIGCAP><HP1>Flow chart for TS_Open (part 1) </HP1> </FIG> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip23B <PICTURE NAME=FLOW23BS> ...skip23B <FIGCAP><HP1>Flow chart for TS_Open (part 2) </HP1> </FIG> <H3>TS_Close(socket) <i1>TS_Close <xmp> OUT status: integer (function value) IN socket: integer </xmp> The socket number becomes unusable in future calls. Any messages attached associated with the socket are returned to the message pool, <hp1>including any passed by the user as a parameter to an oustanding TS_When_Receive request.</hp1> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip24 <PICTURE NAME=FLOW24$S> ...skip24 <FIGCAP><HP1>Flow chart for TS_Close </HP1> </FIG> <H3>TS_Send(socket, data, length) <i1>TS_Send <xmp> ..tb 9 17 25 33 41 49 57 65 73 OUT status: integer (function value) IN socket: integer IN data: pointer to message IN length: integer </xmp> The socket passed must be derived from TS_Open. The data array is in fact a data structure of which the first part is internally used by the RPC system, the second part is free for use by the transport service, and the third part is data belonging to the RPC system, which is actually to be transferred. <dl> <dt>On entry <dd> Within the first part, two of the four fields are set up before TS_send is called: <xmp> ..tb 9 17 25 33 41 49 57 65 73 M_Next is undefined M_Index must be the length of the message to be sent M_Socket must point to the socket to be used. M_Status is undefined </xmp> (Note that this information duplicates, for historical reasons, data in the parameters 'socket' and 'length'. In practice, these fields have turned out more useful than the parameters, so if the system were designed again, TS_SEND would only take one parameter.) <dt>On exit <dd>The pointer to the data may have been changed, so as to point to another message. This allows copying to be reduced, for speed. In the message pointed to be the (new) pointer, <xmp> ..tb 9 17 25 33 41 49 57 65 73 M_Next may be corrupted M_Index may be corrupted (currently) M_Socket points to the socket to be used. M_Status may be corrupted </xmp> </dl> <H3>TS_Receive(socket, data, max_length, xfer_count) <i1>TS_Receive <xmp> ..tb 9 17 25 33 41 49 57 65 73 OUT status: integer (function value) IN socket: integer IN OUT data: RPC_message_pointer IN max_length: integer IN timeout integer </xmp> The socket passed must be derived from TS_Open. The data array is in fact a data structure of which the first part is internally used by the RPC system, the second part is free for use by the transport service, and the third part is data belonging to the RPC system, which is actually to be transferred. A pointer to this data structure is passed by address. The ts_receive routine is free to switch it to a different buffer, to save time. The caller must not, therefore, keep any other record of the location of the original buffer. If the buffer pointer is switched, ts_receive will dispose of (or do something intelligent with) the old buffer. The timeout parameter specifies (nominally in 10us units) a maximum time allowed before reception. A negative value disables the timeout (allows an infinite time). (The unit of 10ms is not a very friendly one, but was chosen in order to give a sufficiently long maximum timeout on a 16 bit system). <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip25a <PICTURE NAME=FLOW25AS> ...skip25a <FIGCAP><HP1>Flow chart for TS_Receive (part 1) </HP1> </FIG> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip25b <PICTURE NAME=FLOW25BS> ...skip25b <FIGCAP><HP1>Flow chart for TS_Receive (part 2) The decision "Duplicate?" in th centre of this chart indicated a call to the function "Duplicate" or "Duplicate_V24". </HP1> </FIG> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip27 <PICTURE NAME=FLOW27$S> ...skip27 <FIGCAP><HP1>Flow chart for Duplicate and Duplicate_V24. Areas of difference between these two modules are indiated by background shading. </HP1> </FIG> <box> The routines below are not necessary in an implementation which is not to support asynchronous servers. </box> <H3>TS_When_Receive(socket, pdata, max_length, proc, param) <i1>TS_When_Receive <xmp> ..tb 9 17 25 33 41 49 57 65 73 OUT status: integer (function value) IN socket: integer IN pdata: pointer to message IN length: integer IN proc: service_procedure (see below) IN user_param: integer </xmp> The socket passed must be derived from TS_Open. The data array is in fact a data structure of which the first part is internally used by the RPC system, the second part is free for use by the transport service, and the third part is data belonging to the RPC system, which is actually to be transferred. This is a receive subroutine needed for asynchronous processing of events. It asks for the procedure "proc" to be called when the next message arrives. The address of a suitable buffer is passed, and its size, and the entry point of the procedure to be called when the read completes. <dl> <dt>On entry <dd> Within the first message, one of the first four fields is set up: <xmp> ..tb 9 17 25 33 41 49 57 65 73 M_Next is undefined M_Index is undefined M_Socket must point to the socket to be used. M_Status is undefined </xmp> <dt>On exit <dd>The pointer to the data may have been changed, so as to point to another message. This allows copying to be reduced, for speed. In the message pointed to be the (new) pointer, <xmp> ..tb 9 17 25 33 41 49 57 65 73 M_Next may be corrupted M_Index may be corrupted M_Socket points to the socket to be used. M_Status may be corrupted </xmp> </dl> The asynchronous service procedure referred to ('proc') takes parameter <xmp> ..tb 9 17 25 33 41 49 57 65 73 IN pdata: pointer to message. </xmp> Within that message (which is not necessarily the same message as was originally passed to TS_When_Receieve), when the 'proc' is called, <xmp> ..tb 9 17 25 33 41 49 57 65 73 M_Next is undefined M_Status is the completion status of the read M_Socket is the socket on which the message arrived M_Index is undefined </xmp> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip26 <PICTURE NAME=FLOW26$S> ...skip26 <FIGCAP><HP1>Flow chart for TS_When_Receive </HP1> </FIG> <H3>TS_My_Address(socket, name) <i1>TS_My_Address This returns the address at which the user may be contacted. <xmp> ..tb 9 17 25 33 41 49 57 65 73 OUT status: integer IN socket: integer OUT name: string </xmp> This is not a very well defined function, in that what can be returned, and it's exact significance, depends on the transport medium being used. Over a serial line, it is impossible to know by what name the line is known at the far end. Over ethernet, though, it is very useful to be able to find one's own ethernet address. In general, the address is returned in the format of an RPC address, as used for TS_Open. <H3>TS_Peer_Address(socket, name) <i1>Address of peer <i1>Peer address This returns the address of the sender of the last received message. <xmp> ..tb 9 17 25 33 41 49 57 65 73 OUT status: integer IN socket: integer OUT name: string </xmp> The same problems exist as for TS_My_Address, in that the information which is available and its significance tend to vary between systems. Again, the rule is that an address which would work with TS_Open should be returned. Over DECNET, for example, one can find the task name but not the TSAP name: in this case an address is returned which would work if the peer task had created a TSAP with the same name as his process. <h2>LANCE ethernet chip routines These routines are written in C. (Files tlance.c, lance.h) They were derived from routines written by L.R.T. Ltd, modified by Ben Segal (CERN/DD/SW), and further modified for inclusion in the RPC project. They do not use the data chaining features of the lance, as RPC buffers are always sufficiently large for a packet. They are aware, on the receive side, of the TS layer's socket format, as they search the socket list for a socket which is waiting for the received packet. These routines have survived a number of releases of the LANCE chip which had more or less obscure bugs. Therefore, they are fairly defensive, and will reinitialise the chip if things go wrong. There is a reasonable amount of trace code built into the module, controleed by an integer variable _debug taking values from 0 to 5 (see the user manual). Trace output from this module is prefixed by "* LANCE: ". The entry points are as follows. Note that as a peculiarity of the C compiler, they must be prefixed by an underscore character if referenced from outside the C world. <h3>bbinit This is a one-time general initialisation. It sets upo data structures in memory, initialises the chip itself, and sets up (via a small piece of assembler) the interrupt vectors to point (via a small piece of assembler) to the ..bf MONO lance_i ..pf interrupt service procedure. On exit from _bbinit, the lance chip is initialised by inactive. <h3>bbopen This routine activates the lance chip. For safety, it resets the buffer system. <h3>bbclose This routine deactivates the LANCE chip. <h3>lance_send This routine takes as one parameter the address of a packet to send. It returns a status when it has succeded, or faileded after several ettempts, to send the packet. The following entry points are used internally to the tlance.c module. <h3>lance_i This is the general interrupt service routine. It displays trace information if errors occur, if the level of _debug is high enough. It calls lance_read if a packets has been received. <h3>lance_read This checks that a packet really has arrived, and that it is good. If so, it scans the queue of sockets waiting for ethernet packets looking for an address and protocol type match. A mask on this check allows wildcard reception by a socket. <h3>getstation This routine is used internally to extract the ethernet address of the card in use from the hardware. <h3>initilise_lance This sets up the data structures and initialises the chip. <H1>Message Formats <i1>Message Format The messages are formatted according to the Courier <BIBREF REFID=COURIER> format. The "extended subset" of that format which is used is described below. The message is described as a stream of 8 bit bytes. When word&hyphen.oriented communication is used, one must ensure that the byte order is preserved. An alternative representation considered was the X.409 standard <i1>X.409 <BIBREF REFID="X409">. It requires more bytes, and so is not appropriate in bandwidth&hyphen.limited systems such as the Valet+, but is becoming accepted as a standard for data representation. It may be supported as an alternative. A further alternative may be the Apollo NCS format, purely for compatibility with NCS. <i1>Apollo NCS <H2>Data type representation <i1>Data representation <i1>Representation, data A list of the data types handled by the RPC system are given in appendix A of the User Manual. Below is listed the way each data type is packed into the message. Bytes (represented by boxes) are sent in the order listed left to right across the page. Note that each type is padded with zeroes so that every element wil start on a 16 bit boundary, according to the Courier spec. This may be suppressed, for efficiency, using the NOALIGN option in the INCLUDE file after running the RPCC and before running the Pascal compiler on the stubs. ..tb 9 17 25 33 41 49 57 65 73 ..bd RPC_BYTE <i1>RPC_Byte ..cc on ..tb set $ ..bx 8 15 / 16 23 $0$Value ..bx off ..bx 8 15 $Value$ (If NOALIGN option used) ..bx off ..cc off ..bd RPC_CHAR <i1>RPC_Char This is sent as an ASCII (or, rather, CCITT International Alphabet 5) character. Translation is performed by the stubs if necessary. ..cc on ..tb 9 17 25 33 41 49 57 65 73 ..bx 8 15 / 16 23 $0$Value ..bx off ..bx 8 15 $Value$$(If NOALIGN option used) ..bx off ..cc off ..bd RPC_SHORT <i1>RPC_Short This may be used to represent a signed (2s complement) or unsigned quantity. (The two's complement notation is so standard that it is not necessary to distinguish between them at this level. If any machine used a different format for negative numbers, we would have to). ..cc on ..bx 8 15 / 16 23 $MSB b1$LSB b0 ..bx off ..cc off Note that the MOST SIGNIFICANT BYTE IS SENT FIRST. If the number is given by X = b0 + (256 * b1) then the order of bytes in the buffer is b1, b0 as shown. ..bd RPC_LONG <i1>RPC_Long This may be used to represent a signed (2s complement) or unsigned quantity. (The two's complement notation is so standard that it is not necessary to distinguish between them at this level. If any machine used a different format for negative numbers). ..cc on ..bx 8 15 / 16 23 / 24 31 / 32 39 $MSB b3$b2$b1$LSB b0 ..bx off ..cc off Note that the MOST SIGNIFICANT BYTE IS SENT FIRST. If the number is given by X = b0 + (256 * b1) + (65536 + b2) + (256*65536* b3) then the order of bytes in the buffer is b3, b2, b1, b0 as shown. ..bd RPC_REAL32 <i1>RPC_Real32 This is sent in IEEE format. (There is no defined format for real numbers in either the Courier or the X.409 recommendation). This, and conversion to and from other 32 bit formats is discussed in <BIBREF REFID="REALS"> Conversion is performed, on the VAX, when the number is (un)packed by the stub. ..cc on ..bx 8 10 15 / 16 18 23 / 24 31 / 32 39 $S exp$e MSB$mantissa$LSB ..bx off ..cc off The first byte sent contains the sign bit (S) in the most significant bit, and then the 7 most significant bits of the exponent. The next byte contains the least significant bit of the exponent (E), and the 7 most significant bits of the mantissa. The next two bytes contain the rest of the mantissa. The sign bit, if set, means that the number is negative, though the mantissa remains uncomplemented. The exponent field is zero to indicate a zero real number. The mantissa is unsigned and normalised, with the leading "1" omitted. The exponent bias is 150. The absolute value of a number is X = 2**(exponent-150) * (mantissa + 2**23) ..BD RPC_STRING <i1>RPC_String This is represented by a two byte length field, followed by a series of one&hyphen.byte characters, followed by a zero byte if this is necessary to leave the total number of bytes even. The lengh field is in RPC_SHORT format. The characters are in the same coding as RPC_CHAR, with conversion being done by the stubs as necessary. ..cc on ..bx 8 15 / 16 23 / 24 31 / 32 39 / 48 55 / 56 63 $Len b1$Len b0$Char 1$Char 2$...$Char n$Zero ..bx off ..cc off In this example the characters are char 1.. char n, where n is odd, so a padding zero byte is required. ..bd Arrays and Records <i1>Arrays, representation <i1>Records, representation Arrays are of fixed, predefined, size, and are packed simply as a series of elements of thier component type. A multidimensional array is treated as an array of arrays, so the last index cycles most rapidly. A record is represented simply by the set of its constituent fields, in the order in which they were declared. ..bd Sequences <i1>Sequences, representation Sequences, in RPCL, are arrays of length unknown at compile time. They are transferred preceded by a count field, in RPC_INTEGER format. ..tb set <H2>X.409 Alternatives <i1>X.409 mapping In an alternative representation, data is represented in X.409 format <BIBREF REFID="X409"> In this case, the message is a series of bytes ("octets"), there being no word alignment. Every data item it represented by an identifying byte, followed by a length field of one or more bytes, followed by the data. The RPCL types map onto X.409 types as follows: RPC_BYTE, RPC_SHORT, RPC_LONG: Integer (Id 02 hex) RPC_CHAR, RPC_STRING: IA5 string (Id 16 hex) ARRAY, SEQUENCE: Sequence (ID 30 hex) Using tagging in this way requires more run&hyphen.time code and execution time, but allows makes a message more readable by a third party unaware of its formatting. <H2>Message content Below we list the contents of a message of each type. Every message begins with a "message type" field, of type RPC_SHORT, which can take the values 0 Call message 1 Reject message 2 Return message There follows a transaction identifier(a 16 bit number) which may be used for sequence checking, but at present is always set to zero. The formats of each type of message are as follows: <H3>Call message <xmp> message_type RPC_short 0 transaction_id RPC_short package_number RPC_long version_number RPC_short procedure_number RPC_short (within the package) </xmp> This header is followed by all the IN and INOUT parameters, in the order specified in the definition file. The transaction ID is only used when the simple protocol is run over a connectionless communications service. The package number and version number are described in the user manual. The procedure number is the index of the procedure within the package, generated by the RPC compiler. <H3>Reject message <xmp> message_type RPC_short 1 transaction_id RPC_short reason RPC_short Courier reason number status RPC_long CERN (VMS like) error code. </xmp> <H3>Return message <xmp> message_type RPC_short 2 transaction_id RPC_short </xmp> This header is followed by the function result (if any), then by all the OUT and INOUT parameters, in the order specified in the definition file. <h2>Coding scheme for RS232 point&hyphen.point messages <i1 see="RS232">V24 <i1>RS232 transmission encoding There are two levels of encoding: the frame formatting (header and checksum), and the byte encoding. In order to send binary byte stream data over 7&hyphen.bit ASCII terminal channels, an ad hoc coding scheme has been implemented, which is described below. The scheme is neither theoretically optimal nor beautiful nor standard, but is simple and relatively efficient. It is designed to get though the filters and conditions imposed on such channels by operating systems and drivers. This coding scheme is optional, and used only if 7 bit communication is requested. The byte encoding scheme allows 8&hyphen.bit bytes to be logically sent. In some cases, they can be sent directly, where both transmitting and receiving operating systems allow it. Where this is not so, a seven bit encoding scheme satisfies the following requirements: <ul> <li>Only seven bit characters are used <li>Control characters (00 to 1F hex) are not used <li>The DELETE character is not used <li>Small vales are transmitted as single bytes </ul> <H3>8&hyphen.bit encoding scheme for bytes In this case, the only formatting necessary is the escaping of the start of message character, and of the escape character. <H3>7&hyphen.bit Encoding scheme for bytes. This scheme was updated by RB in summer 1988. When transmitting a byte, <ol> <li>If the byte is less than 5C hex, add 20 hex to it and transmit it. <li>If the byte is between 5D and B9 hex inclusive, precede it with the character 7D, then subtract 3D from it and send the result. <li>If the byte is between BA and FF hex inclusive, precede it with the character 7E, then subtract 9A from the byte and send the result. </ol> <xmp> x in 0..5C --> x-20 x in 5D..B9 --> 7D, x-3D x in BA..FF --> 7E, x-9A </xmp> When receiving it, the reverse operation is <ol> <li>If the character is less than 7D, subtract 20 from it. <li>If the character is 7D, take the next one and add 3D. <li>If the character is 7E, take the next one and add 9A </ol> <H3>Framing format There are two frame formats. One is a general one, which can reprsent any RPC message. The other is a specially efficient one optimised for a strict subset of possible messages, which will suffice for simple systems. The frame starts with <xmp> Message Type, Transaction ID, </xmp> which, for a Call type message, should be followed by <xmp> Program Number, Version Number, Procedure Number, all the IN and INOUT parameters, </xmp> The following encoding scheme is used: <xmp> 7 6 5 4 3 2 1 0 _____________________ _____________ | M : T : y : p : e | | T : I : D | --------------------- ------------- 0 - Call, long format Packet Number 1 - Reject 2 - Return 4 - ? (Cast) 5 - ? 6 - ? 7 - ? 8 - Call Program 0, short format 9 - Call Program 1, short format n - Call Program n-8, short format 31 - Call Program 23, short format </xmp> This mechanism, allows us to send both the Message Type and Transaction ID in just one byte. In the case where <xmp> ..tb 9 17 25 33 41 49 57 65 73 The message is a CALL and Program Number < 24 and Version Number = 0 and Procedure Number < 256, (1) </xmp> the short form is used. In this case there are just two bytes to send: <xmp> ..tb 9 17 25 33 41 49 57 65 73 The byte above (1 byte) Procedure number (2 bytes) Total: 3 bytes </xmp> If the condition (1) is false we must use the long header format that requires us to send: <xmp> ..tb 9 17 25 33 41 49 57 65 73 The byte above (1 byte), Program Number (4 bytes), Version Number (2 bytes), Procedure Number (2 bytes). Total: 9 bytes </xmp> The equivalent information is stored in memory, and sent over ethernet, etc., as 12 bytes. <i1>Frame format, over RS232 <i1>Short frame format The frame is preceded by a frame start byte, and followed by a checksum. A one byte checksum is used for messages below (???) bytes in length, and for longer messages a 32 byte CRC is used. <H3>Frame reception Any control characters received should be ignored at any time. Any characters received before the start of frame character shall be ignored. <H1>Porting the RPC system into a new environment <i1>Porting the system If a new machine or software environment is to be introduced to a distributed system using RPC, there are three stages of porting of software wich may be necessary. These are the provision of communications support in the new environment, the porting of the RPC RTS itself, and, if required, the porting of the compiler. <H2>Provision of the Communications support <i1>Transport Service, porting <i1 see="Transport Service">Communications Assuming that communications support is available for a suitable medium in the new environment, all that must be done is to provide it in the (relatively simple) form needed by the RTS. In the simplest case (with no asynchronous calls) all that is required are the OPEN, CLOSE, SEND and RECEIVE procedures. One of the existing TSxxx modules may be taken as a template, and used <ol> <li>as an adapter to the existing communications primitives <li>as a switcher to select the communication medium required. </ol> <H2>Porting the Run&hyphen.Time System <i1>RPCRTS, porting This module is written in Pascal, with conditional compilation sections handled by the INCLUDE facility. Porting onto a new environment may require slight additions to the master source file if the Pascal compiler on the new machine has eccentricities not already catered for. A version of the RTS now exists which has been translated into C. This is generated by slightly preprocessing a version of the pascal source, running the 'ptc' converter progrem, and then slightly postprocessing the C source it produces. Therefore, whereas a TS module exists for each target environment, there is only one master RPCRTS file. <H2>Porting the Compiler <i1>Compiler, porting the The same applies to the compiler: it is written in Pascal, with conditional compilation sections handled by the INCLUDE facility. Porting onto a new environment may require slight additions to the master source file if the Pascal compiler on the new machine has eccentricities not already catered for. If cross&hyphen.compilation on an existing machine (for example VMS or Unix 4.2 Bsd) is satisfactory, then porting of the compiler itself will not be necessary. It may also be the case that modifications have to be introduced into the code generator if the stub code requires special facilities for the new environment. In general, no modifications to the compiler at all will be necessary if the port is to a new runtime software environment on a CPU which is already supported. <h1>Code Management <i1>Code management <i1>Source management <i1>CMS <i1>MMS DEC's Code Management System (CMS) is used for all source files, and the Module Management System (MMS) is used to control the generation of code and avoid inconsistencies. Those wishing to change any code in the RPC system should make themselves familiar with both of these. They allow, especially when used in a controlled way, a project to be partially self&hyphen.documenting. The following notes are provided to help the many people working together on projects such as the RPC, CHI etc to find their way around each other's files. The rules given should make this simpler. (They are not all adhered to 100% of the time <h2>Directory management <i1>Directory management Any development directory contains: <dl> <dt>Source files <dd>made by hand, cannot be generated automatically. <dt>Product files <dd>final products, or files used to to make products of other directories. <dt>Byproduct files <dd>made in the process of making other files but not needed by any system. (eg maps, listing files). Not used to make other files <dT>Intermediate files <dd>used in the process of making product files, can be generated automatically. <dt>Disconnected files <dd>Not involved at all in the software generation (e.g. documentation, subdirectories) </dl> The use of MMS allows these files to be distinguished by a newcomer to the project. Note that the same directory is used for the source and object files: the subdirectory structure is used to distinguish different areas of the project, not different types of file. <ol> <li>Every directory with any product files MUST contain a makefile called Makefile (or descrip.mms on VMS). <i1>Makefile <li>Any files in a directory which are final products or used in other directories must be explicitly declared in the Makefile (or descrip.mms) in that directory. <li>The default make (MMS) command must bring ALL product files in the directory up to date. The default target is the first dependency in the file. For example, it might be <xmp> default : xxx, yyy, zzz touch default </xmp> Therefore, one can find out how to make any file required by any other directory, by looking in the Makefile. One can check whether the products are up to date with an MMS/SKIP/NOACT command. <li>The MAKE CLEAN command (MMS CLEAN) should delete all intermediate files (except those which take a ridiculous amount of time to build), and any byproduct files with limited usefulness. Therefore, it is simple to save space in a directory which someone else has left full of intermediate files. <xmp> Example: clean : - delete *.lis.*, *.ccu.*, *.lmp.* </xmp> <li>The MAKE CLEAN_ALL command should delete all intermediate files, any product files of which the distibution copies are kept elsewhere, and all byproduct files. <li>All source files must be in one (and one only!) CMS library. If the CMS library is not the defualt one for that project (say, the first CMS.DIR found going up the directory tree) then the rule for fetching the source file from the correct CMS library must be put into the makefile. Therefore, a modification history for the whole project is <i1>Modification history effectively available. Also, an MMS/CMS command should be able to rebuild all product files from a directory empty except for the makefile. <li>All the software and build mechanisms for a released product must be under project users (eg RPC, FASTBUS, PILS, VALET). Any copies under personal users should be experimental only. </ol> <h2>C Programming Standards <h3>File organisation Experience suggests the following rules are necessary for portability and comprehensibility: <ol> <li>Filenames should have only one dot (for VMS, MSDOS), and a maximum of 8 characters before the dot (for MSDOS, 9 for VM) and 3 characters after the dot (MSDOS). (The limit on the Cray is a maximum of 14 characters including the dot and the extension.) <li> The public include file is the one used by users of the products of the project. It should be referred to using <angle brackets>. Other include files used for development within the project is kept within the project directory and referred to using "double quotes". Those wishing to use a specific development version of the public include file may use the -I (unix) or /INCLUDE= (VMS) options of the C compiler. <li>For any module which may be called from other modules, a .h file should be used to define the entry points and any other definitions needed by the caller. This include file have as name the facility name followed by .h. Any include files used internally, for example for sharing data or structures between modules, should have a different name to the .c files. </ol> <h3>Style <i1>Style, in C Many of the rules below are essential for portability and readability, or compatibility with some utilities, and are not simple a matter of taste. However, there are a few questions of personal preference. When editing anyone's file, please do not change the layout just to suit yourself, unless changing it to the common layout below. The following conventions were adopted in principle for the RPC code, and should be used for new files. At the start of a file, the file header should contain: <ul> <li>An english title for the file (not its name) underlined with ===. <li>On the right, ending in column 80, the file name. <li>After the title, an abstract of the file. <li>A list authors (names and affiliations) so that later modifications and personal comments can be marked simply with initials. <li>A list <hp2>restrictions</hp2> on the use of the module, special consitions for its compilations, limitations as to its portability, etc. <li>A history list, with dates, of any modifications made, initialled by the person responsible. <li>A double row of asterisks down the left hand side to distinguish the block comment from code. </ul> It may be useful to divide the file into sections using form feeds and section titles centered and double underlined, for example: <xmp> Local Coding Macros =================== </xmp> For each function, a function heading should contain: <ul> <li>Before any procedure, a form feed character OR (if the function will fit on the rest of a 66 line page, or immediately after a section heading), two blank lines. <li>An english title for the function (not its name) underlined with dashes and indented. <li>On the right, ending in column 80, the function name. This makes it easier to find when flicking through a listing. <li>After the title, an abstract of the procedure. <li>A list of preconditions which MUST be true on entry into the function. <li>A list of conditions which are true on exit. This includes any change to global data or system state, anything corrupted. <li>Optionally, a list of parameters, if these are not clear from comments against the parameter declarations and in the pre- and post-condistions. <li>A history list, with dates, of any modifications made specifically to this function and not recorded under a general heading for the module. <li>A declaration of the function BOTH in ANSII C AND in Kernighan and Ritchie C. Use the __STDC__ preprocessor token to select one or the other. <li>A double row of asterisks down the left hand side to distinguish any block comment from code. </ul> Below is a small example. <box> <listing> <Form Feed> /* Allocate an RPC message rpc_new() ** ----------------------- ** ** This procedure must be fast, as it is used in many places in the rpcrts.c ** and ts.c modules. For this reason, it keeps a list of allocated messages, ** and does not return them to the heap when they have been used. ** ** On entry, ** p The address of the pointer to be filled in ** bytes Irrelevant. In principle this was to be allowed for a ** message size required, if we had variable sized messages. ** On exit, ** *p is either the address of a message, or zero if there is ** not enough memory. */ /*ARGSUSED*/ #ifdef __STDC__ void rpc_new( rpc_message_pointer *p, /* -> Pointer to be returned */ rpc_integer bytes) /* Size: ignored */ #else void rpc_new(p, bytes) rpc_message_pointer *p; rpc_integer bytes; #endif { if (free_messages == NULL) { *p = (rpc_message_pointer)malloc((unsigned)(sizeof(*(*p)))); } else { *p = free_messages; free_messages = (*p)->m_next; } /* if */ } /* rpc_new */ </listing> </box> Within a function, <ul> <li>Indent by four columns at each open brace "{". <li>Please put the openning brace at the end of the same line as the if, while, etc. <li>Line up the closing brace with the first character of the opening "if", "while", etc.: <hp2>not</hp2> with the enclosed code. <li>Comment the closing brace with the function name (for a function block), or with "if" or "while" etc. For large blocks, give some clue as to which "if", for example "/* if bad status */". When closing an "else" block, give the correct sense of the comment to the closing brace, e.g. "/* else good status */". <li>Leave spaces on either side of = and ==, but not inside round brackets. <li>Do not perform an assignment in the outer level of a conditional expression. For example, <xmp> if (pointer = mallloc(sizeof(*pointer)) { printf("malloc was successful\n"); } </xmp> is confusing to most people and gives a warning with some compilers. </ul> <h3>Use of the preprocessor. <i1>Preprocessor, use of <ul> <li>All macros should be entirely in upper case unless they completely emulate functions, in which case they may be entirely in lower case. <li>When an include file refers to a data type which is a pointer to a structure defined in another module, then a preprocessor token may be used to allow either the relevant include file to be included, or for a dummy declaration (int *) to be made. This allows one to hide the structures when knowledge of them is not required. It allows one to restrict the effort involved in a remake, and to restrict the number of include files distributed. <li>Tokens beginning and ending with two underscores are reserved for interaction with the compiler. Do not use them for anything else. (Apart from causing confusion to people, this upsets the OS9 compiler) </ul> <h3>System-specific code Any <hp2>system-specific code</hp2> may, if large, be consigned to a completely separate module, or may be controlled using macros. In the latter case, no reference to specific machines or operating systems should be put in the text, only in a file called syspec.h. This file defines, for a given system, some alternative macros and flags. For example, code depending on the byte order might use a LSBFIRST flag which in turn is set by syspec.h. The syspec.h file is in fact a set of conditional inclusions of different system_specific files, one for each system. It uses the compiler predefined tokes to select the relevant file. In this way, extra machines can sometimes be added without touching the distributed files, except adding a line to syspec.h. Macros which may be altered by syspec.h should be defined conditionally on their not having been defined before. <xmp> #ifndef PANIC #define PANIC fprint("Panic!\n"); exit(-999); #endif </xmp> <h3>Utilities Compiling a module with an ANSI C compiler is a useful way of checking for nonportable aspects and for bugs, especially if you have included ANSI C headings. The TurboC compiler on MSDOS is fairly fussy, for example. `lint' is a unix tool for checking C files for consistency. It is not perfect, and code does not have to be completely "lint free" but it is is useful, and code should run though line with as few warnings as possible. Note the use of a lint specila comment /*ARGSUSED*/ to prevent lint complaining that a deliberately unused argument had not been referenced. There are a few other similar special comments (See `man lint' on unix.) There are various "prettifiers" which aim to format C files. `indent' from Gnu and unix `cb' are two. These do not do a good enough job: they tend to make a mess of comments, for example. Therefore, do not assume you can clean up your code automatically afterwards! <h2>Software distribution <i1>Distribution, software <i1>Versions, software This scheme roughly follows the DD/OC MODEL scheme, leaving a little more flexibility. A version directory is one whose name is vx_y where the version is version x.y. (eg [RPC.UTIL.V2_1]). <ol> <li>Under a version directory the subdirectory distribution_files should contain all the files to be distributed with the release. <li>Under the version directory the subdirectory MASTERS (eg [RPC.V2_1.MASTERS]) should contain the master sources of any command files, release notes etc applicable only to that release. <li>In the version directory, the BUILD.COM command file must <ol> <li>Copy all files from the development directories and the masters subdirectory into the distribution_files subdirectory <li>Create a VMS instalation kit (.A, if appropriate), or otherwise a backup (.BCK) file containing all the distributed files. </ol> <li>The file SETUP.COM (for VMS software) should be released with the <i1>SETUP.COM files and perform any necessary logon time set&hyphen.up for a user to be able to use the software. The source for this will normally be in the masters subdirectory. </ol> <xmp> ..tb 9 17 25 33 41 49 57 65 73 [...V2_1] | build.com | ___________|___________________ | | MASTERS RELEASE_FILES setup.com all files included in release xxvvv_release_notes </xmp> Note: One needs priviledge to build a VMS/INSTAL set, <i1>VMS/INSTAL so in practice this cannot be done on VXCERN: Nor can an INSTAL procedure be done by a nonpriviledged user who wants to pick up a version of the software on another machine. For systems which do not require priviledge to set up, this simpler SETUP.COM system is used instead. <appendix> <h1>Differences from "hermes" <i1>hermes This section notes some of the differences between the DD/OC RPC system and the "hermes" system from Eric Cooper at Berkley, adapted by Kris Kostro at CERN<BIBREF REFID=HERMES>. The two systems are similar in that they are both remote procedure call systems using an RPC compiler for stub generation. <H2>Parameter passing Whereas the hermes project <BIBREF refid=hermes> is designed to interface to C programs, in DD/OC we (and our users) require remote C, Pascal and FORTRAN and PILS programs. The calling conventions therefore differ. The "hermes" system would only allow data to be returned as a function value, wheras we must allow all combinations of IN, INOUT and OUT parameters. <H2>Linking There is no provision in the hermes system for the linking of distributed system to take place transparently. This is only provided, in the DD/OC system, under VMS. <H2>Description file The hermes system uses a special "MESA" language, rather than Ada. <H2>Message Format The message format in hermes has been adapted from the Courier standard in various ways. <H2>Compiler generation The hermes compiler ("rig") is built using the UNIX tools "lex" and "yacc", and produces output in C. This RPC compiler is written in Pascal, and produces Pascal, FORTRAN, PILS or C output. <H2>Protocol The hermes system's built&hyphen.in protocol uses a concept of a transient connection, which lapses on timeout. The same connection is used by the transaction protocol and by the rpc system. In the OC system, a transport connection is established at program initialisation time, and kept whilst the program is running. <h1>Comparison with Sun XDR RPC <i1>Sun XDR <i1>XDR, Sun <ul> <li> Sun RPC supports a variety of call&hyphen.time authentication schemes. These are relative heavy schemes, based on unix principles. <li> The XDR language used to describe operations and data types is in the C idiom. <li> There is a run time check on the version of RPC protocol, as well as on the versions of the stubs. This is a good idea. <li> The marshalling routines will handle pointers, but not NIL ones. I guess this means they will recreate linked data structures. (but why not NIL pointers?) <li> Following the C idiom, all parameters are considered IN parameters, and the function result may be a complex structure. <li> The marshalling format is different: in particular, everything is a multiple of 32 bits. (With us it is 8 or 16, depending on the ALIGN option). <li> The (un)marshalling routines put (take) data directly to (from) a serial data stream, rather than via a buffer. This is an interesting difference of philosophy: it relies on the Unix concept of bytes stream oriented communication. I feel that in our environments, this would be too slow, as there would be too many calls to the communications services. However, this approach has the advantage of having no limitations imposed by the buffering available within the RPC system. </ul> The conclusion is that Sun RPC was not optimised for very fast calls at run time, as there is quite a large call time overhead. Similarities between the two systems include <ul> <li>The basic RPC concept. <li>The basic byte ordering is the same, and constant. </ul> <h1>Glossary In response to a specific request, here is a list of some of the terms used in this document, with short definitions. <dl> <dt>Application <i1>Application, definition <dd>Code which is directly, rather than indirectly, useful. <dt>AST <i1>AST, definition <dd>Originally, VMS term "Asynchronous System Trap". A user procedure which is executed in the context of a user process, but for which the original activity of that process was interrupted. A call to an AST is forced by the system, just as though the user code had executed a procedure call. See an application note by TBL, or VAX/VMS documentation. <dt>Callback <i1>Callback, definition <dd>Within one remote procedure, the calling of another remote procedure which is executed by the process which called the first remote procedure. See the user manual. <dt>Client <i1>Client, definition <dd>That which uses a service provided by another. In RPC, that which calls a procedure executed by another. See user manual. A process is a client in relation to a particular remote procedure: the same process may be server of some procedures and client of others. <dt>Client Stub <dd>A dummy module emulating a remote procedure on a local machine. See user manual. <dt>Connection <i1>Connection, definition <dd>An agreement between communicating transport services which allows orderly and reliable communication. For the transport service, this involves keeping a certain amount of information about messages sent or received. <hp1>Connection identifier:</hp1> A number by which the user of the Transport Service refers to a particular connection. <hp1>Connection oriented:</hp1> Of a transport service, providing orderly reliable communication by the use of connections. <hp1>Connectionless:</hp1> Not connection oriented. A connectionless communications service normally allows the unrelaible exchange of messages <dt>Datagram <i1>Datagram, definition <dd>A message sent over a connectionless communication service. <dt>Disjoint <dd>Not overlapping. <hp1>Disjoint address spaces:</hp1> address space not accessible from the same process, and therefore not allowing communication using shared memory. <dt>Handle <i1>Handle, definition <dd>A reference number by which a Client Stub refers to its Peer Server Stub. <dt>INCLUDE <i1>INCLUDE preprocessor <dd>A source code preprocesor developed as part of the CERN cross&hyphen.software suite. This allows conditional selection of code sections, and the embedding of code from other files. It is used in the generation of the RPC software for specific target environments. <dt>Link <dd>A term sometimes used for "Connection". <dt>Marshalling <i1>Marshalling, definition. <dd>The action of taking data from the application data area, converting it into a suitable format, and putting it into a message ready for transmission. <dt>Medium <i1>Medium, definition <dd>That through which things travel. In this document, the method of communicaication including all the services provided by the operating system. <dt>Message <i1>Message, definition <dd>1. A logical unit of communication, containing an ordered set of bytes, and having a defined length. 2. A data structure used within the RPC system to store such a message and also some associated information. <dt>Package <i1>Package, definition <dd>An ordered set of procedures and functions. A package is defined by one definition file. A package is the unit of RPC compilation, and a unit of addressing at run time. See the user manual. <dt>Packet <i1>Packet, definition <dd>A unit of communication over a network. A term generally used at the link layer, e.g. for an ethernet packet. Packets tend to have limited sizes. <dt>Packing <dd>A term sometimes used for "Marshalling". "Marshalling" is preferred because "packing" suggests expressing data in a more compact format, as in the pascal "packed array". <dt>Peer <i1>Peer, definition <dd>Another on an equal standing. Partner on the same level. <hp1>To be tried by one's peers:</hp1> to be tried by those of equal social status. <hp1>Peer address:</hp1> The address of the equivalent entity with which one is communicating. <dt>Process <dd>A serial thread of execution by a computer. <dt>Remote <dd>At a distance. In RPC, strictly: in a different process with a disjoint address space. Alternatively, on a different processor. <dt>Run Time Support <dd>A body of software providing to the Stubs those services which they require. <dt>Segmentation <i1>Segmentation, definition <dd>The process or dividing large messages into small packets, and of reassembling them into the original messages. <dt>Server <i1>Server, definition <dd>That which provides a service to another. In RPC, that process which executes a procedure called by another process. <dt>Service <dd>Useful function. <dt>Server Stub <dd>A dummy module emulating a remote calling program on a local machine. See user manual. <dt>Socket <i1>Socket, definition <dd>A channel of communication provided by the RPC transport service. A socket number is generated by the transport layer when a socket is opened by calling TS_Open. The number is quoted for all communications functions referring to that channel. Internally to the TS, the socket number is typically a pointer to a data structure describing the channel. (The term 'channel' is used here only in explanation: it is not used in the RPC system to refer to a socket). (Berkley unix has a 'socket library' with a similar concept.) <dt>Stub <i1>Stub, definition <dd>A dummy module emulating a remote module on a local machine. See user manual. <dt>Task <dd>Common term for Process, e.g. in RMS68K. The term "process" is preferred in this document, but terms monotask and multitask are current. <dt>TS <dd>1. Transport Service. 2. The module of this RPC system which provides a transport service to the RPCRTS module. (See the diagram at the start of Chapter 3) <dt>TSAP <i1>TSAP, definition. <dd>Transport Service Access Point. An ISO term, denoting a thing to which a connection may be made. The name of the TSAP to which you want to connect is what you give to an ISO transport service when asking for a new connection. The DECNET equivalent term is "Decnet object". <dt>Transport Service <i1>Transport Service, definition <dd>The function of reliable communication of information between different processes in disjoint address spaces. <dt>Unit <dd>Old name for a package. See "package". </dl> <backm> <bl compact=1> <BIB ID=COURIER> "Courier: The Remote Procedure Call Protocol", XEROX Corporation, XSIS 038112, December 1981. <BIB ID=X409> CCITT "Red Book" Volume VIII, Recomendation X.409: "Message Handling Systems: Presentation Transfer Syntax and Notation" <BIB ID=HERMES> K. Kostro, CERN/DD/SW, "Portable Communications with Remote Procedure Call Protocols", 14 November 1985 <BIB ID=RPCUSER> T. Berners&hyphen.Lee, A. Pastore DD/OC, "RPC User Manual", August 86 <bib id=REALS> H. von der Schmidt, "32&hyphen.bit Floating Point Formats possibly used in OPAL and conversion between them" HvdS/OPAL 13 May 86 <BIB ID=ASN1> ISO 8824, "Specification of Abstract Syntax Notation One (ASN/1)", International Standards Organistion. <BIB ID=ecma127> European Computer Manufacturers Association, standard ECMA&hyphen.127 "RPC: Basic Remote Procedure Call using ISO Remote Operations" UDC 681.327.8.04/.06 ISO/IEC DIS 10148 <BIB ID=iso8649> ISO 8649, <hp1>Information Processing Systems - Open Systems Interconnection - Association Control Service Element,</hp1> International Standards Organistion. <BIB ID=iso90721> ISO 9072/1, <hp1>Remote Operations (1): Model, Notation & Service Definition</hp1> International Standards Organistion. <BIB ID=iso90722> ISO 9072/2, <hp1>Remote Operations (2): Protocol Specification,</hp1> International Standards Organistion. <BIB ID=NCSRM> Apollo Computer, <hp1>Network Computing System Reference Manual</hp1>, Order number 010200 <BIB ID=NCAPS> L. Zahn, Apollo Computer, <hp1>Network Computing Architecture Protocol Specifications</hp1>, Order number 010201 To find documents, people, programs, etc., related to the DD/OC RPC project under VMS, type <fn> Avaliable on VXCERN. To use on, put the line ..br $ enq*uire:==@disk$d1[timbl.enq]enquire into your login.com file. </fn> "ENQUIRE RPC". </bl> <index ix=1>Index </backm>

..ti set ..tb 9 17 25 33 41 49 57 65 73 <TITLE>RPC Internals ..ce (Formerly, "RPC Implementation Guide") <AUTHOR>T.J. Berners&hyphen.Lee CERN DD/OC <DATE>Version 2.3.2, Last Revised February 1989 <COPYRIGHT>CERN 1986, 87, 88, 89 ..fo on This document provides internal details of the RPC system which are not covered by the RPC User Manual, and which a user should not need to know. In particular, it defines internal interfaces within the RPC system and message formats. ..fo off ..ce on <HP1>(Details about how to acquire a copy of this document are given in an appendix). DOCFIND RPC INTERNALS RPCIMP </HP1> </TITLEP> <TOC> </FRONTM> <BODY> <H1>Introduction ..tb 9 17 25 33 41 49 57 65 73 ..tb set ..hy set endpt 3 ..hy set minpt 3 ..ti set This document was written for those who intend to maintain, modify or port the RPC system, or (to a lesser extent) those who have an academic interest in its internal design. It assumes a prior knowledge of RPC principles, and of the User Manual <BIBREF REFID=RPCUSER> for this system. The amount of overlap with that manual has been reduced as far as possible, but has been unavoidable in some places. This document covers the internal functioning of the RPC system itself. In particular, it defines internal interfaces within the RPC system and message formats. It does not cover code which was developed in order to make some of the target environments more friendly, such as the AST system or the Event Handling system under MoniCa. There is some discussion of design criteria and choices, with comparison with other systems. The level of detail covered has had to be a compromise, in an effort to answer a reasonable number of likely questions in a reasonable space and time. <H1>Design Choices <i1 ix=1>Design choices This section introduces this particular RPC system to those who are familiar with other RPC systems, explaining some of the eccentricities of this system. These have followed from the requirements of the system, in particular the environments in which it has had to run. <h2>Requirements and Constraints <i1 ix=1>Constraints, on design <h3>Target environments RPC is used for a variety of applications in control and data acquisition systems for physics experiments and related test equipment. These systems must be distributed for three reasons: the severe geographical constraints, the system topology required to handle very large data rates, and the fact that the design teams are distributed between collaborating departments. In our case, processors involved are VAX ..bf FOOT; &S'TM. ..pf minicomputers, and 32 bit (M68000 series) and 16 bit (M6809) microprocessors, and personal computers (IBM and Macintosh). The operating environments are varied, and Unix is (at the time of writing) conspicuous by its absence from the online systems. Some of the systems are monotask systems, or (in VMS) systems in which process creation entails a large overhead. This lead to te use in many cases of "AST" (see Glossary) procedures, rather than the frequent forking of processes. <h3>Example applications Examples of the use of RPC include facilities such as the following, which have been provided remotely using RPC by various groups at CERN at elsewhere: <ul> <li>Remote error logging <li>File access <li>Terminal access <li>Graphics packages <li>Menu packages <li>Gas system control and monitoring <li>Control of processor farms <li>Remote program load and control <li>Remote database access <li>Remote FASTBUS access <li>Remote operating system commands <li>etc. </ul> <h3>Communications protocols <i1 ix=1>Protocols, requirements Whereas many commercial and academic RPC systems have been based on the Internet protocols, these we not generally available in most of the online environments. <ul> <li>DECNET ..bf FOOT; &S'TM., ..pf <li>raw Ethernet <li>ISO Class 4 Transport, <li>direct communication over 32 bit parallel cables, <li>RS232 links </ul> are currently used, with the adition of others possible in the future. The system had therefore to be designed to cope with both unreliable datagram and reliable connection oriented services, and to allow a migration path for users as the availability of facilities changed. It is general required to be able to reconfigure an application to run over a different protocol at run time, by changing tables or by control by the program itself. The use of unreliable datagram services was considered to be a temporary phase, and was covered by the invention of an ad hoc protocol summarised later in this document. <h3>Application Languages As FORTRAN has been a favorite language for physicists for a long time, there was a requirement to provide RPC stubs to interface to FORTRAN code. The first RPC application (The Valet-Plus) involved a lot of Pascal code, so Pascal compatibility was also required. Later, the OS9 system became important for users, and only supported C, and the Valet plus environment produced a requirement for stubs in PILS. In some cases (when using the CERN cross software, on the PCs, and originally on VAX/VMS) the support for Pascal was, when the system was developed, much stronger for the support for C. Support for FORTRAN was patchy in the microprocessor and PC areas, especially if it to be mixed with other languages. <h2>Design choices <h3>Implementation Language For the RPC system itself, FORTRAN was rejected as having insufficient power (no pointers or structures) and being insufficiently available. Because of its availablity at the time, Pascal was chosen as the initial implementation language for the system, whereas C would have been more usual. Since that time, support for C oriented environments has been provided, using the help of pascal to C conversion programs. FORTRAN callable stubs were provided for some time by producing "FORTRAN callable" pascal, but eventually it proved simpler to provide stubs in FORTRAN itself, and provide FORTRAN-callable entry points in the run time support. <i1 ix=1>Languages, implemention (The support for N languages on M machines tends to produce N*M mappings between them, e.g. between FORTRAN and C. It is easier to support the mapping in the limited case of run-time library entry points, rether than in the general case of a stub procedure produced by the RPC compiler. One then has only to support N compiler options and M runtime library flavours). The same became true of C and PILS <i1 ix=1>PILS <REFER ID=PILS>, so the RPC compiler now produces four different languages. <h3>Description Language ..cc 6 In our case, we chose an Ada ..bf FOOT; &S'TM. ..pf -like language to describe the procedure interfaces, as it was the only real time language in use locally which provides sufficient information about each parameter, namely its type and direction. For none of the other programming languages used was this true. <h3>Message format Standardisation of the format used in RPC messages is still an open question, despite its obvious advantages for interworking. One of the reasons for this is that different RPC systems have been optimised according to different, conflicting, criteria. There is a trade&hyphen.off, for example, in the coding scheme between generality, economy of message length, and encoding speed. In our case, we chose the Xerox Courier format, in which data sizes are defined at compilation time, and data is marshalled into bytes most significant byte first. The overhead of encoding data according to the X.409 standard was felt to be too high, both in time and space. In particular, transmission over RS232 lines put stringent constraints on the efficiency of encoding. Some systems deliberately check whether the communicating machines happen to be of the same type, and in this case skip the conversion of data into standard form. The time, space and complexity of this was not felt to be justified by the relatively small amount of time spent converting data formats. (This is especially so when the conversion routines called by the stub code are expanded in&hyphen.line by a good optimising compiler such as the VAX/Pascal compiler) <h3>Data Types The primary requirement was for FORTRAN style data types, but the type offered were extended on demand to the full set of types used by Pascal and C, with the exception of variant records (unions). We had to extend the Courier data types to include a real number type (sent in IEEE format). We also found that user program portability is greatly enhanced by the addition of a type which is represented as the native integer on any machine, but is constrained to lie in a 16 bit range, and is passed as 16 bits. The string type proved to have the greatest variation in its local representation, each version of pascal having its own representation, and a semantics slightly different from that of FORTRAN. In general, it was found to be important that the logical type of the data transferred be specified quite independently of the actual local format. The first determines the format of the messages, while the second determines the method by which the data is passed on the stack. The mapping between these two is a function of options supplied to the rpc compiler when the stubs are generated. It is not, for example, so useful to have a data type "zero terminated string": the data type is "string", which is mapped onto a zero&hyphen.terminated string when stubs are generated in C. The same procedure, written in FORTRAN, will use a string descriptor, while in pascal it would use one of the more or less exotic devices built into various compilers, or, for standard pascal, a pair of separate variables for the characters and the length. <h3>Addressing and Identifiers Unlike in some systems, no numbers or identifiers need to be globally unique. <i1>Unique identifiers The procedure number within the package is only known by the client and server stub modules, and is allocated at compilation time. The other three layers are allocated at load time. The package number is generated by, and only relevant in the context of, the server on which it it provided. It could be used as a capability in a capability&hyphen.based authorisation system. The client stub module knows only a logical name for its server counterpart: this is used to look up the full address of the remote package. Although this process currently uses local tables which are set up when the system is configured, provision is made for the search to be extended to a heirarchical sequence of name servers. We find that a distributed system is rarely built as a monolithic whole, but grows by the progressive amalgamation of independently designed sub systems. It is therefore important that a system may function independently as much as possible, while also being part of a larger system. The principle of information hiding in modular software then applies equally to distributed software, and requires that one does not rely on globally unique identifiers or centralised administration. <h3>Call Time vs. Load Time <i1>Call time <i1>Load time Trade&hyphen.offs exist when designing the run&hyphen.time support software for RPC. There is a distinction between systems designed for an occasional isolated call, and those designed to handle large amounts of traffic between particular partners. In our case, the goal was to optimise run&hyphen.time performance by doing as much work as possible at compilation time (type checking, for instance), and at program load time. At load time, any necessary connections are established between tasks. The media to be used, and the remote addresses, are looked up at this time. This allows a system to be reconfigured without recompilation or even relinking the executable images. Load time is also the appropriate time for starting server tasks, and checking client authorisation. Under the VAX/VMS ..bf FOOT; &S'TM. ..pf operating system, PILS, and TurboPascal, facilities exist for the automatic intialisation of software modules when they are loaded. In this case, any stub modules declare themselves to the run&hyphen.time support at load time. The remote binding is then carried out quite invisibly to the programmer, who does not have to write any code <i1>Initialisation, automatic to do it himself. Although it is possible for a program to expressly relink itself to different modules while it is running, in general any remote call is made over an established communication path to an established server module, and can therefore execute with no additional delay. This method of working also allows us to use connection&hyphen.oriented communications services (which happen to be available locally and are at a more advanced stage of standardisation than connectionless services) without a prohibitive overhead. The call&hyphen.time performance is then determined by parameter copying time and by operating system overheads. Without any assembly coding in these areas, a dummy call with n bytes of user data takes, across Ethernet, for example, about (5+0.08n) ms between VAX and an M68000 based system. <h2>Extra Facilities Although the primary goal was simply transparent procedure call between machines, experience has lead to a number of extra features, beyond the basic RPC function. These include <ul> <li>detailed trace outputs about remote calls and their parameters, to help in the debugging of distributed applications. <li> When the confidence in the remote machine, the remote application software, or the communication protocol is low, users have asked for optional application level timeouts on remote operations. This turns out to be important in a real time system for preventing some unforseen fault causing an unduly large amount of the system to hang. In this case, a maximum time allowed for a remote operation may be specified as a compiler option, or in the definition file. See PRAGMA TIMEOUT in the user manual. <i1>Pragmas <i1>Timeouts <li>Some users (especially FORTRAN programmers) have variables whose actual data type vairies at run time. There is provision for users to include their own marshalling algorithms to cope with this (on the assumption that <hp2>some</hp2> algorithm exists for interpreting the data). This also allows related implicit data such as FORTRAN common blocks to be transferred if necessary. <i1>COMMON blocks, transfer of See PRAGMA EXTERNAL_MARSHALLING in the user manual. <li> Also selectable at compilation time is the option of remote operations, once started, continuing concurrently with the calling program. This only applies to procedures which return no data. By adding a little pipelining, this option increases the performance of some control applications. See PRAGMA concurrent in the user manual. <li>Multiple servers: Dynamic Binding. In some cases, a client may have the choice, at run&hyphen.time, of many different, though congruent, server modules. For example, a control program may have access to identical servers on different parts of the apparatus. For these applications, facilities are provided for a client program to reconfigure itself to use different server modules, and to switch the binding rapidly between them. Obviously, in this case, the application program does become aware of the addresses of server modules, so the transparency is reduced. <li>Asynchronous servers. The lack of multitasking in many environments has produced a requirements for a server which will run application code, but then be interrupted at AST (see Glossary) level to service a remotely called procedure. <li>Two forms of reverse call proved to be required. In one ("callback"), a call is made from the server process back to the original client process during the original call. In the other ("CALLER addressing"), the server merely establishes a handle on the caller in order to call him back later asynchronously, outside the bounds of the original call. </ul> <h2>Retrospective comments In considering the design in the light of experience, there are a few areas in which things might usefully have been done differently. <ol> <li>The use of Courier format has been useful only in that it answered some basically arbitrary questions, and was stable. It has not, in practice, been used to interwork with any other systems to date. If one were to change the format, then from the technical point of view, the version number system <fn> The ideal version numbering would involve the specification of three numbers each particular version of the definition file. One would be the version number (V) of the file; one would be the lowest previous version number of server with which a client of version V would be compatible, and the other the lowest (previous) version number of client with which a server of version V would be compatible. The checking of version numbers would involve sending two of these from one side to the other. </fn> could have been more sophisticated, and checked at initialisation time, rather than every call. Also, the TID field of the message belongs to a lower layer of protocol, and so should have been included in an outer layer of message format. <li>The limitation of fixed size buffers has been awkward in some cases. It would have been useful to be able to send arbitrary large messages. There is a question as to whether this would compromise the speed. This is discussed later in more detail, with some workarounds. <li>If we had realised how long the ad hoc datagram protocol would be used, we would perhaps have made it more sophisticated. The deficiencies of this protocol are also discussed later. </ol> A list of possible extensions and improvements to the current system is kept elsewhere, as it changes (in both directions) from time to time. <H1>Operational Overview There follow a few notes on the way a remote call works. Some of this information is duplicated in the user manual. For more detail, see the flowcharts of the procedures in question, and comments in the source code. ..cc on ..fo off ..bx 5 35 / 45 75 ..tb set $ ..tb 20c 40c 60c $Client application$User$Server application $module$package$module $$calls ..bx 20 / 60 ..bx new 40 ..bx off ..bx off $User package$defined by$User package $calls$- definition file -$calls ..bx 20 / 60 ..bx new 10 30 / 50 70 $Client stub$Courier$Server stub $$Level 1 ..bx off ..bx can $RPC_xxx$$RPC_xxx $calls$$calls ..bx on 20 / 60 ..bx new 5 35 / 45 75 $RunTime Support$Courier$RunTime Support $(portable)$$(portable) ..bx off ..sk ..bx can $TS_xxx$$TS_xxx $calls$calls ..bx on 20 / 60 ..sk ..bx new 5 35 / 45 75 ..sk $General Communications$$General Communications $Support$$Support ..bx off ..sk ..bx 5 35 / 45 75 ..br $Specific underlying$(whatever$$Specific underlying $medium$protocol)$medium ..bx off ..fo on ..ce on ..us on ..tb set Software interfaces in an RPC link. <i1 ix=1>Interfaces, diagram of software Vertical interfaces are procedure calls between layers of software; horizontal interfaces are protocols at each level. ..us off ..ce off ..cc off ..tb 9 17 25 33 41 49 57 65 73 <H2>RPC_Open: setting up an RPC link <i1>RPC_Open The addressing and routing decisions which are necessary in order to connect two modules are made, as far as is possible, at initialisation time, in order to allow remote calls to run with as little overhead as possible. This has been referred to elsewhere <BIBREF refid=ecma127> as "semi&hyphen.lazy evaluation of binding". <i1>Binding A procedure is provided to set up an RPC service. It is called by, or on behalf of, the client stub module in order to establish contact with its partner server stub. The package required is identified by a name (a string of characters). The procedure, RPC_Open, is part of the RPC run&hyphen.time support. RPC_Open then returns to the caller a <i1>Handle <hp2>handle</hp2> (an integer) to be used when referencing the package. When RPC_Open is called, it in fact sets up a table within the local RPC RTS which gives, for that "handle", the network address of the relevant remote <i1>handle server, and the package number which will identify the package on the remote machine. The caller is unaware of these values. Under VMS, the logical name table is currently used for name translation, and under MoniCa, the MoniCa symbol table. Both allow interactive setting up using <i1>DEFINE command a DEFINE command. See the user manual for details for other systems. (The translation, or mapping, function is related to the verification of the user's right to access the remote package. (authorisation). Some work has been done on this, including a definition of requirements and a prototype database access implementation. This work has not yet been integrated with the RPC system.) ..bx off The client stub module xxx contains a procedure open_xxx which, when called, will call RPC_Open, with its own logical name, and hence acquire a handle which it can use for future calls. Under VMS, this procedure is flagged as an "initialise" procedure, and called automatically on image load. It is also called automatically under TurboPascal on the IBM PC or the Macintosh, or in PILS. <H2>Using the link: Stub Calls Run&hyphen.Time Support Having packed the buffer, the stub calls the run&hyphen.time support procedure RPC_CALL. It passes across the package handle, and the buffer. <H2>RTS calls Communication Service <i1>RPC_Call The RPC_CALL procedure performs a little more formatting of the message, inserting the reference number of the package on the remote machine. The run&hyphen.time support package uses the package handle to extract this from its tables, and also to get details of the how to contact the remote server. This includes which network to use, if there is more than one (eg RS232 and Ethernet), and any addresses which it will need to use on that network. If the communication service it uses is connection&hyphen.oritented, then it will use a connection which has already been set up by RPC_Open. Standard calling sequences in this area have now been defined by the CATS committee. The TS_xxx routines which it proposed are similar to the TS_xxx routines currently used by this system, and documented in a later section. <H2>Adaptation of communications calling sequence <i1>Transport service, adapter layer A layer of software exists which adapts the standard calling sequence and functionality required by the RPCRTS to the local facilities available is written specially as part of the port to each system. This layer provides four functions: <ul> <li>Selection of the apropriate medium <li> Calling the local routines, OS entry points, etc, to communicate over that medium <li> Handling of multiple simultaneous connections, for connection oriented services. <li>Protocols for ensuring reliability, over unrealiable datagram communication services. </ul> See "Transport Service" index entries. <H2>Server RTS system calls communication service. On the remote ("server") machine, the Run&hyphen.Time Support system is waiting for a message. The calling seqeunce here is again dependent on the communication service used. Depending on the type of server, blocking or asynchronous receive calls are used. <i1>spawning of servers The question of whether the RTS process forks (spawns) off a separate process to handle the call is up to the RTS in question. This is not currently an option. Under VMS, the overhead of process creation is very great. Under monotask systems (PC etc) it is impossible. Under the TCS system (DD/OC MODEL project VMS subtasks), a fork is made for each request. (This has been built up using the RPC_Queue_Server" and RPC_Service" primitives.) <H2>Server RTS system calls Server Stub When a message is received, the server RTS calls the server stub. <i1>Server stub, overview There is one stub procedure for each "package". Inside it is a big CASE statement, ("computed GOTO") to select, on the procedure number, the relevant unpacking/packing code and user procedure. <H2>Server stub calls User routine The User procedure is called in just the same way as it would have been called, were the calling program on the same machine. <H2>The Return of OUT Parameters Before the server stub calls the user routine, in the case of a "concurrent" procedure (not function) with only "IN" parameters, the stub calls RPC_early_return. This sends a reply back to the caller, who can then continue to execute while the called routine continues in paralel. If an error occurs, the stub may set the m_status field in the message to a bad (even) value, and the call will be rejected, with the given as rejection reason. In the normal case, when the stub returns to the RTS, the return parameters, which have all been packed back into the message buffer, are returned to the caller. <H2>Remote Errors <i1>Error codes Errors may occur on the called machine, either in the RTS or in the called routine. In either case, the errors will, where possible, be trapped, and a "reject" response given. In this context it would be very convenient to have a global format for error codes, so that the reason code can be interpreted on the remote machine. The portable RTS always generates VMS&hyphen.style error codes, with a special facility number. The Courier "reject" message type is used for this feature, with the VMS error as a parameter. <H2>Asynchronous Events, Interrupts and Break&hyphen.in In order to allow asynchonous events to be handled remotely, the RTS must allow a client stub so be called as a result of an unsolicited call message arriving. This requires that the communications service be capable of calling the RTS assynchonously, and that a priority channel be allocated to such messages. This is required, for example, in order to implement remote break&hyphen.in in a PILS or MoniCa system. <H2>Closing a service The party who calls RPC_Open is responsible for calling RPC_close when the service is no longer required. <h2>RPC buffers <i1>Message pool <i1>Message format <i1 see="Message">Buffer The RPC system uses a pool of fixed length buffers. The buffer length is defined (by the constant MAXLENGTH in the RPCCONST header file) when the library is built. Currently, the standard version has a buffer length if 1500 bytes, and the "large" version a length of 8000 bytes. <h3>Buffer Format <i1>Message, format in memory <fig place=float frame=none> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip16 <PICTURE NAME=FLOW16$S> ...skip16 <figcap><hp1>The format of an RPC_Message structure in memory. Alternative uses of the same space are separated horizointally. The maximum size of the message depends on the version of the run time system used. </hp1> </fig> The buffer format in memory consists of three parts. The first part is reserved for the internal use of the RPC RTS module and TS. It contains a link (for threading onto lists), a status longword for the general status of the transaction, which may be overwritten at any stage. There is also an index into the third part of the buffer which is used as a read and write pointer by the marshalling routines, and there is the number of the socket over which the call is being made. The second part is reserved for internal use by the transport service. It may be used for formatting a header to the message, where the communications interface cannot gather data from dispersed buffers: this saves the overhead of copying the data into an assembly area. The third part of the message structure is that which is actually transmitted between the parties. It's format is basically Courier standard, and is described later. <h3>Buffer passing and pointer swapping Messages are passed around by pointers, and in general the TS layer reserves the right to switch pointers. That is, the pointer itself is passed VAR, so that, by overwriting the pointer, the TS system can return a different buffer without having to copy the data. Therefore, for instance, in an M68020 system with a LANCE interface chip, the packet is read into a buffer by the LANCE DMA. The buffer is swapped with the empty one provided by the RPC system's READ request, so that the unmarshalling routines take it straight out of that buffer into the user memory area, with no extra copies at all. <h3>Buffer allocation strategy Some systems have more useable heap systems than others. <i1>Heap, use of In cases where the heap does not exist, or where unrealistic constraints are put on its use, the buffers are taken from a static pool of messages (array ..bf MONO static_messages ..pf ). This is controlled by the precompilation option NOHEAP. <i1>NOHEAP precompilation option At initialisation time, the messages in the array are put onto the free list. The size of the initial pool is selected at a compile&hyphen.time as a function of the environment. One must remember, in this context, that the ethernet and RS232 protocols require an extra buffer per socket to store the last call or the last reply sent (or both, if callback occurs). Under MoniCa, the LANCE chip requires (currently 4) buffers as a receive packet pool. In the case where the heap is usable, buffers are allocated from that. Even in this case, once a buffer has been acquired by the RPC system, it is kept permanently. When not in use, it is kept on a free list rather than being returned to the heap. This means that during normal running, buffers can be allocated and deallocated very rapidly, which is essential to the efficiency of the communications code. (Under VMS, Eileen Berman/FNAL found that a large proportion of the time when running over RS232 was actually spent in pascal 'new' when this was not done) <h3>Buffer size limitation When an application requires larger data transfers than the buffer size, it is necessary to get around the buffer size limitation. I note here a few techniques. <ol> <li>One possibility is always to write a loop in the application, to transfer the data in chunks. This was done (See note by Jorgen Petersen) for the HMINI histogram retrieval routines. <li>In the long term, it would be possible to build in segmentation algorithms into the RPC system. Unfortunately, checking explicitly for buffer overflow would slow down the marshalling. In principle, though, the compiler could detect when the parameters could be crossing buffer boundaries, and could (only in those cases) insert code which checks and allows segmentation. This would allow the much more common short RPC to remain optimal. A possible future project. <i1>Message size limitation <li>As buffer sizes become significantly large, the marshalling/unmarshalling time starts to dominate the communication overhead for a fast medium. In the subset of cases in which one is communicating between like machines, it then becomes very much more efficient to transfer the data directly between user memory and user memory, with no intermediate buffering. The CERN Host Interface (CHI), because it can byte swap, in fact allows direct transfer even between machines with different byte ordering. In this case, the (hand modified) stub does an explicit transfer of the data. This is used in the CHI for the remote FASTBUS block operations, when the block size excedes a 1kByte. The buffer size at which one resorts to this technique is a question of trade&hyphen.off between the reduced marshalling time and the increased overhead for one extra DMA. The same technique could be used over DECNET between two VAXes, if required. </ol> <h1>Compiler Overview <i1>Compiler overview <i1 see="Compiler">RPCC The RPCC compiler <fn> This section is in part derived from comments by Antonio Pastore in the source code. </fn> is organized in two parts: <ol> <li>The PARSER <li>The CODE GENERATOR </ol> <h2>Parser <i1>Parser The PARSER checks the input for SYNTACTICAL and LEXICAL errors and produces as output two trees, the TYPE and BLOCK trees. It uses a lexical analyser (GETOKEN) which returns tokens (identifiers, numeric values etc) and checks for reserved keywords of the langage. It can backtrack by one token: this is used for error recovery. The user defined data types are descibed by a list of named types, each of which has a name, and a pointer to the type descriptor tree. A type descriptor tree has as elements a structure which can define a simple type, or a complex type in terms of pointers to the descriptors of its subtypes. Field for each type also exist to define the maximum and minimum sizes of the type, in standard representation, and flags for any special options (external marshalling, etc.). <i1>Block, in RPCC A BLOCK in this context means either a procedure or a function. The BLOCKS tree (pointed to by BLOCKPTR) is a linked list of record each describing a block. Each BLOCK has a NAME, (as given by the user), and a list of parameters. Each record of this list has a NAME, the parameter's name (as given by the user), and a TYPE associated to it. This is a pointer to a type descriptor. The same routine as used to build a TYPE definition is used to analize a parameter declaration inside a BLOCK. <h2>Code generators The CODE GENERATOR reads the BLOCK tree and produces as output two files, the CLIENT and the SERVER stubs. It also optionally produces a declaration file in Pascal or C, with or without a definition of the user's types. Three code generator modules currently exist. One output generator (CODEGEN) will produce Pascal or C stubs; there is also one (CODEFOR) to produce FORTRAN, and one (CODEPILS) for PILS. <i1>Code generators They tend to use similar techniques, although in places the generation of Pascal types proves a simpler task to perform recursively than the generation of C or FORTRAN types. <h2>Source of the compiler The compiler is written with the (rare) use of the INCLUDE preprocessor. This is used to select machine&hyphen.dependent parts, maily in the OPTIONS module. Versions for Bsd 4.2 Unix/Ultrix, for VAXVMS and for the IBM PC can be generated in this way. The source is divided into modules: <dl> <dt>MAIN <dd>The main compiler file, which INCLUDEs each of the following modules in the order given: <dt>DECGLOB <dd>The declarations of constants, data types, and variables for the compiler as a whole. <dt>ERROR <dd>The error reporting and handling code. <dt>DEBUG <dd>The code to list the completed trees created by the compiler. This code is invoked by the debug options. (See user manual). <dt>INIT <dd>General initialisation code. This includes setting up data areas which are effectively constant, as strict Pascal does not provide for constant data declarations. <dt>OPTIONS <dd>This module handles the command line options. It contains implementations of unix style arc and argv functions for every system under which the compiler runs. Also, included are the (machine dependent) file open and file close procedures. <dt>GETOKEN <dd>The lexical analyser. GETOKEN is also the name of the principle procedure of this module. <dt>PARSER <dd>The parser. <dt>CODEFOR <dd>The code generator for FORTRAN stubs. <dt>CODEPILS <dd>The code generator for stubs in PILS. Derived from CODEFOR. PILS has more limited data types than fortran, but better string handling. <dt>CODEGEN <dd>A combined code generator for pascal and C. This module contains a certain amount of special code for different versions of Pascal, especially for the TurboPascal versions. TurboPascal requires UNIT declarations, and IMPLEMENTATION and INTERFACE sections producing separately. </dl> <H1>Transport Service <H2>General <i1>Transport service overview The Transport module (TS) is responsible for all the communications required by the RPC system. As normally most of the communications code is provided by existing operating system facilities, the TS module acts as an adapter layer, to connect the functions required by RPC to the form in which they are available from the operating system. They also perform functions which depend on whether the communication medium is basically connection oriented or connectionless. Where the underlying service is a datagram service, the TS layer must provide such functions as: <ul> <li>Reliability <li>Segmentation </ul> For connection oriented communication, those functions are normally already covered, but the TS layer must now handle: <ul> <li>Listening for connections (multiclient server) <li>Acceptance of connections <li>Actions when peer disconnects. <li>Message boundary marking (if stream oriented) </ul> The connectionless case is in principle more complicated, but the more complicated operating system interface for the connection oriented cases means that neither form of the TS layer is trivial. <h2>Connectionless communication protocol <i1>Protocol, datagram <i1 ix=1>Protocol, datagram The RPC system was originally designed to run using standard protocols, all of which were connection oriented. The built&hyphen.in protocol which was supposed to bridge the gap until these services were available was designed to be as simpple as possible. It does not handle segmentation at all. This means that in any one call over ethernet, the total data in the call or the reply cannot exceed one packet size (1500 bytes). For most applications, this was not considered a problem. The TS layer does use a simple protocol to give reliability in the case of lost packets. The assumption is made that the medium is only rarely unreliable <fn> This is in practice the case with ethernet, or local RS232 connections. In our experience to date, packets over ethernet had only been lost due to receiver overrun (a software problem), but not due to network saturation or packet corruption. For several months, RPC systems ran at CERN assuming complete reliability of the ethernet, without practical problems, before the protocol was introduced. </fn> so no effort was made to make the recovery fast. Different forms of reliability have been considered for RPC systems, different "call semantics". These are: <i1>Call semantics <ul> <li>"At most once": This is basically unreliable remote procedure call, in which one call is sent out, and if it or the reply is lost, no action is taken. <li>"At least once": In this case, one reply is required, and retransmissions are made, but no check is made that a call is not executed twice due to a retransmission. This is fine for idempotent <fn> <i1>Idempotent, definition <i1 ix=1>Idempotent, definition An idempotent opertation is one which may be executed any number (>0) of times, with the same result. Functions without side effects are idempotent, as are procedures which set a global state to a specified absolute value. </fn>operations. <li>"Exactly once": This is the correct semantics for a remote call which is to emulate a local call, idempotent or not. </ul> The TS layer keeps a notional "connection" between the server and any particular client, for the purposes of removing duplicate calls. The protocol used provides "exactly once" semantics except for the first call any connection, for which "at least once" semantics hold. The first call is normally an internal idempotent call (to look up the package number), so this is not a problem. A summary of the protocol is as follows: <ul> <li>Calls sent over a connection are numbered, starting at zero, then cycling between 1 and some maximum number. A call numbered zero is always acceptable - this restarts the sequence (starts a new "connection"). If ever such a packet is received, it is assumed that it is a new instantiation of a client which has sent it. <li>If the client does not receive a reply, it retransmits the call (with the same sequence number). The retransmission is done after a certain time, which doubles each retransmission ("exponential backoff"). <li>If the client receives a reply (return or reject) with the wrong sequence number, it ignores it. <li> If it receives a call with (nonzero) number matching that of the last rely sent, the server retransmits the last reply. Any other call is executed and replied to. <li>A multi&hyphen.client server keeps connections (i.e. a record of the last reply sent) indefinitely. If there were a very large number of possible clients, this would obviously eventully cause the server to run out of memory. In any case, if TS_Close is called, all messages are returned to the pool. </ul> <i1>Retransmission of packets This is not a perfect protocol, but is was felt that it was not worth the effort investing in a more complex one unless a standard existed. Aspects of its simplicity which allow easier implementation include: <ul> <li>The client is the only side which retransmits on timeout. The server is basically passive. <li>The server is not required to receive messages while executing the remote procedure. This allows implementaions on strictly single&hyphen.task systems. </ul> The simplicity of the protocol gives rise to the following restrictions: <ul> <li>The first call must be idempotent. <li>Large numbers of clients could make a server run out of memory <li>There is no way for a client to distinguish between a call which is taking a long time to execute, and a crashed server. </ul> (It is important that the protocol should run on a single task system. In practice, an inefficiency can arrise on such systems, in that ethernet drivers, when there are insufficient buffers, tend to throw away the most recent, rather than the latest, packets. This means that while a procedure is running (and therefore no servicing of incoming packets is possible, that the buffers will fill up with retransmitted calls. In some cases, if the client is faster than the server, then when the server responds, it loses the next call before it can clear out the backlog of retransmitted packets from the last call. This slows the system down by one retransmission on each call following a call which took a long time.) <H1>Software Interfaces and Flow Charts <i1>Interfaces, software, details <i1>Flow charts This chapter outlines the two interfaces: that between the stub and the run&hyphen.time support package, and that between the RTS and the transport service. Note that strings are all passed as 40 character arrays, by address. This means <i1>Strings passed to RTS that the caller must provide a 40 character string, blank filled if necessary. When using FORTRAN on the VAX, the string parameter must be passed using the built&hyphen.in dummy function %REF(thestring) in order to pass the string by address rather than by descriptor. The string length (currently 40), for pascal programmers, is defined in the standard include files as RPC_name_length. The definition of the calling conventions for the routines below must be taken as being the include files which are part of the implementation. These define the calls in terms of the relevant Pascal compiler (VAR or not VAR, etc). Those who program in other languages should ensure that the calling method they use corresponds. The "message" type is defined in the standard RPCTYPES include file issued with the RPC library. This is a Pascal include file, or C include file, depending on the system. Anyone writing code which needs to know the format of this message should either write in Pascal or C, or automate the generation of their include file from the pascal one. <hp1>In the flow charts, boxes drawn with heavy lines indicate procedures expanded elsewhere. </hp1> <H2>Run&hyphen.Time Support Calling Sequences <i1 see="user manual">Calling sequence, for RTS The calling sequences for these routines are given in the User Manual, which should be used for reference. <H3>RPC_Open <i1>RPC_Open Called by client stub, at initialisation. The service_name describes the remote service required. It may be a formatted physical address (program number, network address, etc) or it may be a logical name which is converted by the run&hyphen time system into an appropriate address. The handle returned is used in all future reference to the service by the client. May be called by the user application if he manages his own handles. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip1 <PICTURE NAME=FLOW1$$S> ...skip1 <FIGCAP><HP1>Flow chart of RPC_Open. </HP1> </FIG> RPC_Open, if successful, creates a new record on the client_list. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip29 <PICTURE NAME=FLOW29$S> ...skip29 <FIGCAP><HP1>Format of client_list entries </HP1> </FIG> The work of setting up a client handle is done by the procedure "Setup" which is also used by RPC_Switch. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip2A <PICTURE NAME=FLOW2A$S> ...skip2A <FIGCAP><HP1>Flow chart of Setup (part 1) <i1>Setup, flow chart </HP1> </FIG> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip2A <PICTURE NAME=FLOW2B$S> ...skip2B <FIGCAP><HP1>Flow chart of Setup (part 2) <i1>Setup, flow chart </HP1> </FIG> Find_Remote is a routine which interrogates the remote node to find out the package number of a package for which it only has the name. The call to Find_Remote is in fact a remote call to Find_Local, and is implemented by a standard stub. The stub is written into the RPCRTS module explicitly, as if it were not it would have mutual references with RPCRTS, which would complicate the implementation under strongly hierarchical systems such as TurboPascal. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip21 <PICTURE NAME=FLOW21$S> ...skip21 <FIGCAP><HP1> Find_Local simply looks up the package name and returns the package number </HP1> </FIG> <H3>RPC_Close <i1>RPC_Close Called by client set&hyphen.up when the service is no longer required. The handle value becomes invalid for future use. May be called by the user application if he manages his own handles. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip3 <PICTURE NAME=FLOW3$$S> ...skip3 <FIGCAP><HP1>Flow chart of RPC_Close </HP1> </FIG> <H3>RPC_Call <i1>RPC_Call Called by client stub. On entry, the buffer consists of a header (in which the procedure number has been written) followed the procedure arguments. The count tx_bytes includes only the number of bytes in the procedure argument array: it does not include the header. The timeout parameter specifies (nominally in 10us units) a maximum time allowed for the call. A negative value disabled the timeout (allows an infinite time). The buffer_size parameter indicates the total size in the buffer available for returned arguments (again, exclusive of the header). There are two versions of this routine. RPC_Call itself handles errors, whereas RPC_Call_Status returned error status to te caller. A possibility here is for the RTS to deallocate the buffer, and pass back a pointer to a different one used for recieving the data. This saves time in certain cases, especially if the communications service can accept a user buffer and deallocate it itself after the data has left. Therefore, the caller of RPC_Call must not keep any other record of the address of his buffer. ..us off <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip4A <PICTURE NAME=FLOW4A$S> ...skip4A <FIGCAP><HP1>Flow chart of RPC_Call and RPC_Call_Status. In the case of a local call, the routine Handle_Call is used. RPC_Call_Status returns the status value at the end of this part, wheras RPC_Call continues to treat the error. </HP1> </FIG> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip4B <PICTURE NAME=FLOW4B$S> ...skip4B <FIGCAP><HP1>Flow chart of RPC_Call (continued). </HP1> </FIG> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip5 <PICTURE NAME=FLOW5$$S> ...skip5 <FIGCAP><HP1>Flow chart of Handle Call. This routine is used for local calls as well as by a server of remote calls. </HP1> </FIG> <H3>RPC_Attach_Stub <i1>RPC_Attach_Stub Called by server stub on initialisation. The entry_point parameter passes the actual stub procedure to be attached. The service_name parameter gives the logical name of the server. This may be used by the run&hyphen.time support to update name tables. The service name will be the same for different instances of the same server. The program number returned may be used to disconnect the stub later. Internally, this calls RPC_Specify_Stub. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip6 <PICTURE NAME=FLOW6$$S> ...skip6 <FIGCAP><HP1>Flow chart of RPC_Attach_Stub </HP1> </FIG> This procedure sets up a new server stub record on the list "program_list". <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip28 <PICTURE NAME=FLOW28$S> ...skip28 <FIGCAP><HP1>Format of program_list entries </HP1> </FIG> <H3>RPC_Specify_Stub <i1>RPC_Specify_Stub This is identical to RPC_Attach_Stub, except the program number is generated by the user. This allows him to allocate "well known numbers" to represent particular packages. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip7 <PICTURE NAME=FLOW7$$S> ...skip7 <FIGCAP><HP1>Flow chart of RPC_Specify_Stub </HP1> </FIG> <H3>RPC_Detach_Stub <i1>RPC_Detach_Stub Called by server set&hyphen.up when the server is no longer required. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip8 <PICTURE NAME=FLOW8$$S> ...skip8 <FIGCAP><HP1>Flow chart of RPC_Detach_Stub </HP1> </FIG> <H3>A Server Stub <i1>Server stub, interface IN call_buffer: pointer to message ..bx off Called by the RTS. The length of the stub buffer should be obvious to the stub code, which is aware of its format. In case of error, the stub writes a bad status into the m_status field of the message. <H3>RPC_Early_Return <i1>RPC_Early_Return Called by the server stub, only in the case of a "concurrent" procedure. The message consists of a header (which is left intact by the stub) and a results array, which is filled. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip9 <PICTURE NAME=FLOW9$$S> ...skip9 <FIGCAP><HP1>Flow chart of RPC_Early_Return </HP1> </FIG> <H3>RPC_Loop_Server <i1>RPC_Loop_Server Called by: User server program. The set of clients to be serviced is identified by a string, which may in some cases include wildcards. This procedure is the simplest form of server, as it loops forever, servicing incoming requests. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip10 <PICTURE NAME=FLOW10$S> ...skip10 <FIGCAP><HP1>Flow chart of RPC_Loop_Server </HP1> </FIG> <H3>RPC_Start_Server <i1>RPC_Start_Server Called by: User server program. The set of clients to be serviced is identified by a string, which may in some cases include wildcards. This procedure queues a read request, and has the next message and successive messages serviced at AST level. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip22 <PICTURE NAME=FLOW22$S> ...skip22 <FIGCAP><HP1>Flow chart of RPC_Start_Server </HP1> </FIG> RPC_Start_Server specifies, as the AST procedure to run, RPC_Server_AST. This routine first handles the call, and then queues the next read on the channel. Originally, it used to queue the read in advance, which would speed up the system in some cases, but this causes problems if sophisticated stubs or user routines use the same communication channel while the call is being handled: messages would be received in the wrong order. (See the PIPELINE precompilation option). <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip30 <PICTURE NAME=FLOW30$S> ...skip30 <FIGCAP><HP1>Flow chart of RPC_Server_AST </HP1> </FIG> <H3>RPC_Create_Server <i1>RPC_Create_Server This routine sets up the communication channel for a server, but does not let it run. The socket returned may be used with the routines below to service requests as required. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip20 <PICTURE NAME=FLOW20$S> ...skip20 <FIGCAP><HP1>Flow chart of RPC_Create_Server </HP1> </FIG> <H3>RPC_Accept <i1>RPC_Accept This routine waits for a request on the given socket, and services it according to the currently attached stub modules. The timeout is in units of 10ms. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip11 <PICTURE NAME=FLOW11$S> ...skip11 <FIGCAP><HP1>Flow chart of RPC_Accept </HP1> </FIG> <H3>RPC_Queue_Server <i1>RPC_Queue_Server This routine queues an asynchonous user routine (AST) to be called when the next request arrives on the given socket (see RPC_create_server). It allocates space for the incoming message, and passes the address of this to the AST routine. The user AST routine has entry point astadr and is passed by address one parameter (info). This parameter is suitable to be passed to RPC_Service (see below). Its format is private to the RPC system. This parameter is all that is required by the service routine below: the user program may, however, want to perform some pre&hyphen. or post&hyphen. processing, such as forking off other tasks to handle the requests, or application functions. (For the format of the message buffer, see the rpc$types include file, as this contains several fields used internally by the RPC RTS, which may be changed between versions. It is not recommended that user code uses this buffer directly, but calls RPC_service to have it serviced.) This routine, with RPC_Service, was used to make a multitask server under the MODEL TCS scheduler. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip12 <PICTURE NAME=FLOW12$S> ...skip12 <FIGCAP><HP1>Flow chart of RPC_Queue_Server </HP1> </FIG> <H3>RPC_Service <i1>RPC_Service This is a procedure taking one record parameter as described in the preceding paragraph. It will handle one incoming call message, deallocate the message buffer passed to it, and return. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip13 <PICTURE NAME=FLOW13$S> ...skip13 <FIGCAP><HP1>Flow chart of RPC_Service </HP1> </FIG> <H3>RPC_Configure <i1>RPC_Configure This call allows one client to use more than one identical server in turn. The effect is for the connections to all open servers to be closed and reopened with the translation of their logical names being perfomed according to new values which may have been set up. A client process which wishes to use several identical servers may therefore repeatedly redefine the logicalnames which give the addresses of those servers, and call RPC_configure to cause future calls to be directed to the new servers. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip14 <PICTURE NAME=FLOW14$S> ...skip14 <FIGCAP><HP1>Flow chart of RPC_Configure </HP1> </FIG> <H3>RPC_Switch <i1>RPC_Switch This performs the reconfiguration function on a single package, referenced by a handle. The package must be already open (for the handle to be valid). The value of the handle is unchanged. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip15 <PICTURE NAME=FLOW15$S> ...skip15 <FIGCAP><HP1>Flow chart of RPC_Switch </HP1> </FIG> <H3>RPC_Establish <i1>RPC_Establish This procedure may be used in a client program to handle errors occuring in remote procedure calls. By default, if an unrecoverable error occurs in the communication with the server, the client program is stopped. ($EXIT occues under VAX/VMS, a pascal halt on other systems.) If the user wishes to handle the error himself, he passes the address of his error handler to rpc_establish. If this has been done, his handler will be called <hp1>instead</hp1> of the default handler. In order to revert to the default handler, RPC_establish is called with a value of zero. The user error handler is a routine taking one parameter, which is the message in question. If the user is programming in pascal, he will find the offending error status in the m_status field of the message. RPC_Establish simply places the given value into a global variable. <h3>RPC_Init This is called automatically where possible (e.g. VAX/VMS, TurboPascal, MoniCa). In other cases, it must be called explicitly by the user program. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip17 <PICTURE NAME=FLOW17$S> ...skip17 <FIGCAP><HP1>Flow chart for RPC_Init </HP1> </FIG> <h2>Internal Routines These routines are not normally called from outside the run time support. <h3>RPC_New <i1>RPC_New RPC_New is the routine which produces a new message buffer. It will allocate a message from the pool if the pool is not empty. Depending on whether the heap is in use, it may use the heap if the pool is empty. RPC_New may be called from the transport service as well as from stubs and internally. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip18 <PICTURE NAME=FLOW18$S> ...skip18 <FIGCAP><HP1>Flow chart for RPC_New </HP1> </FIG> <h3>RPC_Dispose <i1>RPC_Dispose, flow chart RPC_Dispose performs the reverse operation to RPC_New, except that it never returns a message to the heap, it keeps it for efficiency. ..go zz <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip19 <PICTURE NAME=FLOW19$S> ...skip19 <FIGCAP><HP1>Flow chart for RPC_Dispose </HP1> </FIG> ...zz <h2>TS layer TS_XXXX procedures <i1>Transport Service, calling sequence The calls used internally by the run&hyphen.time system are currently of the form given below. The status code is returned as a function value in each case. Outline flow charts are given below for some parts of a typical example of a TS adapter layer (TSPM68k) which supports RS232 and Ethernet communication. In some cases, the connection oriented communication handling is also shown. The reader is refered to the source for the system in question for the exact parameter passing conventions, data types, etc. <H3>TS_Open(socket, name) <i1>TS_Open <xmp> OUT status: integer (function value) OUT socket: integer IN name: string </xmp> All the information about the medium and about the protocol, service type and peer address where applicable are coded into the name string. A wildcard syntax in the string signifies connection to a class of peers. In this case the address for transmission is always to be taken from the source of the last received message. A socket number for reference is returned. <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip23A <PICTURE NAME=FLOW23AS> ...skip23A <FIGCAP><HP1>Flow chart for TS_Open (part 1) </HP1> </FIG> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip23B <PICTURE NAME=FLOW23BS> ...skip23B <FIGCAP><HP1>Flow chart for TS_Open (part 2) </HP1> </FIG> <H3>TS_Close(socket) <i1>TS_Close <xmp> OUT status: integer (function value) IN socket: integer </xmp> The socket number becomes unusable in future calls. Any messages attached associated with the socket are returned to the message pool, <hp1>including any passed by the user as a parameter to an oustanding TS_When_Receive request.</hp1> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip24 <PICTURE NAME=FLOW24$S> ...skip24 <FIGCAP><HP1>Flow chart for TS_Close </HP1> </FIG> <H3>TS_Send(socket, data, length) <i1>TS_Send <xmp> ..tb 9 17 25 33 41 49 57 65 73 OUT status: integer (function value) IN socket: integer IN data: pointer to message IN length: integer </xmp> The socket passed must be derived from TS_Open. The data array is in fact a data structure of which the first part is internally used by the RPC system, the second part is free for use by the transport service, and the third part is data belonging to the RPC system, which is actually to be transferred. <dl> <dt>On entry <dd> Within the first part, two of the four fields are set up before TS_send is called: <xmp> ..tb 9 17 25 33 41 49 57 65 73 M_Next is undefined M_Index must be the length of the message to be sent M_Socket must point to the socket to be used. M_Status is undefined </xmp> (Note that this information duplicates, for historical reasons, data in the parameters 'socket' and 'length'. In practice, these fields have turned out more useful than the parameters, so if the system were designed again, TS_SEND would only take one parameter.) <dt>On exit <dd>The pointer to the data may have been changed, so as to point to another message. This allows copying to be reduced, for speed. In the message pointed to be the (new) pointer, <xmp> ..tb 9 17 25 33 41 49 57 65 73 M_Next may be corrupted M_Index may be corrupted (currently) M_Socket points to the socket to be used. M_Status may be corrupted </xmp> </dl> <H3>TS_Receive(socket, data, max_length, xfer_count) <i1>TS_Receive <xmp> ..tb 9 17 25 33 41 49 57 65 73 OUT status: integer (function value) IN socket: integer IN OUT data: RPC_message_pointer IN max_length: integer IN timeout integer </xmp> The socket passed must be derived from TS_Open. The data array is in fact a data structure of which the first part is internally used by the RPC system, the second part is free for use by the transport service, and the third part is data belonging to the RPC system, which is actually to be transferred. A pointer to this data structure is passed by address. The ts_receive routine is free to switch it to a different buffer, to save time. The caller must not, therefore, keep any other record of the location of the original buffer. If the buffer pointer is switched, ts_receive will dispose of (or do something intelligent with) the old buffer. The timeout parameter specifies (nominally in 10us units) a maximum time allowed before reception. A negative value disables the timeout (allows an infinite time). (The unit of 10ms is not a very friendly one, but was chosen in order to give a sufficiently long maximum timeout on a 16 bit system). <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip25a <PICTURE NAME=FLOW25AS> ...skip25a <FIGCAP><HP1>Flow chart for TS_Receive (part 1) </HP1> </FIG> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip25b <PICTURE NAME=FLOW25BS> ...skip25b <FIGCAP><HP1>Flow chart for TS_Receive (part 2) The decision "Duplicate?" in th centre of this chart indicated a call to the function "Duplicate" or "Duplicate_V24". </HP1> </FIG> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip27 <PICTURE NAME=FLOW27$S> ...skip27 <FIGCAP><HP1>Flow chart for Duplicate and Duplicate_V24. Areas of difference between these two modules are indiated by background shading. </HP1> </FIG> <box> The routines below are not necessary in an implementation which is not to support asynchronous servers. </box> <H3>TS_When_Receive(socket, pdata, max_length, proc, param) <i1>TS_When_Receive <xmp> ..tb 9 17 25 33 41 49 57 65 73 OUT status: integer (function value) IN socket: integer IN pdata: pointer to message IN length: integer IN proc: service_procedure (see below) IN user_param: integer </xmp> The socket passed must be derived from TS_Open. The data array is in fact a data structure of which the first part is internally used by the RPC system, the second part is free for use by the transport service, and the third part is data belonging to the RPC system, which is actually to be transferred. This is a receive subroutine needed for asynchronous processing of events. It asks for the procedure "proc" to be called when the next message arrives. The address of a suitable buffer is passed, and its size, and the entry point of the procedure to be called when the read completes. <dl> <dt>On entry <dd> Within the first message, one of the first four fields is set up: <xmp> ..tb 9 17 25 33 41 49 57 65 73 M_Next is undefined M_Index is undefined M_Socket must point to the socket to be used. M_Status is undefined </xmp> <dt>On exit <dd>The pointer to the data may have been changed, so as to point to another message. This allows copying to be reduced, for speed. In the message pointed to be the (new) pointer, <xmp> ..tb 9 17 25 33 41 49 57 65 73 M_Next may be corrupted M_Index may be corrupted M_Socket points to the socket to be used. M_Status may be corrupted </xmp> </dl> The asynchronous service procedure referred to ('proc') takes parameter <xmp> ..tb 9 17 25 33 41 49 57 65 73 IN pdata: pointer to message. </xmp> Within that message (which is not necessarily the same message as was originally passed to TS_When_Receieve), when the 'proc' is called, <xmp> ..tb 9 17 25 33 41 49 57 65 73 M_Next is undefined M_Status is the completion status of the read M_Socket is the socket on which the message arrived M_Index is undefined </xmp> <FIG PLACE=FLOAT FRAME=NONE> ..if "&SYSTERMT" ne "APA6670" and "&SYSTERMT" ne "I3820" ..th ..go skip26 <PICTURE NAME=FLOW26$S> ...skip26 <FIGCAP><HP1>Flow chart for TS_When_Receive </HP1> </FIG> <H3>TS_My_Address(socket, name) <i1>TS_My_Address This returns the address at which the user may be contacted. <xmp> ..tb 9 17 25 33 41 49 57 65 73 OUT status: integer IN socket: integer OUT name: string </xmp> This is not a very well defined function, in that what can be returned, and it's exact significance, depends on the transport medium being used. Over a serial line, it is impossible to know by what name the line is known at the far end. Over ethernet, though, it is very useful to be able to find one's own ethernet address. In general, the address is returned in the format of an RPC address, as used for TS_Open. <H3>TS_Peer_Address(socket, name) <i1>Address of peer <i1>Peer address This returns the address of the sender of the last received message. <xmp> ..tb 9 17 25 33 41 49 57 65 73 OUT status: integer IN socket: integer OUT name: string </xmp> The same problems exist as for TS_My_Address, in that the information which is available and its significance tend to vary between systems. Again, the rule is that an address which would work with TS_Open should be returned. Over DECNET, for example, one can find the task name but not the TSAP name: in this case an address is returned which would work if the peer task had created a TSAP with the same name as his process. <h2>LANCE ethernet chip routines These routines are written in C. (Files tlance.c, lance.h) They were derived from routines written by L.R.T. Ltd, modified by Ben Segal (CERN/DD/SW), and further modified for inclusion in the RPC project. They do not use the data chaining features of the lance, as RPC buffers are always sufficiently large for a packet. They are aware, on the receive side, of the TS layer's socket format, as they search the socket list for a socket which is waiting for the received packet. These routines have survived a number of releases of the LANCE chip which had more or less obscure bugs. Therefore, they are fairly defensive, and will reinitialise the chip if things go wrong. There is a reasonable amount of trace code built into the module, controleed by an integer variable _debug taking values from 0 to 5 (see the user manual). Trace output from this module is prefixed by "* LANCE: ". The entry points are as follows. Note that as a peculiarity of the C compiler, they must be prefixed by an underscore character if referenced from outside the C world. <h3>bbinit This is a one-time general initialisation. It sets upo data structures in memory, initialises the chip itself, and sets up (via a small piece of assembler) the interrupt vectors to point (via a small piece of assembler) to the ..bf MONO lance_i ..pf interrupt service procedure. On exit from _bbinit, the lance chip is initialised by inactive. <h3>bbopen This routine activates the lance chip. For safety, it resets the buffer system. <h3>bbclose This routine deactivates the LANCE chip. <h3>lance_send This routine takes as one parameter the address of a packet to send. It returns a status when it has succeded, or faileded after several ettempts, to send the packet. The following entry points are used internally to the tlance.c module. <h3>lance_i This is the general interrupt service routine. It displays trace information if errors occur, if the level of _debug is high enough. It calls lance_read if a packets has been received. <h3>lance_read This checks that a packet really has arrived, and that it is good. If so, it scans the queue of sockets waiting for ethernet packets looking for an address and protocol type match. A mask on this check allows wildcard reception by a socket. <h3>getstation This routine is used internally to extract the ethernet address of the card in use from the hardware. <h3>initilise_lance This sets up the data structures and initialises the chip. <H1>Message Formats <i1>Message Format The messages are formatted according to the Courier <BIBREF REFID=COURIER> format. The "extended subset" of that format which is used is described below. The message is described as a stream of 8 bit bytes. When word&hyphen.oriented communication is used, one must ensure that the byte order is preserved. An alternative representation considered was the X.409 standard <i1>X.409 <BIBREF REFID="X409">. It requires more bytes, and so is not appropriate in bandwidth&hyphen.limited systems such as the Valet+, but is becoming accepted as a standard for data representation. It may be supported as an alternative. A further alternative may be the Apollo NCS format, purely for compatibility with NCS. <i1>Apollo NCS <H2>Data type representation <i1>Data representation <i1>Representation, data A list of the data types handled by the RPC system are given in appendix A of the User Manual. Below is listed the way each data type is packed into the message. Bytes (represented by boxes) are sent in the order listed left to right across the page. Note that each type is padded with zeroes so that every element wil start on a 16 bit boundary, according to the Courier spec. This may be suppressed, for efficiency, using the NOALIGN option in the INCLUDE file after running the RPCC and before running the Pascal compiler on the stubs. ..tb 9 17 25 33 41 49 57 65 73 ..bd RPC_BYTE <i1>RPC_Byte ..cc on ..tb set $ ..bx 8 15 / 16 23 $0$Value ..bx off ..bx 8 15 $Value$ (If NOALIGN option used) ..bx off ..cc off ..bd RPC_CHAR <i1>RPC_Char This is sent as an ASCII (or, rather, CCITT International Alphabet 5) character. Translation is performed by the stubs if necessary. ..cc on ..tb 9 17 25 33 41 49 57 65 73 ..bx 8 15 / 16 23 $0$Value ..bx off ..bx 8 15 $Value$$(If NOALIGN option used) ..bx off ..cc off ..bd RPC_SHORT <i1>RPC_Short This may be used to represent a signed (2s complement) or unsigned quantity. (The two's complement notation is so standard that it is not necessary to distinguish between them at this level. If any machine used a different format for negative numbers, we would have to). ..cc on ..bx 8 15 / 16 23 $MSB b1$LSB b0 ..bx off ..cc off Note that the MOST SIGNIFICANT BYTE IS SENT FIRST. If the number is given by X = b0 + (256 * b1) then the order of bytes in the buffer is b1, b0 as shown. ..bd RPC_LONG <i1>RPC_Long This may be used to represent a signed (2s complement) or unsigned quantity. (The two's complement notation is so standard that it is not necessary to distinguish between them at this level. If any machine used a different format for negative numbers). ..cc on ..bx 8 15 / 16 23 / 24 31 / 32 39 $MSB b3$b2$b1$LSB b0 ..bx off ..cc off Note that the MOST SIGNIFICANT BYTE IS SENT FIRST. If the number is given by X = b0 + (256 * b1) + (65536 + b2) + (256*65536* b3) then the order of bytes in the buffer is b3, b2, b1, b0 as shown. ..bd RPC_REAL32 <i1>RPC_Real32 This is sent in IEEE format. (There is no defined format for real numbers in either the Courier or the X.409 recommendation). This, and conversion to and from other 32 bit formats is discussed in <BIBREF REFID="REALS"> Conversion is performed, on the VAX, when the number is (un)packed by the stub. ..cc on ..bx 8 10 15 / 16 18 23 / 24 31 / 32 39 $S exp$e MSB$mantissa$LSB ..bx off ..cc off The first byte sent contains the sign bit (S) in the most significant bit, and then the 7 most significant bits of the exponent. The next byte contains the least significant bit of the exponent (E), and the 7 most significant bits of the mantissa. The next two bytes contain the rest of the mantissa. The sign bit, if set, means that the number is negative, though the mantissa remains uncomplemented. The exponent field is zero to indicate a zero real number. The mantissa is unsigned and normalised, with the leading "1" omitted. The exponent bias is 150. The absolute value of a number is X = 2**(exponent-150) * (mantissa + 2**23) ..BD RPC_STRING <i1>RPC_String This is represented by a two byte length field, followed by a series of one&hyphen.byte characters, followed by a zero byte if this is necessary to leave the total number of bytes even. The lengh field is in RPC_SHORT format. The characters are in the same coding as RPC_CHAR, with conversion being done by the stubs as necessary. ..cc on ..bx 8 15 / 16 23 / 24 31 / 32 39 / 48 55 / 56 63 $Len b1$Len b0$Char 1$Char 2$...$Char n$Zero ..bx off ..cc off In this example the characters are char 1.. char n, where n is odd, so a padding zero byte is required. ..bd Arrays and Records <i1>Arrays, representation <i1>Records, representation Arrays are of fixed, predefined, size, and are packed simply as a series of elements of thier component type. A multidimensional array is treated as an array of arrays, so the last index cycles most rapidly. A record is represented simply by the set of its constituent fields, in the order in which they were declared. ..bd Sequences <i1>Sequences, representation Sequences, in RPCL, are arrays of length unknown at compile time. They are transferred preceded by a count field, in RPC_INTEGER format. ..tb set <H2>X.409 Alternatives <i1>X.409 mapping In an alternative representation, data is represented in X.409 format <BIBREF REFID="X409"> In this case, the message is a series of bytes ("octets"), there being no word alignment. Every data item it represented by an identifying byte, followed by a length field of one or more bytes, followed by the data. The RPCL types map onto X.409 types as follows: RPC_BYTE, RPC_SHORT, RPC_LONG: Integer (Id 02 hex) RPC_CHAR, RPC_STRING: IA5 string (Id 16 hex) ARRAY, SEQUENCE: Sequence (ID 30 hex) Using tagging in this way requires more run&hyphen.time code and execution time, but allows makes a message more readable by a third party unaware of its formatting. <H2>Message content Below we list the contents of a message of each type. Every message begins with a "message type" field, of type RPC_SHORT, which can take the values 0 Call message 1 Reject message 2 Return message There follows a transaction identifier(a 16 bit number) which may be used for sequence checking, but at present is always set to zero. The formats of each type of message are as follows: <H3>Call message <xmp> message_type RPC_short 0 transaction_id RPC_short package_number RPC_long version_number RPC_short procedure_number RPC_short (within the package) </xmp> This header is followed by all the IN and INOUT parameters, in the order specified in the definition file. The transaction ID is only used when the simple protocol is run over a connectionless communications service. The package number and version number are described in the user manual. The procedure number is the index of the procedure within the package, generated by the RPC compiler. <H3>Reject message <xmp> message_type RPC_short 1 transaction_id RPC_short reason RPC_short Courier reason number status RPC_long CERN (VMS like) error code. </xmp> <H3>Return message <xmp> message_type RPC_short 2 transaction_id RPC_short </xmp> This header is followed by the function result (if any), then by all the OUT and INOUT parameters, in the order specified in the definition file. <h2>Coding scheme for RS232 point&hyphen.point messages <i1 see="RS232">V24 <i1>RS232 transmission encoding There are two levels of encoding: the frame formatting (header and checksum), and the byte encoding. In order to send binary byte stream data over 7&hyphen.bit ASCII terminal channels, an ad hoc coding scheme has been implemented, which is described below. The scheme is neither theoretically optimal nor beautiful nor standard, but is simple and relatively efficient. It is designed to get though the filters and conditions imposed on such channels by operating systems and drivers. This coding scheme is optional, and used only if 7 bit communication is requested. The byte encoding scheme allows 8&hyphen.bit bytes to be logically sent. In some cases, they can be sent directly, where both transmitting and receiving operating systems allow it. Where this is not so, a seven bit encoding scheme satisfies the following requirements: <ul> <li>Only seven bit characters are used <li>Control characters (00 to 1F hex) are not used <li>The DELETE character is not used <li>Small vales are transmitted as single bytes </ul> <H3>8&hyphen.bit encoding scheme for bytes In this case, the only formatting necessary is the escaping of the start of message character, and of the escape character. <H3>7&hyphen.bit Encoding scheme for bytes. This scheme was updated by RB in summer 1988. When transmitting a byte, <ol> <li>If the byte is less than 5C hex, add 20 hex to it and transmit it. <li>If the byte is between 5D and B9 hex inclusive, precede it with the character 7D, then subtract 3D from it and send the result. <li>If the byte is between BA and FF hex inclusive, precede it with the character 7E, then subtract 9A from the byte and send the result. </ol> <xmp> x in 0..5C --> x-20 x in 5D..B9 --> 7D, x-3D x in BA..FF --> 7E, x-9A </xmp> When receiving it, the reverse operation is <ol> <li>If the character is less than 7D, subtract 20 from it. <li>If the character is 7D, take the next one and add 3D. <li>If the character is 7E, take the next one and add 9A </ol> <H3>Framing format There are two frame formats. One is a general one, which can reprsent any RPC message. The other is a specially efficient one optimised for a strict subset of possible messages, which will suffice for simple systems. The frame starts with <xmp> Message Type, Transaction ID, </xmp> which, for a Call type message, should be followed by <xmp> Program Number, Version Number, Procedure Number, all the IN and INOUT parameters, </xmp> The following encoding scheme is used: <xmp> 7 6 5 4 3 2 1 0 _____________________ _____________ | M : T : y : p : e | | T : I : D | --------------------- ------------- 0 - Call, long format Packet Number 1 - Reject 2 - Return 4 - ? (Cast) 5 - ? 6 - ? 7 - ? 8 - Call Program 0, short format 9 - Call Program 1, short format n - Call Program n-8, short format 31 - Call Program 23, short format </xmp> This mechanism, allows us to send both the Message Type and Transaction ID in just one byte. In the case where <xmp> ..tb 9 17 25 33 41 49 57 65 73 The message is a CALL and Program Number < 24 and Version Number = 0 and Procedure Number < 256, (1) </xmp> the short form is used. In this case there are just two bytes to send: <xmp> ..tb 9 17 25 33 41 49 57 65 73 The byte above (1 byte) Procedure number (2 bytes) Total: 3 bytes </xmp> If the condition (1) is false we must use the long header format that requires us to send: <xmp> ..tb 9 17 25 33 41 49 57 65 73 The byte above (1 byte), Program Number (4 bytes), Version Number (2 bytes), Procedure Number (2 bytes). Total: 9 bytes </xmp> The equivalent information is stored in memory, and sent over ethernet, etc., as 12 bytes. <i1>Frame format, over RS232 <i1>Short frame format The frame is preceded by a frame start byte, and followed by a checksum. A one byte checksum is used for messages below (???) bytes in length, and for longer messages a 32 byte CRC is used. <H3>Frame reception Any control characters received should be ignored at any time. Any characters received before the start of frame character shall be ignored. <H1>Porting the RPC system into a new environment <i1>Porting the system If a new machine or software environment is to be introduced to a distributed system using RPC, there are three stages of porting of software wich may be necessary. These are the provision of communications support in the new environment, the porting of the RPC RTS itself, and, if required, the porting of the compiler. <H2>Provision of the Communications support <i1>Transport Service, porting <i1 see="Transport Service">Communications Assuming that communications support is available for a suitable medium in the new environment, all that must be done is to provide it in the (relatively simple) form needed by the RTS. In the simplest case (with no asynchronous calls) all that is required are the OPEN, CLOSE, SEND and RECEIVE procedures. One of the existing TSxxx modules may be taken as a template, and used <ol> <li>as an adapter to the existing communications primitives <li>as a switcher to select the communication medium required. </ol> <H2>Porting the Run&hyphen.Time System <i1>RPCRTS, porting This module is written in Pascal, with conditional compilation sections handled by the INCLUDE facility. Porting onto a new environment may require slight additions to the master source file if the Pascal compiler on the new machine has eccentricities not already catered for. A version of the RTS now exists which has been translated into C. This is generated by slightly preprocessing a version of the pascal source, running the 'ptc' converter progrem, and then slightly postprocessing the C source it produces. Therefore, whereas a TS module exists for each target environment, there is only one master RPCRTS file. <H2>Porting the Compiler <i1>Compiler, porting the The same applies to the compiler: it is written in Pascal, with conditional compilation sections handled by the INCLUDE facility. Porting onto a new environment may require slight additions to the master source file if the Pascal compiler on the new machine has eccentricities not already catered for. If cross&hyphen.compilation on an existing machine (for example VMS or Unix 4.2 Bsd) is satisfactory, then porting of the compiler itself will not be necessary. It may also be the case that modifications have to be introduced into the code generator if the stub code requires special facilities for the new environment. In general, no modifications to the compiler at all will be necessary if the port is to a new runtime software environment on a CPU which is already supported. <h1>Code Management <i1>Code management <i1>Source management <i1>CMS <i1>MMS DEC's Code Management System (CMS) is used for all source files, and the Module Management System (MMS) is used to control the generation of code and avoid inconsistencies. Those wishing to change any code in the RPC system should make themselves familiar with both of these. They allow, especially when used in a controlled way, a project to be partially self&hyphen.documenting. The following notes are provided to help the many people working together on projects such as the RPC, CHI etc to find their way around each other's files. The rules given should make this simpler. (They are not all adhered to 100% of the time <h2>Directory management <i1>Directory management Any development directory contains: <dl> <dt>Source files <dd>made by hand, cannot be generated automatically. <dt>Product files <dd>final products, or files used to to make products of other directories. <dt>Byproduct files <dd>made in the process of making other files but not needed by any system. (eg maps, listing files). Not used to make other files <dT>Intermediate files <dd>used in the process of making product files, can be generated automatically. <dt>Disconnected files <dd>Not involved at all in the software generation (e.g. documentation, subdirectories) </dl> The use of MMS allows these files to be distinguished by a newcomer to the project. Note that the same directory is used for the source and object files: the subdirectory structure is used to distinguish different areas of the project, not different types of file. <ol> <li>Every directory with any product files MUST contain a makefile called Makefile (or descrip.mms on VMS). <i1>Makefile <li>Any files in a directory which are final products or used in other directories must be explicitly declared in the Makefile (or descrip.mms) in that directory. <li>The default make (MMS) command must bring ALL product files in the directory up to date. The default target is the first dependency in the file. For example, it might be <xmp> default : xxx, yyy, zzz touch default </xmp> Therefore, one can find out how to make any file required by any other directory, by looking in the Makefile. One can check whether the products are up to date with an MMS/SKIP/NOACT command. <li>The MAKE CLEAN command (MMS CLEAN) should delete all intermediate files (except those which take a ridiculous amount of time to build), and any byproduct files with limited usefulness. Therefore, it is simple to save space in a directory which someone else has left full of intermediate files. <xmp> Example: clean : - delete *.lis.*, *.ccu.*, *.lmp.* </xmp> <li>The MAKE CLEAN_ALL command should delete all intermediate files, any product files of which the distibution copies are kept elsewhere, and all byproduct files. <li>All source files must be in one (and one only!) CMS library. If the CMS library is not the defualt one for that project (say, the first CMS.DIR found going up the directory tree) then the rule for fetching the source file from the correct CMS library must be put into the makefile. Therefore, a modification history for the whole project is <i1>Modification history effectively available. Also, an MMS/CMS command should be able to rebuild all product files from a directory empty except for the makefile. <li>All the software and build mechanisms for a released product must be under project users (eg RPC, FASTBUS, PILS, VALET). Any copies under personal users should be experimental only. </ol> <h2>C Programming Standards <h3>File organisation Experience suggests the following rules are necessary for portability and comprehensibility: <ol> <li>Filenames should have only one dot (for VMS, MSDOS), and a maximum of 8 characters before the dot (for MSDOS, 9 for VM) and 3 characters after the dot (MSDOS). (The limit on the Cray is a maximum of 14 characters including the dot and the extension.) <li> The public include file is the one used by users of the products of the project. It should be referred to using <angle brackets>. Other include files used for development within the project is kept within the project directory and referred to using "double quotes". Those wishing to use a specific development version of the public include file may use the -I (unix) or /INCLUDE= (VMS) options of the C compiler. <li>For any module which may be called from other modules, a .h file should be used to define the entry points and any other definitions needed by the caller. This include file have as name the facility name followed by .h. Any include files used internally, for example for sharing data or structures between modules, should have a different name to the .c files. </ol> <h3>Style <i1>Style, in C Many of the rules below are essential for portability and readability, or compatibility with some utilities, and are not simple a matter of taste. However, there are a few questions of personal preference. When editing anyone's file, please do not change the layout just to suit yourself, unless changing it to the common layout below. The following conventions were adopted in principle for the RPC code, and should be used for new files. At the start of a file, the file header should contain: <ul> <li>An english title for the file (not its name) underlined with ===. <li>On the right, ending in column 80, the file name. <li>After the title, an abstract of the file. <li>A list authors (names and affiliations) so that later modifications and personal comments can be marked simply with initials. <li>A list <hp2>restrictions</hp2> on the use of the module, special consitions for its compilations, limitations as to its portability, etc. <li>A history list, with dates, of any modifications made, initialled by the person responsible. <li>A double row of asterisks down the left hand side to distinguish the block comment from code. </ul> It may be useful to divide the file into sections using form feeds and section titles centered and double underlined, for example: <xmp> Local Coding Macros =================== </xmp> For each function, a function heading should contain: <ul> <li>Before any procedure, a form feed character OR (if the function will fit on the rest of a 66 line page, or immediately after a section heading), two blank lines. <li>An english title for the function (not its name) underlined with dashes and indented. <li>On the right, ending in column 80, the function name. This makes it easier to find when flicking through a listing. <li>After the title, an abstract of the procedure. <li>A list of preconditions which MUST be true on entry into the function. <li>A list of conditions which are true on exit. This includes any change to global data or system state, anything corrupted. <li>Optionally, a list of parameters, if these are not clear from comments against the parameter declarations and in the pre- and post-condistions. <li>A history list, with dates, of any modifications made specifically to this function and not recorded under a general heading for the module. <li>A declaration of the function BOTH in ANSII C AND in Kernighan and Ritchie C. Use the __STDC__ preprocessor token to select one or the other. <li>A double row of asterisks down the left hand side to distinguish any block comment from code. </ul> Below is a small example. <box> <listing> <Form Feed> /* Allocate an RPC message rpc_new() ** ----------------------- ** ** This procedure must be fast, as it is used in many places in the rpcrts.c ** and ts.c modules. For this reason, it keeps a list of allocated messages, ** and does not return them to the heap when they have been used. ** ** On entry, ** p The address of the pointer to be filled in ** bytes Irrelevant. In principle this was to be allowed for a ** message size required, if we had variable sized messages. ** On exit, ** *p is either the address of a message, or zero if there is ** not enough memory. */ /*ARGSUSED*/ #ifdef __STDC__ void rpc_new( rpc_message_pointer *p, /* -> Pointer to be returned */ rpc_integer bytes) /* Size: ignored */ #else void rpc_new(p, bytes) rpc_message_pointer *p; rpc_integer bytes; #endif { if (free_messages == NULL) { *p = (rpc_message_pointer)malloc((unsigned)(sizeof(*(*p)))); } else { *p = free_messages; free_messages = (*p)->m_next; } /* if */ } /* rpc_new */ </listing> </box> Within a function, <ul> <li>Indent by four columns at each open brace "{". <li>Please put the openning brace at the end of the same line as the if, while, etc. <li>Line up the closing brace with the first character of the opening "if", "while", etc.: <hp2>not</hp2> with the enclosed code. <li>Comment the closing brace with the function name (for a function block), or with "if" or "while" etc. For large blocks, give some clue as to which "if", for example "/* if bad status */". When closing an "else" block, give the correct sense of the comment to the closing brace, e.g. "/* else good status */". <li>Leave spaces on either side of = and ==, but not inside round brackets. <li>Do not perform an assignment in the outer level of a conditional expression. For example, <xmp> if (pointer = mallloc(sizeof(*pointer)) { printf("malloc was successful\n"); } </xmp> is confusing to most people and gives a warning with some compilers. </ul> <h3>Use of the preprocessor. <i1>Preprocessor, use of <ul> <li>All macros should be entirely in upper case unless they completely emulate functions, in which case they may be entirely in lower case. <li>When an include file refers to a data type which is a pointer to a structure defined in another module, then a preprocessor token may be used to allow either the relevant include file to be included, or for a dummy declaration (int *) to be made. This allows one to hide the structures when knowledge of them is not required. It allows one to restrict the effort involved in a remake, and to restrict the number of include files distributed. <li>Tokens beginning and ending with two underscores are reserved for interaction with the compiler. Do not use them for anything else. (Apart from causing confusion to people, this upsets the OS9 compiler) </ul> <h3>System-specific code Any <hp2>system-specific code</hp2> may, if large, be consigned to a completely separate module, or may be controlled using macros. In the latter case, no reference to specific machines or operating systems should be put in the text, only in a file called syspec.h. This file defines, for a given system, some alternative macros and flags. For example, code depending on the byte order might use a LSBFIRST flag which in turn is set by syspec.h. The syspec.h file is in fact a set of conditional inclusions of different system_specific files, one for each system. It uses the compiler predefined tokes to select the relevant file. In this way, extra machines can sometimes be added without touching the distributed files, except adding a line to syspec.h. Macros which may be altered by syspec.h should be defined conditionally on their not having been defined before. <xmp> #ifndef PANIC #define PANIC fprint("Panic!\n"); exit(-999); #endif </xmp> <h3>Utilities Compiling a module with an ANSI C compiler is a useful way of checking for nonportable aspects and for bugs, especially if you have included ANSI C headings. The TurboC compiler on MSDOS is fairly fussy, for example. `lint' is a unix tool for checking C files for consistency. It is not perfect, and code does not have to be completely "lint free" but it is is useful, and code should run though line with as few warnings as possible. Note the use of a lint specila comment /*ARGSUSED*/ to prevent lint complaining that a deliberately unused argument had not been referenced. There are a few other similar special comments (See `man lint' on unix.) There are various "prettifiers" which aim to format C files. `indent' from Gnu and unix `cb' are two. These do not do a good enough job: they tend to make a mess of comments, for example. Therefore, do not assume you can clean up your code automatically afterwards! <h2>Software distribution <i1>Distribution, software <i1>Versions, software This scheme roughly follows the DD/OC MODEL scheme, leaving a little more flexibility. A version directory is one whose name is vx_y where the version is version x.y. (eg [RPC.UTIL.V2_1]). <ol> <li>Under a version directory the subdirectory distribution_files should contain all the files to be distributed with the release. <li>Under the version directory the subdirectory MASTERS (eg [RPC.V2_1.MASTERS]) should contain the master sources of any command files, release notes etc applicable only to that release. <li>In the version directory, the BUILD.COM command file must <ol> <li>Copy all files from the development directories and the masters subdirectory into the distribution_files subdirectory <li>Create a VMS instalation kit (.A, if appropriate), or otherwise a backup (.BCK) file containing all the distributed files. </ol> <li>The file SETUP.COM (for VMS software) should be released with the <i1>SETUP.COM files and perform any necessary logon time set&hyphen.up for a user to be able to use the software. The source for this will normally be in the masters subdirectory. </ol> <xmp> ..tb 9 17 25 33 41 49 57 65 73 [...V2_1] | build.com | ___________|___________________ | | MASTERS RELEASE_FILES setup.com all files included in release xxvvv_release_notes </xmp> Note: One needs priviledge to build a VMS/INSTAL set, <i1>VMS/INSTAL so in practice this cannot be done on VXCERN: Nor can an INSTAL procedure be done by a nonpriviledged user who wants to pick up a version of the software on another machine. For systems which do not require priviledge to set up, this simpler SETUP.COM system is used instead. <appendix> <h1>Differences from "hermes" <i1>hermes This section notes some of the differences between the DD/OC RPC system and the "hermes" system from Eric Cooper at Berkley, adapted by Kris Kostro at CERN<BIBREF REFID=HERMES>. The two systems are similar in that they are both remote procedure call systems using an RPC compiler for stub generation. <H2>Parameter passing Whereas the hermes project <BIBREF refid=hermes> is designed to interface to C programs, in DD/OC we (and our users) require remote C, Pascal and FORTRAN and PILS programs. The calling conventions therefore differ. The "hermes" system would only allow data to be returned as a function value, wheras we must allow all combinations of IN, INOUT and OUT parameters. <H2>Linking There is no provision in the hermes system for the linking of distributed system to take place transparently. This is only provided, in the DD/OC system, under VMS. <H2>Description file The hermes system uses a special "MESA" language, rather than Ada. <H2>Message Format The message format in hermes has been adapted from the Courier standard in various ways. <H2>Compiler generation The hermes compiler ("rig") is built using the UNIX tools "lex" and "yacc", and produces output in C. This RPC compiler is written in Pascal, and produces Pascal, FORTRAN, PILS or C output. <H2>Protocol The hermes system's built&hyphen.in protocol uses a concept of a transient connection, which lapses on timeout. The same connection is used by the transaction protocol and by the rpc system. In the OC system, a transport connection is established at program initialisation time, and kept whilst the program is running. <h1>Comparison with Sun XDR RPC <i1>Sun XDR <i1>XDR, Sun <ul> <li> Sun RPC supports a variety of call&hyphen.time authentication schemes. These are relative heavy schemes, based on unix principles. <li> The XDR language used to describe operations and data types is in the C idiom. <li> There is a run time check on the version of RPC protocol, as well as on the versions of the stubs. This is a good idea. <li> The marshalling routines will handle pointers, but not NIL ones. I guess this means they will recreate linked data structures. (but why not NIL pointers?) <li> Following the C idiom, all parameters are considered IN parameters, and the function result may be a complex structure. <li> The marshalling format is different: in particular, everything is a multiple of 32 bits. (With us it is 8 or 16, depending on the ALIGN option). <li> The (un)marshalling routines put (take) data directly to (from) a serial data stream, rather than via a buffer. This is an interesting difference of philosophy: it relies on the Unix concept of bytes stream oriented communication. I feel that in our environments, this would be too slow, as there would be too many calls to the communications services. However, this approach has the advantage of having no limitations imposed by the buffering available within the RPC system. </ul> The conclusion is that Sun RPC was not optimised for very fast calls at run time, as there is quite a large call time overhead. Similarities between the two systems include <ul> <li>The basic RPC concept. <li>The basic byte ordering is the same, and constant. </ul> <h1>Glossary In response to a specific request, here is a list of some of the terms used in this document, with short definitions. <dl> <dt>Application <i1>Application, definition <dd>Code which is directly, rather than indirectly, useful. <dt>AST <i1>AST, definition <dd>Originally, VMS term "Asynchronous System Trap". A user procedure which is executed in the context of a user process, but for which the original activity of that process was interrupted. A call to an AST is forced by the system, just as though the user code had executed a procedure call. See an application note by TBL, or VAX/VMS documentation. <dt>Callback <i1>Callback, definition <dd>Within one remote procedure, the calling of another remote procedure which is executed by the process which called the first remote procedure. See the user manual. <dt>Client <i1>Client, definition <dd>That which uses a service provided by another. In RPC, that which calls a procedure executed by another. See user manual. A process is a client in relation to a particular remote procedure: the same process may be server of some procedures and client of others. <dt>Client Stub <dd>A dummy module emulating a remote procedure on a local machine. See user manual. <dt>Connection <i1>Connection, definition <dd>An agreement between communicating transport services which allows orderly and reliable communication. For the transport service, this involves keeping a certain amount of information about messages sent or received. <hp1>Connection identifier:</hp1> A number by which the user of the Transport Service refers to a particular connection. <hp1>Connection oriented:</hp1> Of a transport service, providing orderly reliable communication by the use of connections. <hp1>Connectionless:</hp1> Not connection oriented. A connectionless communications service normally allows the unrelaible exchange of messages <dt>Datagram <i1>Datagram, definition <dd>A message sent over a connectionless communication service. <dt>Disjoint <dd>Not overlapping. <hp1>Disjoint address spaces:</hp1> address space not accessible from the same process, and therefore not allowing communication using shared memory. <dt>Handle <i1>Handle, definition <dd>A reference number by which a Client Stub refers to its Peer Server Stub. <dt>INCLUDE <i1>INCLUDE preprocessor <dd>A source code preprocesor developed as part of the CERN cross&hyphen.software suite. This allows conditional selection of code sections, and the embedding of code from other files. It is used in the generation of the RPC software for specific target environments. <dt>Link <dd>A term sometimes used for "Connection". <dt>Marshalling <i1>Marshalling, definition. <dd>The action of taking data from the application data area, converting it into a suitable format, and putting it into a message ready for transmission. <dt>Medium <i1>Medium, definition <dd>That through which things travel. In this document, the method of communicaication including all the services provided by the operating system. <dt>Message <i1>Message, definition <dd>1. A logical unit of communication, containing an ordered set of bytes, and having a defined length. 2. A data structure used within the RPC system to store such a message and also some associated information. <dt>Package <i1>Package, definition <dd>An ordered set of procedures and functions. A package is defined by one definition file. A package is the unit of RPC compilation, and a unit of addressing at run time. See the user manual. <dt>Packet <i1>Packet, definition <dd>A unit of communication over a network. A term generally used at the link layer, e.g. for an ethernet packet. Packets tend to have limited sizes. <dt>Packing <dd>A term sometimes used for "Marshalling". "Marshalling" is preferred because "packing" suggests expressing data in a more compact format, as in the pascal "packed array". <dt>Peer <i1>Peer, definition <dd>Another on an equal standing. Partner on the same level. <hp1>To be tried by one's peers:</hp1> to be tried by those of equal social status. <hp1>Peer address:</hp1> The address of the equivalent entity with which one is communicating. <dt>Process <dd>A serial thread of execution by a computer. <dt>Remote <dd>At a distance. In RPC, strictly: in a different process with a disjoint address space. Alternatively, on a different processor. <dt>Run Time Support <dd>A body of software providing to the Stubs those services which they require. <dt>Segmentation <i1>Segmentation, definition <dd>The process or dividing large messages into small packets, and of reassembling them into the original messages. <dt>Server <i1>Server, definition <dd>That which provides a service to another. In RPC, that process which executes a procedure called by another process. <dt>Service <dd>Useful function. <dt>Server Stub <dd>A dummy module emulating a remote calling program on a local machine. See user manual. <dt>Socket <i1>Socket, definition <dd>A channel of communication provided by the RPC transport service. A socket number is generated by the transport layer when a socket is opened by calling TS_Open. The number is quoted for all communications functions referring to that channel. Internally to the TS, the socket number is typically a pointer to a data structure describing the channel. (The term 'channel' is used here only in explanation: it is not used in the RPC system to refer to a socket). (Berkley unix has a 'socket library' with a similar concept.) <dt>Stub <i1>Stub, definition <dd>A dummy module emulating a remote module on a local machine. See user manual. <dt>Task <dd>Common term for Process, e.g. in RMS68K. The term "process" is preferred in this document, but terms monotask and multitask are current. <dt>TS <dd>1. Transport Service. 2. The module of this RPC system which provides a transport service to the RPCRTS module. (See the diagram at the start of Chapter 3) <dt>TSAP <i1>TSAP, definition. <dd>Transport Service Access Point. An ISO term, denoting a thing to which a connection may be made. The name of the TSAP to which you want to connect is what you give to an ISO transport service when asking for a new connection. The DECNET equivalent term is "Decnet object". <dt>Transport Service <i1>Transport Service, definition <dd>The function of reliable communication of information between different processes in disjoint address spaces. <dt>Unit <dd>Old name for a package. See "package". </dl> <backm> <bl compact=1> <BIB ID=COURIER> "Courier: The Remote Procedure Call Protocol", XEROX Corporation, XSIS 038112, December 1981. <BIB ID=X409> CCITT "Red Book" Volume VIII, Recomendation X.409: "Message Handling Systems: Presentation Transfer Syntax and Notation" <BIB ID=HERMES> K. Kostro, CERN/DD/SW, "Portable Communications with Remote Procedure Call Protocols", 14 November 1985 <BIB ID=RPCUSER> T. Berners&hyphen.Lee, A. Pastore DD/OC, "RPC User Manual", August 86 <bib id=REALS> H. von der Schmidt, "32&hyphen.bit Floating Point Formats possibly used in OPAL and conversion between them" HvdS/OPAL 13 May 86 <BIB ID=ASN1> ISO 8824, "Specification of Abstract Syntax Notation One (ASN/1)", International Standards Organistion. <BIB ID=ecma127> European Computer Manufacturers Association, standard ECMA&hyphen.127 "RPC: Basic Remote Procedure Call using ISO Remote Operations" UDC 681.327.8.04/.06 ISO/IEC DIS 10148 <BIB ID=iso8649> ISO 8649, <hp1>Information Processing Systems - Open Systems Interconnection - Association Control Service Element,</hp1> International Standards Organistion. <BIB ID=iso90721> ISO 9072/1, <hp1>Remote Operations (1): Model, Notation & Service Definition</hp1> International Standards Organistion. <BIB ID=iso90722> ISO 9072/2, <hp1>Remote Operations (2): Protocol Specification,</hp1> International Standards Organistion. <BIB ID=NCSRM> Apollo Computer, <hp1>Network Computing System Reference Manual</hp1>, Order number 010200 <BIB ID=NCAPS> L. Zahn, Apollo Computer, <hp1>Network Computing Architecture Protocol Specifications</hp1>, Order number 010201 To find documents, people, programs, etc., related to the DD/OC RPC project under VMS, type <fn> Avaliable on VXCERN. To use on, put the line ..br $ enq*uire:==@disk$d1[timbl.enq]enquire into your login.com file. </fn> "ENQUIRE RPC". </bl> <index ix=1>Index </backm>