--- /dev/null
+ The citadel.pam configuration file has been updated for Red Hat 7.1.
+If you have such a system, it should Just Work; if you don't, you're going to
+have to tweak it, preferably BEFORE you do a make install. See below. Even
+if you have Red Hat 7.1, you should look at the file anyway and understand how
+it affects your system security. The original PAM.txt is included below:
+
+
+ Citadel 5.53 and later include support for Pluggable Authentication
+Modules (PAM.) However, we don't recommend enabling this feature (./configure
+--with-pam) unless you understand exactly how it will affect your system's
+security. Specifically, the system administrator must supply a configuration
+for every authentication service which uses PAM. We have automated this process
+for Linux by supplying a file which is placed in /etc/pam.d during the
+installation process, but not on other systems, for 2 reasons:
+
+ 1) Other systems don't have /etc/pam.d; instead they use one big
+configuration file, usually /etc/pam.conf. It's not quite as trivial to
+automatically modify this file in a safe and secure fashion.
+
+ 2) Other systems have a different set of available authentication
+modules; they are likely to lack all three of the ones we use in the Linux
+configuration. We don't have enough information about the features of the
+authentication modules on other platforms to be able to provide secure
+configurations.
+
+ That said, the configuration we've provided should work on at least
+Red Hat Linux 4.2-5.2, (although we don't recommend building Citadel on Red
+Hat 4.x due to libc thread-safety issues) and if you understand PAM
+configuration on your system, feel free to build Citadel with PAM support,
+as long as you realize that YOU'RE ON YOUR OWN.
--- /dev/null
+
+BUILDING THE CITADEL SYSTEM WITH PRIVATE LIBRARIES
+--------------------------------------------------
+
+This method is known to work on Linux and FreeBSD systems. It is a way of
+building the Citadel system with its own private copies of Berkeley DB and
+libical. This avoids conflicts with any other version of these libraries
+which may already exist on your host system.
+
+You can perform your builds in any directory (such as /usr/src or even your
+home directory). The target directories will be:
+
+* /usr/local/citadel (Citadel server, text client, utilities)
+* /usr/local/webcit (the WebCit front end)
+* /usr/local/ctdlsupport (libical, libdb, and their headers, etc.)
+
+The behavior of Easy Install is based upon this methodology.
+
+1. Unpack the Berkeley DB tarball. chdir to the "build_unix" directory
+ and build the package with these commands:
+
+ ../dist/configure --prefix=/usr/local/ctdlsupport
+ make
+ make install
+
+2. Unpack the libical tarball and build it with these commands:
+
+ ./configure --prefix=/usr/local/ctdlsupport
+ make
+ make install
+
+3. Set these environment variables for the rest of the build.
+ (This assumes you're using the 'bash' shell. Otherwise you're on your own.)
+
+export CFLAGS='-I/usr/local/ctdlsupport/include'
+export CPPFLAGS='-I/usr/local/ctdlsupport/include'
+export LDFLAGS='-L/usr/local/ctdlsupport/lib -Wl,--rpath -Wl,/usr/local/ctdlsupport/lib'
+
+ -L tells the build system where to find libraries during the build process,
+while -Wl,--rpath inserts that same library path into the Citadel binaries
+so they know where to find the libs at runtime. Since we can't depend on
+the correct version of Berkeley DB already being on the system, this is how
+we carry our own along. It's better than static linking everything.
+
+4. If LDAP support is required, unpack the OpenLDAP tarball and build with:
+
+./configure --prefix=/usr/local/ctdlsupport --with-db=/usr/local/ctdlsupport
+make
+make depend
+make install
+
+5. Now you're ready to build the Citadel server. Unpack the Citadel tarball
+ and build it with these commands:
+
+./configure --prefix=/usr/local/citadel --with-db=/usr/local/ctdlsupport
+make
+make install
+
+ ** NOTE: if you already have a Citadel server in /usr/local/citadel, change
+ the 'make install' to 'make upgrade'. And I'm sure you remembered to shut
+ down your Citadel service and make a backup before starting, right?
+
+6. Finally, unpack the WebCit tarball and build it with these commands:
+
+./configure --prefix=/usr/local/webcit
+make
+make install
+
+
+
+All of your software is now in place. Run /usr/local/citadel/setup to configure
+the Citadel server, and /usr/local/webcit/setup to configure WebCit. If you are
+on a Linux machine, setup will tweak /etc/inittab to automatically start the
+services. If you are on a FreeBSD machine, you will need to manually configure
+your startup scripts to start the services.
--- /dev/null
+Some notes on the build process...
+
+ Oct 28 01:57 1998 from LoanShark @uncnsrd
+
+ autodependency generation is implemented by generating a bunch of .d
+files (with another suffix rule) using $(CC) - M and "-include"-ing them
+from the main Makefile. the .d files are made to depend on the .c files
+they correspond to, and the referenced header files, so they're updated as
+needed. the only version of make that I know for sure works with this is
+GNU make, but the Makefile should still work on other versions of Unix
+make, just without the autodependency stuff.
+
+
+ Oct 28 20:49 1998 from LoanShark @uncnsrd
+one thing I forgot to mention about the autodependency generation: for a
+file to be included in the autodepend process, it must be listed in the
+SOURCES macro near the top of Makefile.in.
+ also, I've added an 'install' target to the makefile. this will make it
+possible to build RPM's, and bring the installation process closer to that
+of other packages, which if you're using the new configure script goes
+like this:
+ ./configure [--enable-ansi-color] [--disable-autologin] [--prefix=/foo]
+
+ --prefix specifies the location Citadel will run from,
+/usr/local/citadel by default.
+ to build a highly optimized version without debugging symbols, you could
+ do something like this (with the bourne shell):
+ CC=egcs CFLAGS='-O6 -fomit-frame-pointer' ./configure
+ after configuring, simply type `make', then su to a user who has
+appropriate permissions, and `make install'. then run setup as usual.
+ are there any other areas that need to be cleared up?
+
--- /dev/null
+====== How Threads & IO for clients work ======
+Citserver is a multi thread process. It has four types of threads:
+ - Workers - these are anonymous threads which can do any of the server related functions like Imapserver; smtpserver; citadel protocol and so on.
+ - Housekeeping - once per minute one Worker becomes the Housekeeping thread which does recurrent jobs like database maintenance and so on.
+ - IO - one thread is spawned to become the IO-Thread at startup. Inside it lives the event-queue which is implemented by libev.
+ - DB-IO - one thread is spawned to become the database IO thread. Inside it lives another event-queue, which is also implemented by libev.
+
+
+===== Worker Threads =====
+The workers live blocking on selects to server sockets. Once IO on a server socket becomes available, one thread awakes, and takes the task.
+It is then assigned a CitContext structure on a Thread global var (accessible via the CC macro), which then becomes his personality and controls what the thread is going to do with the IO,
+and which server code (Imap/SMTP/Citadel/...) is going to evaluate the IO and reply to it.
+
+===== Housekeeping Thread =====
+Once every minute one Worker becomes something special: the Housekeeping Thread. Only one Houskekeeper can exist at once. As long as one Housekeeper is running no other is started.
+Jobs done by the Housekeeper are registered via the <>-API. The current list:
+
+ - DB maintenance; flush redo logs etc.
+ - fulltext index (this one is a little problematic here, since it may be long running and block other housekeeping tasks for a while)
+ - Networking-Queue
+ - Citadel Networking Queue aggregator (Blocks NTT)
+ - Citadel Inbound Queue operator (Blocks NTT)
+ - Citadel Networking Queue
+ - SMTP-Client-Queue
+ - RSS-Aggregator scheduling
+ - HTTP-Message Notification Queue (Extnotify)
+ - POP3Client Queue
+
+The queues operate over their respective queue; which may be implemented with a private room (smtp-Queue, Extnotify Client), per room configs (RSS-Client; POP3Client), or an aggregate of Rooms/Queue/Roomconfigs (Citadel Networker).
+
+
+==== Networking Queue ====
+The networking queue opperats over an aggregate of per room network configs and messages in rooms that are newer than a remembered whatermark (netconfigs.txt#lastsent).
+All messages Newer than the watermark are then processed:
+ - for all mailinglist recipients an entry is added to the mailqueue
+ - for all mailinglist digest recipients a digest file is written, or once per day scheduled to
+ - for all Citadel networking nodes a spool file is written for later evaluation.
+
+==== Citadel Networking Queue aggregator ====
+The aggregator iterates over the spool directory, and combines all per message spool files to one big file which is to be sent by the Citadel Networking Queue later.
+It blocks the NTT-List for the remote party while doing so; if the NTT-List is blocked the file is skipped for next operation. Once all files are successfully processed, all files not associated with a networking config are flushed.
+
+==== Citadel Inbound Queue operator ====
+The inbound queue operator scans for new inbound spool files, processes them, splits them into messages which are then posted into their respective rooms.
+
+
+==== other Queues ====
+Once these Queue-operators have identified a job to be performed like
+ - talking to another Citadel Node to transfer spool files back and forth
+ - sending mails via SMTP
+ - polling remote POP3-Servers for new mails
+ - fetching RSS-Feeds via HTTP
+ - Notifying external facilities for new messages in inboxes via HTTP
+
+They create a Context for that Job (AsyncIO struct) plus their own Private data, plus a CitContext which is run under the privileges of a SYS_* user.
+HTTP Jobs just schedule the HTTP-Job, and are then called back on their previously registered hook once the HTTP-Request is done.
+
+All other have to implement their own IO controled business logic.
+
+
+===== IO-Thread =====
+The IO-Event-queue lives in this thread. Code running in this thread has to obey several rules.
+ - it mustn't do blocking operations (like disk IO or nameserver lookups other than the c-ares facilities)
+ - it mustn't use blocking Sockets
+ - other threads mustn't access memory belonging to an IO job
+ - New IO contexts have to be registered via QueueEventContext() from other threads; Its callback are then called from whithin the event queue to start its logic.
+ - New HTTP-IO contexts have to be registered via QueueCurlContext()
+ - once you've registered a context, you must not access its memory anymore to avoid race conditions.
+
+==== Logic in event_client ====
+InitIOStruct() which prepares an AsyncIO struct for later use before using QueueEventContext() to activate it.
+Here we offer the basics for I/O; Hostname lookus, connecting remote parties, basic read and write operation control flows.
+This logic communicates with the next layer via callback hooks. Before running hooks or logging CC is set from AsyncIO->CitContext
+Most clients are stacked that way:
+
+<event base> --- <IO grammer logic> -- <state machine dispatcher> -- <state hooks>
+
+=== event base ===
+While the event base is responsible for reading and writing, it just does so without the understanding of the protocol it speaks. It at most knows about chunksises to send/read, or when a line is sent / received.
+=== IO grammer logic ===
+The IO grammer logic is implemented for each client. It understands the grammer of the protocol spoken; i.e. the SMTP implementation knows that it has to read status lines as long as there are 3 digits followed by a dash (i.e. in the ehlo reply). Once the grammer logic has decided a communication state is completed, it calls the next layer.
+
+=== state machine dispatcher ===
+The state machine dispatcher knows which part of the application logic is currently operated. Once its called, it dispatches to the right read/write hook. Read/write hooks may in term call the dispatcher again, if they want to divert to another part of the logic.
+
+=== state hooks ===
+The state hooks or handlers actualy PARSE the IO-Buffer and evaluate what next action should be taken;
+There could be an error from the remote side so the action has to be aborted, or usually simply the next state can be reached.
+
+
+=== surrounding businesslogic ===
+Depending on the protocol there has to be surrounding business logic, which handles i.e. nameserver lookups, which ips to connect, what to do on a connection timeout, or doing local database IO.
+
+===== DB-IO-Thread =====
+Since lookups / writes to the database may take time, this mustn't be done inside of the IO-Eventloop. Therefore this other thread & event queue is around.
+Usually a context should respect that there may be others around also wanting to do DB IO, so it shouldn't do endless numbers of requests.
+While its ok to do _some_ requests in a row (like deleting some messages, updating messages, etc.),
+loops over unknown numbers of queries should be unrolled and be done one query at a time (i.e. looking up messages in the 'already known table')
+
+=== Transitioning back & forth ===
+If the application logic has found that a transition is to be made from the event IO to the DB IO thread. Do so by calling DBQueueEventContext() (DB -> IO) EventQueueDBOperation() (IO->DB) and pass its return value up the row to enable the controll logic to do the actual transition for you.
--- /dev/null
+ Citadel Server Extension API Documentation
+ ---------------------------------------------
+
+ This is a VERY INCOMPLETE documentation of the API for extending the
+Citadel server using dynamically loaded modules. It really isn't an API at
+all, but rather a list of some of the functions available in the server which
+are likely to be of use to module writers.
+
+ The current trend is to move as much stuff as possible out of the server
+proper and into loadable modules. This makes the code much easier to read and
+understand.
+
+ Expect this document to become more complete over time, as both the API and
+the person documenting it have a chance to mature a bit. :)
+
+
+
+ USER RELATED FUNCTIONS
+ ----------------------
+
+ The fundamental user data is stored in "struct ctdluser" which is defined
+in citadel.h. The following functions are available:
+
+
+ int getuser(struct ctdluser *usbuf, char name[])
+
+ Given the name of a requested user and a buffer to store the user
+record in, getuser() will search the userlog for the named user and load its
+data into the buffer. getuser() returns 0 upon success or a nonzero error
+code if the requested operation could not be performed.
+
+
+ void putuser(struct ctdluser *usbuf, char *name)
+
+ After reading in a user record with getuser() and perhaps modifying the data
+in some way, a program may use putuser() to write it back to disk.
+
+
+ int lgetuser(struct ctdluser *usbuf, char *name)
+ void lputuser(struct ctdluser *usbuf, char *name)
+
+ If critical-section operation is required, this pair of calls may be used.
+They function the same as getuser() and putuser(), except that lgetuser()
+locks the user file immediately after retrieving the record and lputuser()
+unlocks it. This will guarantee that no other threads manipulate the same
+user record at the same time.
+
+ NOTE: do NOT attempt to combine the locking lgetuser/lputuser calls with any
+other locking calls in this API. Attempting to obtain concurrent locks on
+multiple files may result in a deadlock condition which would freeze the
+entire server.
+
+
+ void ForEachUser(void (*CallBack)(struct ctdluser *EachUser))
+
+ This allows a user-supplied function to be called once for each user on
+the system. The user-supplied function will be called with a pointer to a
+user structure as its only argument.
+
+
+ int getuserbynumber(struct ctdluser *usbuf, long int number)
+
+ getuserbynumber() functions similarly to getuser(), except that it is
+supplied with a user number rather than a name. Calling this function
+results in a sequential search of the user file, so use it sparingly if at
+all.
+
+
+ int purge_user(char *pname)
+
+ This function deletes the named user off the system and erases all related
+objects: bio, photo, etc. It returns 0 upon success or a nonzero error code
+if the requested operation could not be performed.
+
+
+
+ HOW TO REGISTER FUNCTION HOOKS
+ ------------------------------
+
+ The truly powerful part of the Citadel API is the ability for extensions to
+register "hooks" -- user-supplied functions will be called while the server
+is performing various tasks. Here are the API calls to register hooks:
+
+
+ void CtdlRegisterProtoHook(void (*handler)(char *), char *cmd, char *desc)
+ void CtdlUnregisterProtoHook(void (*handler)(char *), char *cmd)
+
+ CtdlRegisterProtoHook() adds a new server command to the system. The
+handler function should accept a single string parameter, which will be set
+to a string containing any parameters the client software sent along with
+the server command. "cmd" should be the four-character mnemonic the server
+command is known by, and "desc" is a description of the new command.
+
+ CtdlUnregisterProtoHook() removes a server command from the system. It
+must be called with the same handler and cmd which were previously registered.
+
+
+ void CtdlRegisterCleanupHook(void *fcn_ptr)
+ void CtdlUnregisterCleanupHook(void *fcn_ptr)
+
+ CtdlRegisterCleanupHook() registers a new function to be called whenever the
+server is shutting down. Cleanup functions accept no parameters.
+
+ CtdlUnregsiterCleanupHook() removes a cleanup function from the system. It
+must be called with the same fcn_ptr which was previously registered.
+
+
+void CtdlRegisterSessionHook(void *fcn_ptr, int EventType)
+void CtdlUnregisterSessionHook(void *fcn_ptr, int EventType)
+
+ CtdlRegisterSessionHook() registers a session hook. Session hooks accept
+no parameters. There are multiple types of session hooks; the server
+extension registers which one it is interested in by setting the value of
+EventType. The available session hook types are:
+
+#define EVT_STOP 0 /* Session is terminating */
+#define EVT_START 1 /* Session is starting */
+#define EVT_LOGIN 2 /* A user is logging in */
+#define EVT_NEWROOM 3 /* Changing rooms */
+#define EVT_LOGOUT 4 /* A user is logging out */
+#define EVT_SETPASS 5 /* Setting or changing password */
+#define EVT_CMD 6 /* Called after each server command */
+#define EVT_RWHO 7 /* An RWHO command is being executed */
+#define EVT_ASYNC 8 /* Doing asynchronous message */
+
+#define EVT_TIMER 50 /* Timer events are called once per minute */
+#define EVT_HOUSE 51 /* Housekeeping event */
+
+ CtdlUnregisterSessionHook() removes a session hook. It must be called with
+the same fcn_ptr and EventTYpe which were previously registered.
+
+
+void CtdlRegisterUserHook(void *fcn_ptr, int EventType)
+void CtdlUnregisterUserHook(void *fcn_ptr, int EventType)
+
+ CtdlRegisterUserHook() registers a user hook. User hooks accept two
+parameters: a string pointer containing the user name, and a long which *may*
+contain a user number (only applicable for certain types of hooks). The
+available user hook types are:
+
+#define EVT_PURGEUSER 100 /* Deleting a user */
+#define EVT_OUTPUTMSG 101 /* Outputting a message */
+
+ CtdlUnregisterUserHook() removes a user hook from the system. It must be
+called with the same fcn_ptr and EventType which were previously registered.
+
+
+ void CtdlRegisterLogHook(void (*fcn_ptr) (char *), int loglevel)
+ void CtdlUnregisterLogHook(void (*fcn_ptr) (char *), int loglevel)
+
+ CtdlRegisterLogHook() adds a new logging function to the system. The
+handler function should accept a single string as a parameter. Logging
+functions can be used to implement additional logging facilities in
+addition to the Citadel trace file, which is output on stderr, or
+redirected to a file with the -t command line option. The handler
+function will be called if the loglevel is greater than or equal to
+loglevel.
+
+ Security considerations: Logs may contain plain text passwords and
+other sensitive information. It is your responsibility to ensure that
+your logs have appropriate access protection. The Citadel trace file
+is readable only by the superuser when the -t option is used.
+
+ CtdlUnregisterLogHook() removes a logging function from the system. It
+must be called with the same fcn_ptr and loglevel which were previously
+registered.
+
+
+ void CtdlRegisterMessageHook(int (*handler) (struct CtdlMessage *),
+ int EventType)
+ void CtdlUnregisterMessageHook(int (*handler) (struct CtdlMessage *),
+ int EventType)
+
+
+ CtdlRegisterMessageHook() registers a function with the message
+handling subsystem. This function will be called with one parameter,
+a pointer to a CtdlMessage structure, when the message triggers an event
+of type EventType. The registered function should return non zero if it
+has handled the event to prevent other hook functions from also processing
+the event.
+
+ CtdlUnregisterMessageHook() removes a function from the list of registered
+message handlers. To successfully remove a function registered with
+CtdlRegisterMessageHook() CtdlUnregisterMessageHook() must be called with
+the same parameters.
+
+ Possible values for EventType are:
+ EVT_BEFOREREAD Called after a message is loaded from disk but before
+it is presented for reading.
+ EVT_BEFORESAVE Called before the message is saved to disk. returning
+non zero for this event will prevent the message being saved to disk in the
+normal manner.
+ EVT_AFTERSAVE Called after the message is saved to disk but before
+any IGnet spooling is carried out.
+ EVT_SMTPSCAN Called during the SMTP reception of a message after the
+message is received and before the response is sent to the sender. This is
+intended for spam filters and virus checkers. A positive return code will
+cause the message to be rejected by the SMTP server.
+
+
+ void CtdlRegisterRoomHook(int (*fcn_ptr) (char *))
+ void CtdlUnregisterRoomHook(int (*fcn_ptr) (char *))
+
+ Register or remove a function with the room processing system.
+Registered functions are called in the order they are registered when a message
+is added to a room. This allows modules such as Sieve to process new messages
+appearing in a room.
+
+
+ void CtdlRegisterXmsgHook(int (*fcn_ptr) (char *, char *, char *), int order)
+ void CtdlUnregisterXmsgHook(int (*fcn_ptr) (char *, char *, char *), int order)
+
+ Please write documentation for me!
+
+
+ void CtdlRegisterServiceHook(int tcp_port, char *sockpath,
+ void (*h_greeting_function) (void),
+ void (*h_command_function) (void))
+ void CtdlUnregisterServiceHook(int tcp_port, char *sockpath,
+ void (*h_greeting_function) (void),
+ void (*h_command_function) (void))
+
+ Please write documentation for me!
+
+
+ FUNCTIONS WHICH MANIPULATE USER/ROOM RELATIONSHIPS
+
+ void CtdlGetRelationship(struct visit *vbuf,
+ struct ctdluser *rel_user,
+ struct ctdlroom *rel_room);
+ void CtdlSetRelationship(struct visit *newvisit,
+ struct ctdluser *rel_user,
+ struct ctdlroom *rel_room);
+
+ These functions get/set a "struct visit" structure which may contain
+information about the relationship between a user and a room. Specifically:
+
+struct visit {
+ char v_roomname[20];
+ long v_generation;
+ long v_lastseen;
+ unsigned int v_flags;
+ };
+
+#define V_FORGET 1 /* User has zapped this room */
+#define V_LOCKOUT 2 /* User is locked out of this room */
+#define V_ACCESS 4 /* Access is granted to this room */
+
+ Don't change v_roomname or v_generation; they're used to identify the room
+being referred to. A room is unique to the system by its combination of room
+name and generation number. If a new room is created with the same name as
+a recently deleted room, it will have a new generation number, and therefore
+stale "visit" records will not be applied (and will eventually be purged).
+
+ v_lastseen contains the number of the newest message the user has read in
+this room. Any existing messages higher than this number can be considered
+as "new messages."
+
+ v_flags contains information regarding access to the room.
+
+
+
+ int CtdlRoomAccess(struct ctdlroom *roombuf, struct ctdluser *userbuf)
+
+ This is a convenience function which uses CtdlGetRelationship() to determine
+whether a user has access to a room. It returns a bucket of bits which may
+contain:
+
+#define UA_INUSE 1 /* Room exists */
+#define UA_KNOWN 2 /* Room is in user's Known list */
+#define UA_GOTOALLOWED 4 /* User may <.G>oto this room */
+#define UA_HASNEWMSGS 8 /* Room contains new messages */
+#define UA_ZAPPED 16 /* User has forgotten this room */
+
+
+
+
+ ROOM RELATED FUNCTIONS
+ ----------------------
+
+
+unsigned create_room(char *new_room_name,
+ int new_room_type,
+ char *new_room_pass,
+ int new_room_floor,
+ int really_create)
+
+ This function is used to create a new room. new_room_name should be set to
+the desired name for the new room.
+
+ new_room_type should be set to one of the following values:
+ 0 = public room
+ 1 = guess-name room
+ 2 = passworded room
+ 3 = invitation-only room
+ 4 = personal (mailbox) room
+ 5 = personal (mailbox) room, and new_room_name already contains
+ the namespace prefix (use with caution!)
+
+ new_room_pass should be set to the desired password for the room (obviously
+this is only valid for passworded rooms).
+
+ If the room is really to be created, set really_create to 1. Otherwise, the
+caller may merely check to see if it's possible to create the room without
+actually creating it by setting really_create to 0.
+
+ create_room() returns the flags associated with the new room (as in the
+data structure item room.QRflags). If the room cannot be created (for
+example, a room with the name already exists), it returns 0.
+
+
--- /dev/null
+The totally incomplete guide to Citadel internals
+-----------------------------------------------------
+-----------------------------------------------------
+
+Citadel has evolved quite a bit since its early days, and the data structures
+have evolved with it. This document provides a rough overview of how the
+system works internally. For details you're going to have to dig through the
+code, but this'll get you started.
+
+
+DATABASE TABLES
+---------------
+As you probably already know by now, Citadel uses a group of tables stored
+with a record manager (usually Berkeley DB). Since we're using a record
+manager rather than a relational database, all record structures are managed
+by Citadel. Here are some of the tables we keep on disk:
+
+
+USER RECORDS
+------------
+This table contains all user records. It's indexed by
+user name (translated to lower case for indexing purposes). The records in
+this file look something like this:
+
+struct ctdluser { /* User record */
+ int version; /* Cit vers. which created this rec */
+ uid_t uid; /* Associate with a unix account? */
+ char password[32]; /* password (for Citadel-only users)*/
+ unsigned flags; /* See US_ flags below */
+ long timescalled; /* Total number of logins */
+ long posted; /* Number of messages posted (ever) */
+ CIT_UBYTE axlevel; /* Access level */
+ long usernum; /* User number (never recycled) */
+ time_t lastcall; /* Last time the user called */
+ int USuserpurge; /* Purge time (in days) for user */
+ char fullname[64]; /* Name for Citadel messages & mail */
+};
+
+ Most fields here should be fairly self-explanatory. The ones that might
+deserve some attention are:
+
+ - *uid* -- if uid is not the same as the *unix uid* Citadel is running as, then the
+ account is assumed to belong to the user on the underlying Unix system with
+ that uid. This allows us to require the user's OS password instead of having
+ a separate Citadel password.
+
+ - *usernum* -- these are assigned sequentially, and **NEVER REUSED**. This is
+ important because it allows us to use this number in other data structures
+ without having to worry about users being added/removed later on, as you'll
+ see later in this document.
+
+
+ROOM RECORDS
+------------
+These are room records. There is a room record for every room on the
+system, public or private or mailbox. It's indexed by room name (also in
+lower case for easy indexing) and it contains records which look like this:
+
+ struct ctdlroom {
+ char QRname[ROOMNAMELEN]; /* Name of room */
+ char QRpasswd[10]; /* Only valid if it's a private rm */
+ long QRroomaide; /* User number of room aide */
+ long QRhighest; /* Highest message NUMBER in room */
+ time_t QRgen; /* Generation number of room */
+ unsigned QRflags; /* See flag values below */
+ char QRdirname[15]; /* Directory name, if applicable */
+ long QRinfo; /* Info file update relative to msgs*/
+ char QRfloor; /* Which floor this room is on */
+ time_t QRmtime; /* Date/time of last post */
+ struct ExpirePolicy QRep; /* Message expiration policy */
+ long QRnumber; /* Globally unique room number */
+ char QRorder; /* Sort key for room listing order */
+ unsigned QRflags2; /* Additional flags */
+ int QRdefaultview; /* How to display the contents */
+ };
+
+Again, mostly self-explanatory. Here are the interesting ones:
+
+*QRnumber* is a globally unique room ID, while QRgen is the "generation number"
+of the room (it's actually a timestamp). The two combined produce a unique
+value which identifies the room. The reason for two separate fields will be
+explained below when we discuss the visit table. For now just remember that
+*QRnumber* remains the same for the duration of the room's existence, and QRgen
+is timestamped once during room creation but may be restamped later on when
+certain circumstances exist.
+
+FLOORTAB
+--------
+Floors. This is so simplistic it's not worth going into detail about, except
+to note that we keep a reference count of the number of rooms on each floor.
+
+MSGLISTS
+--------
+Each record in this table consists of a bunch of message numbers
+which represent the contents of a room. A message can exist in more than one
+room (for example, a mail message with multiple recipients -- 'single instance
+store'). This table is never, ever traversed in its entirety. When you do
+any type of read operation, it fetches the msglist for the room you're in
+(using the room's ID as the index key) and then you can go ahead and read
+those messages one by one.
+
+Each room is basically just a list of message numbers. Each time
+we enter a new message in a room, its message number is appended to the end
+of the list. If an old message is to be expired, we must delete it from the
+message base. Reading a room is just a matter of looking up the messages
+one by one and sending them to the client for display, printing, or whatever.
+
+
+VISIT
+-----
+This is the tough one. Put on your thinking cap and grab a fresh cup of
+coffee before attempting to grok the visit table.
+
+This table contains records which establish the relationship between users
+and rooms. Its index is a hash of the user and room combination in question.
+When looking for such a relationship, the record in this table can tell the
+server things like "this user has zapped this room," "this user has access to
+this private room," etc. It's also where we keep track of which messages
+the user has marked as "old" and which are "new" (which are not necessarily
+contiguous; contrast with older Citadel implementations which simply kept a
+"last read" pointer).
+
+
+Here's what the records look like:
+
+ struct visit {
+ long v_roomnum;
+ long v_roomgen;
+ long v_usernum;
+ long v_lastseen;
+ unsigned int v_flags;
+ char v_seen[SIZ];
+ int v_view;
+ };
+
+ #define V_FORGET 1 /* User has zapped this room */
+ #define V_LOCKOUT 2 /* User is locked out of this room */
+ #define V_ACCESS 4 /* Access is granted to this room */
+
+This table is indexed by a concatenation of the first three fields. Whenever
+we want to learn the relationship between a user and a room, we feed that
+data to a function which looks up the corresponding record. The record is
+designed in such a way that an "all zeroes" record (which is what you get if
+the record isn't found) represents the default relationship.
+
+With this data, we now know which private rooms we're allowed to visit: if
+the *V_ACCESS* bit is set, the room is one which the user knows, and it may
+appear in his/her known rooms list. Conversely, we also know which rooms the
+user has zapped: if the *V_FORGET* flag is set, we relegate the room to the
+zapped list and don't bring it up during new message searches. It's also
+worth noting that the *V_LOCKOUT* flag works in a similar way to administratively
+lock users out of rooms.
+
+Implementing the "cause all users to forget room" command, then, becomes very
+simple: we simply change the generation number of the room by putting a new
+timestamp in the *QRgen* field. This causes all relevant visit records to
+become irrelevant, because they appear to point to a different room. At the
+same time, we don't lose the messages in the room, because the msglists table
+is indexed by the room number (*QRnumber*), which never changes.
+
+*v_seen* contains a string which represents the set of messages in this room
+which the user has read (marked as 'seen' or 'old'). It follows the same
+syntax used by IMAP and NNTP. When we search for new messages, we simply
+return any messages that are in the room that are **not** represented by this
+set. Naturally, when we do want to mark more messages as seen (or unmark
+them), we change this string. Citadel BBS client implementations are naive
+and think linearly in terms of "everything is old up to this point," but IMAP
+clients want to have more granularity.
+
+
+DIRECTORY
+---------
+This table simply maps Internet e-mail addresses to Citadel network addresses
+for quick lookup. It is generated from data in the Global Address Book room.
+
+USETABLE
+--------
+This table keeps track of message ID's of messages arriving over a network,
+to prevent duplicates from being posted if someone misconfigures the network
+and a loop is created. This table goes unused on a non-networked Citadel.
+
+THE MESSAGE STORE
+-----------------
+This is where all message text is stored. It's indexed by message number:
+give it a number, get back a message. Messages are numbered sequentially, and
+the message numbers are never reused.
+
+We also keep a "metadata" record for each message. This record is also stored
+in the msgmain table, using the index (0 - msgnum). We keep in the metadata
+record, among other things, a reference count for each message. Since a
+message may exist in more than one room, it's important to keep this reference
+count up to date, and to delete the message from disk when the reference count
+reaches zero.
+
+#Here's the format for the message itself:
+
+ - Each message begins with an 0xFF 'start of message' byte.
+
+ - The next byte denotes whether this is an anonymous message. The codes
+ available are *MES_NORMAL*, *MES_ANON*, or *MES_AN2* (defined in citadel.h).
+
+ - The third byte is a "message type" code. The following codes are defined:
+ - 0 - "Traditional" Citadel format. Message is to be displayed "formatted."
+ - 1 - Plain pre-formatted ASCII text (otherwise known as text/plain)
+ - 4 - MIME formatted message. The text of the message which follows is
+ expected to begin with a "Content-type:" header.
+
+ - After these three opening bytes, the remainder of
+ the message consists of a sequence of character strings. Each string
+ begins with a type byte indicating the meaning of the string and is
+ ended with a null. All strings are printable ASCII: in particular,
+ all numbers are in ASCII rather than binary. This is for simplicity,
+ both in implementing the system and in implementing other code to
+ work with the system. For instance, a database driven off Citadel archives
+ can do wildcard matching without worrying about unpacking binary data such
+ as message ID's first. To provide later downward compatability
+ all software should be written to IGNORE fields not currently defined.
+
+
+#The type bytes currently defined are:
+
+
+| BYTE | Enum | NW | Mnemonic | Enum / Comments
+|-------|-------------------|------|----------------|---------------------------------------------------------
+| A | eAuthor | from | Author | *eAuthor*
+| | | | | Name of originator of message.
+| B | eBig\_message | | Big message | *eBig\_message*
+| | | | | This is a flag which indicates that the message is
+| | | | | big, and Citadel is storing the body in a separate
+| | | | | record. You will never see this field because the
+| | | | | internal API handles it.
+| C | eRemoteRoom | | RemoteRoom | *eRemoteRoom*
+| | | | | when sent via Citadel Networking, this is the room
+| | | | | its going to be put on the remote site.
+| D | eDestination | | Destination | *eDestination*
+| | | | | Contains name of the system this message should
+| | | | | be sent to, for mail routing (private mail only).
+| E | eExclusiveID | exti | Exclusive ID | *eExclusiveID*
+| | | | | A persistent alphanumeric Message ID used for
+| | | | | network replication. When a message arrives that
+| | | | | contains an Exclusive ID, any existing messages which
+| | | | | contain the same Exclusive ID and are *older* than this
+| | | | | message should be deleted. If there exist any messages
+| | | | | with the same Exclusive ID that are *newer*, then this
+| | | | | message should be dropped.
+| F | erFc822Addr | rfca | rFc822 address | *erFc822Addr*
+| | | | | For Internet mail, this is the delivery address of the
+| | | | | message author.
+| H | eHumanNode | hnod | Human node name| *eHumanNode*
+| | | | | Human-readable name of system message originated on.
+| I | emessageId | msgn | Message ID | *emessageId*
+| | | | | An RFC822-compatible message ID for this message.
+| J | eJournal | jrnl | Journal | *eJournal*
+| | | | | The presence of this field indicates that the message
+| | | | | is disqualified from being journaled, perhaps because
+| | | | | it is itself a journalized message and we wish to
+| | | | | avoid double journaling.
+| K | eReplyTo | rep2 | Reply-To | *eReplyTo*
+| | | | | the Reply-To header for mailinglist outbound messages
+| L | eListID | list | List-ID | *eListID*
+| | | | | Mailing list identification, as per RFC 2919
+| M | eMesageText | text | Message Text | *eMesageText*
+| | | | | Normal ASCII, newlines seperated by CR's or LF's,
+| | | | | null terminated as always.
+| N | eNodeName | node | Nodename | *eNodeName*
+| | | | | Contains node name of system message originated on.
+| O | eOriginalRoom | room | Room | *eOriginalRoom* - Room of origin.
+| P | eMessagePath | path | Path | *eMessagePath*
+| | | | | Complete path of message, as in the UseNet news
+| | | | | standard. A user should be able to send Internet mail
+| | | | | to this path. (Note that your system name will not be
+| | | | | tacked onto this until you're sending the message to
+| | | | | someone else)
+| R | eRecipient | rcpt | Recipient | *eRecipient* - Only present in Mail messages.
+| S | eSpecialField | spec | Special field | *eSpecialField*
+| | | | | Only meaningful for messages being spooled over a
+| | | | | network. Usually means that the message isn't really
+| | | | | a message, but rather some other network function:
+| | | | | -> "S" followed by "FILE" (followed by a null, of
+| | | | | course) means that the message text is actually an
+| | | | | IGnet/Open file transfer. (OBSOLETE)
+| | | | | -> "S" followed by "CANCEL" means that this message
+| | | | | should be deleted from the local message base once
+| | | | | it has been replicated to all network systems.
+| T | eTimestamp | time | date/Time | *eTimestamp*
+| | | | | Unix timestamp containing the creation date/time of
+| | | | | the message.
+| U | eMsgSubject | subj | sUbject | *eMsgSubject* - Optional.
+| | | | | Developers may choose whether they wish to
+| | | | | generate or display subject fields.
+| V | eenVelopeTo | nvto | enVelope-to | *eenVelopeTo*
+| | | | | The recipient specified in incoming SMTP messages.
+| W | eWeferences | wefw | Wefewences | *eWeferences*
+| | | | | Previous message ID's for conversation threading. When
+| | | | | converting from RFC822 we use References: if present, or
+| | | | | In-Reply-To: otherwise.
+| | | | | (Who in extnotify spool messages which don't need to know
+| | | | | other message ids)
+| Y | eCarbonCopY | cccc | carbon copY | *eCarbonCopY*
+| | | | | Optional, and only in Mail messages.
+| % | eHeaderOnly | nhdr | oNlyHeader | we will just be sending headers. for the Wire protocol only.
+| % | eFormatType | type | type | type of citadel message: (Wire protocol only)
+| | | | | FMT\_CITADEL 0 Citadel vari-format (proprietary)
+| | | | | FMT\_FIXED 1 Fixed format (proprietary)
+| | | | | FMT\_RFC822 4 Standard (headers are in M field)
+| % | eMessagePart | part | emessagePart | *eMessagePart* is the id of this part in the mime hierachy
+| % | eSubFolder | suff | eSubFolder | descend into a mime sub container
+| % | ePevious | pref | ePevious | exit a mime sub container
+| 0 | eErrorMsg | | Error | *eErrorMsg*
+| | | | | This field is typically never found in a message on
+| | | | | disk or in transit. Message scanning modules are
+| | | | | expected to fill in this field when rejecting a message
+| | | | | with an explanation as to what happened (virus found,
+| | | | | message looks like spam, etc.)
+| 1 | eSuppressIdx | | suppress index | *eSuppressIdx*
+| | | | | The presence of this field indicates that the message is
+| | | | | disqualified from being added to the full text index.
+| 2 | eExtnotify | | extnotify | *eExtnotify* - Used internally by the serv_extnotify module.
+| 3 | eVltMsgNum | | msgnum | *eVltMsgNum*
+| | | | | Used internally to pass the local message number in the
+| | | | | database to after-save hooks. Discarded afterwards.
+
+EXAMPLE
+-------
+Let *<FF>* be a *0xFF* byte, and *<0>* be a null *(0x00)* byte. Then a message
+which prints as...
+
+ Apr 12, 1988 23:16 From Test User In Network Test> @lifesys (Life Central)
+ Have a nice day!
+
+might be stored as...
+
+ <FF><40><0>I12345<0>Pneighbor!lifesys!test_user<0>T576918988<0> (continued)
+ -----------|Mesg ID#|--Message Path---------------|--Date------
+
+ AThe Test User<0>ONetwork Test<0>Nlifesys<0>HLife Central<0>MHave a nice day!<0>
+ |-----Author-----|-Room name-----|-nodename-|Human Name-|--Message text-----
+
+Weird things can happen if fields are missing, especially if you use the
+networker. But basically, the date, author, room, and nodename may be in any
+order. But the leading fields and the message text must remain in the same
+place. The H field looks better when it is placed immediately after the N
+field.
+
+
+EUID (EXCLUSIVE MESSAGE ID'S)
+-----------------------------
+This is where the groupware magic happens. Any message in any room may have
+a field called the Exclusive message *ID*, or *EUID*. We keep an index in the
+table *CDB_EUIDINDEX* which knows the message number of any item that has an
+*EUID*. This allows us to do two things:
+
+ - If a subsequent message arrives with the same *EUID*, it automatically
+ *deletes* the existing one, because the new one is considered a replacement
+ for the existing one.
+ - If we know the *EUID* of the item we're looking for, we can fetch it by *EUID*
+ and get the most up-to-date version, even if it's been updated several times.
+
+This functionality is made more useful by server-side hooks. For example,
+when we save a vCard to an address book room, or an iCalendar item to a
+calendar room, our server modules detect this condition, and automatically set
+the *EUID* of the message to the *UUID* of the *vCard* or *iCalendar* item.
+Therefore when you save an updated version of an address book entry or
+a calendar item, the old one is automatically deleted.
+
+NETWORKING (REPLICATION)
+------------------------
+Citadel nodes network by sharing one or more rooms. Any Citadel node
+can choose to share messages with any other Citadel node, through the sending
+of spool files. The sending system takes all messages it hasn't sent yet, and
+spools them to the recieving system, which posts them in the rooms.
+
+The *EUID* discussion above is extremely relevant, because *EUID* is carried over
+the network as well, and the replacement rules are followed over the network
+as well. Therefore, when a message containing an *EUID* is saved in a networked
+room, it replaces any existing message with the same *EUID* *on every node in
+the network*.
+
+Complexities arise primarily from the possibility of densely connected
+networks: one does not wish to accumulate multiple copies of a given
+message, which can easily happen. Nor does one want to see old messages
+percolating indefinitely through the system.
+
+This problem is handled by keeping track of the path a message has taken over
+the network, like the UseNet news system does. When a system sends out a
+message, it adds its own name to the bang-path in the *<P>* field of the
+message. If no path field is present, it generates one.
+
+With the path present, all the networker has to do to assure that it doesn't
+send another system a message it's already received is check the <P>ath field
+for that system's name somewhere in the bang path. If it's present, the system
+has already seen the message, so we don't send it.
+
+We also keep a small database, called the "use table," containing the ID's of
+all messages we've seen recently. If the same message arrives a second or
+subsequent time, we will find its ID in the use table, indicating that we
+already have a copy of that message. It will therefore be discarded.
+
+The above discussion should make the function of the fields reasonably clear:
+
+ o Travelling messages need to carry original message-id, system of origin,
+ date of origin, author, and path with them, to keep reproduction and
+ cycling under control.
+
+(Uncoincidentally) the format used to transmit messages for networking
+purposes is precisely that used on disk, serialized. The current
+distribution includes serv_network.c, which is basically a database replicator;
+please see network.txt on its operation and functionality (if any).
+
+PORTABILITY ISSUES
+------------------
+Citadel is 64-bit clean, architecture-independent, and Year 2000
+compliant. The software should compile on any POSIX compliant system with
+a full pthreads implementation and TCP/IP support. In the future we may
+try to port it to non-POSIX systems as well.
+
+On the client side, it's also POSIX compliant. The client even seems to
+build ok on non-POSIX systems with porting libraries (such as Cygwin).
+
+SUPPORTING PRIVATE MAIL
+-----------------------
+Can one have an elegant kludge? This must come pretty close.
+
+Private mail is sent and recieved in the *Mail>* room, which otherwise
+behaves pretty much as any other room. To make this work, we have a
+separate Mail> room for each user behind the scenes. The actual room name
+in the database looks like *"0000001234.Mail"* (where *'1234'* is the user
+number) and it's flagged with the *QR_MAILBOX* flag. The user number is
+stripped off by the server before the name is presented to the client. This
+provides the ability to give each user a separate namespace for mailboxes
+and personal rooms.
+
+This requires a little fiddling to get things just right. For example,
+*make_message()* has to be kludged to ask for the name of the recipient
+of the message whenever a message is entered in *Mail>*. But basically
+it works pretty well, keeping the code and user interface simple and
+regular.
+
+PASSWORDS AND NAME VALIDATION
+-----------------------------
+This has changed a couple of times over the course of Citadel's history. At
+this point it's very simple, again due to the fact that record managers are
+used for everything. The user file (user) is indexed using the user's
+name, converted to all lower-case. Searching for a user, then, is easy. We
+just lowercase the name we're looking for and query the database. If no
+match is found, it is assumed that the user does not exist.
+
+This makes it difficult to forge messages from an existing user. (Fine
+point: nonprinting characters are converted to printing characters, and
+leading, trailing, and double blanks are deleted.)
--- /dev/null
+Description of the custom MIME type "application/x-citadel-delivery-list"
+
+ This MIME type may be found in the outbound queue room on Citadel systems,
+which is typically called "__CitadelSMTPspoolout__". The room is flagged as
+a "system" room, which means that it's completely hidden to everyone, although
+an Admin can get to it if the full name is specified (but unlike a normal
+hidden room, it does not appear in the known rooms list on subsequent visits).
+
+ Messages in this format contain delivery instructions. Therefore, for each
+message in the queue to be delivered to one or more recipients, there will be
+*two* messages in the room: one containing the actual message, and the other
+containing delivery instructions. It is expected that the instructions
+message may be replaced at any time (using an Exclusive ID field) if delivery
+to only some of the recipients has been accomplished.
+
+ Citadel keeps reference counts of each message on disk. Therefore if a
+message contains a mixture of local and remote recipients, there may be two
+or more references to the message itself, one of them being the one in the
+queue.
+
+ A delivery list contains one or more lines of text, each line containing
+a single instruction (usually a recipient). Fields are separated by the
+vertical bar character ("|") and there will always be at least one field on
+each line.
+
+ -- Once per Queue-Item --
+
+ INSTRUCTION: msgid
+ SYNTAX: msgid|0000000
+ DESCRIPTION:
+ Defines the actual message for which we are providing delivery
+ instructions. This instruction must precede all the others. When
+ all deliveries have either succeeded or failed, both the instructions and
+ the copy of the message itself in the queue room should be deleted. The
+ second parameter specifies the message ID in the local database.
+
+
+ INSTRUCTION: submitted
+ SYNTAX: submitted|999999999
+ DESCRIPTION:
+ Contains a timestamp designating when this message was first entered into
+ the system.
+
+
+ INSTRUCTION: attempted
+ SYNTAX: attempted|999999999
+ DESCRIPTION:
+ Contains a timestamp designating the date/time of the last delivery
+ attempt.
+
+
+ INSTRUCTION: retry
+ SYNTAX: retry|9999999
+ DESCRIPTION:
+ Citadel does not retry SMTP delivery at a fixed interval. Instead, it
+starts at a nominal interval (15 minutes by default) and then doubles the
+interval after each failed attempt. This instruction contains the interval
+which should currently be followed (and doubled again, if the next delivery
+fails).
+
+
+ INSTRUCTION: bounceto
+ SYNTAX: bounceto|Big Bad Sender[@host]
+ DESCRIPTION:
+ Where to send "bounce" messages (delivery status notifications). The
+ contents of the second field are supplied as the "recipient" field to
+ CtdlSaveMsg(), and therefore may be a local username, a user on another
+ Citadel, or an Internet e-mail address.
+
+
+ INSTRUCTION: envelope_from
+ SYNTAX: envelope_from|blurdybloop@example.com
+ DESCRIPTION:
+ Sets a value to be used as the envelope sender during the 'MAIL FROM:'
+ phase of the SMTP transaction. If an envelope sender is not supplied,
+ one is extracted from the message body.
+
+ INSTRUCTION: source_room
+ SYNTAX: source_room|Citadel Support
+ DESCRIPTION:
+ when sending mailinglist rooms this contains the source room for displaying
+ and messaging purposes.
+
+
+
+
+-- Once per Remote-part per Queue-Item --
+
+ INSTRUCTION: remote
+ SYNTAX: remote|friko@mumjiboolean.com|0|delivery status message
+ DESCRIPTION:
+ Names a recipient on a remote system to which the message should be
+ delivered. The third parameter may contain any of the following values:
+ 0 = No delivery has yet been attempted
+ 2 = Delivery was successful
+ 3 = transient error state; like connection failure or DNS lookup failure
+ 4 = A transient error was experienced ... try again later
+ 5 = Delivery to this address failed permanently. The error message
+ should be placed in the fourth field so that a bounce message may
+ be generated.
--- /dev/null
+Every client for the Citadel system is capable of identifying itself with
+a developer code, client code, and client version. Here are the known values
+for these fields.
+
+
+Developer codes (and where known, client codes)
+
+ 0 The Citadel core development team
+ (See copyright.txt for the list of people involved in the project)
+ Here are the client ID's we've used:
+
+ Client Name Status Description
+ 0 Citadel deployed Citadel text based client
+ 1 lcit dead Curses-based text client
+ 2 WinCit legacy Visual Basic client for Win16
+ 4 WebCit deployed Web-based access to Citadel
+ 5 Shaggy dead Java-powered client
+ 6 Daphne dead wx multiplatform "fat" client
+ 7 Shaggy (new) unknown Java-powered client
+ <chilly@uncensored.citadel.org>
+ 8 ctdlphp deployed PHP interface to Citadel
+ 9 z-push in development Citadel backend for Z-Push
+
+ 1 Brian Ledbetter <brian@shadowcom.net>
+
+ Client Name Status Description
+ 0 libCxClient deployed Client-side API library
+ 1 Infusion deployed Groupware for Citadel
+
+ 2 OST Systems, Inc.
+ Matthew Scott <mscott@springfieldtech.com>
+
+ Client Name Status Description
+ 0 CivisLink development Python Citadel protocol library
+
+ 3 Jesse Vincent <jrvincent@wesleyan.edu>
+
+ 4 Brian Costello <btx@calyx.net>
+
+ 5 Robert Abatecola <robert@tsgus.com>
+
+ 6 Bastille <tdf@area51.v-wave.com>
+
+ 7 Walden Leverich <waldenl@techsoftinc.com>
+
+ 8 Michael Hampton <error@citadel.org>
+
+ Client Name Status Description
+ 0 libcitadel in development Client-side API library
+ 1 D.O.C. dead Dave's Own Citadel look-alike
+ 8 stress deployed Citadel server stress tester
+ 16 Courier in development GTK+ Unix/Win32 Client
+
+ 31 Stuart Cianos <scianos@alphavida.com>
+
+ 69 Anticlimactic Teleservices
+ Don Kimberlin (err head) <donkimberlin@hotmail.com>
+
+ 71 Robert Barta <rho@cpan.org>
+
+ 177 Kevin Fisher <kgf@golden.net>
+
+ 226 David Given <dg@cowlark.com>
--- /dev/null
+=====Logging =====
+Note: the logging scheme is not yet persistant all over the citserver.
+
+citadel server has several components that emmit different log strings. Its designed so you can filter for specific sessions.
+
+Since there are several sources for numbering schemes, several numbers need to be employed to make one job / session completely traceable.
+The syntax is in general like CC[%d]
+
+
+=== CC - Citadel Context ===
+All contexts that are visible to the RWHO command have a citadel context. The ID of such a context is generated from a source that can grow and shrink again. Thus CC may not be uniq to one session.
+
+Citadel may have several kind of sessions:
+ - sessions generated by external connections; they have a CC.
+ - sessions spawned by repetive tasks; they at first may have their own id-range, then later on an IO (AsyncIO) and CC (Citadel Context)
+
+=== Identifieing running sessions ===
+The RWHO command will list you the currently active CCs:
+
+# sendcommand RWHO
+sendcommand: started (pid=14528) connecting to Citadel server at /var/run/citadel/citadel-admin.socket
+200 potzblitz Citadel server ADMIN CONNECTION ready.
+RWHO
+100
+576|SYS_rssclient||http://xkcd.org/rss.xml||0||.||||0
+1965|willi|Mail|PotzBlitz|IMAP session|1346695500| |.||||1
+sendcommand: processing ended.
+
+576 is the CC connected to this client. with
+
+grep 'CC\[576\]' /var/log/syslog
+
+will show you the loglines from all contexts that had the ID 576; the one named above is an RSS aggregator:
+
+Jul 3 09:07:01 PotzBlitz citserver[1435]: EVCURL:IO[576]CC[576] error description: The requested URL returned error: 404
+Jul 3 09:07:01 PotzBlitz citserver[1435]: EVCURL:IO[576]CC[576] error performing request: HTTP response code said error
+Jul 3 09:07:01 PotzBlitz citserver[1435]: IO[576]CC[576][29]RSSPneed a 200, got a 404 !
+
+
+grep 'CC\[1965\]' /var/log/syslog
+will show you the loglines from all contexts that had the ID 1965; the one named above is an inbound IMAP Connection:
+
+Jul 3 09:05:01 PotzBlitz citserver[12826]: IMAPCC[1965] <plain_auth>
+Jul 3 09:05:01 PotzBlitz citserver[12826]: CC[1965]<Lobby> 144 new of 144 total messages
+Jul 3 09:05:02 PotzBlitz citserver[12826]: CC[1965]<0000000010.Mail> 1 new of 17 total messages
+
+=== Numbering and prefixes ===
+
+While external sessions will constantly show the same CC (or maybe add their own name space as above with IMAPCC) Its important to understand that not all numbers are available in all situations with sessions from repetive tasks; thus you may need several filters to gather all logmessages belonging to one job.
+
+Generaly once per minute one worker becomes an housekeeping thread; This housekeeper scans the whole system for jobs to start.
+Once a job is identified, its detection is logged; neither a CC nor IO are already assigned to this job yet;
+but it already has its uniq number assigned to it which is specific to the type of task.
+After the preanalysis of this job is done, it may become an AsyncIO job with a Citadel Context. The job now lives in one of the eventqueues, waiting for external file IO or Database IO or performing some calculation of a new state.
+Some basic IO debug logging may not facilitate the CC or the Job-ID; especialy during HTTP or DNS requests.
+
+=== Debug logging ===
+By default citserver will only log information about errors or traffic analysis related errors. If you want to track all a job does, you can enable more logging:
+
+sendcommand LOGP (LOG Print; show status of logs; the following list may increase over time) :
+
+sendcommand: started (pid=19897) connecting to Citadel server at /var/run/citadel/citadel-admin.socket
+200 potzblitz Citadel server ADMIN CONNECTION ready.
+LOGP
+100 Log modules enabled:
+modules|0 <- module loading; unloading etc.
+
+rssclient|0 <- RSS aggregator client debugging; on message level
+RSSAtomParser|0 <- debugging the parsing of ATOM / RSS feeds; once a message is isolated, the above logging comes into place again.
+curl|0 <- HTTP request logging; i.e. querieing the external RSS resources by http is logged with this.
+
+networkqueue|0 <- working through rooms, finding jobs to schedule; reading network spool files, writing them; etc.
+networktalkingto|0 <- locked list of network interconnections to circumvent race conditions by simultaneous spooling / sending / receiving
+networkclient|0 <- outbound networking connections; this has an IO context.
+
+cares|0 <- asynchroneous nameserver lookup
+eventloop|0 <- adding jobs; connecting; timeouts; etc.
+eventloopbacktrace|0 <- add stacktraces in certain places; very noisy.
+
+sieve|0 <- sieve filtering of mailboxes
+messages|0 <- message processing; loading / printing / saving
+
+imapsrv|0 <- Imap connections
+serv_xmpp|0 <- XMPP connections
+pop3client|0 <- pop3 aggregators
+
+smtpeventclient|0 <- outbound SMTP connections
+
+sendcommand: processing ended.
+
+If the second column is 1, that debug log is enabled.
+
+You can enable the logging by 3 ways:
+ - sendcommand 'LOGS cares|1' (live)
+ - via webcit->aide->Enable/Disable logging of the server components; in an interactive manner
+ - via environment variable 'CITADEL_LOGDEBUG' as a coma separated list of facilities to enable; before starting citserver
+ - via commandline parameter '-x' as a coma separated list of facilities to enable; please note that this switch used to set the loglevel previously.
+
+
+
+eventloop,eventloopbacktrace,cares
+ - IO the AsyncIO ID
+ - IOQ (Queueing of events into eventqueues)
+
+modules:
+ - modules - module init / registering specific stuff
+
+smtpeventclient
+ - S[QueueID][N] where the Queue-ID is the number of the Message to be sent, and N the SMTP-job-ID
+ - SMTPCQ infrastructure logging when the housekeeper is checking for SMTP-Delivery jobs to spawn
+
+imapsrv
+ - IMAPCC: imap client contexts; commands sent to us from Imap-Clients
+
+
+networkqueue
+ - CC -
+
+networkclient
+ - CC, IO - the context
+ - NW[nodename][networkerid] the name of the node we're trying to talk to; the networker connection counter uniq per attempt
+
+curl
+ - CURL - logging where we don't have a context
+ - CC, IO - the context
+
+RSSClient
+ - IO, CC, N, 'RSS'; IO & CC not always available.
+ - 'RSSP' - the rss and atom parser
+
+
+pop3client
+ - POP3 IO, CC, N - N the number of that connection
+
+sieve
+ - sieve doesn't have CC or IOs, its just ran one at a time serverwide.
+
+
+xmpp
+ - XMPP - whether CC could & should be logged has to be researched and added.
+
+Context
+ - 'Context:' context related logging - server infrastructure, cleanup of contexts etc.
+
+messages
+ - MSG - CC - the context that is manipulating this message.
+
+
+
--- /dev/null
+
+ Description of the files in the "netconfigs" directory
+
+ These files contain a set of network configurations for a room. They are
+stored in the directory $CTDLDIR/netconfigs and are named according to each
+room's internal ID number. When a room is deleted, its network configuration
+file is deleted as well.
+
+ The configuration file contains one or more lines of text, each line
+containing a configuration option. These lines may specify message pointers,
+room sharing instructions, mailing list recipients, etc. Fields are separated
+by the vertical bar character ("|") and there will always be at least one
+field on each line.
+
+
+ INSTRUCTION: lastsent
+ SYNTAX: lastsent|0000000
+ DESCRIPTION:
+ Defines the *local* message number of the last message in this room which
+we have performed outbound network processing on. Any batch job which sends
+out messages should do stuff.
+
+
+ INSTRUCTION: listrecp
+ SYNTAX: listrecp|friko@mumjiboolean.com
+ DESCRIPTION:
+ Defines a recipient to whom all messages in this room should be sent. This
+is used for "list serve" applications.
+
+
+ INSTRUCTION: digestrecp
+ SYNTAX: digestrecp|friko@mumjiboolean.com
+ DESCRIPTION:
+ Defines a recipient to whom all messages in this room should be sent. This
+is used for "list serve" applications. The difference between listrecps and
+digestrecps is that the latter will have messages embedded inside a message
+sent by the listserver. The message will appear to be sent by the room's
+e-mail address instead of the sender's e-mail address.
+
+
+ INSTRUCTION: ignet_push_share
+ SYNTAX: ignet_push_share|uncnsrd
+ (or) ignet_push_share|uncnsrd|Foo Bar Baz
+ DESCRIPTION:
+ Specifies that the second argument is the name of a neighboring node on an
+IGnet (Citadel networking) network, to which this room should be pushed
+(spooled). Conceptually, this node will have a corresponding record pushing
+data in the other direction.
+
+ If the third argument is present, it is the name of the corresponding
+room on the remote node. This allows a room to be shared even when the room
+name is different on each node. Such a configuration *must* be configured
+mutually: each node must know the name of the room on the other.
+
+
+ INSTRUCTION: subpending
+ SYNTAX: subpending|friko@mumjiboolean.com|listrecp|A234Z|1234567890|http://foo.com/lists
+ "Subscription pending" for the specified address. This means that
+someone has requested to subscribe an e-mail address (in this case,
+friko@mumjiboolean.com) to the list. The third parameter is either "list"
+or "digest" to specify a normal subscription or a digest subscription.
+The fourth parameter is an authentication token which is generated by the server
+and e-mailed to the specified address. When the user comes back and supplies
+this token, the subscription is complete. The fifth parameter is a simple
+timestamp, so that we may purge old records which were never confirmed.
+
+ The sixth field is the URL of the web page used to enter the subscription
+request, minus any parameters.
+
+
+ INSTRUCTION: unsubpending
+ SYNTAX: unsubpending|friko@mumjiboolean.com|A234Z|1234567890|http://foo.com/lists
+ Similar to the 'subpending' command, except this one is for unsubscribe
+requests. The same rules apply with regard to the token and the web page.
+
+
+ INSTRUCTION: participate
+ SYNTAX: participate|list-addr@example.com
+ This allows a room to be a *subscriber* to an external mailing list
+somewhere. The room's email address should be subscribed to the list, and the
+external listserv should be told "do not send me my own messages." Any
+messages which originated on Citadel (as opposed to messages which arrived via
+email, such as those from the list) will be sent to the list, with the display
+name of the author kept intact, but the email address changed to the address of
+the room.
+
+ This is somewhat finicky, which is why it is not officially supported, and
+why there are no facilities in the user interface to configure it. If you want
+to use this option you are on your own.
+
+
+ INSTRUCTION: pop3client
+ SYNTAX: pop3client|hostname|username|password|keep|interval
+ Periodically fetch messages from a remote POP3 account and store them in this
+room. Messages will be deleted from the remote account once successfully stored.
+if Keep is set, messages won't be erased on the remote host.
+Interval can specify a bigger value than the system configs value.
--- /dev/null
+
+Package builders: you can set the following environment variables
+before running setup in quiet mode (setup -q) in order to keep it
+quiet but still do setup.
+
+First, please set CITADEL_INSTALLER to "yes"
+
+Then set:
+
+CITADEL Directory in which Citadel is installed
+
+SLAPD_BINARY Location of slapd (optional)
+LDAP_CONFIG Location of slapd.conf (optional)
+
+SUPPORT Location of libraries, etc. (optional)
+ (i.e. /usr/local/ctdlsupport)
+
+CTDL_DIALOG Location of the 'dialog' program (optional)
+
+ACT_AS_MTA When other MTA's (Postfix, etc.) are found, setup
+ will offer to disable them in order to keep them
+ from interfering with Citadel. Set this variable to
+ "yes" to always disable the other MTA without asking,
+ or to "no" to never disable the other MTA.
+
+CREATE_INITTAB_ENTRY Set to "yes" to automatically create an entry in
+ /etc/inittab to start the Citadel server, if needed.
+
+CREATE_XINETD_ENTRY Set to "yes" to automatically configure xinetd to route
+ telnet connections into a Citadel client, if needed.
+
+SYSADMIN_NAME Name of system administrator.
+
+NO_INIT_SCRIPTS Set to non-null to suppress the creation of init
+ scripts when running on a SysV-init based host.
--- /dev/null
+====== SMTP Client ======
+===== source files involved: =====
+
+event_client.h - AsyncIO structure, interfaces to event Queue
+
+
+modules/smtp/smtp_clienthandlers.h - SMTP-Client Context struct & functions.
+modules/smtp/smtpqueue.h - parsing of Queue mails & respective structures
+
+modules/smtp/serv_smtpqueue.c - queue handling executed by housekeeping thread; de/serialization of Queue-control-messages
+modules/smtp/serv_smtpeventclient.c - business logic
+modules/smtp/smtp_clienthandlers.c - Protocoll logic
+
+msgbase.c - writes spool messages + spoolcontrol files
+
+event_client.c - base event functions
+
+modules/eventclient/serv_eventclient.c - the event queue
+modules/c-ares-dns/serv_c-ares-dns.c - nameserver lookup
+
+===== Spool Controll =====
+Citadel stores two messages for mail deliveries; the spool control message format is described in delivery-list.txt; its generated in msgbase.c, and de/serialized in serv_smtpqueue.c.
+The for Spool-control structures are kept in memory as OneQueItem per Spool-Control-File, and MailQEntry per Recipient of the message.
+For each MailQEntry which is still to be delivered, one SmtpOutMsg (containing the AsyncIO context) is spawned. The Jobs are processed in paralell, each job finished triggers an update of the spool control message, so we don't accidently deliver a message twice to one recipient. If the last recipient has succeeded, or permanently failed, the spool-file and the mail-file are removed from the spool room.
+
+===== Business Logic for one SMTP Connection =====
+For each SMTP client one SmtpOutMsg is created as Context; we now live inside the IO-Eventloop. SMTP requires to attempt to connect different servers in different situations; not all errors from remote sides are permanent, or fatal for one delivery attempt. One delivery attempt may involve talking to several SMTP-Servers; maybe the first isn't active or currently not accepting messages.
+If citadel runs in relay mode, only the relays are connected; no MX lookup is done.
+
+ - MX nameserver lookup; may give several mailservers; IPv4 and IPv6 addresses may be there. If no MX is found, the server is contacted directly.
+ - Now we have a number of Mailservers we have to iterate over; This is the entry for all subsequent tries for delivery; FailOneAttempt() will decide what to do next.
+ - one server has to be resolved; A or AAAA record in case of IPv4 or IPv6
+ - we try to connect the server nonblocking; if we fail to connect in a given time range, the next server has to be tried.
+ - we now have to wait for the server greeting; we musn't say anything before the server sent a complete line. A timeout may occur, the next server has to be tried, if.
+ - Now the handlers start to control the logic. A timeout may occur between each of them; Servers may reply with fatal or non fatal or temporary error codes; depending on them
+ - we fail the conversation with this server, and try the next
+ - we fail this delivery attempt, and retry later in the next queue run.
+ - we succeed the delivery of this message.
--- /dev/null
+SYSTEM TESTING PROPOSAL
+
+This document is intended as a discussion of possible automated tests. It does
+not describe any existing tests.
+
+
+---
+
+
+First we should create a client that leverages expect (or something similar) to
+perform automated testing of the client interface. Tests can be written as
+expect scripts.
+
+Each system being tested will need to create an aide level user for the test
+client to connect as.
+
+The test client will create another user to carry out the tests. This allows the
+aide level user to vary the level of the test user and check the access level
+code.
+
+----
+
+
+For a first step each test site should create a test user that can send internet
+mail.
+This test user needs some sieve rules to forward mail around (eventually this
+will be created automatically by the test client). These rules will forward mail
+to other test users at other sites participating in the test system and to a
+networked room.
+Each system participating in the test should share some rooms.
+
+The idea is:
+ 1. A test site posts a message to its test user using citmail or some
+other email prog.
+ 2. The sieve rules forward the message to each of the other test users
+at the other sites.
+ 3. The sieve rules for the other test users detect that the message was
+forwarded to them and they file it into a networked room
+ 4. By virtue of the networked room the message returns to the
+originating system where the administrator can see it.
+
+Once I (davew) have written my module to alter the message body we can have it
+add text to the message to indicate the full path of the message.
+
+
--- /dev/null
+How "views" work in Citadel
+---------------------------
+
+ There's no need to be rigid and stupid about how different rooms are presented
+in a Citadel client. And we don't enforce things either. But there's a need
+to make things look the way the user wants to see them. For example, we might
+always choose to see a room full of private mail as a summary (one line per
+message) rather than always dumping out the entire contents like we do on a
+typical message room. An address book room might look better as a tabbed view
+or something, rather than as a bunch of messages with vCards attached to them.
+
+ This is why we define "views" for a room. It gives the client software a
+hint as to how to display the contents of a room. This is kept on a per-user
+basis by storing it in the 'visit' record for a particular room/user
+combination. It is visit.v_view and is an integer. Naturally, there also
+needs to be a default, for users who have never visited the room before. This
+is in the room record as room.QRdefaultview (and is also an integer).
+
+ In recent versions of Citadel, the view for a room also defines when and how
+it is indexed. For example, mailboxes and bulletin boards don't need to have
+an euid index, but address books and calendars do.
+
+ The values currently defined are:
+
+#define VIEW_BBS 0 /* Bulletin board view */
+#define VIEW_MAILBOX 1 /* Mailbox summary */
+#define VIEW_ADDRESSBOOK 2 /* Address book view */
+#define VIEW_CALENDAR 3 /* Calendar view */
+#define VIEW_TASKS 4 /* Tasks view */
+#define VIEW_NOTES 5 /* Notes view */
+#define VIEW_WIKI 6 /* Wiki view */
+
+++ /dev/null
- The citadel.pam configuration file has been updated for Red Hat 7.1.
-If you have such a system, it should Just Work; if you don't, you're going to
-have to tweak it, preferably BEFORE you do a make install. See below. Even
-if you have Red Hat 7.1, you should look at the file anyway and understand how
-it affects your system security. The original PAM.txt is included below:
-
-
- Citadel 5.53 and later include support for Pluggable Authentication
-Modules (PAM.) However, we don't recommend enabling this feature (./configure
---with-pam) unless you understand exactly how it will affect your system's
-security. Specifically, the system administrator must supply a configuration
-for every authentication service which uses PAM. We have automated this process
-for Linux by supplying a file which is placed in /etc/pam.d during the
-installation process, but not on other systems, for 2 reasons:
-
- 1) Other systems don't have /etc/pam.d; instead they use one big
-configuration file, usually /etc/pam.conf. It's not quite as trivial to
-automatically modify this file in a safe and secure fashion.
-
- 2) Other systems have a different set of available authentication
-modules; they are likely to lack all three of the ones we use in the Linux
-configuration. We don't have enough information about the features of the
-authentication modules on other platforms to be able to provide secure
-configurations.
-
- That said, the configuration we've provided should work on at least
-Red Hat Linux 4.2-5.2, (although we don't recommend building Citadel on Red
-Hat 4.x due to libc thread-safety issues) and if you understand PAM
-configuration on your system, feel free to build Citadel with PAM support,
-as long as you realize that YOU'RE ON YOUR OWN.
+++ /dev/null
-
-BUILDING THE CITADEL SYSTEM WITH PRIVATE LIBRARIES
---------------------------------------------------
-
-This method is known to work on Linux and FreeBSD systems. It is a way of
-building the Citadel system with its own private copies of Berkeley DB and
-libical. This avoids conflicts with any other version of these libraries
-which may already exist on your host system.
-
-You can perform your builds in any directory (such as /usr/src or even your
-home directory). The target directories will be:
-
-* /usr/local/citadel (Citadel server, text client, utilities)
-* /usr/local/webcit (the WebCit front end)
-* /usr/local/ctdlsupport (libical, libdb, and their headers, etc.)
-
-The behavior of Easy Install is based upon this methodology.
-
-1. Unpack the Berkeley DB tarball. chdir to the "build_unix" directory
- and build the package with these commands:
-
- ../dist/configure --prefix=/usr/local/ctdlsupport
- make
- make install
-
-2. Unpack the libical tarball and build it with these commands:
-
- ./configure --prefix=/usr/local/ctdlsupport
- make
- make install
-
-3. Set these environment variables for the rest of the build.
- (This assumes you're using the 'bash' shell. Otherwise you're on your own.)
-
-export CFLAGS='-I/usr/local/ctdlsupport/include'
-export CPPFLAGS='-I/usr/local/ctdlsupport/include'
-export LDFLAGS='-L/usr/local/ctdlsupport/lib -Wl,--rpath -Wl,/usr/local/ctdlsupport/lib'
-
- -L tells the build system where to find libraries during the build process,
-while -Wl,--rpath inserts that same library path into the Citadel binaries
-so they know where to find the libs at runtime. Since we can't depend on
-the correct version of Berkeley DB already being on the system, this is how
-we carry our own along. It's better than static linking everything.
-
-4. If LDAP support is required, unpack the OpenLDAP tarball and build with:
-
-./configure --prefix=/usr/local/ctdlsupport --with-db=/usr/local/ctdlsupport
-make
-make depend
-make install
-
-5. Now you're ready to build the Citadel server. Unpack the Citadel tarball
- and build it with these commands:
-
-./configure --prefix=/usr/local/citadel --with-db=/usr/local/ctdlsupport
-make
-make install
-
- ** NOTE: if you already have a Citadel server in /usr/local/citadel, change
- the 'make install' to 'make upgrade'. And I'm sure you remembered to shut
- down your Citadel service and make a backup before starting, right?
-
-6. Finally, unpack the WebCit tarball and build it with these commands:
-
-./configure --prefix=/usr/local/webcit
-make
-make install
-
-
-
-All of your software is now in place. Run /usr/local/citadel/setup to configure
-the Citadel server, and /usr/local/webcit/setup to configure WebCit. If you are
-on a Linux machine, setup will tweak /etc/inittab to automatically start the
-services. If you are on a FreeBSD machine, you will need to manually configure
-your startup scripts to start the services.
+++ /dev/null
-Some notes on the build process...
-
- Oct 28 01:57 1998 from LoanShark @uncnsrd
-
- autodependency generation is implemented by generating a bunch of .d
-files (with another suffix rule) using $(CC) - M and "-include"-ing them
-from the main Makefile. the .d files are made to depend on the .c files
-they correspond to, and the referenced header files, so they're updated as
-needed. the only version of make that I know for sure works with this is
-GNU make, but the Makefile should still work on other versions of Unix
-make, just without the autodependency stuff.
-
-
- Oct 28 20:49 1998 from LoanShark @uncnsrd
-one thing I forgot to mention about the autodependency generation: for a
-file to be included in the autodepend process, it must be listed in the
-SOURCES macro near the top of Makefile.in.
- also, I've added an 'install' target to the makefile. this will make it
-possible to build RPM's, and bring the installation process closer to that
-of other packages, which if you're using the new configure script goes
-like this:
- ./configure [--enable-ansi-color] [--disable-autologin] [--prefix=/foo]
-
- --prefix specifies the location Citadel will run from,
-/usr/local/citadel by default.
- to build a highly optimized version without debugging symbols, you could
- do something like this (with the bourne shell):
- CC=egcs CFLAGS='-O6 -fomit-frame-pointer' ./configure
- after configuring, simply type `make', then su to a user who has
-appropriate permissions, and `make install'. then run setup as usual.
- are there any other areas that need to be cleared up?
-
+++ /dev/null
-====== How Threads & IO for clients work ======
-Citserver is a multi thread process. It has four types of threads:
- - Workers - these are anonymous threads which can do any of the server related functions like Imapserver; smtpserver; citadel protocol and so on.
- - Housekeeping - once per minute one Worker becomes the Housekeeping thread which does recurrent jobs like database maintenance and so on.
- - IO - one thread is spawned to become the IO-Thread at startup. Inside it lives the event-queue which is implemented by libev.
- - DB-IO - one thread is spawned to become the database IO thread. Inside it lives another event-queue, which is also implemented by libev.
-
-
-===== Worker Threads =====
-The workers live blocking on selects to server sockets. Once IO on a server socket becomes available, one thread awakes, and takes the task.
-It is then assigned a CitContext structure on a Thread global var (accessible via the CC macro), which then becomes his personality and controls what the thread is going to do with the IO,
-and which server code (Imap/SMTP/Citadel/...) is going to evaluate the IO and reply to it.
-
-===== Housekeeping Thread =====
-Once every minute one Worker becomes something special: the Housekeeping Thread. Only one Houskekeeper can exist at once. As long as one Housekeeper is running no other is started.
-Jobs done by the Housekeeper are registered via the <>-API. The current list:
-
- - DB maintenance; flush redo logs etc.
- - fulltext index (this one is a little problematic here, since it may be long running and block other housekeeping tasks for a while)
- - Networking-Queue
- - Citadel Networking Queue aggregator (Blocks NTT)
- - Citadel Inbound Queue operator (Blocks NTT)
- - Citadel Networking Queue
- - SMTP-Client-Queue
- - RSS-Aggregator scheduling
- - HTTP-Message Notification Queue (Extnotify)
- - POP3Client Queue
-
-The queues operate over their respective queue; which may be implemented with a private room (smtp-Queue, Extnotify Client), per room configs (RSS-Client; POP3Client), or an aggregate of Rooms/Queue/Roomconfigs (Citadel Networker).
-
-
-==== Networking Queue ====
-The networking queue opperats over an aggregate of per room network configs and messages in rooms that are newer than a remembered whatermark (netconfigs.txt#lastsent).
-All messages Newer than the watermark are then processed:
- - for all mailinglist recipients an entry is added to the mailqueue
- - for all mailinglist digest recipients a digest file is written, or once per day scheduled to
- - for all Citadel networking nodes a spool file is written for later evaluation.
-
-==== Citadel Networking Queue aggregator ====
-The aggregator iterates over the spool directory, and combines all per message spool files to one big file which is to be sent by the Citadel Networking Queue later.
-It blocks the NTT-List for the remote party while doing so; if the NTT-List is blocked the file is skipped for next operation. Once all files are successfully processed, all files not associated with a networking config are flushed.
-
-==== Citadel Inbound Queue operator ====
-The inbound queue operator scans for new inbound spool files, processes them, splits them into messages which are then posted into their respective rooms.
-
-
-==== other Queues ====
-Once these Queue-operators have identified a job to be performed like
- - talking to another Citadel Node to transfer spool files back and forth
- - sending mails via SMTP
- - polling remote POP3-Servers for new mails
- - fetching RSS-Feeds via HTTP
- - Notifying external facilities for new messages in inboxes via HTTP
-
-They create a Context for that Job (AsyncIO struct) plus their own Private data, plus a CitContext which is run under the privileges of a SYS_* user.
-HTTP Jobs just schedule the HTTP-Job, and are then called back on their previously registered hook once the HTTP-Request is done.
-
-All other have to implement their own IO controled business logic.
-
-
-===== IO-Thread =====
-The IO-Event-queue lives in this thread. Code running in this thread has to obey several rules.
- - it mustn't do blocking operations (like disk IO or nameserver lookups other than the c-ares facilities)
- - it mustn't use blocking Sockets
- - other threads mustn't access memory belonging to an IO job
- - New IO contexts have to be registered via QueueEventContext() from other threads; Its callback are then called from whithin the event queue to start its logic.
- - New HTTP-IO contexts have to be registered via QueueCurlContext()
- - once you've registered a context, you must not access its memory anymore to avoid race conditions.
-
-==== Logic in event_client ====
-InitIOStruct() which prepares an AsyncIO struct for later use before using QueueEventContext() to activate it.
-Here we offer the basics for I/O; Hostname lookus, connecting remote parties, basic read and write operation control flows.
-This logic communicates with the next layer via callback hooks. Before running hooks or logging CC is set from AsyncIO->CitContext
-Most clients are stacked that way:
-
-<event base> --- <IO grammer logic> -- <state machine dispatcher> -- <state hooks>
-
-=== event base ===
-While the event base is responsible for reading and writing, it just does so without the understanding of the protocol it speaks. It at most knows about chunksises to send/read, or when a line is sent / received.
-=== IO grammer logic ===
-The IO grammer logic is implemented for each client. It understands the grammer of the protocol spoken; i.e. the SMTP implementation knows that it has to read status lines as long as there are 3 digits followed by a dash (i.e. in the ehlo reply). Once the grammer logic has decided a communication state is completed, it calls the next layer.
-
-=== state machine dispatcher ===
-The state machine dispatcher knows which part of the application logic is currently operated. Once its called, it dispatches to the right read/write hook. Read/write hooks may in term call the dispatcher again, if they want to divert to another part of the logic.
-
-=== state hooks ===
-The state hooks or handlers actualy PARSE the IO-Buffer and evaluate what next action should be taken;
-There could be an error from the remote side so the action has to be aborted, or usually simply the next state can be reached.
-
-
-=== surrounding businesslogic ===
-Depending on the protocol there has to be surrounding business logic, which handles i.e. nameserver lookups, which ips to connect, what to do on a connection timeout, or doing local database IO.
-
-===== DB-IO-Thread =====
-Since lookups / writes to the database may take time, this mustn't be done inside of the IO-Eventloop. Therefore this other thread & event queue is around.
-Usually a context should respect that there may be others around also wanting to do DB IO, so it shouldn't do endless numbers of requests.
-While its ok to do _some_ requests in a row (like deleting some messages, updating messages, etc.),
-loops over unknown numbers of queries should be unrolled and be done one query at a time (i.e. looking up messages in the 'already known table')
-
-=== Transitioning back & forth ===
-If the application logic has found that a transition is to be made from the event IO to the DB IO thread. Do so by calling DBQueueEventContext() (DB -> IO) EventQueueDBOperation() (IO->DB) and pass its return value up the row to enable the controll logic to do the actual transition for you.
+++ /dev/null
- Citadel Server Extension API Documentation
- ---------------------------------------------
-
- This is a VERY INCOMPLETE documentation of the API for extending the
-Citadel server using dynamically loaded modules. It really isn't an API at
-all, but rather a list of some of the functions available in the server which
-are likely to be of use to module writers.
-
- The current trend is to move as much stuff as possible out of the server
-proper and into loadable modules. This makes the code much easier to read and
-understand.
-
- Expect this document to become more complete over time, as both the API and
-the person documenting it have a chance to mature a bit. :)
-
-
-
- USER RELATED FUNCTIONS
- ----------------------
-
- The fundamental user data is stored in "struct ctdluser" which is defined
-in citadel.h. The following functions are available:
-
-
- int getuser(struct ctdluser *usbuf, char name[])
-
- Given the name of a requested user and a buffer to store the user
-record in, getuser() will search the userlog for the named user and load its
-data into the buffer. getuser() returns 0 upon success or a nonzero error
-code if the requested operation could not be performed.
-
-
- void putuser(struct ctdluser *usbuf, char *name)
-
- After reading in a user record with getuser() and perhaps modifying the data
-in some way, a program may use putuser() to write it back to disk.
-
-
- int lgetuser(struct ctdluser *usbuf, char *name)
- void lputuser(struct ctdluser *usbuf, char *name)
-
- If critical-section operation is required, this pair of calls may be used.
-They function the same as getuser() and putuser(), except that lgetuser()
-locks the user file immediately after retrieving the record and lputuser()
-unlocks it. This will guarantee that no other threads manipulate the same
-user record at the same time.
-
- NOTE: do NOT attempt to combine the locking lgetuser/lputuser calls with any
-other locking calls in this API. Attempting to obtain concurrent locks on
-multiple files may result in a deadlock condition which would freeze the
-entire server.
-
-
- void ForEachUser(void (*CallBack)(struct ctdluser *EachUser))
-
- This allows a user-supplied function to be called once for each user on
-the system. The user-supplied function will be called with a pointer to a
-user structure as its only argument.
-
-
- int getuserbynumber(struct ctdluser *usbuf, long int number)
-
- getuserbynumber() functions similarly to getuser(), except that it is
-supplied with a user number rather than a name. Calling this function
-results in a sequential search of the user file, so use it sparingly if at
-all.
-
-
- int purge_user(char *pname)
-
- This function deletes the named user off the system and erases all related
-objects: bio, photo, etc. It returns 0 upon success or a nonzero error code
-if the requested operation could not be performed.
-
-
-
- HOW TO REGISTER FUNCTION HOOKS
- ------------------------------
-
- The truly powerful part of the Citadel API is the ability for extensions to
-register "hooks" -- user-supplied functions will be called while the server
-is performing various tasks. Here are the API calls to register hooks:
-
-
- void CtdlRegisterProtoHook(void (*handler)(char *), char *cmd, char *desc)
- void CtdlUnregisterProtoHook(void (*handler)(char *), char *cmd)
-
- CtdlRegisterProtoHook() adds a new server command to the system. The
-handler function should accept a single string parameter, which will be set
-to a string containing any parameters the client software sent along with
-the server command. "cmd" should be the four-character mnemonic the server
-command is known by, and "desc" is a description of the new command.
-
- CtdlUnregisterProtoHook() removes a server command from the system. It
-must be called with the same handler and cmd which were previously registered.
-
-
- void CtdlRegisterCleanupHook(void *fcn_ptr)
- void CtdlUnregisterCleanupHook(void *fcn_ptr)
-
- CtdlRegisterCleanupHook() registers a new function to be called whenever the
-server is shutting down. Cleanup functions accept no parameters.
-
- CtdlUnregsiterCleanupHook() removes a cleanup function from the system. It
-must be called with the same fcn_ptr which was previously registered.
-
-
-void CtdlRegisterSessionHook(void *fcn_ptr, int EventType)
-void CtdlUnregisterSessionHook(void *fcn_ptr, int EventType)
-
- CtdlRegisterSessionHook() registers a session hook. Session hooks accept
-no parameters. There are multiple types of session hooks; the server
-extension registers which one it is interested in by setting the value of
-EventType. The available session hook types are:
-
-#define EVT_STOP 0 /* Session is terminating */
-#define EVT_START 1 /* Session is starting */
-#define EVT_LOGIN 2 /* A user is logging in */
-#define EVT_NEWROOM 3 /* Changing rooms */
-#define EVT_LOGOUT 4 /* A user is logging out */
-#define EVT_SETPASS 5 /* Setting or changing password */
-#define EVT_CMD 6 /* Called after each server command */
-#define EVT_RWHO 7 /* An RWHO command is being executed */
-#define EVT_ASYNC 8 /* Doing asynchronous message */
-
-#define EVT_TIMER 50 /* Timer events are called once per minute */
-#define EVT_HOUSE 51 /* Housekeeping event */
-
- CtdlUnregisterSessionHook() removes a session hook. It must be called with
-the same fcn_ptr and EventTYpe which were previously registered.
-
-
-void CtdlRegisterUserHook(void *fcn_ptr, int EventType)
-void CtdlUnregisterUserHook(void *fcn_ptr, int EventType)
-
- CtdlRegisterUserHook() registers a user hook. User hooks accept two
-parameters: a string pointer containing the user name, and a long which *may*
-contain a user number (only applicable for certain types of hooks). The
-available user hook types are:
-
-#define EVT_PURGEUSER 100 /* Deleting a user */
-#define EVT_OUTPUTMSG 101 /* Outputting a message */
-
- CtdlUnregisterUserHook() removes a user hook from the system. It must be
-called with the same fcn_ptr and EventType which were previously registered.
-
-
- void CtdlRegisterLogHook(void (*fcn_ptr) (char *), int loglevel)
- void CtdlUnregisterLogHook(void (*fcn_ptr) (char *), int loglevel)
-
- CtdlRegisterLogHook() adds a new logging function to the system. The
-handler function should accept a single string as a parameter. Logging
-functions can be used to implement additional logging facilities in
-addition to the Citadel trace file, which is output on stderr, or
-redirected to a file with the -t command line option. The handler
-function will be called if the loglevel is greater than or equal to
-loglevel.
-
- Security considerations: Logs may contain plain text passwords and
-other sensitive information. It is your responsibility to ensure that
-your logs have appropriate access protection. The Citadel trace file
-is readable only by the superuser when the -t option is used.
-
- CtdlUnregisterLogHook() removes a logging function from the system. It
-must be called with the same fcn_ptr and loglevel which were previously
-registered.
-
-
- void CtdlRegisterMessageHook(int (*handler) (struct CtdlMessage *),
- int EventType)
- void CtdlUnregisterMessageHook(int (*handler) (struct CtdlMessage *),
- int EventType)
-
-
- CtdlRegisterMessageHook() registers a function with the message
-handling subsystem. This function will be called with one parameter,
-a pointer to a CtdlMessage structure, when the message triggers an event
-of type EventType. The registered function should return non zero if it
-has handled the event to prevent other hook functions from also processing
-the event.
-
- CtdlUnregisterMessageHook() removes a function from the list of registered
-message handlers. To successfully remove a function registered with
-CtdlRegisterMessageHook() CtdlUnregisterMessageHook() must be called with
-the same parameters.
-
- Possible values for EventType are:
- EVT_BEFOREREAD Called after a message is loaded from disk but before
-it is presented for reading.
- EVT_BEFORESAVE Called before the message is saved to disk. returning
-non zero for this event will prevent the message being saved to disk in the
-normal manner.
- EVT_AFTERSAVE Called after the message is saved to disk but before
-any IGnet spooling is carried out.
- EVT_SMTPSCAN Called during the SMTP reception of a message after the
-message is received and before the response is sent to the sender. This is
-intended for spam filters and virus checkers. A positive return code will
-cause the message to be rejected by the SMTP server.
-
-
- void CtdlRegisterRoomHook(int (*fcn_ptr) (char *))
- void CtdlUnregisterRoomHook(int (*fcn_ptr) (char *))
-
- Register or remove a function with the room processing system.
-Registered functions are called in the order they are registered when a message
-is added to a room. This allows modules such as Sieve to process new messages
-appearing in a room.
-
-
- void CtdlRegisterXmsgHook(int (*fcn_ptr) (char *, char *, char *), int order)
- void CtdlUnregisterXmsgHook(int (*fcn_ptr) (char *, char *, char *), int order)
-
- Please write documentation for me!
-
-
- void CtdlRegisterServiceHook(int tcp_port, char *sockpath,
- void (*h_greeting_function) (void),
- void (*h_command_function) (void))
- void CtdlUnregisterServiceHook(int tcp_port, char *sockpath,
- void (*h_greeting_function) (void),
- void (*h_command_function) (void))
-
- Please write documentation for me!
-
-
- FUNCTIONS WHICH MANIPULATE USER/ROOM RELATIONSHIPS
-
- void CtdlGetRelationship(struct visit *vbuf,
- struct ctdluser *rel_user,
- struct ctdlroom *rel_room);
- void CtdlSetRelationship(struct visit *newvisit,
- struct ctdluser *rel_user,
- struct ctdlroom *rel_room);
-
- These functions get/set a "struct visit" structure which may contain
-information about the relationship between a user and a room. Specifically:
-
-struct visit {
- char v_roomname[20];
- long v_generation;
- long v_lastseen;
- unsigned int v_flags;
- };
-
-#define V_FORGET 1 /* User has zapped this room */
-#define V_LOCKOUT 2 /* User is locked out of this room */
-#define V_ACCESS 4 /* Access is granted to this room */
-
- Don't change v_roomname or v_generation; they're used to identify the room
-being referred to. A room is unique to the system by its combination of room
-name and generation number. If a new room is created with the same name as
-a recently deleted room, it will have a new generation number, and therefore
-stale "visit" records will not be applied (and will eventually be purged).
-
- v_lastseen contains the number of the newest message the user has read in
-this room. Any existing messages higher than this number can be considered
-as "new messages."
-
- v_flags contains information regarding access to the room.
-
-
-
- int CtdlRoomAccess(struct ctdlroom *roombuf, struct ctdluser *userbuf)
-
- This is a convenience function which uses CtdlGetRelationship() to determine
-whether a user has access to a room. It returns a bucket of bits which may
-contain:
-
-#define UA_INUSE 1 /* Room exists */
-#define UA_KNOWN 2 /* Room is in user's Known list */
-#define UA_GOTOALLOWED 4 /* User may <.G>oto this room */
-#define UA_HASNEWMSGS 8 /* Room contains new messages */
-#define UA_ZAPPED 16 /* User has forgotten this room */
-
-
-
-
- ROOM RELATED FUNCTIONS
- ----------------------
-
-
-unsigned create_room(char *new_room_name,
- int new_room_type,
- char *new_room_pass,
- int new_room_floor,
- int really_create)
-
- This function is used to create a new room. new_room_name should be set to
-the desired name for the new room.
-
- new_room_type should be set to one of the following values:
- 0 = public room
- 1 = guess-name room
- 2 = passworded room
- 3 = invitation-only room
- 4 = personal (mailbox) room
- 5 = personal (mailbox) room, and new_room_name already contains
- the namespace prefix (use with caution!)
-
- new_room_pass should be set to the desired password for the room (obviously
-this is only valid for passworded rooms).
-
- If the room is really to be created, set really_create to 1. Otherwise, the
-caller may merely check to see if it's possible to create the room without
-actually creating it by setting really_create to 0.
-
- create_room() returns the flags associated with the new room (as in the
-data structure item room.QRflags). If the room cannot be created (for
-example, a room with the name already exists), it returns 0.
-
-
+++ /dev/null
-The totally incomplete guide to Citadel internals
------------------------------------------------------
------------------------------------------------------
-
-Citadel has evolved quite a bit since its early days, and the data structures
-have evolved with it. This document provides a rough overview of how the
-system works internally. For details you're going to have to dig through the
-code, but this'll get you started.
-
-
-DATABASE TABLES
----------------
-As you probably already know by now, Citadel uses a group of tables stored
-with a record manager (usually Berkeley DB). Since we're using a record
-manager rather than a relational database, all record structures are managed
-by Citadel. Here are some of the tables we keep on disk:
-
-
-USER RECORDS
-------------
-This table contains all user records. It's indexed by
-user name (translated to lower case for indexing purposes). The records in
-this file look something like this:
-
-struct ctdluser { /* User record */
- int version; /* Cit vers. which created this rec */
- uid_t uid; /* Associate with a unix account? */
- char password[32]; /* password (for Citadel-only users)*/
- unsigned flags; /* See US_ flags below */
- long timescalled; /* Total number of logins */
- long posted; /* Number of messages posted (ever) */
- CIT_UBYTE axlevel; /* Access level */
- long usernum; /* User number (never recycled) */
- time_t lastcall; /* Last time the user called */
- int USuserpurge; /* Purge time (in days) for user */
- char fullname[64]; /* Name for Citadel messages & mail */
-};
-
- Most fields here should be fairly self-explanatory. The ones that might
-deserve some attention are:
-
- - *uid* -- if uid is not the same as the *unix uid* Citadel is running as, then the
- account is assumed to belong to the user on the underlying Unix system with
- that uid. This allows us to require the user's OS password instead of having
- a separate Citadel password.
-
- - *usernum* -- these are assigned sequentially, and **NEVER REUSED**. This is
- important because it allows us to use this number in other data structures
- without having to worry about users being added/removed later on, as you'll
- see later in this document.
-
-
-ROOM RECORDS
-------------
-These are room records. There is a room record for every room on the
-system, public or private or mailbox. It's indexed by room name (also in
-lower case for easy indexing) and it contains records which look like this:
-
- struct ctdlroom {
- char QRname[ROOMNAMELEN]; /* Name of room */
- char QRpasswd[10]; /* Only valid if it's a private rm */
- long QRroomaide; /* User number of room aide */
- long QRhighest; /* Highest message NUMBER in room */
- time_t QRgen; /* Generation number of room */
- unsigned QRflags; /* See flag values below */
- char QRdirname[15]; /* Directory name, if applicable */
- long QRinfo; /* Info file update relative to msgs*/
- char QRfloor; /* Which floor this room is on */
- time_t QRmtime; /* Date/time of last post */
- struct ExpirePolicy QRep; /* Message expiration policy */
- long QRnumber; /* Globally unique room number */
- char QRorder; /* Sort key for room listing order */
- unsigned QRflags2; /* Additional flags */
- int QRdefaultview; /* How to display the contents */
- };
-
-Again, mostly self-explanatory. Here are the interesting ones:
-
-*QRnumber* is a globally unique room ID, while QRgen is the "generation number"
-of the room (it's actually a timestamp). The two combined produce a unique
-value which identifies the room. The reason for two separate fields will be
-explained below when we discuss the visit table. For now just remember that
-*QRnumber* remains the same for the duration of the room's existence, and QRgen
-is timestamped once during room creation but may be restamped later on when
-certain circumstances exist.
-
-FLOORTAB
---------
-Floors. This is so simplistic it's not worth going into detail about, except
-to note that we keep a reference count of the number of rooms on each floor.
-
-MSGLISTS
---------
-Each record in this table consists of a bunch of message numbers
-which represent the contents of a room. A message can exist in more than one
-room (for example, a mail message with multiple recipients -- 'single instance
-store'). This table is never, ever traversed in its entirety. When you do
-any type of read operation, it fetches the msglist for the room you're in
-(using the room's ID as the index key) and then you can go ahead and read
-those messages one by one.
-
-Each room is basically just a list of message numbers. Each time
-we enter a new message in a room, its message number is appended to the end
-of the list. If an old message is to be expired, we must delete it from the
-message base. Reading a room is just a matter of looking up the messages
-one by one and sending them to the client for display, printing, or whatever.
-
-
-VISIT
------
-This is the tough one. Put on your thinking cap and grab a fresh cup of
-coffee before attempting to grok the visit table.
-
-This table contains records which establish the relationship between users
-and rooms. Its index is a hash of the user and room combination in question.
-When looking for such a relationship, the record in this table can tell the
-server things like "this user has zapped this room," "this user has access to
-this private room," etc. It's also where we keep track of which messages
-the user has marked as "old" and which are "new" (which are not necessarily
-contiguous; contrast with older Citadel implementations which simply kept a
-"last read" pointer).
-
-
-Here's what the records look like:
-
- struct visit {
- long v_roomnum;
- long v_roomgen;
- long v_usernum;
- long v_lastseen;
- unsigned int v_flags;
- char v_seen[SIZ];
- int v_view;
- };
-
- #define V_FORGET 1 /* User has zapped this room */
- #define V_LOCKOUT 2 /* User is locked out of this room */
- #define V_ACCESS 4 /* Access is granted to this room */
-
-This table is indexed by a concatenation of the first three fields. Whenever
-we want to learn the relationship between a user and a room, we feed that
-data to a function which looks up the corresponding record. The record is
-designed in such a way that an "all zeroes" record (which is what you get if
-the record isn't found) represents the default relationship.
-
-With this data, we now know which private rooms we're allowed to visit: if
-the *V_ACCESS* bit is set, the room is one which the user knows, and it may
-appear in his/her known rooms list. Conversely, we also know which rooms the
-user has zapped: if the *V_FORGET* flag is set, we relegate the room to the
-zapped list and don't bring it up during new message searches. It's also
-worth noting that the *V_LOCKOUT* flag works in a similar way to administratively
-lock users out of rooms.
-
-Implementing the "cause all users to forget room" command, then, becomes very
-simple: we simply change the generation number of the room by putting a new
-timestamp in the *QRgen* field. This causes all relevant visit records to
-become irrelevant, because they appear to point to a different room. At the
-same time, we don't lose the messages in the room, because the msglists table
-is indexed by the room number (*QRnumber*), which never changes.
-
-*v_seen* contains a string which represents the set of messages in this room
-which the user has read (marked as 'seen' or 'old'). It follows the same
-syntax used by IMAP and NNTP. When we search for new messages, we simply
-return any messages that are in the room that are **not** represented by this
-set. Naturally, when we do want to mark more messages as seen (or unmark
-them), we change this string. Citadel BBS client implementations are naive
-and think linearly in terms of "everything is old up to this point," but IMAP
-clients want to have more granularity.
-
-
-DIRECTORY
----------
-This table simply maps Internet e-mail addresses to Citadel network addresses
-for quick lookup. It is generated from data in the Global Address Book room.
-
-USETABLE
---------
-This table keeps track of message ID's of messages arriving over a network,
-to prevent duplicates from being posted if someone misconfigures the network
-and a loop is created. This table goes unused on a non-networked Citadel.
-
-THE MESSAGE STORE
------------------
-This is where all message text is stored. It's indexed by message number:
-give it a number, get back a message. Messages are numbered sequentially, and
-the message numbers are never reused.
-
-We also keep a "metadata" record for each message. This record is also stored
-in the msgmain table, using the index (0 - msgnum). We keep in the metadata
-record, among other things, a reference count for each message. Since a
-message may exist in more than one room, it's important to keep this reference
-count up to date, and to delete the message from disk when the reference count
-reaches zero.
-
-#Here's the format for the message itself:
-
- - Each message begins with an 0xFF 'start of message' byte.
-
- - The next byte denotes whether this is an anonymous message. The codes
- available are *MES_NORMAL*, *MES_ANON*, or *MES_AN2* (defined in citadel.h).
-
- - The third byte is a "message type" code. The following codes are defined:
- - 0 - "Traditional" Citadel format. Message is to be displayed "formatted."
- - 1 - Plain pre-formatted ASCII text (otherwise known as text/plain)
- - 4 - MIME formatted message. The text of the message which follows is
- expected to begin with a "Content-type:" header.
-
- - After these three opening bytes, the remainder of
- the message consists of a sequence of character strings. Each string
- begins with a type byte indicating the meaning of the string and is
- ended with a null. All strings are printable ASCII: in particular,
- all numbers are in ASCII rather than binary. This is for simplicity,
- both in implementing the system and in implementing other code to
- work with the system. For instance, a database driven off Citadel archives
- can do wildcard matching without worrying about unpacking binary data such
- as message ID's first. To provide later downward compatability
- all software should be written to IGNORE fields not currently defined.
-
-
-#The type bytes currently defined are:
-
-
-| BYTE | Enum | NW | Mnemonic | Enum / Comments
-|-------|-------------------|------|----------------|---------------------------------------------------------
-| A | eAuthor | from | Author | *eAuthor*
-| | | | | Name of originator of message.
-| B | eBig\_message | | Big message | *eBig\_message*
-| | | | | This is a flag which indicates that the message is
-| | | | | big, and Citadel is storing the body in a separate
-| | | | | record. You will never see this field because the
-| | | | | internal API handles it.
-| C | eRemoteRoom | | RemoteRoom | *eRemoteRoom*
-| | | | | when sent via Citadel Networking, this is the room
-| | | | | its going to be put on the remote site.
-| D | eDestination | | Destination | *eDestination*
-| | | | | Contains name of the system this message should
-| | | | | be sent to, for mail routing (private mail only).
-| E | eExclusiveID | exti | Exclusive ID | *eExclusiveID*
-| | | | | A persistent alphanumeric Message ID used for
-| | | | | network replication. When a message arrives that
-| | | | | contains an Exclusive ID, any existing messages which
-| | | | | contain the same Exclusive ID and are *older* than this
-| | | | | message should be deleted. If there exist any messages
-| | | | | with the same Exclusive ID that are *newer*, then this
-| | | | | message should be dropped.
-| F | erFc822Addr | rfca | rFc822 address | *erFc822Addr*
-| | | | | For Internet mail, this is the delivery address of the
-| | | | | message author.
-| H | eHumanNode | hnod | Human node name| *eHumanNode*
-| | | | | Human-readable name of system message originated on.
-| I | emessageId | msgn | Message ID | *emessageId*
-| | | | | An RFC822-compatible message ID for this message.
-| J | eJournal | jrnl | Journal | *eJournal*
-| | | | | The presence of this field indicates that the message
-| | | | | is disqualified from being journaled, perhaps because
-| | | | | it is itself a journalized message and we wish to
-| | | | | avoid double journaling.
-| K | eReplyTo | rep2 | Reply-To | *eReplyTo*
-| | | | | the Reply-To header for mailinglist outbound messages
-| L | eListID | list | List-ID | *eListID*
-| | | | | Mailing list identification, as per RFC 2919
-| M | eMesageText | text | Message Text | *eMesageText*
-| | | | | Normal ASCII, newlines seperated by CR's or LF's,
-| | | | | null terminated as always.
-| N | eNodeName | node | Nodename | *eNodeName*
-| | | | | Contains node name of system message originated on.
-| O | eOriginalRoom | room | Room | *eOriginalRoom* - Room of origin.
-| P | eMessagePath | path | Path | *eMessagePath*
-| | | | | Complete path of message, as in the UseNet news
-| | | | | standard. A user should be able to send Internet mail
-| | | | | to this path. (Note that your system name will not be
-| | | | | tacked onto this until you're sending the message to
-| | | | | someone else)
-| R | eRecipient | rcpt | Recipient | *eRecipient* - Only present in Mail messages.
-| S | eSpecialField | spec | Special field | *eSpecialField*
-| | | | | Only meaningful for messages being spooled over a
-| | | | | network. Usually means that the message isn't really
-| | | | | a message, but rather some other network function:
-| | | | | -> "S" followed by "FILE" (followed by a null, of
-| | | | | course) means that the message text is actually an
-| | | | | IGnet/Open file transfer. (OBSOLETE)
-| | | | | -> "S" followed by "CANCEL" means that this message
-| | | | | should be deleted from the local message base once
-| | | | | it has been replicated to all network systems.
-| T | eTimestamp | time | date/Time | *eTimestamp*
-| | | | | Unix timestamp containing the creation date/time of
-| | | | | the message.
-| U | eMsgSubject | subj | sUbject | *eMsgSubject* - Optional.
-| | | | | Developers may choose whether they wish to
-| | | | | generate or display subject fields.
-| V | eenVelopeTo | nvto | enVelope-to | *eenVelopeTo*
-| | | | | The recipient specified in incoming SMTP messages.
-| W | eWeferences | wefw | Wefewences | *eWeferences*
-| | | | | Previous message ID's for conversation threading. When
-| | | | | converting from RFC822 we use References: if present, or
-| | | | | In-Reply-To: otherwise.
-| | | | | (Who in extnotify spool messages which don't need to know
-| | | | | other message ids)
-| Y | eCarbonCopY | cccc | carbon copY | *eCarbonCopY*
-| | | | | Optional, and only in Mail messages.
-| % | eHeaderOnly | nhdr | oNlyHeader | we will just be sending headers. for the Wire protocol only.
-| % | eFormatType | type | type | type of citadel message: (Wire protocol only)
-| | | | | FMT\_CITADEL 0 Citadel vari-format (proprietary)
-| | | | | FMT\_FIXED 1 Fixed format (proprietary)
-| | | | | FMT\_RFC822 4 Standard (headers are in M field)
-| % | eMessagePart | part | emessagePart | *eMessagePart* is the id of this part in the mime hierachy
-| % | eSubFolder | suff | eSubFolder | descend into a mime sub container
-| % | ePevious | pref | ePevious | exit a mime sub container
-| 0 | eErrorMsg | | Error | *eErrorMsg*
-| | | | | This field is typically never found in a message on
-| | | | | disk or in transit. Message scanning modules are
-| | | | | expected to fill in this field when rejecting a message
-| | | | | with an explanation as to what happened (virus found,
-| | | | | message looks like spam, etc.)
-| 1 | eSuppressIdx | | suppress index | *eSuppressIdx*
-| | | | | The presence of this field indicates that the message is
-| | | | | disqualified from being added to the full text index.
-| 2 | eExtnotify | | extnotify | *eExtnotify* - Used internally by the serv_extnotify module.
-| 3 | eVltMsgNum | | msgnum | *eVltMsgNum*
-| | | | | Used internally to pass the local message number in the
-| | | | | database to after-save hooks. Discarded afterwards.
-
-EXAMPLE
--------
-Let *<FF>* be a *0xFF* byte, and *<0>* be a null *(0x00)* byte. Then a message
-which prints as...
-
- Apr 12, 1988 23:16 From Test User In Network Test> @lifesys (Life Central)
- Have a nice day!
-
-might be stored as...
-
- <FF><40><0>I12345<0>Pneighbor!lifesys!test_user<0>T576918988<0> (continued)
- -----------|Mesg ID#|--Message Path---------------|--Date------
-
- AThe Test User<0>ONetwork Test<0>Nlifesys<0>HLife Central<0>MHave a nice day!<0>
- |-----Author-----|-Room name-----|-nodename-|Human Name-|--Message text-----
-
-Weird things can happen if fields are missing, especially if you use the
-networker. But basically, the date, author, room, and nodename may be in any
-order. But the leading fields and the message text must remain in the same
-place. The H field looks better when it is placed immediately after the N
-field.
-
-
-EUID (EXCLUSIVE MESSAGE ID'S)
------------------------------
-This is where the groupware magic happens. Any message in any room may have
-a field called the Exclusive message *ID*, or *EUID*. We keep an index in the
-table *CDB_EUIDINDEX* which knows the message number of any item that has an
-*EUID*. This allows us to do two things:
-
- - If a subsequent message arrives with the same *EUID*, it automatically
- *deletes* the existing one, because the new one is considered a replacement
- for the existing one.
- - If we know the *EUID* of the item we're looking for, we can fetch it by *EUID*
- and get the most up-to-date version, even if it's been updated several times.
-
-This functionality is made more useful by server-side hooks. For example,
-when we save a vCard to an address book room, or an iCalendar item to a
-calendar room, our server modules detect this condition, and automatically set
-the *EUID* of the message to the *UUID* of the *vCard* or *iCalendar* item.
-Therefore when you save an updated version of an address book entry or
-a calendar item, the old one is automatically deleted.
-
-NETWORKING (REPLICATION)
-------------------------
-Citadel nodes network by sharing one or more rooms. Any Citadel node
-can choose to share messages with any other Citadel node, through the sending
-of spool files. The sending system takes all messages it hasn't sent yet, and
-spools them to the recieving system, which posts them in the rooms.
-
-The *EUID* discussion above is extremely relevant, because *EUID* is carried over
-the network as well, and the replacement rules are followed over the network
-as well. Therefore, when a message containing an *EUID* is saved in a networked
-room, it replaces any existing message with the same *EUID* *on every node in
-the network*.
-
-Complexities arise primarily from the possibility of densely connected
-networks: one does not wish to accumulate multiple copies of a given
-message, which can easily happen. Nor does one want to see old messages
-percolating indefinitely through the system.
-
-This problem is handled by keeping track of the path a message has taken over
-the network, like the UseNet news system does. When a system sends out a
-message, it adds its own name to the bang-path in the *<P>* field of the
-message. If no path field is present, it generates one.
-
-With the path present, all the networker has to do to assure that it doesn't
-send another system a message it's already received is check the <P>ath field
-for that system's name somewhere in the bang path. If it's present, the system
-has already seen the message, so we don't send it.
-
-We also keep a small database, called the "use table," containing the ID's of
-all messages we've seen recently. If the same message arrives a second or
-subsequent time, we will find its ID in the use table, indicating that we
-already have a copy of that message. It will therefore be discarded.
-
-The above discussion should make the function of the fields reasonably clear:
-
- o Travelling messages need to carry original message-id, system of origin,
- date of origin, author, and path with them, to keep reproduction and
- cycling under control.
-
-(Uncoincidentally) the format used to transmit messages for networking
-purposes is precisely that used on disk, serialized. The current
-distribution includes serv_network.c, which is basically a database replicator;
-please see network.txt on its operation and functionality (if any).
-
-PORTABILITY ISSUES
-------------------
-Citadel is 64-bit clean, architecture-independent, and Year 2000
-compliant. The software should compile on any POSIX compliant system with
-a full pthreads implementation and TCP/IP support. In the future we may
-try to port it to non-POSIX systems as well.
-
-On the client side, it's also POSIX compliant. The client even seems to
-build ok on non-POSIX systems with porting libraries (such as Cygwin).
-
-SUPPORTING PRIVATE MAIL
------------------------
-Can one have an elegant kludge? This must come pretty close.
-
-Private mail is sent and recieved in the *Mail>* room, which otherwise
-behaves pretty much as any other room. To make this work, we have a
-separate Mail> room for each user behind the scenes. The actual room name
-in the database looks like *"0000001234.Mail"* (where *'1234'* is the user
-number) and it's flagged with the *QR_MAILBOX* flag. The user number is
-stripped off by the server before the name is presented to the client. This
-provides the ability to give each user a separate namespace for mailboxes
-and personal rooms.
-
-This requires a little fiddling to get things just right. For example,
-*make_message()* has to be kludged to ask for the name of the recipient
-of the message whenever a message is entered in *Mail>*. But basically
-it works pretty well, keeping the code and user interface simple and
-regular.
-
-PASSWORDS AND NAME VALIDATION
------------------------------
-This has changed a couple of times over the course of Citadel's history. At
-this point it's very simple, again due to the fact that record managers are
-used for everything. The user file (user) is indexed using the user's
-name, converted to all lower-case. Searching for a user, then, is easy. We
-just lowercase the name we're looking for and query the database. If no
-match is found, it is assumed that the user does not exist.
-
-This makes it difficult to forge messages from an existing user. (Fine
-point: nonprinting characters are converted to printing characters, and
-leading, trailing, and double blanks are deleted.)
+++ /dev/null
-Description of the custom MIME type "application/x-citadel-delivery-list"
-
- This MIME type may be found in the outbound queue room on Citadel systems,
-which is typically called "__CitadelSMTPspoolout__". The room is flagged as
-a "system" room, which means that it's completely hidden to everyone, although
-an Admin can get to it if the full name is specified (but unlike a normal
-hidden room, it does not appear in the known rooms list on subsequent visits).
-
- Messages in this format contain delivery instructions. Therefore, for each
-message in the queue to be delivered to one or more recipients, there will be
-*two* messages in the room: one containing the actual message, and the other
-containing delivery instructions. It is expected that the instructions
-message may be replaced at any time (using an Exclusive ID field) if delivery
-to only some of the recipients has been accomplished.
-
- Citadel keeps reference counts of each message on disk. Therefore if a
-message contains a mixture of local and remote recipients, there may be two
-or more references to the message itself, one of them being the one in the
-queue.
-
- A delivery list contains one or more lines of text, each line containing
-a single instruction (usually a recipient). Fields are separated by the
-vertical bar character ("|") and there will always be at least one field on
-each line.
-
- -- Once per Queue-Item --
-
- INSTRUCTION: msgid
- SYNTAX: msgid|0000000
- DESCRIPTION:
- Defines the actual message for which we are providing delivery
- instructions. This instruction must precede all the others. When
- all deliveries have either succeeded or failed, both the instructions and
- the copy of the message itself in the queue room should be deleted. The
- second parameter specifies the message ID in the local database.
-
-
- INSTRUCTION: submitted
- SYNTAX: submitted|999999999
- DESCRIPTION:
- Contains a timestamp designating when this message was first entered into
- the system.
-
-
- INSTRUCTION: attempted
- SYNTAX: attempted|999999999
- DESCRIPTION:
- Contains a timestamp designating the date/time of the last delivery
- attempt.
-
-
- INSTRUCTION: retry
- SYNTAX: retry|9999999
- DESCRIPTION:
- Citadel does not retry SMTP delivery at a fixed interval. Instead, it
-starts at a nominal interval (15 minutes by default) and then doubles the
-interval after each failed attempt. This instruction contains the interval
-which should currently be followed (and doubled again, if the next delivery
-fails).
-
-
- INSTRUCTION: bounceto
- SYNTAX: bounceto|Big Bad Sender[@host]
- DESCRIPTION:
- Where to send "bounce" messages (delivery status notifications). The
- contents of the second field are supplied as the "recipient" field to
- CtdlSaveMsg(), and therefore may be a local username, a user on another
- Citadel, or an Internet e-mail address.
-
-
- INSTRUCTION: envelope_from
- SYNTAX: envelope_from|blurdybloop@example.com
- DESCRIPTION:
- Sets a value to be used as the envelope sender during the 'MAIL FROM:'
- phase of the SMTP transaction. If an envelope sender is not supplied,
- one is extracted from the message body.
-
- INSTRUCTION: source_room
- SYNTAX: source_room|Citadel Support
- DESCRIPTION:
- when sending mailinglist rooms this contains the source room for displaying
- and messaging purposes.
-
-
-
-
--- Once per Remote-part per Queue-Item --
-
- INSTRUCTION: remote
- SYNTAX: remote|friko@mumjiboolean.com|0|delivery status message
- DESCRIPTION:
- Names a recipient on a remote system to which the message should be
- delivered. The third parameter may contain any of the following values:
- 0 = No delivery has yet been attempted
- 2 = Delivery was successful
- 3 = transient error state; like connection failure or DNS lookup failure
- 4 = A transient error was experienced ... try again later
- 5 = Delivery to this address failed permanently. The error message
- should be placed in the fourth field so that a bounce message may
- be generated.
+++ /dev/null
-Every client for the Citadel system is capable of identifying itself with
-a developer code, client code, and client version. Here are the known values
-for these fields.
-
-
-Developer codes (and where known, client codes)
-
- 0 The Citadel core development team
- (See copyright.txt for the list of people involved in the project)
- Here are the client ID's we've used:
-
- Client Name Status Description
- 0 Citadel deployed Citadel text based client
- 1 lcit dead Curses-based text client
- 2 WinCit legacy Visual Basic client for Win16
- 4 WebCit deployed Web-based access to Citadel
- 5 Shaggy dead Java-powered client
- 6 Daphne dead wx multiplatform "fat" client
- 7 Shaggy (new) unknown Java-powered client
- <chilly@uncensored.citadel.org>
- 8 ctdlphp deployed PHP interface to Citadel
- 9 z-push in development Citadel backend for Z-Push
-
- 1 Brian Ledbetter <brian@shadowcom.net>
-
- Client Name Status Description
- 0 libCxClient deployed Client-side API library
- 1 Infusion deployed Groupware for Citadel
-
- 2 OST Systems, Inc.
- Matthew Scott <mscott@springfieldtech.com>
-
- Client Name Status Description
- 0 CivisLink development Python Citadel protocol library
-
- 3 Jesse Vincent <jrvincent@wesleyan.edu>
-
- 4 Brian Costello <btx@calyx.net>
-
- 5 Robert Abatecola <robert@tsgus.com>
-
- 6 Bastille <tdf@area51.v-wave.com>
-
- 7 Walden Leverich <waldenl@techsoftinc.com>
-
- 8 Michael Hampton <error@citadel.org>
-
- Client Name Status Description
- 0 libcitadel in development Client-side API library
- 1 D.O.C. dead Dave's Own Citadel look-alike
- 8 stress deployed Citadel server stress tester
- 16 Courier in development GTK+ Unix/Win32 Client
-
- 31 Stuart Cianos <scianos@alphavida.com>
-
- 69 Anticlimactic Teleservices
- Don Kimberlin (err head) <donkimberlin@hotmail.com>
-
- 71 Robert Barta <rho@cpan.org>
-
- 177 Kevin Fisher <kgf@golden.net>
-
- 226 David Given <dg@cowlark.com>
+++ /dev/null
-=====Logging =====
-Note: the logging scheme is not yet persistant all over the citserver.
-
-citadel server has several components that emmit different log strings. Its designed so you can filter for specific sessions.
-
-Since there are several sources for numbering schemes, several numbers need to be employed to make one job / session completely traceable.
-The syntax is in general like CC[%d]
-
-
-=== CC - Citadel Context ===
-All contexts that are visible to the RWHO command have a citadel context. The ID of such a context is generated from a source that can grow and shrink again. Thus CC may not be uniq to one session.
-
-Citadel may have several kind of sessions:
- - sessions generated by external connections; they have a CC.
- - sessions spawned by repetive tasks; they at first may have their own id-range, then later on an IO (AsyncIO) and CC (Citadel Context)
-
-=== Identifieing running sessions ===
-The RWHO command will list you the currently active CCs:
-
-# sendcommand RWHO
-sendcommand: started (pid=14528) connecting to Citadel server at /var/run/citadel/citadel-admin.socket
-200 potzblitz Citadel server ADMIN CONNECTION ready.
-RWHO
-100
-576|SYS_rssclient||http://xkcd.org/rss.xml||0||.||||0
-1965|willi|Mail|PotzBlitz|IMAP session|1346695500| |.||||1
-sendcommand: processing ended.
-
-576 is the CC connected to this client. with
-
-grep 'CC\[576\]' /var/log/syslog
-
-will show you the loglines from all contexts that had the ID 576; the one named above is an RSS aggregator:
-
-Jul 3 09:07:01 PotzBlitz citserver[1435]: EVCURL:IO[576]CC[576] error description: The requested URL returned error: 404
-Jul 3 09:07:01 PotzBlitz citserver[1435]: EVCURL:IO[576]CC[576] error performing request: HTTP response code said error
-Jul 3 09:07:01 PotzBlitz citserver[1435]: IO[576]CC[576][29]RSSPneed a 200, got a 404 !
-
-
-grep 'CC\[1965\]' /var/log/syslog
-will show you the loglines from all contexts that had the ID 1965; the one named above is an inbound IMAP Connection:
-
-Jul 3 09:05:01 PotzBlitz citserver[12826]: IMAPCC[1965] <plain_auth>
-Jul 3 09:05:01 PotzBlitz citserver[12826]: CC[1965]<Lobby> 144 new of 144 total messages
-Jul 3 09:05:02 PotzBlitz citserver[12826]: CC[1965]<0000000010.Mail> 1 new of 17 total messages
-
-=== Numbering and prefixes ===
-
-While external sessions will constantly show the same CC (or maybe add their own name space as above with IMAPCC) Its important to understand that not all numbers are available in all situations with sessions from repetive tasks; thus you may need several filters to gather all logmessages belonging to one job.
-
-Generaly once per minute one worker becomes an housekeeping thread; This housekeeper scans the whole system for jobs to start.
-Once a job is identified, its detection is logged; neither a CC nor IO are already assigned to this job yet;
-but it already has its uniq number assigned to it which is specific to the type of task.
-After the preanalysis of this job is done, it may become an AsyncIO job with a Citadel Context. The job now lives in one of the eventqueues, waiting for external file IO or Database IO or performing some calculation of a new state.
-Some basic IO debug logging may not facilitate the CC or the Job-ID; especialy during HTTP or DNS requests.
-
-=== Debug logging ===
-By default citserver will only log information about errors or traffic analysis related errors. If you want to track all a job does, you can enable more logging:
-
-sendcommand LOGP (LOG Print; show status of logs; the following list may increase over time) :
-
-sendcommand: started (pid=19897) connecting to Citadel server at /var/run/citadel/citadel-admin.socket
-200 potzblitz Citadel server ADMIN CONNECTION ready.
-LOGP
-100 Log modules enabled:
-modules|0 <- module loading; unloading etc.
-
-rssclient|0 <- RSS aggregator client debugging; on message level
-RSSAtomParser|0 <- debugging the parsing of ATOM / RSS feeds; once a message is isolated, the above logging comes into place again.
-curl|0 <- HTTP request logging; i.e. querieing the external RSS resources by http is logged with this.
-
-networkqueue|0 <- working through rooms, finding jobs to schedule; reading network spool files, writing them; etc.
-networktalkingto|0 <- locked list of network interconnections to circumvent race conditions by simultaneous spooling / sending / receiving
-networkclient|0 <- outbound networking connections; this has an IO context.
-
-cares|0 <- asynchroneous nameserver lookup
-eventloop|0 <- adding jobs; connecting; timeouts; etc.
-eventloopbacktrace|0 <- add stacktraces in certain places; very noisy.
-
-sieve|0 <- sieve filtering of mailboxes
-messages|0 <- message processing; loading / printing / saving
-
-imapsrv|0 <- Imap connections
-serv_xmpp|0 <- XMPP connections
-pop3client|0 <- pop3 aggregators
-
-smtpeventclient|0 <- outbound SMTP connections
-
-sendcommand: processing ended.
-
-If the second column is 1, that debug log is enabled.
-
-You can enable the logging by 3 ways:
- - sendcommand 'LOGS cares|1' (live)
- - via webcit->aide->Enable/Disable logging of the server components; in an interactive manner
- - via environment variable 'CITADEL_LOGDEBUG' as a coma separated list of facilities to enable; before starting citserver
- - via commandline parameter '-x' as a coma separated list of facilities to enable; please note that this switch used to set the loglevel previously.
-
-
-
-eventloop,eventloopbacktrace,cares
- - IO the AsyncIO ID
- - IOQ (Queueing of events into eventqueues)
-
-modules:
- - modules - module init / registering specific stuff
-
-smtpeventclient
- - S[QueueID][N] where the Queue-ID is the number of the Message to be sent, and N the SMTP-job-ID
- - SMTPCQ infrastructure logging when the housekeeper is checking for SMTP-Delivery jobs to spawn
-
-imapsrv
- - IMAPCC: imap client contexts; commands sent to us from Imap-Clients
-
-
-networkqueue
- - CC -
-
-networkclient
- - CC, IO - the context
- - NW[nodename][networkerid] the name of the node we're trying to talk to; the networker connection counter uniq per attempt
-
-curl
- - CURL - logging where we don't have a context
- - CC, IO - the context
-
-RSSClient
- - IO, CC, N, 'RSS'; IO & CC not always available.
- - 'RSSP' - the rss and atom parser
-
-
-pop3client
- - POP3 IO, CC, N - N the number of that connection
-
-sieve
- - sieve doesn't have CC or IOs, its just ran one at a time serverwide.
-
-
-xmpp
- - XMPP - whether CC could & should be logged has to be researched and added.
-
-Context
- - 'Context:' context related logging - server infrastructure, cleanup of contexts etc.
-
-messages
- - MSG - CC - the context that is manipulating this message.
-
-
-
+++ /dev/null
-
- Description of the files in the "netconfigs" directory
-
- These files contain a set of network configurations for a room. They are
-stored in the directory $CTDLDIR/netconfigs and are named according to each
-room's internal ID number. When a room is deleted, its network configuration
-file is deleted as well.
-
- The configuration file contains one or more lines of text, each line
-containing a configuration option. These lines may specify message pointers,
-room sharing instructions, mailing list recipients, etc. Fields are separated
-by the vertical bar character ("|") and there will always be at least one
-field on each line.
-
-
- INSTRUCTION: lastsent
- SYNTAX: lastsent|0000000
- DESCRIPTION:
- Defines the *local* message number of the last message in this room which
-we have performed outbound network processing on. Any batch job which sends
-out messages should do stuff.
-
-
- INSTRUCTION: listrecp
- SYNTAX: listrecp|friko@mumjiboolean.com
- DESCRIPTION:
- Defines a recipient to whom all messages in this room should be sent. This
-is used for "list serve" applications.
-
-
- INSTRUCTION: digestrecp
- SYNTAX: digestrecp|friko@mumjiboolean.com
- DESCRIPTION:
- Defines a recipient to whom all messages in this room should be sent. This
-is used for "list serve" applications. The difference between listrecps and
-digestrecps is that the latter will have messages embedded inside a message
-sent by the listserver. The message will appear to be sent by the room's
-e-mail address instead of the sender's e-mail address.
-
-
- INSTRUCTION: ignet_push_share
- SYNTAX: ignet_push_share|uncnsrd
- (or) ignet_push_share|uncnsrd|Foo Bar Baz
- DESCRIPTION:
- Specifies that the second argument is the name of a neighboring node on an
-IGnet (Citadel networking) network, to which this room should be pushed
-(spooled). Conceptually, this node will have a corresponding record pushing
-data in the other direction.
-
- If the third argument is present, it is the name of the corresponding
-room on the remote node. This allows a room to be shared even when the room
-name is different on each node. Such a configuration *must* be configured
-mutually: each node must know the name of the room on the other.
-
-
- INSTRUCTION: subpending
- SYNTAX: subpending|friko@mumjiboolean.com|listrecp|A234Z|1234567890|http://foo.com/lists
- "Subscription pending" for the specified address. This means that
-someone has requested to subscribe an e-mail address (in this case,
-friko@mumjiboolean.com) to the list. The third parameter is either "list"
-or "digest" to specify a normal subscription or a digest subscription.
-The fourth parameter is an authentication token which is generated by the server
-and e-mailed to the specified address. When the user comes back and supplies
-this token, the subscription is complete. The fifth parameter is a simple
-timestamp, so that we may purge old records which were never confirmed.
-
- The sixth field is the URL of the web page used to enter the subscription
-request, minus any parameters.
-
-
- INSTRUCTION: unsubpending
- SYNTAX: unsubpending|friko@mumjiboolean.com|A234Z|1234567890|http://foo.com/lists
- Similar to the 'subpending' command, except this one is for unsubscribe
-requests. The same rules apply with regard to the token and the web page.
-
-
- INSTRUCTION: participate
- SYNTAX: participate|list-addr@example.com
- This allows a room to be a *subscriber* to an external mailing list
-somewhere. The room's email address should be subscribed to the list, and the
-external listserv should be told "do not send me my own messages." Any
-messages which originated on Citadel (as opposed to messages which arrived via
-email, such as those from the list) will be sent to the list, with the display
-name of the author kept intact, but the email address changed to the address of
-the room.
-
- This is somewhat finicky, which is why it is not officially supported, and
-why there are no facilities in the user interface to configure it. If you want
-to use this option you are on your own.
-
-
- INSTRUCTION: pop3client
- SYNTAX: pop3client|hostname|username|password|keep|interval
- Periodically fetch messages from a remote POP3 account and store them in this
-room. Messages will be deleted from the remote account once successfully stored.
-if Keep is set, messages won't be erased on the remote host.
-Interval can specify a bigger value than the system configs value.
+++ /dev/null
-
-Package builders: you can set the following environment variables
-before running setup in quiet mode (setup -q) in order to keep it
-quiet but still do setup.
-
-First, please set CITADEL_INSTALLER to "yes"
-
-Then set:
-
-CITADEL Directory in which Citadel is installed
-
-SLAPD_BINARY Location of slapd (optional)
-LDAP_CONFIG Location of slapd.conf (optional)
-
-SUPPORT Location of libraries, etc. (optional)
- (i.e. /usr/local/ctdlsupport)
-
-CTDL_DIALOG Location of the 'dialog' program (optional)
-
-ACT_AS_MTA When other MTA's (Postfix, etc.) are found, setup
- will offer to disable them in order to keep them
- from interfering with Citadel. Set this variable to
- "yes" to always disable the other MTA without asking,
- or to "no" to never disable the other MTA.
-
-CREATE_INITTAB_ENTRY Set to "yes" to automatically create an entry in
- /etc/inittab to start the Citadel server, if needed.
-
-CREATE_XINETD_ENTRY Set to "yes" to automatically configure xinetd to route
- telnet connections into a Citadel client, if needed.
-
-SYSADMIN_NAME Name of system administrator.
-
-NO_INIT_SCRIPTS Set to non-null to suppress the creation of init
- scripts when running on a SysV-init based host.
+++ /dev/null
-====== SMTP Client ======
-===== source files involved: =====
-
-event_client.h - AsyncIO structure, interfaces to event Queue
-
-
-modules/smtp/smtp_clienthandlers.h - SMTP-Client Context struct & functions.
-modules/smtp/smtpqueue.h - parsing of Queue mails & respective structures
-
-modules/smtp/serv_smtpqueue.c - queue handling executed by housekeeping thread; de/serialization of Queue-control-messages
-modules/smtp/serv_smtpeventclient.c - business logic
-modules/smtp/smtp_clienthandlers.c - Protocoll logic
-
-msgbase.c - writes spool messages + spoolcontrol files
-
-event_client.c - base event functions
-
-modules/eventclient/serv_eventclient.c - the event queue
-modules/c-ares-dns/serv_c-ares-dns.c - nameserver lookup
-
-===== Spool Controll =====
-Citadel stores two messages for mail deliveries; the spool control message format is described in delivery-list.txt; its generated in msgbase.c, and de/serialized in serv_smtpqueue.c.
-The for Spool-control structures are kept in memory as OneQueItem per Spool-Control-File, and MailQEntry per Recipient of the message.
-For each MailQEntry which is still to be delivered, one SmtpOutMsg (containing the AsyncIO context) is spawned. The Jobs are processed in paralell, each job finished triggers an update of the spool control message, so we don't accidently deliver a message twice to one recipient. If the last recipient has succeeded, or permanently failed, the spool-file and the mail-file are removed from the spool room.
-
-===== Business Logic for one SMTP Connection =====
-For each SMTP client one SmtpOutMsg is created as Context; we now live inside the IO-Eventloop. SMTP requires to attempt to connect different servers in different situations; not all errors from remote sides are permanent, or fatal for one delivery attempt. One delivery attempt may involve talking to several SMTP-Servers; maybe the first isn't active or currently not accepting messages.
-If citadel runs in relay mode, only the relays are connected; no MX lookup is done.
-
- - MX nameserver lookup; may give several mailservers; IPv4 and IPv6 addresses may be there. If no MX is found, the server is contacted directly.
- - Now we have a number of Mailservers we have to iterate over; This is the entry for all subsequent tries for delivery; FailOneAttempt() will decide what to do next.
- - one server has to be resolved; A or AAAA record in case of IPv4 or IPv6
- - we try to connect the server nonblocking; if we fail to connect in a given time range, the next server has to be tried.
- - we now have to wait for the server greeting; we musn't say anything before the server sent a complete line. A timeout may occur, the next server has to be tried, if.
- - Now the handlers start to control the logic. A timeout may occur between each of them; Servers may reply with fatal or non fatal or temporary error codes; depending on them
- - we fail the conversation with this server, and try the next
- - we fail this delivery attempt, and retry later in the next queue run.
- - we succeed the delivery of this message.
+++ /dev/null
-SYSTEM TESTING PROPOSAL
-
-This document is intended as a discussion of possible automated tests. It does
-not describe any existing tests.
-
-
----
-
-
-First we should create a client that leverages expect (or something similar) to
-perform automated testing of the client interface. Tests can be written as
-expect scripts.
-
-Each system being tested will need to create an aide level user for the test
-client to connect as.
-
-The test client will create another user to carry out the tests. This allows the
-aide level user to vary the level of the test user and check the access level
-code.
-
-----
-
-
-For a first step each test site should create a test user that can send internet
-mail.
-This test user needs some sieve rules to forward mail around (eventually this
-will be created automatically by the test client). These rules will forward mail
-to other test users at other sites participating in the test system and to a
-networked room.
-Each system participating in the test should share some rooms.
-
-The idea is:
- 1. A test site posts a message to its test user using citmail or some
-other email prog.
- 2. The sieve rules forward the message to each of the other test users
-at the other sites.
- 3. The sieve rules for the other test users detect that the message was
-forwarded to them and they file it into a networked room
- 4. By virtue of the networked room the message returns to the
-originating system where the administrator can see it.
-
-Once I (davew) have written my module to alter the message body we can have it
-add text to the message to indicate the full path of the message.
-
-
+++ /dev/null
-How "views" work in Citadel
----------------------------
-
- There's no need to be rigid and stupid about how different rooms are presented
-in a Citadel client. And we don't enforce things either. But there's a need
-to make things look the way the user wants to see them. For example, we might
-always choose to see a room full of private mail as a summary (one line per
-message) rather than always dumping out the entire contents like we do on a
-typical message room. An address book room might look better as a tabbed view
-or something, rather than as a bunch of messages with vCards attached to them.
-
- This is why we define "views" for a room. It gives the client software a
-hint as to how to display the contents of a room. This is kept on a per-user
-basis by storing it in the 'visit' record for a particular room/user
-combination. It is visit.v_view and is an integer. Naturally, there also
-needs to be a default, for users who have never visited the room before. This
-is in the room record as room.QRdefaultview (and is also an integer).
-
- In recent versions of Citadel, the view for a room also defines when and how
-it is indexed. For example, mailboxes and bulletin boards don't need to have
-an euid index, but address books and calendars do.
-
- The values currently defined are:
-
-#define VIEW_BBS 0 /* Bulletin board view */
-#define VIEW_MAILBOX 1 /* Mailbox summary */
-#define VIEW_ADDRESSBOOK 2 /* Address book view */
-#define VIEW_CALENDAR 3 /* Calendar view */
-#define VIEW_TASKS 4 /* Tasks view */
-#define VIEW_NOTES 5 /* Notes view */
-#define VIEW_WIKI 6 /* Wiki view */
-
projects / citadel.git / commitdiff