------------------------------------------------------
- The totally incomplete guide to Citadel/UX internals
+ The totally incomplete guide to Citadel internals
------------------------------------------------------
Citadel has evolved quite a bit since its early days, and the data structures
code, but this'll get you started.
- Database tables
-
+ DATABASE TABLES
+ ---------------
As you probably already know by now, Citadel uses a group of tables stored
with a record manager (usually Berkeley DB). Since we're using a record
struct ctdluser { /* User record */
int version; /* Cit vers. which created this rec */
uid_t uid; /* Associate with a unix account? */
- char password[32]; /* password (for BBS-only users) */
+ char password[32]; /* password (for Citadel-only users)*/
unsigned flags; /* See US_ flags below */
long timescalled; /* Total number of logins */
long posted; /* Number of messages posted (ever) */
time_t lastcall; /* Last time the user called */
int USuserpurge; /* Purge time (in days) for user */
char fullname[64]; /* Name for Citadel messages & mail */
- CIT_UBYTE USscreenwidth; /* Screen width (for textmode users)*/
- CIT_UBYTE USscreenheight; /* Screen height(for textmode users)*/
};
Most fields here should be fairly self-explanatory. The ones that might
usernum -- these are assigned sequentially, and NEVER REUSED. This is
important because it allows us to use this number in other data structures
-without having to worry about users being added/removed later on, as you'll
+ without having to worry about users being added/removed later on, as you'll
see later in this document.
- The screenwidth and screenheight fields are almost never used anymore. Back
-when people were calling into dialup systems we had no way of knowing the
-user's screen dimensions, but modern networks almost always transmit this
-information so we set it up dynamically.
-
ROOM RECORDS
------------
as message ID's first. To provide later downward compatability
all software should be written to IGNORE fields not currently defined.
- The type bytes currently defined are:
-
-BYTE Mnemonic Comments
-
-A Author Name of originator of message.
-B Big message This is a flag which indicates that the message is
- big, and Citadel is storing the body in a separate
- record. You will never see this field because the
- internal API handles it.
-D Destination Contains name of the system this message should
- be sent to, for mail routing (private mail only).
-E Extended ID A persistent alphanumeric Message ID used for
- network replication. When a message arrives that
- contains an Extended ID, any existing messages which
- contain the same Extended ID and are *older* than this
- message should be deleted. If there exist any messages
- with the same Extended ID that are *newer*, then this
- message should be dropped.
-F rFc822 address For Internet mail, this is the delivery address of the
- message author.
-H HumanNodeName Human-readable name of system message originated on.
-I Original ID A 32-bit integer containing the message ID on the
- system the message *originated* on.
-M Message Text Normal ASCII, newlines seperated by CR's or LF's,
- null terminated as always.
-N Nodename Contains node name of system message originated on.
-O Room Room of origin.
-P Path Complete path of message, as in the UseNet news
- standard. A user should be able to send Internet mail
- to this path. (Note that your system name will not be
- tacked onto this until you're sending the message to
- someone else)
-R Recipient Only present in Mail messages.
-S Special field Only meaningful for messages being spooled over a
- network. Usually means that the message isn't really
- a message, but rather some other network function:
- -> "S" followed by "FILE" (followed by a null, of
- course) means that the message text is actually an
- IGnet/Open file transfer.
- -> "S" followed by "CANCEL" means that this message
- should be deleted from the local message base once
- it has been replicated to all network systems.
-T Date/Time A 32-bit integer containing the date and time of
- the message in standard UNIX format (the number
- of seconds since January 1, 1970 GMT).
-U Subject Optional. Developers may choose whether they wish to
- generate or display subject fields. Citadel/UX does
- not generate them, but it does print them when found.
-0 Error This field is typically never found in a message on
- disk or in transit. Message scanning modules are
- expected to fill in this field when rejecting a message
- with an explanation as to what happened (virus found,
- message looks like spam, etc.)
+ The type bytes currently defined are:
+
+BYTE Enum Mnemonic Enum / Comments
+
+A eAuthor Author eAuthor
+ Name of originator of message.
+B eBig_message Big message eBig_message
+ This is a flag which indicates that the message is
+ big, and Citadel is storing the body in a separate
+ record. You will never see this field because the
+ internal API handles it.
+C eRemoteRoom RemoteRoom eRemoteRoom
+ when sent via Citadel Networking, this is the room
+ its going to be put on the remote site.
+D eDestination Destination eDestination
+ Contains name of the system this message should
+ be sent to, for mail routing (private mail only).
+E eExclusiveID Exclusive ID eExclusiveID
+ A persistent alphanumeric Message ID used for
+ network replication. When a message arrives that
+ contains an Exclusive ID, any existing messages which
+ contain the same Exclusive ID and are *older* than this
+ message should be deleted. If there exist any messages
+ with the same Exclusive ID that are *newer*, then this
+ message should be dropped.
+F erFc822Addr rFc822 address erFc822Addr
+ For Internet mail, this is the delivery address of the
+ message author.
+H eHumanNode Human node name eHumanNode
+ Human-readable name of system message originated on.
+I emessageId Message ID emessageId
+ An RFC822-compatible message ID for this message.
+J eJournal Journal eJournal
+ The presence of this field indicates that the message
+ is disqualified from being journaled, perhaps because
+ it is itself a journalized message and we wish to
+ avoid double journaling.
+K eReplyTo Reply-To eReplyTo
+ the Reply-To header for mailinglist outbound messages
+L eListID List-ID eListID
+ Mailing list identification, as per RFC 2919
+M eMesageText Message Text eMesageText
+ Normal ASCII, newlines seperated by CR's or LF's,
+ null terminated as always.
+N eNodeName Nodename eNodeName
+ Contains node name of system message originated on.
+O eOriginalRoom Room eOriginalRoom - Room of origin.
+P eMessagePath Path eMessagePath
+ Complete path of message, as in the UseNet news
+ standard. A user should be able to send Internet mail
+ to this path. (Note that your system name will not be
+ tacked onto this until you're sending the message to
+ someone else)
+R eRecipient Recipient eRecipient - Only present in Mail messages.
+S eSpecialField Special field eSpecialField
+ Only meaningful for messages being spooled over a
+ network. Usually means that the message isn't really
+ a message, but rather some other network function:
+ -> "S" followed by "FILE" (followed by a null, of
+ course) means that the message text is actually an
+ IGnet/Open file transfer. (OBSOLETE)
+ -> "S" followed by "CANCEL" means that this message
+ should be deleted from the local message base once
+ it has been replicated to all network systems.
+T eTimestamp date/Time eTimestamp
+ Unix timestamp containing the creation date/time of
+ the message.
+U eMsgSubject sUbject eMsgSubject - Optional.
+ Developers may choose whether they wish to
+ generate or display subject fields.
+V eenVelopeTo enVelope-to eenVelopeTo
+ The recipient specified in incoming SMTP messages.
+W eWeferences Wefewences eWeferences
+ Previous message ID's for conversation threading. When
+ converting from RFC822 we use References: if present, or
+ In-Reply-To: otherwise.
+ (Who in extnotify spool messages which don't need to know
+ other message ids)
+Y eCarbonCopY carbon copY eCarbonCopY
+ Optional, and only in Mail messages.
+0 eErrorMsg Error eErrorMsg
+ This field is typically never found in a message on
+ disk or in transit. Message scanning modules are
+ expected to fill in this field when rejecting a message
+ with an explanation as to what happened (virus found,
+ message looks like spam, etc.)
+1 eSuppressIdx suppress index eSuppressIdx
+ The presence of this field indicates that the message is
+ disqualified from being added to the full text index.
+2 eExtnotify extnotify eExtnotify - Used internally by the serv_extnotify module.
+3 eVltMsgNum msgnum eVltMsgNum
+ Used internally to pass the local message number in the
+ database to after-save hooks. Discarded afterwards.
EXAMPLE
Let <FF> be a 0xFF byte, and <0> be a null (0x00) byte. Then a message
which prints as...
-Apr 12, 1988 23:16 From Test User In Network Test> @lifesys (Life BBS)
+Apr 12, 1988 23:16 From Test User In Network Test> @lifesys (Life Central)
Have a nice day!
might be stored as...
<FF><40><0>I12345<0>Pneighbor!lifesys!test_user<0>T576918988<0> (continued)
-----------|Mesg ID#|--Message Path---------------|--Date------
-AThe Test User<0>ONetwork Test<0>Nlifesys<0>HLife BBS<0>MHave a nice day!<0>
+AThe Test User<0>ONetwork Test<0>Nlifesys<0>HLife Central<0>MHave a nice day!<0>
|-----Author-----|-Room name-----|-nodename-|Human Name-|--Message text-----
Weird things can happen if fields are missing, especially if you use the
field.
+ EUID (EXCLUSIVE MESSAGE ID'S)
+ -----------------------------
+
+ This is where the groupware magic happens. Any message in any room may have
+a field called the Exclusive message ID, or EUID. We keep an index in the
+table CDB_EUIDINDEX which knows the message number of any item that has an
+EUID. This allows us to do two things:
+
+ 1. If a subsequent message arrives with the same EUID, it automatically
+*deletes* the existing one, because the new one is considered a replacement
+for the existing one.
+ 2. If we know the EUID of the item we're looking for, we can fetch it by EUID
+and get the most up-to-date version, even if it's been updated several times.
+
+ This functionality is made more useful by server-side hooks. For example,
+when we save a vCard to an address book room, or an iCalendar item to a
+calendar room, our server modules detect this condition, and automatically set
+the EUID of the message to the UUID of the vCard or iCalendar item. Therefore
+when you save an updated version of an address book entry or a calendar item,
+the old one is automatically deleted.
- Networking
+ NETWORKING (REPLICATION)
+ ------------------------
Citadel nodes network by sharing one or more rooms. Any Citadel node
can choose to share messages with any other Citadel node, through the sending
of spool files. The sending system takes all messages it hasn't sent yet, and
spools them to the recieving system, which posts them in the rooms.
+The EUID discussion above is extremely relevant, because EUID is carried over
+the network as well, and the replacement rules are followed over the network
+as well. Therefore, when a message containing an EUID is saved in a networked
+room, it replaces any existing message with the same EUID *on every node in
+the network*.
+
Complexities arise primarily from the possibility of densely connected
networks: one does not wish to accumulate multiple copies of a given
message, which can easily happen. Nor does one want to see old messages
With the path present, all the networker has to do to assure that it doesn't
send another system a message it's already received is check the <P>ath field
for that system's name somewhere in the bang path. If it's present, the system
-has already seen the message, so we don't send it. (Note that the current
-implementation does not allow for "loops" in the network -- if you build your
-net this way you will see lots of duplicate messages.)
+has already seen the message, so we don't send it.
+
+We also keep a small database, called the "use table," containing the ID's of
+all messages we've seen recently. If the same message arrives a second or
+subsequent time, we will find its ID in the use table, indicating that we
+already have a copy of that message. It will therefore be discarded.
The above discussion should make the function of the fields reasonably clear:
please see network.txt on its operation and functionality (if any).
-
- Portability issues
+ PORTABILITY ISSUES
+ ------------------
- Citadel/UX is 64-bit clean, architecture-independent, and Year 2000
+ Citadel is 64-bit clean, architecture-independent, and Year 2000
compliant. The software should compile on any POSIX compliant system with
a full pthreads implementation and TCP/IP support. In the future we may
try to port it to non-POSIX systems as well.
-
-
- SUPPORTING PRIVATE MAIL
+ SUPPORTING PRIVATE MAIL
+ -----------------------
Can one have an elegant kludge? This must come pretty close.
- PASSWORDS AND NAME VALIDATION
+ PASSWORDS AND NAME VALIDATION
+ -----------------------------
This has changed a couple of times over the course of Citadel's history. At
this point it's very simple, again due to the fact that record managers are
projects / citadel.git / blobdiff