1 The totally incomplete guide to Citadel internals
2 -----------------------------------------------------
4 Citadel has evolved quite a bit since its early days, and the data structures
5 have evolved with it. This document provides a rough overview of how the
6 system works internally. For details you're going to have to dig through the
7 code, but this'll get you started.
12 As you probably already know by now, Citadel uses a group of tables stored
13 with a record manager (usually Berkeley DB). Since we're using a record
14 manager rather than a relational database, all record structures are managed
15 by Citadel. Here are some of the tables we keep on disk:
20 This table contains all user records. It's indexed by
21 user name (translated to lower case for indexing purposes). The records in
22 this file look something like this:
24 struct ctdluser { // User record
25 int version; // Citadel version. which created this record
26 uid_t uid; // Associate with a unix account?
27 char password[32]; // Account password (unless using external authentication)
28 unsigned flags; // See US_ flags below
29 CIT_UBYTE axlevel; // Access level
30 long usernum; // User number (never recycled)
31 time_t lastcall; // Last time the user called
32 int USuserpurge; // Purge time (in days) for user
33 char fullname[64]; // Name for Citadel messages & mail
36 Most fields here should be fairly self-explanatory. The ones that might
37 deserve some attention are:
39 - `uid` -- if uid is not the same as the *unix uid* Citadel is running as, then the
40 account is assumed to belong to the user on the underlying Unix system with
41 that uid. This allows us to require the user's OS password instead of having
42 a separate Citadel password.
44 - `usernum` -- these are assigned sequentially, and **NEVER REUSED**. This is
45 important because it allows us to use this number in other data structures
46 without having to worry about users being added/removed later on, as you'll
47 see later in this document.
52 These are room records. There is a room record for every room on the
53 system, public or private or mailbox. It's indexed by room name (also in
54 lower case for easy indexing) and it contains records which look like this:
57 char QRname[ROOMNAMELEN]; /* Name of room */
58 char QRpasswd[10]; /* Only valid if it's a private rm */
59 long QRroomaide; /* User number of room aide */
60 long QRhighest; /* Highest message NUMBER in room */
61 time_t QRgen; /* Generation number of room */
62 unsigned QRflags; /* See flag values below */
63 char QRdirname[15]; /* Directory name, if applicable */
64 long QRinfo; /* Info file update relative to msgs*/
65 char QRfloor; /* Which floor this room is on */
66 time_t QRmtime; /* Date/time of last post */
67 struct ExpirePolicy QRep; /* Message expiration policy */
68 long QRnumber; /* Globally unique room number */
69 char QRorder; /* Sort key for room listing order */
70 unsigned QRflags2; /* Additional flags */
71 int QRdefaultview; /* How to display the contents */
74 Again, mostly self-explanatory. Here are the interesting ones:
76 `QRnumber` is a globally unique room ID, while `QRgen` is the "generation number"
77 of the room (it's actually a timestamp). The two combined produce a unique
78 value which identifies the room. The reason for two separate fields will be
79 explained below when we discuss the visit table. For now just remember that
80 `QRnumber` remains the same for the duration of the room's existence, and `QRgen`
81 is timestamped once during room creation but may be restamped later on when
82 certain circumstances exist.
86 Floors. This is so simplistic it's not worth going into detail about, except
87 to note that we keep a reference count of the number of rooms on each floor.
91 Each record in this table consists of a bunch of message numbers
92 which represent the contents of a room. A message can exist in more than one
93 room (for example, a mail message with multiple recipients -- 'single instance
94 store'). This table is never, ever traversed in its entirety. When you do
95 any type of read operation, it fetches the msglist for the room you're in
96 (using the room's ID as the index key) and then you can go ahead and read
97 those messages one by one.
99 Each room is basically just a list of message numbers. Each time
100 we enter a new message in a room, its message number is appended to the end
101 of the list. If an old message is to be expired, we must delete it from the
102 message base. Reading a room is just a matter of looking up the messages
103 one by one and sending them to the client for display, printing, or whatever.
108 This is the tough one. Put on your thinking cap and grab a fresh cup of
109 coffee before attempting to grok the visit table.
111 This table contains records which establish the relationship between users
112 and rooms. Its index is a hash of the user and room combination in question.
113 When looking for such a relationship, the record in this table can tell the
114 server things like "this user has zapped this room," "this user has access to
115 this private room," etc. It's also where we keep track of which messages
116 the user has marked as "old" and which are "new" (which are not necessarily
117 contiguous; contrast with older Citadel implementations which simply kept a
118 "last read" pointer).
121 Here's what the records look like:
128 unsigned int v_flags;
133 #define V_FORGET 1 /* User has zapped this room */
134 #define V_LOCKOUT 2 /* User is locked out of this room */
135 #define V_ACCESS 4 /* Access is granted to this room */
137 This table is indexed by a concatenation of the first three fields. Whenever
138 we want to learn the relationship between a user and a room, we feed that
139 data to a function which looks up the corresponding record. The record is
140 designed in such a way that an "all zeroes" record (which is what you get if
141 the record isn't found) represents the default relationship.
143 With this data, we now know which private rooms we're allowed to visit: if
144 the `V_ACCESS` bit is set, the room is one which the user knows, and it may
145 appear in his/her known rooms list. Conversely, we also know which rooms the
146 user has zapped: if the `V_FORGET` flag is set, we relegate the room to the
147 zapped list and don't bring it up during new message searches. It's also
148 worth noting that the `V_LOCKOUT` flag works in a similar way to administratively
149 lock users out of rooms.
151 Implementing the "cause all users to forget room" command, then, becomes very
152 simple: we simply change the generation number of the room by putting a new
153 timestamp in the `QRgen` field. This causes all relevant visit records to
154 become irrelevant, because they appear to point to a different room. At the
155 same time, we don't lose the messages in the room, because the msglists table
156 is indexed by the room number (`QRnumber`), which never changes.
158 `v_seen` contains a string which represents the set of messages in this room
159 which the user has read (marked as 'seen' or 'old'). It follows the same
160 syntax used by IMAP and NNTP. When we search for new messages, we simply
161 return any messages that are in the room that are **not** represented by this
162 set. Naturally, when we do want to mark more messages as seen (or unmark
163 them), we change this string. Citadel BBS client implementations are naive
164 and think linearly in terms of "everything is old up to this point," but IMAP
165 clients want to have more granularity.
170 This table simply maps Internet e-mail addresses to Citadel network addresses
171 for quick lookup. It is generated from data in the Global Address Book room.
175 This table keeps track of message ID's of messages arriving over a network,
176 to prevent duplicates from being posted if someone misconfigures the network
177 and a loop is created. This table goes unused on a non-networked Citadel.
181 This is where all message text is stored. It's indexed by message number:
182 give it a number, get back a message. Messages are numbered sequentially, and
183 the message numbers are never reused.
185 We also keep a "metadata" record for each message. This record is also stored
186 in the msgmain table, using the index (0 - msgnum). We keep in the metadata
187 record, among other things, a reference count for each message. Since a
188 message may exist in more than one room, it's important to keep this reference
189 count up to date, and to delete the message from disk when the reference count
192 NoSQL pros might recognize the message store as a "document database". Each
193 record is an arbitrary collection of keys and values, some of which may
194 appear all the time, and some which might appear infrequently.
196 # Here's the format for the message itself:
198 - Each message begins with an 0xFF 'start of message' byte.
200 - The next byte denotes whether this is an anonymous message. The codes
201 available are `MES_NORMAL`, `MES_ANON`, or `MES_AN2` (defined in `citadel.h`).
203 - The third byte is a "message type" code. The following codes are defined:
204 - 0 - "Traditional" Citadel format. Message is to be displayed "formatted."
205 - 1 - Plain pre-formatted ASCII text (otherwise known as text/plain)
206 - 4 - MIME formatted message. The text of the message which follows is
207 expected to begin with a "Content-type:" header.
209 - After these three opening bytes, the remainder of
210 the message consists of a sequence of character strings. Each string
211 begins with a type byte indicating the meaning of the string and is
212 ended with a null. All strings are printable ASCII: in particular,
213 all numbers are in ASCII rather than binary. This is for simplicity,
214 both in implementing the system and in implementing other code to
215 work with the system. For instance, a database driven off Citadel archives
216 can do wildcard matching without worrying about unpacking binary data such
217 as message ID's first. To provide later downward compatability
218 all software should be written to IGNORE fields not currently defined.
221 # The type bytes currently defined are:
224 | BYTE | Enum | NW | Mnemonic | Enum / Comments
225 |-------|-------------------|------|----------------|---------------------------------------------------------
226 | A | eAuthor | from | Author | The display name of the Author of the message.
227 | B | eBig_message | | Big message | This is a flag which indicates that the message is
228 | | | | | big, and Citadel is storing the body in a separate
229 | | | | | record. You will never see this field because the
230 | | | | | internal API handles it.
231 | E | eExclusiveID | exti | Exclusive ID | A persistent alphanumeric Message ID used for
232 | | | | | replication control. When a message arrives that
233 | | | | | contains an Exclusive ID, any existing messages which
234 | | | | | contain the same Exclusive ID and are *older* than this
235 | | | | | message should be deleted. If there exist any messages
236 | | | | | with the same Exclusive ID that are *newer*, then this
237 | | | | | message should be dropped.
238 | F | erFc822Addr | rfca | rFc822 address | email address or user principal name of the message
240 | I | emessageId | msgn | Message ID | An RFC822-compatible message ID for this message.
242 | J | eJournal | jrnl | Journal | The presence of this field indicates that the message
243 | | | | | is disqualified from being journaled, perhaps because
244 | | | | | it is itself a journalized message and we wish to
245 | | | | | avoid double journaling.
246 | K | eReplyTo | rep2 | Reply-To | the Reply-To header for mailinglist outbound messages
247 | L | eListID | list | List-ID | Mailing list identification, as per RFC 2919
248 | M | eMesageText | text | Message Text | Normal ASCII, newlines seperated by CR's or LF's,
249 | | | | | null terminated as always.
250 | O | eOriginalRoom | room | Room | Room of origin.
251 | P | eMessagePath | path | Path | Complete path of message, as in the UseNet news
252 | | | | | standard. A user should be able to send Internet mail
253 | | | | | to this path. (Note that your system name will not be
254 | | | | | tacked onto this until you're sending the message to
255 | | | | | someone else)
256 | R | eRecipient | rcpt | Recipient | Only present in Mail messages.
257 | T | eTimestamp | time | date/Time | Unix timestamp containing the creation date/time of
258 | | | | | the message.
259 | U | eMsgSubject | subj | sUbject | Message subject. Optional.
260 | | | | | Developers may choose whether they wish to
261 | | | | | generate or display subject fields.
262 | V | eenVelopeTo | nvto | enVelope-to | The recipient specified in incoming SMTP messages.
263 | W | eWeferences | wefw | Wefewences | Previous message ID's for conversation threading. When
264 | | | | | converting from RFC822 we use References: if present, or
265 | | | | | In-Reply-To: otherwise.
266 | | | | | (Who in extnotify spool messages which don't need to know
267 | | | | | other message ids)
268 | Y | eCarbonCopY | cccc | carbon copY | Carbon copy (CC) recipients.
269 | | | | | Optional, and only in Mail messages.
270 | % | eHeaderOnly | nhdr | oNlyHeader | we will just be sending headers. for the Wire protocol only.
271 | % | eFormatType | type | type | type of citadel message: (Wire protocol only)
272 | | | | | FMT_CITADEL 0 Citadel vari-format (proprietary)
273 | | | | | FMT_FIXED 1 Fixed format (proprietary)
274 | | | | | FMT_RFC822 4 Standard (headers are in M field)
275 | % | eMessagePart | part | emessagePart | eMessagePart is the id of this part in the mime hierachy
276 | % | eSubFolder | suff | eSubFolder | descend into a mime sub container
277 | % | ePevious | pref | ePevious | exit a mime sub container
278 | 0 | eErrorMsg | | Error | This field is typically never found in a message on
279 | | | | | disk or in transit. Message scanning modules are
280 | | | | | expected to fill in this field when rejecting a message
281 | | | | | with an explanation as to what happened (virus found,
282 | | | | | message looks like spam, etc.)
283 | 1 | eSuppressIdx | | suppress index | The presence of this field indicates that the message is
284 | | | | | disqualified from being added to the full text index.
285 | 2 | eExtnotify | | extnotify | Used internally by the serv_extnotify module.
286 | 3 | eVltMsgNum | | msgnum | Used internally to pass the local message number in the
287 | | | | | database to after-save hooks. Discarded afterwards.
288 | | | locl | | The presence of this field indicates that the message
289 | | | | | is believed to have originated on the local Citadel node,
290 | | | | | not as an inbound email or some other outside source.
294 Let `<FF>` be a `0xFF` byte, and `<0>` be a null `(0x00)` byte. Then a message
297 Apr 12, 1988 23:16 From Test User In Network Test> @lifesys (Life Central)
300 might be stored as...
302 <FF><40><0>I12345<0>Pneighbor!lifesys!test_user<0>T576918988<0> (continued)
303 -----------|Mesg ID#|--Message Path---------------|--Date------
305 AThe Test User<0>ONetwork Test<0>Nlifesys<0>HLife Central<0>MHave a nice day!<0>
306 |-----Author-----|-Room name-----|-nodename-|Human Name-|--Message text-----
308 Weird things can happen if fields are missing, especially if you use the
309 networker. But basically, the date, author, room, and nodename may be in any
310 order. But the leading fields and the message text must remain in the same
311 place. The H field looks better when it is placed immediately after the N
315 EUID (EXCLUSIVE MESSAGE ID'S)
316 -----------------------------
317 This is where the groupware magic happens. Any message in any room may have
318 a field called the Exclusive message *ID*, or *EUID*. We keep an index in the
319 table `CDB_EUIDINDEX` which knows the message number of any item that has an
320 *EUID*. This allows us to do two things:
322 - If a subsequent message arrives with the same *EUID*, it automatically
323 *deletes* the existing one, because the new one is considered a replacement
324 for the existing one.
325 - If we know the *EUID* of the item we're looking for, we can fetch it by *EUID*
326 and get the most up-to-date version, even if it's been updated several times.
328 This functionality is made more useful by server-side hooks. For example,
329 when we save a vCard to an address book room, or an iCalendar item to a
330 calendar room, our server modules detect this condition, and automatically set
331 the *EUID* of the message to the *UUID* of the *vCard* or *iCalendar* item.
332 Therefore when you save an updated version of an address book entry or
333 a calendar item, the old one is automatically deleted.
337 Citadel is 32/64 bit clean and architecture-independent. The software is
338 developed and primarily run on the Linux operating system (which uses the
339 Linux kernel) but it should compile and run on any reasonably POSIX
342 On the client side, it's also POSIX compliant. The client even seems to
343 build ok on non-POSIX systems with porting libraries (such as Cygwin and
346 SUPPORTING PRIVATE MAIL
347 -----------------------
348 Can one have an elegant kludge? This must come pretty close.
350 Private mail is sent and recieved in the `Mail>` room, which otherwise
351 behaves pretty much as any other room. To make this work, we have a
352 separate Mail> room for each user behind the scenes. The actual room name
353 in the database looks like `0000001234.Mail` (where `1234` is the user
354 number) and it's flagged with the `QR_MAILBOX` flag. The user number is
355 stripped off by the server before the name is presented to the client. This
356 provides the ability to give each user a separate namespace for mailboxes
359 This requires a little fiddling to get things just right. For example,
360 `make_message()` has to be kludged to ask for the name of the recipient
361 of the message whenever a message is entered in `Mail>`. But basically
362 it works pretty well, keeping the code and user interface simple and
365 PASSWORDS AND NAME VALIDATION
366 -----------------------------
367 This has changed a couple of times over the course of Citadel's history. At
368 this point it's very simple, again due to the fact that record managers are
369 used for everything. The user file (user) is indexed using the user's
370 name, converted to all lower-case. Searching for a user, then, is easy. We
371 just lowercase the name we're looking for and query the database. If no
372 match is found, it is assumed that the user does not exist.
374 This makes it difficult to forge messages from an existing user. (Fine
375 point: nonprinting characters are converted to printing characters, and
376 leading, trailing, and double blanks are deleted.)