Message Journaling with the Citadel System

A technical guide

Copyright ©2006 by the Citadel development team.

Table of Contents


Introduction

There are many different environments in which the Citadel system is useful for messaging and collaboration tasks. Naturally, many sites use Citadel as their organization's primary email server. Some of these organizations are corporations which operate in industries that are subject to regulations which dictate that all electronic messages must be archived for some period of time.

Message journaling is a function of the Citadel system which provides the ability to record all messages which are processed by the server. Journaling, by itself, is not a message archival service, nor should one attempt to treat it as one. We recommend that you combine Citadel journaling with a third-party archival product or service, in order to assemble a data retention solution that will meet the requirements of your organization.

Technical overview of Citadel journaling

In order to keep the configuration options simple and straightforward, Citadel implements only one variety of journaling. It is generally known in the messaging universe as "envelope journaling." As you are probably already aware, one cannot trust the headers of any particular message to be a reliable indication of which users have read it. In addition to the declared sender and recipients, there may also be blind carbon copy (BCC) recipients, recipients on one or more mailing lists, and even recipients who are not listed due to forged or badly formed headers.

The format of a journalized message, therefore, contains extra information about who actually sent and/or received the original message. The very first portion of the journalized message lists the sender, message ID, and recipients. It should be noted that the listed Sender and Recipients are those which were relevant to delivery on the Citadel server performing the journaling of the message. If the message arrived via SMTP from the outside world, any recipients not on the local Citadel server would not be listed, because they were not involved in the transaction which delivered the message to the server.

The original message is then included, as an attachment of type message/rfc822, to the journal message.

This logic provides the ability to answer these questions:

Either question can be answered by searching your archive for the message in question and reviewing the names and addresses listed in the journalized messages.

Additional resource utilization

On a server with message journaling enabled, each message must pass through the submit chain twice: once for the delivery of the original message, and once for the delivery of the journalized message. You must plan accordingly for the additional overhead which will be incurred. Nearly twice as much CPU, memory, and disk space will be consumed by journaling.

Despite the extra processing, however, it should be noted that the Citadel server performs journaling as a deferred background operation. This allows interactive users to continue along instead of waiting for the server to synchronously journal each message that is submitted.

Where should I send the journalized messages?

As previously noted, any valid email address may be used as the destination for journalized messages. Where you send your journalized messages will likely depend upon how you are going to archive them.

Some archival systems contain their own email addresses. With these products and services, you are expected to email your journalized messages directly to a specific address. This is typically referred to as "push" journaling because the transfer is initiated by your server. If you intend to use this type of system, the required configuration is obvious: the journal destination is the email address of your archival system.

Other archival systems are built around the idea that you should provide a dedicated email address on your server, and the archival system periodically uses POP3 (or POP3S, the SSL-encrypted version of POP3) to poll the mailbox for new journalized messages. Upon successful retrieval of new messages, the archival system deletes the journalized messages from your server. This is typically referred to as "pull" journaling because the transfer is initiated externally. If you intend to use this type of system, you should create a dedicated account on your Citadel server for this purpose, and then configure that account as the journal destination. No real users should ever log in to this account. It exists solely as a drop box for journalized messages.

In rare instances, you may wish to send the journalized messages to an address on some other node on a private Citadel network. This could be useful if, for example, one server is used to aggregate messages journalized from several other servers. It might also be useful if, for some reason, you require a dedicated server for journaling.

It should also be noted that the message data format contains a field which declares that a message is actually a journalized copy of some other message. The Citadel system will not journalize any message which carries this flag. This prevents messages from being double journalized (or worse, caught in an infinite loop of journaling).

How to configure journaling

After you have formulated a strategy for combining journaling with some third-party archival product or service as part of a compliance solution, the actual configuration is very easy. You only need to set the following three configuration items in Citadel:

From the text-based Citadel client, these options may be found in the .Aide System-configuration General command. You will see the following options among the many other options in this dialog:

Perform journaling of email messages [Yes]: 
Perform journaling of non-email messages [Yes]: 
Email destination of journalized messages [example@example.com]:
"Non-email messages" refers to any message which has no recipients. This includes posts to public and private rooms, calendar items, address book entries, notes, tasks, and any other data types which may be introduced in the future.

Configuration of journaling using WebCit is even easier. Log in with your web browser and then select Administration --> Edit site-wide configuration --> Indexing/Journaling, and you will find the same three options.

It is not necessary to restart the Citadel server when you change your journaling options. Changes will take effect immediately.

Disclaimer

The message journaling feature of Citadel is only one component which may be utilized as part of a larger solution for regulatory compliance. The Citadel developers do not make any guarantee, expressed or implied, that this functionality by itself will cause your organization's email services to be sufficiently archived. It is up to you to design your complete archival solution, to test it, and to evaluate its ability to comply with whatever regulations are relevant to your organization.