Rules Du Jour

Για όσους χρησιμοποιούν κανόνες του SpamAssassin από το SpamAssassin Rules Emporium το Rules Du Jour ήταν ένα πολύ χρήσιμο εργαλείο επικαιροποίησης. Γράφω “ήταν” γιατί μετά το τελευταίο autoupdate έλαβα αυτό το email:

Rules Du Jour has an update available. New version of Rules Du Jour is # Version 1.31 NOTICE! Rules du jour is no longer being maintained. As the author of RDJ, I recommend switching to the official update method for spamassassin, sa-update. However, if you have a patch that is generally applicable, I will be happy to check it out and apply it for general consumption. The rules_du_jour script will remain hosted at sandgnat.com for the forseeable future (I will not knowingly break your working setups!). Additionally, if you are interested in maintaining rules dj jour for a longer period of time, please contact me.”

Ώρα να κάνουμε ένα γύρο όλους τους server και να αλλάξουμε το configuration…

How to update SARE rulesets via Apache SpamAssassin’s sa-update

$#discard and ruleset 0

Note to self: The Sendmail Installation and Operation Guide (at least version 1.25 of doc/op.ps) clearly states:

“The mailer with the special name “discard” causes any mail sent to it to be discarded but otherwise treated as though it were successfully delivered. This mailer cannot be used in ruleset 0, only in the various address checking rulesets.”

So, instead of writing something like this:

LOCAL_RULE_0
R$={NIL}  < @ $=w . > $*        $#discard $: $1

you need to write something like this:

LOCAL_RULE_0
R$={NIL}  < @ $=w . > $*        $#local $: bit-bucket

otherwise you get beaten by “buildaddr: unknown mailer discard” errors, even though $#discard is very well known (to you; not to ruleset 0).

The above example assumes that $={NIL} is a class that contains usernames for which we do not want to accept any email and bit-bucket is an entry in /etc/mail/aliases:

bit-bucket: /dev/null

Yes, it is possible to achieve the same thing using FEATURE(`virtusertable’), but you can use this hack as a guide when you have more complex situations, where you may need to decide programmagically on whether to discard the email or not.

ENISA 2007 study on security and anti-spam measures implemented by European providers

“Providers of electronic communication services are a vital element of the security chain when citizens and enterprises connect to Internet. ENISA studies provide an overview of the measures they take to secure their services and to fight against spam. The aim of this work is promoting best practices.

* ENISA 2007 study on security and anti-spam measures implemented by European’s providers.”

Ενδιαφέροντα σημεία προσοχής εν τάχει:

  • 56% των ISP θεωρούν το spam απειλή.
  • 80% των ISP προσπαθούν να έχουν σωστά DNS PTR records.
  • Το ένα τρίτο των ISP έχει business contigency plan και disaster recovery plan.
  • 90% των ISP προσφέρουν antispam filtering χωρίς χρέωση στους πελάτες τους.
  • 68% των ISP μπλοκάρουν είτε την κίνηση SMTP είτε όλη την κίνηση IP από πηγή προσδιοριζόμενη ως πηγή spam.
  • 81% των ISP έχει υλοποιήσει SMTP-AUTH.
  • 82% των ISP χρησιμοποιούν blacklisting.
  • 50% των ISP χρησιμοποιούν greylisting.
  • 75% των ISP κάνουν content based filtering.
  • Οι μισοί ISP έχουν θέσει περιορισμούς στον όγκο του εξερχόμενου email.
  • 60% των ISP εκτελεί virus scan στην εξερχόμενη αλληλογραφία.
  • 50% των ISP κάνουν block στο port 25 και μόνοι οι μισοί από αυτούς υλοποιούν Message Submission (port 587).

[via]

WTF? $25 per PTR RR per month?

Can this be an explanation of why many “eastern” mail servers do not have a valid PTR record? This is what came up on RFCI-discuss last week:

“If they want to set a PTR for an IP,they must ask to their ISP.Ok ISP can do that,but they require addtional money.It’s about 25 USD per month for a PTR record for each IP.”

Boy do I hate such business practices*. Some months ago I was forced to stop using FEATURE(`require_rdns’) because I had users complaining about communications with servers located at Middle Eastern countries.

Yeah I know, it is long time due for another custom hack dealing with exceptions.


[*] – Led by internet tourists no doubt.

More fun with message threading

When I try to write email-related code and the result fails my expectations, I use my plan B: Write it in c-client. I suppose the fact that I do not start with c-client from the beginning is a result of suffering from the Not Invented Here Syndrome.

The other day I was trying to decipher the semantics of Thread-Index: and Thread-Topic: since it seems that Microsoft has not placed any public information on them. Apostolos suggested that Thread-Index: takes BASE64 values, to which I replied negatively. After all, decoding

AcdyY+a08VX8xfobTsy61v9NHPZ7QA==

using perl -MMIME::Base64 -ne ‘print decode_base64($_);’ does not produce anything meaningful.

However I dug a little bit more, following this piece of advice from the imap-protocol list:

“Look at the evolution source code, it contains quite a bit of
information on this.”

camel-exchange-folder.c from the Evolution Exchange package reveals the following gem:

/* A new post to a folder gets a 27-byte-long thread index. (The value
 * is apparently unique but meaningless.) Each reply to a post gets a
 * 32-byte-long thread index whose first 27 bytes are the same as the
 * parent's thread index. Each reply to any of those gets a
 * 37-byte-long thread index, etc. The Thread-Index header contains a
 * base64 representation of this value.
 */

[ Update: Message Threading in Public Folders ]

Enough with trying to work with Thread-Index: then! JWZ has documented a very effective algorithm for message threading and c-client implements it (read docs/draft/sort.txt from the source code distribution):

SEARCHPGM *spg;
THREADNODE *thr;
:
spg = mail_newsearchpgm();
thr = mail_thread(ds, "REFERENCES", NIL, spg, 0);
walk_thread(thr, NIL);

(You are advised to read docs/internal.txt.)

The “REFERENCES” argument to mail_thread() instructs it to use jwz’s algorithm. The other option is to use “ORDEREDSUBJECT” (or as draft-ietf-imapext-sort-19.txt calls it: “poor man’s threading”). walk_thread() just prints the edges of the graph (actually it is a tree):

void
walk_thread(THREADNODE *thr, THREADNODE *prev)
{
        if (thr) {
                if (prev) {
                        printf("%d %d\n", prev->num, thr->num);
                }

                if (thr->next) {
                        walk_thread(thr->next, thr);
                } else {
                        printf("%d NIL\n", thr->num);
                }

                if (thr->branch) {
                        walk_thread(thr->branch, prev);
                }
        }

        return;
}

You may wish to use the output of the above routine (slightly modified) and feed it to dot, so that you can have an image display of the threads in the email collection that you study.

What is left to discuss a little bit more, is the THREADNODE structure: You can go from a THREADNODE to its first child via the next pointer (thr->next in the above example). If the THREADNODE has two children, then the second is a branch of the first (thr->next->branch). It if has three, the third is a branch of the second child (thr->next->branch->branch) and so on.

re: public spammer

Ο Allu Fun Marx σχολιάζει την διαφημιστική καμπάνια των Public που έβαλε στην πρίζα πολλούς bloggers. Μου ξύπνησε όμως τη διάθεση για κράξιμο, αιτία της οποίας ήταν ο ακόλουθος θόρυβος στο INBOX μου:

“Γειά σου Γιώργο,

Α! Δεν είσαι εντάξει… Ο φίλος σου ο Λουκάς (user@some.where.gr), σου έχει στείλει πρόσκληση και εσύ ούτε καν μπήκες στο κόπο να δεις τι παίζει! Μεγάλο λάθος… δωράκι ήθελε να σου στείλει, δίνοντας σου συμμετοχή στο θεϊκό διαγωνισμό της XX για ένα laptop ή μια οθόνη LCD και να κερδίσει και ο ίδιος μια συμμετοχή.
Μπες, δες και διάλεξε δωράκι, έστω και με μία φόρα μπορεί να κερδίσεις.
Αν θές λοιπόν, να κερδίσεις εσύ και ο φίλος σου πάτα εδώ.

Αν επιθυμείς να μην ξαναλάβεις email από το friendscounter πάτα εδώ.

Η επιτυχία των ανθρώπων πίσω από το friendscounter.gr είναι τεράστια. Δεν “έσπασαν” κανένα κανόνα κι όμως μπόρεσαν να έχουν στα χέρια τους μια ωραιότατη συλλογή από working email addresses (που συνειδητά υπέβαλλαν φίλοι μας) για τις οποίες έχουν και added value: σύνδεση ανάμεσα στις διευθύνσεις αυτές. Και δεν έχει καμία σημασία εάν εσύ επιθυμείς να λάβεις ξανά email από το friendscounter ή όχι. Η σύνδεση υπάρχει. Και να βρίσω το Λουκά, αυτό δεν γυρίζει πίσω.

Δε μπορώ παρά να βγάλω το καπέλο στους outsmarters. Με χαντρούλες κερδίζουν σε πληροφορία. Όπως γράφει και  ο Ματθαίος: “Είσαι τα δεδομένα σου.  Μην τα εκχωρείς“. Και τέλος πάντων εάν επιθυμείς να εκχωρείς τα δικά σου, μην εκχωρείς τα δικά μου.

(In-Reaction-To:)

Trying to make use of Outlook’s Thread-Index: header

tl;dr Finally the format of the Thread-Index: header is documented!

Recently I was in a situation where I had to reconstruct a thread of email messages using the Thread-Index: header which is used by Microsoft’s products, instead of the standard way of threading using Message-Id:, References: and In-Reply-To:

The truth is that I was really frustrated, thinking that Microsoft was breaking the standards using custom headers that do not begin with X- but as Dan Bernstein points out:

822 promised that the IETF would never define field names beginning with X-. It did not prohibit use of non-X names by other organizations.”

Which means that Microsoft is allowed to add Thread-Index: (and Thread-Topic:) without breaking any standards. On the other hand Microsoft does not document anywhere (at least anywhere I looked and I looked plenty) how Thread-Index: is calculated and how it can be decoded to be made useful by any other application, any other than Outlook that is.

After some experimenting and a little bit of reverse engineering I’ve reached to the following results:

  • Thread-Topic: preserves the original subject of the thread, that is the Subject: but stripped from any Fw: or Re: prefixes.
  • Thread-Index: is used in a way similar to In-Reply-To: and References: Assuming that the first message in a thread has a:
    Thread-Index: AcdyY+a08VX8xfobTsy61v9NHPZ7QA==

    and the next in thread:

    Thread-Index: AcdyY+a08VX8xfobTsy61v9NHPZ7QAAAiXbA

    while a third one:

    Thread-Index: AcdyY+a08VX8xfobTsy61v9NHPZ7QAAAiXbAAAXP5fw=

    and a fourth one:

    Thread-Index: AcdyY+a08VX8xfobTsy61v9NHPZ7QAAAiXbAAAXP5fwAABGXGw==

    the pattern that decides the threading seems obvious; I have not yet found out what the single or double equal sign suffix means.

If only Microsoft could make such simple information available! Think of all the lost work hours! Only after I had resolved my problem did I find out about these guys, who had arrived on similar conclusions about the usage of Thread-Index:

Update #1: You may be interested to read the next episode.

Update #2: Yes, I keep refusing the BASE64 explanation. This is because what the BASE64 value decodes to is something either meaningless, or without known semantics.

Update #3: From the GNOME documentation: The value is apparently unique but has no meaning we know of. That is why I refuse the BASE64 explanation. It looks like a BASE64 string and it can get decoded into a string of bytes that one can represent as a number. But the questions remain unanswered: How is the first 27-byte long value chosen? Why every “next” value in a thread 5 bytes longer than the previous one? How are these 5 bytes chosen? The decoded value of an undocumented BASE64 string remains undocumented, hence it may not even be a BASE64 string at all (and may only coincidentally look like one).


The example Thread-Index: headers are taken from the MediaDefender Defenders site

Mulberry opensourced!

Φρέσκο, φρέσκο από το INBOX μου:

Hi folks,
All the Mulberry code is now available as open source under an Apache 2 license. Details here:

<http://trac.mulberrymail.com/mulberry/wiki/opensource>

A mailing list has been setup for anyone interested in working on the Mulberry code. You can subscribe to the list at:

<http://lists.mulberrymail.com/mailman/listinfo/mulberry-dev>


Cyrus Daboo

Bye bye Thunderbird…