being more flexible than FEATURE(compat_check)

A user at ServerFault asked how to restrict a user to send mail only to local addresses. Normally in sendmail, user / sender filtering decisions are done using FEATURE(compat_check), but while it does provide flexibility on deciding on specific pairs which are entries in /etc/mail/access, for more flexible stuff you have to write your own version of the check_compat rule set.

check_compat‘s workspace is a string that contains the addresses given in the MAIL FROM: and RCPT TO: SMTP dialog, separated by a $|. Whenever one works with addreses in sendmail, one has to canonify them, but since whatever rule set is called within another rule set always takes one argument (workspace) we have to use macros to store the canonified addresses before proceeding to any pattern matching. So first we have to declare the macros in our sendmail.mc:

LOCAL_CONFIG
Kput macro
D{put1}empty1
D{put2}empty2

The above snippet has declared a map (named put) and two macros that we will use to store the canonified addresses (named put1 and put2) initialized to some non empty bogus value. Since the workspace for check_compat is in the form sender address $| recipient address, we canonify the recipient address first:

Scheck_compat
R$* $| $*               $: $1 $| $>canonify $2
R$* $| $*               $: $(put {put2} $@ $2 $) $1

Up to here the rule set puts the canonified mail address for the recipient in ${put2} and returns the sender address (the last $1 in the second line) for further processing. Therefore we are now ready to repeat the process and store the canonified sender address in ${put1}:

R$*             $: $>canonify $1
R$*             $: $(put {put1} $@ $1 $)

Macro operations return an empty string so now we have to retrieve the addresses from the macros and reconstruct a canonified workspace for any further processing:

R$*             $: $&{put1} $| $&{put2}

This results in the workspace now being in the canonified form of:

sender < @ sender . domain . > $| recipient < @ recipient . domain . >

regardless of the multitude of ways one can express an email address in. This is why we need canonification in the first place: There are many ways one can enter an address in MAIL FROM: and RCPT TO: and canonification returns an address in a single format that all the other rule sets can work with.

Now if someone wants to restrict where a user sends mail based on MAIL FROM: and the recipient domain, one can add the following lines in check_compat:

# Now we can filter on sender and recipient
Ruser < @ $=w . > $| $* < $=w . >        $#OK
Ruser < @ $=w . > $| $*                  $#discard $: $2

The above silently discards email not directed to the local domains (Class $=w). If you want to test your rule sets (sendmail -bt) you have to keep in mind that sendmail’s test mode interprets $| as two characters, so you have to use a “translate hack”:

LOCAL_RULESETS
STranslate
R$* $$| $*    $: $1 $| $2

Now you can check check_compat by typing:

# sendmail -bt
> Translate,check_compat sender@address,recipient@address

and watch what happens. As always keep in mind that in sendmail.mc the left hand side of the rules is separated from the right hand side with tabs, not spaces. So do not copy-paste. Type the code instead. Next you need to compile your sendmail.cf and restart sendmail. In Debian as root run sendmailconfig to do this.

My eyes hurt! Can it be done another way?

Of course! You can install MIMEDefang together with sendmail and modify filter_recipient to your liking. Depending your operating system / distribution you have to check whether you need to enable filter_recipient or not. In Debian you have to edit /etc/default/mimedefang and restart the MIMEDefang daemon. After enabling it, you need to add in /etc/mail/mimedefang-filter your version for filter_recipient:

sub filter_recipient {
  my ($recipient, $sender, $ip, $hostname, $first, $helo, $rcpt_mailer, $rcpt_host, $rcpt_addr) = @_;

  $sender =~ s/^\<//;
  $sender =~ s/\>$//;
  $sender = lc $sender;
  
  $recipient=~ s/^\<//;
  $recipient=~ s/\>$//;
  $recipient = lc $recipient;

  # Put your conditions here
  ...

  return('CONTINUE', "ok");
}

You need to reload mimedefang-filter after editing this, so as root run (in Debian) /etc/init.d/mimedefang reload and check your logfiles for any errors.

sendmail sender queue groups

Sendmail provides for queue groups where one can have messages that stay in queue be placed in separate queues which are treated differently according to rules described in the queuegroup ruleset. FEATURE(queuegroup) helps managing such queues via the access database but unfortunately deals only with recipient addresses. But what if one wants to place messages in a separate (slower) queue based on sender’s address?

QUEUE_GROUP(`newsletter', `N=10, I=31m, P=/storage/queues/n.*')dnl

LOCAL_RULESETS
Squeuegroup
R$*             $: $>canonify $&f
R$* < @ $* > $*         $: $1
Rowner-newsletter         $# newsletter

The above trick does not make use of the access database. In fact you must not use FEATURE(queuegroup) in your sendmail.mc with it. The queuegroup ruleset is called with the recipient address as an argument. The first line replaces it with the sender’s address ($&f) canonified. In this particular newsletter case, we are only interested in the left hand side of the email address ($1). Others may be interested in the sender’s domain ($2). The third line checks to see whether the left hand side matches what we expect (owner-newsletter) and if so, it selects the corresponding queue. Otherwise the default queue, named mqueue, is selected.

For a more complete ruleset that can treat combinations of senders and recipients and via the access database see “Sendmail Extended Queue Groups“.

check_compat vs MIMEDefang

We have a user that wishes to have messages sent from sender@host-xyzw.etp.eu.example.com discarded by our mailservers. The natural choice for such blocks seems to be FEATURE(compat_check). In fact we had a number of other users with similar requests that were serviced this way. The problem in this case was that the xyzw part of host-xyzw.etp.eu.example.com was not constant or predictable and finite. Naturally I thought that a local version of the check_compat ruleset would suffice, since $*.eu.etp.example.com matches all possible such hostnames. But it seems that according to the bat book this cannot be done while also using FEATURE(compat_check):

Note that although with V8.12 and later you can still write your own check_compat rule set, doing so has been made unnecessary by the FEATURE(compat_check) (ยง7.5.7 on page 288). But also note that, as of V8.12, you cannot both declare the FEATURE(compat_check) and use this check_compat rule set.

Since I did not wish to tamper with our sendmail.mc this time, MIMEDefang came to the rescue: filter_relay is called with arguments both the sender and the receiver and that took care of it. But again, had I chosen to write this using sendmail’s language, it might have looked ugly, but it would also have been a one-liner (ugly but elegant in its own way).

mail hosted at Google, web server elsewhere

This post aims to cover two sets of questions that frequently appear on Serverfault:

“I have the email of my organization hosted at Google and the web server at a hosting provider. When the web server sends email (when a form is completed for example), email is received by everyone except when the recipient is in our domain. Then sendmail tries to deliver locally and not over at Google”. Or, “certain recipients, including Google, reject email from the web server (or servers withing our LAN) as spam”.

There are answers at Serverfault recommending the use of ssmtp in order to forward all sending email via Google, but this requires SMTP authentication and a password saved in a file.

For the purposes of this post the domain example.com will be used.

Configure SPF for example.com

SPF is framework that allows the domain name owners notify the world who they believe the appropriate servers sending mail on behalf of their domain are. Google support pages note that the SPF record should at least be in the form of v=spf1 include:_spf.google.com ~all. However, it is also needed that server.example.org be able to send email on behalf of example.com. So the appropriate record becomes:

v=spf1 a:server.example.org include:_spf.google.com ~all

Note: example.org is not the same domain as example.com

Configure sendmail for server.example.org

example.com is included in /etc/mail/local-host-names, which means that server.example.org treats this a local domain and will try to deliver locally, instead of Google. The following additions to the sendmail configuration file (sendmail.mc) take care of this:

LOCAL_CONFIG
Kbestmx bestmx -T.TMP

LOCAL_RULE_0
R $* < @ example.com. > $*
    $#esmtp $@ [$(bestmx example.com. $)] $: $1 < @ example.com. > $2

The line is broken in two for readability. As always remember that the LHS and the RHS of the rule are separated with tabs and not spaces. So do not copy-paste. Build and install sendmail.cf, restart sendmail and check.

I would welcome additions on how the same can be achieved with postfix or exim.

An alternative to FEATURE(mailertable)

Using FEATURE(mailertable) one can instruct sendmail to route email for certain destination via a specific relay. A mailertable is essentially a static map that instructs sendmail where to route email for certain destinations ignoring DNS MX RRs (or other information). Example:

yahoo.com   smtp:[server.example.com]
yahoo.com.hk   smtp:[server.example.com]
yahoo.com.mx   smtp:[server.example.com]
yahoo.com.br   smtp:[server.example.com]
yahoo.com.cn   smtp:[server.example.com]
yahoo.com.sg   smtp:[server.example.com]

Why would one want to do that? Your customers may have been hit by a botnet and as a result your outgoing mail server may have sent enormous amount of spam. Since most high-profile mail hubs use some kind of reputation scheme on the IP addresses that contact them, it is quite probable that your outgoing mail server is experiencing delays, or worse denied delivery despite the fact that in the meantime you have done your best to stop the botnet and clear your queues. I know for it has happened to me.

A mailertable is a quick solution to route email through another mail server just for recipient domains that implement such policies. But it is far from perfect for the Postmaster has no way to know all the domains that Yahoo! Mail in the above example hosts in order to construct a mailer table. Luckily, when high-profile mail hubs (like Gmail, Yahoo! Mail and Hotmail) implement good patterns on their DNS MX RRs, a programmatic (instead of a static) solution can be deployed:

LOCAL_CONFIG
Kbestmx bestmx -T.TMP

LOCAL_RULE_0
R$+ < @ $+ > $*         $: $(bestmx $2 $: NOTFOUND $) $| $1 < @ $2 > $3
R$+.hotmail.com. $| $+ < @ $+ > $*      $#esmtp $@ [server.example.com] $: $2 < @ $3 > $4
R$+ $| $+ < @ $+ > $*   $: $2 < @ $3 > $4

In the above snippet, any email that is directed to a domain that is served by Hotmail’s servers is routed via server.example.com. For the record, our outgoing webmail server achieved a senderscore of 50, and although a filter stopped the plaque, Hotmail silently discarded email originating from it. Using the above solution restored communications for our users.

Using bestmx for discarding outgoing email

The following ruleset discards email that originates from domains for which we are not best MX. It is meant to be applied on outgoing email servers:

LOCAL_CONFIG
Kbestmx bestmx -T.TMP

LOCAL_RULESETS
SLocal_check_mail
R$*                               $: $>canonify $1
# You may (or may not) want to comment the following line
R < @ >                           $#OK
R$* < @ $+. > $*          $1 < @ $2 > $3
R$* < @ $+ > $*                   $: $2
# Short circuit certain domains (and host names)
Rexample.com                           $#OK
R$* . example.com                      $#OK
R$*                               $: $(bestmx $1 $: NO $)
# If a temporary error occurs, do not block
R$*.TMP                           $#OK
Rserver.example.com.          $#OK
R$*                               $#discard $: $1

This works for as long as spammers do not use domains for which they do not control the DNS zones. If they do control the DNS zones they can easily add your relays as MX to them. In such cases the above ruleset must be modified to lookup the name servers for domains that server.example.com is best MX and then decide to discard. However the above trick erased thousands of outgoing spams yesterday.

PS: Like I posted on twitter: I rewrote the above filter in ~35 lines of Perl (subroutine filter_sender for MIMEDefang’s mimedefang-filter). The sendmail version is both more compact and readable (at least to me).

on picking an MTA

Sometimes I get asked on what is my MTA (Mail Transfer Agent) of choice. Almost always I am asking “What do you want to do with it?”. Personally, in most places I install sendmail. There are cases (cases where one would use FEATURE(nullclient) or similar) where I install nullmailer, for I find it unnecessary to run sendmail.

People sometimes ask me why do I choose sendmail and not Postfix (or Qmail in the old days) or even Exim since we are running a mostly Debian shop. Leaving the monolithic argument aside (which is kind of funny when most people that use it are using a monolithic kernel OS anyway) I am using sendmail because of its expressive power. I can find a way to express what I am thinking (filtering, routing, etc) in its modem noise of a programming language or milters like MIMEDefang (IIRC, there’s a wonderful PDF presentation by Ricudis on the Turing completeness of the sendmail.cf language but I have no link to it).

It is not that I have not used other MTAs. Hell, I was even running Postfix alpha versions right after it was renamed from VMailer. And occasionally I am running MeTA1 instances. But I always return to sendmail. If it does not suit you, it is OK. Pick the one MTA that can help you build the setup that you have in mind, be it Exim, Postfix, netqmail, commercial software like Exchange or CommuniGate, whatever. If it works for you and your team, then it is the right choice. Endless debates are for people who have too much free time.

However, if there is one recommendation that I can share, this is it: If you are serious about email (routing) invest some time reading the bat book. He who can understand a complex piece of software like sendmail, can guide himself through any email system.

(triggered by a brief conversation I had with a friend this afternoon)

sendmail load configuration

This post is about a neat trick that I have not seen many times discussed. According to the configuration README the default values for controlling load averages are:

  • confQUEUE_LA (QueueLA) Load average at which queue-only function kicks in. Default values is (8 * numproc) where numproc is the number of processors online (if that can be determined).
  • confREFUSE_LA (RefuseLA) Load average at which incoming SMTP connections are refused. Default values is (12 * numproc) where numproc is the number of processors online (if that can be determined).

However in “Sendmail Theory and Practice” (I am a proud owner of both editions) Paul Vixie and Fred Avolio propose a different approach:

“Astute readers will note that the value shown for Ox (QueueLA) is larger than the value shown for OX (RefuseLA), and that this is opposite from the configuration files you may have seen elsewhere. Setting them as shown here gives Sendmail a range of load average in which it is capable of delivering messages from its queue but incapable of receiving new messages. This is intentional. If you set Ox to be less than OX, Sendemail has instead a range of load average in which it can receive new mail (thus adding to the queue) but cannot deliver any queued mail. We believe that mail queues should become smaller or stay the same size when the load average is high. After watching our large mail gateway computers melt down many times over the years, we have learned that it is better to let other hosts’ mail stay where it was -on other hosts- when our load average is high, than to accept it even though we don’t plan to do anything with it until load average becomes low again.”

In other words although the defaults suggest otherwise, it may be wiser to have QueueLA > RefuseLA. This piece of advice is on both the 1995 (1st) and 2002 (2nd) editions of the book. A pearl that comes from 1995 that is still relevant.

sendmail: Should I use $*, $+ or $- ?

When writing a sendmail.mc rule, you can use some operators on the left hand side, like $- (match exactly one token), $+ (match one or more tokens) and $* (match zero or more tokens). You may find yourself in a situation where for example you want to use a certain delivery agent for some of your users. Normally you would write something like the following:

LOCAL_CONFIG
Kmitsos btree -m -a.mitsos /etc/mail/mitsosusers

LOCAL_RULE_0
:
R$-  < @ $=w . > $*              $: $(mitsos $1 $)  $3
R$- . mitsos  $*     $#mitsos $: $1

The above example seems to be correct, right? But what if you have a user in mitsosusers that contains a dot (.) in the user name (for example yiorgos.adamopoulos)? Because the . is a token separator (see the OperatorChars definition in your sendmail.cf), $- will not match the name. So the correct ruleset in this case is:

R$+  < @ $=w . > $*              $: $(mitsos $1 $)  $3
R$+ . mitsos  $*     $#mitsos $: $1

I got bitten by this sometime ago, and that is why I am sharing it.

Poor man’s milter-ahead

I have blogged before that the reason that I like MIMEDefang is that it gives the Postmaster a Perl interpreter (a programming language that is) and a library of functions that can be used to filter and manipulate incoming and outgoing email.

Of the functions available I believe that md_check_against_smtp_server() deserves special mention since it can be used to quickly implement a poor man’s milter-ahead or milter-sender. Of course milter-ahead implements many features (caching among others), but with some effort most (if not all) of the functionality can be implemented withing mimedefang-filter.

Then again, milter-ahead does not cost much (€90) even for small organizations, so the not invented here syndrome can be supressed.