Read the source Luke – part 1

A fellow (and very competent) postmaster of an extremely high traffic ISP (by Greek standards) dropped me a note asking me whether I had seen the following error message:

Jan 25 11:07:43 XXXXXXXXXXX sm-mta[21622]: l0M7JEYx016921: SYSERR(root): Error in ldap_search using user@YYYYYYYYYYYY.gr in map ldapmra: Unknown error 325

Being the LDAP hater that I am, I had never encountered it. Googling arround did not result in anything meaningful. So the next step was to read the source code:

First I downloaded and uncpaked the latest sendmail (8.13.8) and openldap (2.3.32) sources. Now where do I start? The error message points where:

Inside the sendmail-8.13.8 directory:

$ grep -r "Error in ldap_search" .
./sendmail/map.c: syserr("Error in ldap_search using %s in map %s",
./sendmail/map.c: syserr("451 4.3.5 Error in ldap_search using %s in map %s",

syserr() is called when sm_ldap_search() fails and errno is set by calling sm_ldap_geterrno():

errno = sm_ldap_geterrno(lmap->ldap_ld) + E_LDAPBASE;

What is E_LDAPBASE? We find it defined in include/sm/errstring.h:

#define E_LDAPBASE (E_PSEUDOBASE + 70)

OK, now what is E_PSEUDOBASE? We find it defined again in include/sm/errstring.h:

#ifndef E_PSEUDOBASE
# define E_PSEUDOBASE 256
#endif /* ! E_PSEUDOBASE */

Hmm… the valuse of E_LDAPBASE is 256 + 70 = 326? Now does this not look familiar? So what makes sm_ldap_geterrno() return -1?

$ grep -r ^sm_ldap_geterrno .
./libsm/ldap.c:sm_ldap_geterrno(ld)

sm_ldap_geterrno() basically calls:

(void) ldap_get_option(ld, LDAP_OPT_ERROR_NUMBER, &err);

and returns the value of err.

Now we switch to the openldap-2.3.32 source directory:

$ grep -r ^ldap_get_option .
:
./libraries/libldap/options.c:ldap_get_option(

from which we find out that in our case it returns the following value:

* (int *) outvalue = ld->ld_errno;

So who sets the value of ld->ld_errno to -1? Back in the sendmail sources we see that we followed this branch of the code because of a call to sm_ldap_search():

$ grep -r ^sm_ldap_search() .
./libsm/ldap.c:sm_ldap_search(lmap, key)

sm_ldap_search returns the value of a call to ldap_search(). Switching back to openldap’s sources:

$ grep -r ^ldap_search .
:
./libraries/libldap/search.c:ldap_search(
:

ldap_search() returns the result of ldap_send_initial_request():

$ grep -r ^ldap_send_initial_request
./libraries/libldap/request.c:ldap_send_initial_request(

which in turns returns the result of ldap_server_request():

$ grep -r ^ldap_send_server_request .
./libraries/libldap/request.c:ldap_send_server_request(

It sets ld_errno to -1 in two cases. The first is when it sets it to LDAP_SERVER_DOWN (which is #defined as (-1) in ldap.h) and when ldap_int_flush_request() fails returning -1:

$ grep -r ^ldap_int_flush_request .
./libraries/libldap/request.c:ldap_int_flush_request(

ldap_int_flush_request() returns -1 when ber_flush() fails and sets ld_errno to LDAP_SERVER_DOWN.

So there:

Unknown error 325 means that your sendmail cannot talk to your LDAP server because sendmail thinks that slapd is down.

Or not?

(part2)

smrsh and Debian sendmail

If you run sendmail from the Debian (Etch) packages and you use FEATURE(`smrsh’) then you must put the symlinks of the commands that you want smrsh to execute under /etc/mail/smrsh and not under /usr/lib/sm.bin . The Debian manpage is not particularly helpful on this.

From sendmail’s configuration README:

smrsh		Use the SendMail Restricted SHell (smrsh) provided
		with the distribution instead of /bin/sh for mailing
		to programs.  This improves the ability of the local
		system administrator to control what gets run via
		e-mail.  If an argument is provided it is used as the
		pathname to smrsh; otherwise, the path defined by
		confEBINDIR is used for the smrsh binary -- by default,
		/usr/libexec/smrsh is assumed.

sendmailX renamed to MeTA1

From Claus Assmann:

Here's the latest status update:

For various reasons, the software has been renamed to MeTA1.
The mailing lists will be renamed accordingly:

domain:
	sendmailx.org	-> MeTA1.org
localparts:
	smx-developers	-> MeTA1-developers
	smx-design	-> MeTA1-design

For now, I just copy all addresses from the old lists to the new
list. If you don't want to participate any longer, please contact
me directly.

As you probably noticed, I took some time off from working on the
MTA (only fixing bugs if a problem showed up), but now I'm back and
one of my current projects is to look into integrating Arena as
scripting language (see some earlier mails about this topic).

I will send out some more information "soon" on the new list,
hopefully you will adjust your filters by then.

More information about MeTA1 and the Arena scripting language.

Avoiding FEATURE(`nullclient’)

From the sendmail cf/README:

nullclient	This is a special case -- it creates a configuration file
		containing nothing but support for forwarding all mail to a
		central hub via a local SMTP-based network.  The argument
		is the name of that hub.
		The only other feature that should be used in conjunction
		with this one is FEATURE(`nocanonify').  No mailers
		should be defined.  No aliasing or forwarding is done.

However, you do not have to run sendmail1 on a server unless it is your SMTP server. Sendmail has FEATURE(`nullclient’) for that. Qmail has something similar. But there is nothing simplest than this alternative:

Enter nullmailer:

This is nullmailer, a sendmail/qmail/etc replacement MTA for hosts which relay to a fixed set of smart relays. It is designed to be simple to configure, secure, and easily extendable.

Nullmailer is very easy to configure compared to any of the well known F/OSS SMTP servers. It is not an SMTP server. It is a /usr/lib/sendmail2 replacement for your servers or unix shell machines.

Installing nullmailer on a Debian system simply means:

# apt-get install nullmailer
# dpkg-reconfigure nullmailer

and you are all set3. If you are on another system the worst case senario requires that you compile it by hand. However, this is not a difficult task even for the inexpirienced SysAdmin, since the code is compact and easy to read and the directions on how to compile / install fairly simple to follow.

[1] Or postfix, or exim, or qmail
[2] Yes, there was a time that the sendmail binary lived in /usr/lib
[3] Usually this is followed by a dpkg –purge exim4-base exim4-config

configuring sendmail for your unix desktop

Mulberry, (or Thunderbird,or any other GUI email client) needs an SMTP (“outgoing”) mail server which it uses to send your emails. Normally this is set to what your ISP (or administrator) has told you to be. However, sometimes your outgoing email server maybe unavailable1 (which is furstrating, since most people expect email to be delivered instantly, or at least leave their system2 instantly). So why not run a mail queue on your own and let your desktop route your email through the outgoing email server when it becomes available again? Here is a small sendmail.mc that should work with your desktop unix machine3:

VERSIONID(`2006/08/23/00')dnl
OSTYPE(`debian')dnl
DOMAIN(`debian-mta')dnl

define(`SMART_HOST', `[outgoing.example.com]')dnl
define(`confCW_FILE', `-o /etc/mail/local-host-names')dnl
FEATURE(`use_cw_file')dnl

FEATURE(`no_default_msa')dnl
DAEMON_OPTIONS(`Name=MTA-v4, Addr=127.0.0.1, Port=smtp')dnl
DAEMON_OPTIONS(`Name=MSP-v4, Addr=127.0.0.1, Port=submission')dnl

MAILER(`local')dnl
MAILER(`smtp')dnl

LOCAL_RULE_0
# LHS is separated from RHS with tabs, not whitespaces
R$- <@ $=w . >          john.doe < @ example.com. >

LOCAL_RULE_0 makes sure that any email sent from this machine is sent as john.doe@example.com. If this is not expected behavior delete LOCAL_RULE_0 or use FEATURE(nullclient).

[1] Unavailable does not mean neccessarily down. It may mean that some kind of rate-limiting is in place.
[2] Where system for such users is only their MUA (Mail User Agent).
[3] The example is Debian/Ubuntu-centric, but can be adapted to any system that runs sendmail

bogusmx.rfc-ignorant.org

RFCI publishes a number of usefull lists that can be used to effectively filter incoming email. One of the most usefull is bogusmx.rfc-ignorant.org which essentialy is a DNSBL that contains domain names with bogus MX servers (like localhost, 0.0.0.0 and so on):

” If any publicly listed MX record for domain contains a hostname which points to bogus IP address space, such as those documented in RFC 3330, or if the domain contains an MX RR that points to an IP address, in violation of RFC 1035 or if the domain has MX RRs which point to hostnames which themselves do not have an associated A record (including MXs which return an NXDOMAIN, or which are CNAMEs)”.

If you are a sendmail user,the trouble with bogusmx.rfc-ignorant.org is that you cannot use it with FEATURE(dnsbl). So you may need to write your own sendmail ruleset. Like this:

LOCAL_CONFIG
# Normally these should not be defined, since Kdnsbl does exactly what
# Kbmx_check does, and Kmacro does what Kbmx_macro does.
Kbmx_check dns -R A -a.FOUND -T.TMP
Kbmx_macro macro

LOCAL_RULESETS
SLocal_check_mail
R$*			$: $>canonify $1
R$* < @ $+ . > $*	$1 < @ $2 > $3
R$* < @ $+ > $*		$: $2
R$*			$: $(bmx_macro {Bmx} $@ $1 $) $1
# The next line is broken in two for readability
R$*			$: $(bmx_check $&{Bmx}.bogusmx.rfc-ignorant.org. $: $1.NOTFOUND $)
# The next line is broken in two for readability
R$* . FOUND		$#error $@ 5.7.1 $: Mail from $&{Bmx} refused.It is listed in bogusmx.rfc-ignorant.org.

You can grab the above sendmail code fragment from here. Always be careful if copy-pasting because sendmail uses tabs and not spaces to distinguish between LHS and RHS.

An alternative on how to use domain based blacklist zones is shown by RFCI here (and also has pointers for mail servers other than sendmail).

virbl.tee.gr

Actually my previous post, together with me using virbl.dnsbl.bit.nl made me think how one can share such data between many email servers under the same administrative domain. Well the answer is fairly easy:

Publish them using rbldnsd. Rbldnsd reads text files, so all you have to do is dump the B-Tree data to a text file of type ip4tset (read rbldnsd(8)):

makemap -u btree /var/cache/local/virbl/virbl.db | awk '{print $1}'

Then instruct sendmail / postfix / whatever email server you are using to use the zone you publish with rbldnsd as a DNSBL.

At this time we have a rate of blocking ~3500 connections daily from infected machines with no complaints by any user (ours or remote).

You may use virbl.tee.gr at your own risk. Machines are listed in and delisted from it automatically. Any machine that gets listed is delisted within an hour. Sending viral email to our mail servers results in relisting it. It’s governing policy is described at http://www.postmaster.tee.gr/ in Greek.

An interesting observation I have made is that (infected) machines that target one set of mail servers do not necessarily target another set (eg. your servers) within the hour. So instead of using virbl.tee.gr (if you find it a good idea) it might suit you better to implement a similar scheme for your servers.

MIMEDefang and virii

OK so you use MIMEDefang together with ClamAV[*] to check incoming messages for viral content. But given the fact that an infected machine will bomb you with many many messages, why should you check every message sent for a given time window? This is what I came up with:

The default mimedefang-filter(5) has the following check which discards viral messages:

if ($FoundVirus) {
  md_graphdefang_log('virus', $VirusName, $RelayAddr);
  md_syslog('warning', "Discarding because of virus $VirusName");
  return action_discard();
}

Changing it to:

if ($FoundVirus) {
  # OK log $RelayAddr
  # If you are on a Debian-like system you have to put
  # use DB_File in /etc/mail/mimedefang.pl.conf
  # otherwise you have to put it somewhere in mimedefang-filter
  my %vbl;
  my $now;
  tie %vbl, 'DB_File', "/var/cache/local/virbl/virbl.db", O_CREAT|O_RDWR, 0644, $DB_BTREE or die;
  $now = time;
  $vbl{$RelayAddr} = $now;
  untie %vbl;
  md_graphdefang_log('virus', $VirusName, $RelayAddr);
  md_syslog('warning', "Discarding because of virus $VirusName");
  return action_discard();
}

logs $RelayAddr (the IP address of the infected machine) together with a timestamp in a BerkeleyDB B-Tree. In our example this is /var/cache/local/virbl/virbl.db. You have to make this file writeable by the user that runs MIMEDefang on your system. And now using the following sendmail.mc code one can block this IP address prior to inspecting the message content:

LOCAL_CONFIG
# .db is appended by sendmail automagically
Kvirbl btree -a.FOUND /var/cache/local/virbl/virbl

LOCAL_RULESETS
# Always remember:  In sendmail the LHS and the RHS of the sendmail.mc/.cf is 
# separated with tabs and not spaces.  So do not copy-paste this fragment,
# type it.
SLocal_check_relay
R$*                     $: $&{client_addr}
R$*                     $: $(virbl $1 $: $1.NOTFOUND $)
# The next line broken in two for readability
R$* . FOUND             $#error $@ 5.7.1 $: You have sent us mail containing
           a virus and are blocked from our systems for an hour.

So now you need an expiration proccess. How long shall these IP addresses remain in your database? I keep them for one hour. It seems to be a reasonable default. A simple expiry script is the following perl snippet:

#!/usr/bin/perl
use DB_File;
$db = shift or die;
$threshold = shift or die;
tie %d, 'DB_File', $db, O_RDONLY, 0644, $DB_BTREE;
$now = time;
foreach $i (keys %d) {
        $diff = $now - $d{$i};
        if ($diff > $threshold) {
                delete $d{$i};
        }
}
untie %d;

You can run this script from cron every ten minutes or so. I’ve written my expiry program in C and run it every two minutes. If you also want to do this, you have to remember that the perl snippet on mimedefang-filter that logs $RelayAddr and the timestamp stores the timestamp as a string and not as an integer.

[*] There exist many HOWTOs on how to setup MIMEDefang to work with ClamAV. Just use Google.

secsup.org

Due to a customer having being stuck with a virus I found out this today:

http://www.secsup.org/complaints/

So I added these lines in our outgoing mail servers' sendmail.mc (for the future):

#Remember that sendmail distinguishes between LHS and RHS with TABS not spaces
LOCAL_CONFIG
F{MCI_DOMAINS} -o /etc/mail/mci-domains

LOCAL_RULESETS
SLocal_check_rcpt
R$*         $: $>canonify $1
R$- < @ $={MCI_DOMAINS} . > $*       $#error $@ 5.1.3 $: "Cannot send mail there.  See http://www.secsup.org/complaints/"

with /etc/mail/mci-domains containing:

mcimail.com
internetmci.com
networkmci.com
internetmci.net
networkmci.net

Using clamscan inside /etc/procmailrc

[ Originally I wrote this in June 2, 2004 ]

If for whatever reason you want to use ClamAV with sendmail and cannot use the clamav-milter (like when working with RedHat ES and its stock sendmail), using procmail as your local delivery agent might help. This is a very simple /etc/procmailrc:

# adamo, Thu May 27 16:26:48 EEST 2004

# Set the default path
PATH=/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin

# Uncomment this to monitor the recipes
#LOGFILE=/var/log/procmail.log

# NEVER forget the last dash (-) on the follwoing command
:0 Wc: /var/tmp/clamscan.${PPID}.$$.lock
| /usr/local/bin/clamscan --stdout --tempdir=/var/tmp --quiet --mbox -

# Delete emails that contain virii
:0 e: /var/tmp/formail.${PPID}.$$.lock
/dev/null

# Tag emails that contain virii
## :0 eWf: /var/tmp/formail.${PPID}.$$.lock
## | /usr/bin/formail -i "Subject: VIRUS FOUND -- PLEASE DELETE ME"

See also at the ClamAV wiki: ClamAndProcmail.