Tag: email

email is still your best backup social network

We communicate through a host of applications using our own devices. In my phone I count 29 installed. Granted, not all of them are in frequent use, but all of them are installed for a reason: Some contact of mine is using some application, be it much popular, or less, or even something like Amazon Chime which you need to converse with AWS people.

Sometimes, infrastructures supporting those applications suffer outages and depending the userbase, pain gets expressed in Twitter, Facebook or LinkedIn, or even internal jokes like "Slack is down, let’s increase productivity until it gets up".

So it occurred to me, that even though every 5 years or so we see a new application claiming to solve texting, email is still the best backup solution:

  • even though it is mostly controlled by 5 big walled gardens, they are not one garden and you can always run your own with some varying success of compatibility if you wish to undertake the pain.
  • while I used to complain about "email not being FTP", it is still used as a file transfer mechanism.
  • while I used to complain about top posting in replies (I used to prefer inline replies), regardless of how ugly it may seem at times, you have a complete log of the communication history.
  • you can use your email store as document database (yes it was not designed for this, I hated the fact, but let’s move on; people search their emails to find stuff they remember being there).
  • while I used to complain a lot about HTML email, you still have the ability to at least send as plain or flashy emails as you like and they will be received and read (in the majority of cases) as you expect.
  • you are not confined to a single application for reading and sending email. People still write mail clients.
  • almost any booking / purchasing system you use sends you an email of what you transacted about. They do not seek your Skype, Messenger, Discord or what. They fall back to your best backup social network: email.
  • while it is not instant messaging, as I still tell people, most emails arrive soon enough to feel like it is.
  • it can be a 1:1 communication medium, 1 to many (a newsletter), or even many to many (mailing lists, Reply All).
  • spammers and other cyber villains still attack through it, so it is in wide use, even if it is not the cool kid in town.

I’ve come to terms with this kind of underground success: It does everything almost good enough and in the background. It is nobody’s first choice, but it is everybody’s common denominator because of the good enough factor.

Now if only I could persuade my children to fill the Subject: line when emailing me…

It turns out we do pay postage for our email

Well, not everybody, but some of us we do. Let me explain myself:

12 years ago after reading about TipJoy, a Y Combinator startup, I thought that this might be a scheme that could be used to force mass senders to pay something in order to ensure (via the investment cost to post to my inbox). I just thought of it kind the wrong way, and not in terms of snail-mail. I thought the recipient should be paid to read the email. But anyway TipJoy folded and life did its own thing and this was an idea left to collect dust.

It turns out that a whole industry focused on email delivery sprang in the meantime. IP reputation became a thing and the small server you operated from your house was part of a dialup or DSL pool that was used by spambots. No matter your intentions and your rigor in setting up your email server, your sender’s reputation was next to nothing. The same held true for your ISPs outgoing mail server too. The same is (still) true if you try to setup a VM to a cloud provider (if you’re allowed to send outgoing email at all). Once you needed to know about SMTP and POP3 only (IMAP too if you wanted to be fancier). Now you needed to learn new stuff too, SPF, DKIM, greylisting, RBL, DMARC, the list goes on. That’s how the sending industry was formed, providing guaranteed delivery, metrics like open-rate and more complex analytics.

Here I am now, some years later, and I am running a small mailing list for the Greek Database research community (I am not a database researcher, just a friend of the community). This mailing list always had reachability problems, even when I was running it on infrastructure I totally controlled when I was working on an ISP. Spam bots and inexperienced users always resulted in some sort of blacklisting that a single person postmaster team struggled to handle. There were delivery problems with a lot of personal time devoted to unblock them.

Since 2014 that I quit the postmaster business, I am running the list using Mailgun (It could have been any other M3AAWG member, I just picked them because of Bob Metcalfe mentioning them on twitter sometime). They used to provide a free service and the list was well within those limits, but they changed that. So there’s a monthly cost that varies from $1 to $10 depending the traffic. Delivery has been stellar ever since I switched to Mailgun and the few issues the mailing list had, were glitches from my side.

So it turns out that you do pay some postage to send your email after-all and the M3AAWG members are the couriers.

email will never die

These thoughts are triggered by the hey release, but they are otherwise unrelated.

Every now and then, an application comes along, grabs substantial userbase and claims to have solved the “issue” with email, texting and whatever memorabilia you want to share with your friends. Provided you’re part of their locked garden. Various pricing models are applied here. Math your heart out.

However, here is the thing: each one of them, requires your lock-in to that application. That’s how I ended up with 16 texting (or similar) applications on my phone. Because my contacts, personal and customers demand their favorite platform for my attention.

And that is why email will never die. It works most of the time and it does not matter who your provider is: Gmail, Yahoo, Hotmail, Protonmail, Fastmail, a local Exchange server, some postfix running on an RPi. It does not matter. Granted, when your message does not go through, there may be insurmountable hoops to jump, but in the end, email is still the ubiquitous text application that works for everyone, everywhere between big walled gardens and even your own backyard if you care to run it on your own. And that is why, even though email is not instant messaging, people (OK not my kids) still treat it like it being such.

I am still undecided whether to invest in Hey. The application looks good, the web client also, but the cost of switching from my current walled garden (that also offers some identity services) is big.

How I run a small (discussion) mailing list

I used to run a fairly large (for two Ops people) mail system until 2014. I did it all, sendmail, MIMEDefang, my own milters, SPF, and a bunch of other accompanying acronyms and technologies. I’ve stopped doing that now. The big companies have both the money and the workforce to do it better. But I still run a small mailing list. It used to run on majordomo but then it started having issues with modern Perl and I moved it to Mailman. Which is kind of an overkill for just a list with 300 or so people on. So for a time I even run it manually using free GSuite (yes there was a time it was free) hosting.

Lately I wanted to really run it automated again. So I thought I should try something different that would also contribute to the general civility and high SNR of it. I’d been lurking around the picolisp mailing list for years and thought I should use it. Because it comes with a mailing list program:

#!bin/picolisp lib.l
# 19apr17abu
# (c) Software Lab. Alexander Burger

# Configuration
(setq
   *MailingList "foobar@example.com"
   *SpoolFile "/var/mail/foobar"
   *MailingDomain "server.example.com"
   *Mailings (make (in "/home/foobar/Mailings" (while (line T) (link @))))
   *SmtpHost "localhost"
   *SmtpPort 25 )

# Process mails
(loop
   (when (gt0 (car (info *SpoolFile)))
      (protect
         (in *SpoolFile
            (unless (= "From" (till " " T))
               (quit "Bad mbox file") )
            (char)
            (while (setq *From (lowc (till " " T)))
               (line)  # Skip rest of line and "\r\n"
               (off
                  *Name *Subject *Date *MessageID *InReplyTo *MimeVersion
                  *ContentType *ContentTransferEncoding *ContentDisposition *UserAgent )
               (while (trim (split (line) " "))
                  (let L @
                     (while (and (sub? (peek) " \t") (char))  # Skip WSP
                        (conc L (trim (split (line) " "))) )
                     (setq *Line (glue " " (cdr L)))
                     (case (pack (car L))
                        ("From:" (setq *Name *Line))
                        ("Subject:" (setq *Subject *Line))
                        ("Date:" (setq *Date *Line))
                        ("Message-ID:" (setq *MessageID *Line))
                        ("In-Reply-To:" (setq *InReplyTo *Line))
                        ("MIME-Version:" (setq *MimeVersion *Line))
                        ("Content-Type:" (setq *ContentType *Line))
                        ("Content-Transfer-Encoding:" (setq *ContentTransferEncoding *Line))
                        ("Content-Disposition:" (setq *ContentDisposition *Line))
                        ("User-Agent:" (setq *UserAgent *Line)) ) ) )
               (if (nor (member *From *Mailings) (= "subscribe" (lowc *Subject)))
                  (out "/dev/null" (echo "^JFrom ") (msg *From " discarded"))
                  (unless (setq *Sock (connect *SmtpHost *SmtpPort))
                     (quit "Can't connect to SMTP server") )
                  (unless
                     (and
                        (pre? "220 " (in *Sock (line T)))
                        (out *Sock (prinl "HELO " *MailingDomain "^M"))
                        (pre? "250 " (in *Sock (line T)))
                        (out *Sock (prinl "MAIL FROM:" *MailingList "^M"))
                        (pre? "250 " (in *Sock (line T))) )
                     (quit "Can't HELO") )
                  (when (= "subscribe" (lowc *Subject))
                     (push1 '*Mailings *From)
                     (out "Mailings" (mapc prinl *Mailings)) )
                  (for To *Mailings
                     (out *Sock (prinl "RCPT TO:" To "^M"))
                     (unless (pre? "250 " (in *Sock (line T)))
                        (msg T " can't mail") ) )
                  (when (and (out *Sock (prinl "DATA^M")) (pre? "354 " (in *Sock (line T))))
                     (out *Sock
                        (prinl "From: " (or *Name *From) "^M")
                        (prinl "Sender: " *MailingList "^M")
                        (prinl "Reply-To: " *MailingList "^M")
                        (prinl "To: " *MailingList "^M")
                        (prinl "Subject: " *Subject "^M")
                        (and *Date (prinl "Date: " @ "^M"))
                        (and *MessageID (prinl "Message-ID: " @ "^M"))
                        (and *InReplyTo (prinl "In-Reply-To: " @ "^M"))
                        (and *MimeVersion (prinl "MIME-Version: " @ "^M"))
                        (and *ContentType (prinl "Content-Type: " @ "^M"))
                        (and *ContentTransferEncoding (prinl "Content-Transfer-Encoding: " @ "^M"))
                        (and *ContentDisposition (prinl "Content-Disposition: " @ "^M"))
                        (and *UserAgent (prinl "User-Agent: " @ "^M"))
                        (prinl "^M")
                        (cond
                           ((= "subscribe" (lowc *Subject))
                              (prinl "Hello " (or *Name *From) " :-)^M")
                              (prinl "You are now subscribed^M")
                              (prinl "****^M^J^M") )
                           ((= "unsubscribe" (lowc *Subject))
                              (out "Mailings"
                                 (mapc prinl (del *From '*Mailings)) )
                              (prinl "Good bye " (or *Name *From) " :-(^M")
                              (prinl "You are now unsubscribed^M")
                              (prinl "****^M^J^M") ) )
                        (echo "^JFrom ")
                        (prinl "^J-- ^M")
                        (prinl "UNSUBSCRIBE: mailto:" *MailingList "?subject=Unsubscribe^M")
                        (prinl ".^M")
                        (prinl "QUIT^M") ) )
                  (close *Sock) ) ) )
         (out *SpoolFile (rewind)) ) )
   (call "fetchmail" "-as")
   (wait `(* 4 60 1000)) )

# vi:et:ts=3:sw=3

You do not have to understand a lot of Lisp to know how this is configured.
– Line 7 is where you put your mailing list’s address
– Line 8 is where incoming mail for the mailing list is saved. If you have a mail server and you run this there, just decide to run the list as a plain user and point to the user’s mailbox under /var/mail. Or, if you run fetchmail point to the file fetchmail saves incoming mail. See line 96 for that.
– Related to the above: Since I run this on my mail server, I delete line 96.
– Line 9 is my machine’s HELO/EHLO name.
– Line 10 is where the list membership is saved.
– Line 11 is the outgoing mail server.
– Line 12 is the outgoing’s mail server SMTP listening port.

Since my own incoming mail server is on a cloud provider and does not enjoy good sending reputation, I am making use of mailgun as a forwarding service. My mail server forwards to mailgun and mailgun delivers to the recipients. If you need to run your own small mail server there are tons of tutorials out there. While I am a die-hard sendmail person, I am running Postfix these days.

The final issue that remains is how to run the mailing list processor. mailing is a console program and you need to run it as a daemon somehow. Docker to the rescuse. I am building an image with the following Dockerfile:

FROM debian:buster
RUN apt-get update && apt-get install -y picolisp
WORKDIR /usr/src
COPY ./mailing .
CMD ./mailing

and I am executing this with:

docker run -d --name=foobar --restart=unless-stopped -v /var/mail/foobar:/var/mail/foobar -v /home/foobar/Mailings:/home/foobar/Mailings foobar_image

I can even inspect the logs with docker logs foobar and have made it resilient through reboots with one command.

There. I hope this gives you some ideas on how to run your own (small) mailing list. With some tinkering, you do not even need to run your own incoming mail server. It can be a mailbox in Gmail or elsewhere, you fetch mails with fetchmail locally, and submit to a relay service and done. Also note: Relay services require payment after some threshold.

addressing

I’ve started reading John Day’s Patterns in Network Architecture and during the first pages it makes strong references to Saltzer’s 1982 paper. Why would I bring this up? Well, I just heard Surprisingly Awesome‘s episode on Postal codes where they deal with two countries (Lebanon and Mongolia) with almost non-existent addressing plans. Here is what an addressing plan should give you:

  • a name identifies what you want,
  • an address identifies where it is, and
  • a route identifies a way to get there

Day makes the case that we usually use that address of a network element in the same way that we use its name also which is an error, since by moving an element elsewhere in the network, we need to change its name also.  You on the other hand do not change your name when you change your home address. You used to change your phone number, but even that has become equally portable.

In places where no stable addressing system exists the courier is required to build a mental representation of the routes in their area of delivery, based on landmarks, trees, neon signs, whatever can help to make the delivery. In Mongolia this is solved differently: When something arrives at the post office, they call you back and you go and pick it up.

Enter the NAC. What is it exactly? It is an effort to map longitude and latitude to a more memorable representation using the base 30 number system using digits and capital letters. Borrowing from Wikipedia, the NAC for the centre of the city of Brussels is HBV6R RG77T. Compact, accurate, but not quite memorable.

what3words seems to be a service set to solve this since with their solution a unique combination of just 3 words identifies a 3m x 3m square anywhere on the planet. For example, roughly the same place as above is described as october.donor.outlined. I admit, this is much easier to type in a GPS (or tell Siri).

However, I am still surprised that nobody ever thought of using IPv6 for this (maybe somebody has? Please tell me). Given the abundance that the 128bits give us, we could have indexed every square meter on the surface of the planet and make it addressable. Oh, the directories we could have built on top of that. But I have no fear. It is quite probable that much of the inhabited First World’s surface will be pingable in the foreseeable future. The IoT will make sure of that.

 

So now we get spammed via Github :)

So the following popped up in my mailbox:

It is nice and joyful to see your profile on https://github.com and i thought is beautiful to make you a friend,looking for a good friendship, mail me to my box, because i am not often here on xxx@yyy.zzz for more introduction and i will send you my picture.

I guess it was bound to happen. Anywhere you allow messaging to occur, spam will follow. I only wonder why it took so long to happen via a popular service like this.

A sure way to get into a spam blacklist

Well here is one: Trying to deceive me. No I do not believe the error message returned from your remove.php:

There was an error processing your request; please manually send an email to with Unsubscribe as its subject

And you know why is that I do not believe you? Because I never subscribed via this email address.  In your newsletter you say you want to disrupt the industry (which industry you leave it vague). So far you’re only annoying.

milter-greylist

After years of using graymilter (with a series of local hacks) I switched to milter-greylist.

After it run for a few days:

# Summary: 149173 records, 137182 greylisted, 11991 whitelisted, 0 tarpitted

and with only a few tweaks in its configuration:

racl whitelist domain google.com
racl whitelist domain googlemail.com
racl whitelist domain gmail.com
racl whitelist domain yahoo.com
racl whitelist domain hotmail.com
racl whitelist domain live.com
racl whitelist domain outlook.com
racl whitelist domain amazon.com
racl whitelist domain ebay.com
#racl whitelist domain gr

racl greylist default

You can apt-get install milter-greylist (which makes maintenance through OS upgrades manageble) and it has all the features that I would love to add in my series of hacks to graymilter but never got around to doing so.

(previous)