Tag: System Administration

Management and the Public Sector

Overheard this weekend:

The Number One rule of management in the Greek Public Sector is: “If you want something done, assign it to someone who already has too many assignments and is overloaded”.

This closely follows my decade long observation that management works with those who work, leaving the rest at peace.

The Deadline – A Novel about Project Management

Dimitris sent me “The Deadline” as a gift for my birthday. Written by Tom DeMarco (author of “Peopleware“) it is a novel that aims to introduce the reader to the complicate and cruel world of software project management. It also explains why most software projects fail. Clearly. In a buy-this-book-for-your-manager-to-open-his-eyes way. Team formation, design, quality control, unrealistic deadlines, goals and schedules, it is all in there. So if you need psychological support when a project goes bad, you should read the book. It is a good bus read.

It is also a book that opens doors to new worlds. Thanks to the book I learned about the adventures of Mr. Tompkins by George Gamow in which he aims to explain modern scientific theories to a popular audience. I see my stack of unread books getting higher again. I also learned about iThink which seems pretty cool (but then again I find Systems Thinking interesting enough). Pity though that iThink costs as much as it does (should I write my half-baked hack of systems thinking software? Damn! When I cannot buy, I try to write code instead and thus pay in time).

What would I change in the book? I would completely discard the very last chapter. Totally unnecessary. But no harm done, since the story is only the vehicle for the project management message and the message does get through. I’ve been lucky enough to have worked with managers like Mr. Tompkins; for this I want to end this post with the very first notes in Mr. Tompkins’s journal:

Four essentials of Good Management:

  • Get the right people
  • Match them to the right jobs
  • Keep them motivated
  • Help their teams to jell and stay jelled

(All the rest is Administrivia)

Amen to that!

The story of Server 54

I’m sure I’ve blogged about this before, but I cannot find it right now. Anyway the following tweet:

Beaker RT @etherealmind: Existential Angst 4 Network Engineers: If a Network Device isn’t Monitored, does it really exist? < Does when it goes down

brought to my attention by @DrInfoSec triggered my memory to recall the story of Server 54. A story that I reproduce here thanks to the Internet Archive:

The University of North Carolina has finally found a network server that, although missing for four years, hasn’t missed a packet in all that time. Try as they might, university administrators couldn’t find the server. Working with Novell Inc. (stock: NOVL), IT workers tracked it down by meticulously following cable until they literally ran into a wall. The server had been mistakenly sealed behind drywall by maintenance workers.

Digging a little bit more, one can find a few more discussions on Server 54.

reboot to fix

Appart from its historic value, “The Hacker Crackdown” is full of gems, like:

Starting over from scratch will generally rid the switch of any software problems that may have developed in the course of running the system. Bugs that arise will be simply wiped out by this process. It is a clever idea. This process of automatically re-booting from scratch is known as the “normal fault recovery routine”.

So you see this was not Bill’s idea in the first place.

Personally I hate reboot-to-fix. Rebooting must be a final(?) solution which in fact not only puts the problem under the carpet, but also deprives one of the possibility (and sometimes data) of finding out what the cause of the problem is. It is performed under pressure, under hurry and usually with no data at hand to replicate the problem and study it in a test environment and with some peace of mind. “Make the danm thing work first; find out what happened later! We’re losing money!” Downtime is an option and so routers and servers get reloaded… I will not sit in an ivory tower though pointing fingers, for I’ve practiced this “problem solving” technique a number of times.

Reboot-to-fix comes with a price: While at times it seems like a time saver, it only pushes forward in time the manifestation of the problem. At a later (and more inconvenient) time. And then it stops looking like a time saver. And if doing the same thing over and over expecting different results can be considered as a sign of paranoia, reboot-to-fix is just another sign of that.

Update: Some 12 days later, Paul Venezia wrote “When in doubt, reboot? Not Unix boxes“. Cool stuff!

TCPMUX – a mostly overlooked TCP service

TCPMUX is described in RFC-1078 (written some 20 years ago). A reference implementation by Network Wizards can be found at ftp://ftp.nw.com/nw/software/tcpmux.c . It is also implemented in DragonFlyBSD’s inetd, NetBSD’s inetd and FreeBSD’s inetd. OpenBSD does not support for it.

The Protocol

A TCP client connects to a foreign host on TCP port 1. It sends the service name followed by a carriage-return line-feed . The service name is never case sensitive. The server replies with a single character indicating positive (“+”) or negative (“-“) acknowledgment, immediately followed by an optional message of explanation, terminated with a . If the reply was positive, the selected protocol begins; otherwise the connection is closed.

The 15+ years I have been a sysadmin I have never seen anyone making a use of it, which is a pity: Most of the time I see fellow sysadmins who want to write a custom daemon, either write it as a standalone server (usually starting with passivesock.c or passiveTCP.c from Comer’s Internetworking with TCP/IP vol.3), or writing is as a simple stdin/stdout application that is started via inetd. The most trivial problem is sometimes more than trivial:

– What port will this application run on?

It seems that 65535 ports is a lot of freedom to choose from and most people want to use “interesting” port numbers (for any definition of interesting). Add firewall policies and router access lists in the picture, you can have a non-technical deadlock in no time!

TCPMUX might be a choice to help simplify / avoid such situations. Any service that supports TCPMUX listens on port 1/tcp and can be forked by inetd(8) (either internally or externally with the help of a tiny server). After all, it can be considered as an “inetd inside inetd” (the classic inetd responding to requests on a port, TCPMUX responding to requests based on the name of the service) and even if you do not want to use TCPMUX, a similar (homegrown) solution might be the answer to keeping your packet filters lean and less complex. It does not have to be less complex than it has to be though. The Wikipedia article on tcpmux clearly identifies risks that come with deploying it. Personally, I view tcpmux as an old and simple TCP RPC mechanism.

Appendix: tcpmux.c

Since the Network Wizards site seems to be down / taken over by some other entity, here is the original tcpmux daemon code (also at github https://github.com/a-yiorgos/tcpmux ):

Continue reading “TCPMUX – a mostly overlooked TCP service” →