Tag: System Administration

Learn to say “No”

Users consider their needs top priority. Not only that, but when they pick up the phone or press the send button of their email client, they demand immediate service. System Administrators on the other hand are trained (over time) to objectively distinguish between real emergencies (threats to the organization’s business operation if not dealt with) and the rest.

So whenever an urgent situation arises, step back and ask yourself:

– Urgent for whom?
– Why is this urgent?
– Is there a process missing here??

These are important questions, especially if there exists no process covering the situation. Organizations have written workflows that define processes, but operate on the evolution of those rules which are mostly undocumented. If you identify a missing process, your reaction to the matter will create a process, no matter what. Solve the problem as a fireman and you have just created a process with your name hardcoded in it. Not your team, your name. People will look for you.

Identify that it is about a missing process problem that needs to be fixed and everybody will promise to you that it will be dealt with. Only it will not. The next time it arises, they will come back to you, because you did it the first time and now you are (informally) in charge of “those things”.

This is neither good for you nor for your employer. So the need to say “No, I will not fix that this way. Create a process and I will” arises. It is to the benefit of your employer to do so. It makes certain that for this particular situation they are not depended from a single person (you). It also protects your time, weekends and vacation. Explain this to upper management. Learn to say “No” in a productive way. Make sure this is not misunderstood as BOFHiness from your part. Put a price tag on what it means not doing it the formal way.

As a System Administrator you do not only manage the computers in your organization. You manage the people using them too. You manage human-computer systems. And whenever there is a void in the workflow, you need to do your best to create a process. Otherwise it will be created without your intervention. You do not want that. You are the System manager.

Eliminating unnecessary processes is part of the job too, but this is maybe for another blog post.

You’re welcome

Today is the 11th Annual System Administrator Appreciation Day. Make some time and go thank your system administrator. It will buy you 364 days of less BOFHiness and non-grumpy support. It will actually show to your system administrator that his unpaid overtime and weekends in the terminal room, are really worth the effort.

Not only is System Administration a thankless job, it is also considered an unneeded burden, a low profile and accordingly paid service. There are even people under the illusion that cloud computing will make system administration obsolete. For them I paraphrase what Bob Lucky wrote in May 1998: System Administration will be in danger of shrinking into a neutron star of infinite weight and importance, but invisible to the known universe. Do you really want to make a handful of us that important for you?

Who do you think is going to administer the Cloud? A system adminisrator. And, again to paraphrase Bob Lucky from the same article, that System Administrator will be extremely well paid, and his or her every demand will be immediately satisfied. That engineer will be the last keeper of the secret of the (cloud) universe: The Root Password.

To those who really do not see it, the Cloud does not make System Administration obsolete. It only changes the way it is practiced. There are Systems built on and into the Cloud that need administration. What the developers of those systems do not see, is themselves becoming the system administrators of the systems (with increasing complexity and dependencies that) they build.

Go and thank your system administrator today. Really do. He does provide you with at least an above the average service. For if he does not, you cannot even begin to do your job, no matter how important (or urgent) you think it is.

You’re welcome.

Cybersyn

Πρωτοδιάβασα για το Cybersyn σε αυτό το blog post του Mike Trick. Πρόκειται για ένα τηλεπικοινωνιακό δίκτυο που έστησε στη Χιλή η Κυβέρνηση του Αλιέντε με μηχανές telex (που βρήκε σε αχρηστία από την προηγούμενη Κυβέρνηση). Project manager του έργου ήταν ο Stafford Beer (ένας πρωτοπόρος της Επιχειρησιακής Έρευνας που πίστευε στο συνδιασμό Cybernetics και managemet). Το έργο έτρεξε από το 1970 έως και το πραξικόπημα του 1973.

Όπως φαίνεται και από το άρθρο του Guardian η μεγαλύτερη επιτυχία του έργου ήρθε στην απεργία του 1972 με 50000 φορτηγατζήδες να κλείνουν δρόμους:

Food and fuel supplies threatened to run out. Then the government realised that Cybersyn offered a way of outflanking the strikers. The telexes could be used to obtain intelligence about where scarcities were worst, and where people were still working who could alleviate them. The control rooms in Santiago were staffed day and night. People slept in them – even government ministers. “The rooms came alive in the most extraordinary way,” says Espejo. “We felt that we were in the centre of the universe.” The strike failed to bring down Allende.

Θυμίζει κάτι αυτό; Το Cybersyn και 200 φορτηγά πάντως τα έβγαλαν πέρα. Στη δικιά μας περίπτωση τα φορτηγά και τα βυτία επιτάσσονται, αλλά ακούω πως δεν είναι εύκολο να ελεγχθεί εάν θα τηρηθεί η επίταξη καθώς δεν υπάρχει ο κατάλληλος μηχανισμός (δεν ξέρω, μπορεί να είναι και λανθασμένη η εντύπωση αυτή).

Μέχρι να εκδοθεί από το MIT Press το “Cybernetic Socialism“, το οποίο και θα περιγράφει την ιστορία του Cybersyn, ίσως θα είναι χρήσιμο να μεταφραστούν από κάποιον τα τμήματα της τεκμηρίωσής του που είναι στα Ισπανικά.

Μπορεί να μάθουμε και κάτι.

Update: Ένα λεπτομερές blog post για το Cybersyn.

on team formation

In a meeting today a friend (quietly) observed that opening a process to a wider audience very fast may compromise the very process that interests the intended audience. I replied back with the thesis that:

“Whenever data increases, quality drops (for any quality metric)”

I first heard that thesis 15+ years back in a meeting about data warehouse quality. Usually when few people get together for a certain task it goes like this:

Small team, with people working towards similar goals

Increase the number of participants and you get something like this:

More people join the party, and things get interesting

Add a political twist and some power-play (personal or between organizations) and you get this:

Politics and power-plays set the project's final course (do nothing)

This is to be expected. David Alan Grier in “The Dictator and the Web Design” (IEEE/Computer, May 2009) notes:

“Traditional management theories identify such fights as the second part of a four-stage development process for small groups, the forming-storming-norming-performing steps that psychologist Bruce Tuckman identified in the 1960s. “Group members become hostile toward one another as a means of expressing their individuality and resisting the formation of group structure,” Tuckman claimed.

In Tuckman”s model, committee members must go through a period in which they express their objections to the collaboration in emotional terms (the storming stage) before they can learn to work together (norming) and actually accomplish their goals (performing).”

So there, as long as performance does not go “our way”, quality drops. By the way, this also explains why Panathinaikos B.C. prevails over Olympiacos B.C. in Greek A1. They both have excellent players, but Panathinaikos make sure that all are focused to the same direction. They are a team performing, while the others are still forming.

A simple pf.conf trick

Sometimes when installing new rules in pf.conf from a remote location your connection to the firewall gets dropped. To avoid this you can fire up screen (or tmux if you like) and then execute:

# sleep 5; pfctl -Fall -f /etc/pf.conf

That way you have enough time to detach from screen, logout and then login again to check the proper application of the new rules. Of course there is always the possibility of not being able to reconnect due to logic errors in pf.conf, but hey he who made no mistakes never built anything.

bnx2 and Debian

One way to install Debian on a machine that requires the bnx2 network driver, is to download the firmware, place it on a USB stick and continue as instructed by the Debian Installer. Another quick trick is to use a USB ethernet card and proceed with installing Debian. Then apt-get install firmware-bnx2 and reconfigure the network interfaces appropriately.

system error 1326

Note to self: When CreateProcessWithLogonW returns a 1326 system error (login error basicaly), you can always spawn Sysinternals’ PsExec.

While RunAs accepts the Administrator’s password from the terminal input by design, PsExec accepts it from a command line switch, which means that the source for the password can be the terminal, a file (encrypted symmetrically or not), a window application, etc. Way more flexible, but potentially insecure, especially if the administrator password is stored somewhere in the filesystem unencrypted.