I think I’ve read about this on sage-members (it is also quite possible that I’ve blogged about it, but a quick search did not reveal anything):

You’ve got a major upgrade ahead of you, one that might take too long to complete and on top of that, the company (your employer) cannot halt while you are at the task. So do you schedule to start the upgrade on Friday evening, or on Sunday morning?

For years I used to opt for Friday evening. But it seems that I was lucky. For as I read in sage-members, what if the upgrade does not complete and you need support? Do you have (verified) support 24×7 for everything involved in the process, including hardware, software, personnel (if shifts are needed)? Even if you do, have you ever tested them? Are the support people you contact on weekends of the quality you expect or simply note takers so that you get an open ticket and a checklist while an actual solution may arrive on Monday evening? What if a simple fan fails and you need to replace it?

Start on Sunday mornings. As a bonus you get a full day for rest and mental preparation.

The recent sport related (but unsporting) events bring to mind the point I was trying to make in my previous post: That an organization must rely on its people following rules and processes and not on their display of filotimo (which must be saved for extreme circumstances only).

A decision was made to have the grass surface of the field in the Olympic Stadium of Athens replaced. The works begun and reports in the press showed progress. However, a pump was broken while the person responsible for it was on leave. This event went unnoticed until a friendly match was given between Panathinaikos and Genoa C.F.C. In this match, Djibril Cisse, Panathinaikos’ main fire power was injured. So were two players of Genoa. The pitch was declared unusable and will be replaced after the U2 concert in the Stadium (September 3).

Tomorrow AEK, who also use the Stadium as home, is supposed to play against Dundee United for the Europa League competition. Only now they face the problem of having to find a home stadium for the match to be played. In a controversial for some fans agreement, they decided to use the Nea Smyrni Stadium, home of Panionios FC. Angered by the agreement, Panionios’ fans entered the pitch and made it virtually unusable, not only for AEK, but for Panionios’ home game in Saturday too! Now AEK is supposed to defend its win, using Karaiskakis Stadium, Olympiacos‘ home, with no fans on their side- only fans of Dundee United who traveled from Scotland will watch the game.

A broken pump while a single person was on leave has lead to two damaged soccer fields and a team not having the support of their fans while giving an international competition game at home. This displays the complexity and inter-connectivity between systems in this world in weird and unforeseen ways, where the law of unintended consequence strikes, with a seemingly low priority glitch creating so much havoc because “the system” could not deal with (or even detect) it.

Users consider their needs top priority. Not only that, but when they pick up the phone or press the send button of their email client, they demand immediate service. System Administrators on the other hand are trained (over time) to objectively distinguish between real emergencies (threats to the organization’s business operation if not dealt with) and the rest.

So whenever an urgent situation arises, step back and ask yourself:

– Urgent for whom?
– Why is this urgent?
– Is there a process missing here??

These are important questions, especially if there exists no process covering the situation. Organizations have written workflows that define processes, but operate on the evolution of those rules which are mostly undocumented. If you identify a missing process, your reaction to the matter will create a process, no matter what. Solve the problem as a fireman and you have just created a process with your name hardcoded in it. Not your team, your name. People will look for you.

Identify that it is about a missing process problem that needs to be fixed and everybody will promise to you that it will be dealt with. Only it will not. The next time it arises, they will come back to you, because you did it the first time and now you are (informally) in charge of “those things”.

This is neither good for you nor for your employer. So the need to say “No, I will not fix that this way. Create a process and I will” arises. It is to the benefit of your employer to do so. It makes certain that for this particular situation they are not depended from a single person (you). It also protects your time, weekends and vacation. Explain this to upper management. Learn to say “No” in a productive way. Make sure this is not misunderstood as BOFHiness from your part. Put a price tag on what it means not doing it the formal way.

As a System Administrator you do not only manage the computers in your organization. You manage the people using them too. You manage human-computer systems. And whenever there is a void in the workflow, you need to do your best to create a process. Otherwise it will be created without your intervention. You do not want that. You are the System manager.

Eliminating unnecessary processes is part of the job too, but this is maybe for another blog post.