Users consider their needs top priority. Not only that, but when they pick up the phone or press the send button of their email client, they demand immediate service. System Administrators on the other hand are trained (over time) to objectively distinguish between real emergencies (threats to the organization’s business operation if not dealt with) and the rest.
So whenever an urgent situation arises, step back and ask yourself:
– Urgent for whom?
– Why is this urgent?
– Is there a process missing here??
These are important questions, especially if there exists no process covering the situation. Organizations have written workflows that define processes, but operate on the evolution of those rules which are mostly undocumented. If you identify a missing process, your reaction to the matter will create a process, no matter what. Solve the problem as a fireman and you have just created a process with your name hardcoded in it. Not your team, your name. People will look for you.
Identify that it is about a missing process problem that needs to be fixed and everybody will promise to you that it will be dealt with. Only it will not. The next time it arises, they will come back to you, because you did it the first time and now you are (informally) in charge of “those things”.
This is neither good for you nor for your employer. So the need to say “No, I will not fix that this way. Create a process and I will” arises. It is to the benefit of your employer to do so. It makes certain that for this particular situation they are not depended from a single person (you). It also protects your time, weekends and vacation. Explain this to upper management. Learn to say “No” in a productive way. Make sure this is not misunderstood as BOFHiness from your part. Put a price tag on what it means not doing it the formal way.
As a System Administrator you do not only manage the computers in your organization. You manage the people using them too. You manage human-computer systems. And whenever there is a void in the workflow, you need to do your best to create a process. Otherwise it will be created without your intervention. You do not want that. You are the System manager.
Eliminating unnecessary processes is part of the job too, but this is maybe for another blog post.
Managing organized complexity 
Well put and well said, but a System Admin is able to create processes or contribute to process creation when the rest of the IT Management or upper management environment is mature enough.
Having the luck to visit all kinds of IT/MIS departments, I regret to say that noone is used to processes and incident management (not talking about security incident here, its another bleeding story) and firefighting is a commonality.
I am all for the knowledge you are transfering to the John Does in the IT who are right now striding floors up and down to click an “OK” button or to do a reboot on an old Windows 2000 workstation, but they also need to understand that this change takes bold actions and it needs support from the management.
Just my 2 eurocents (the last I have till my next paycheck :) ) and thanks for posting this great article…
There is also the problem of office politics, so maybe the boss does see your point but doesn’t like to lose the ability to call you on holidays and in the middle of the night, since it’s convenient for him or his superiors. A process will force people to change habits and they don’t like that.
For example, try the Greek public sector and see where *demanding* a process gets you. Some people take it as blackmail or a power grab on your part and you can end up in a VERY bad place.
In my experience there are 2 baseline processes we all should have and push for above all else:
1. A single input for tech requests: a ticketing system attached to an email works best, in my experience. A policy of “Only phone me if your mail isn’t working, otherwise mail support@…” can be enforced by you personally under the boss’ radar in most cases.
2. No user should have any kind of administrative access to their workstation, otherwise refuse support completely for that workstation. If you lose this one, you’ve lost it all.
In my opinion, if you are unable to push for and implement these two, there is no point discussing processes in your organization at all.
Agreed. There is an aspect though, that requires emphasizing: System Administrators do not “serve” systems or work-flows. They serve people, their users. When an “emergency” comes up, you can be firefigher, BOFH, apathetic (this is how “build me a work-flow, then I’ll do it” sounds to me) or facilitate any action required to have progress towards resolving the issue at hand. This might not be in your job description, maybe it’s actually your manager’s job. But as soon as you embrace the mentality that you serve people, the rest becomes irrelevant.
Interesting (and long) comments. Thank you!
For most organizations there are two classes of users: Internal and external. Serving internal users is IT support (and a whole can of worms). These users, the hardware and the software comprise the system that we operate. So in our case at least, my loyalty is at the 110+K members that we serve, and not at the ~300 people of various working relationships that work for the system too. We serve them and we owe it to them to have the correct processes in place. Sometimes this may mean that a fire that can be put off in a day may take a month to be dealt with, but this happens only once and the cause never emerges again. In the long run the members win. Otherwise they may have to wait for certain employees to return from vacation for example. Scale changes both forward planning and the perception on things which may seem easy for a small input.
Corfiot, like me, works in the Greek Public Sector. This environment is notorious for having no processes for anything and a process for everything at the same time. It is also common expectation that you are not supposed to follow the rules “because it is urgent” only to find out how this conveniently backfires on you (and you only) when this bypass does not bring any result. As a system administrator I have enough responsibility without authority in my hands. More I do not need.
Conveniently though, the Greek Public Sector has the best trouble ticketing system in the country: Protocol. It is slow, but everything submitted to it must be dealt with. If it is not in the Protocol it does not exist (no we do not work this way internally in our department, but certain stuff has to go through it).
I’ve been called a blackmailer once in 2007 by upper management. They failed to realize that I was protecting their ass at the moment. Interestingly they also failed to get re-elected. Do I want to be placed in the same position again? No. Have I learned something by this incident? Yes: Do whatever it takes to fill in the missing processes. Everything else will backfire on you harder sooner or later.
Some background on this post: Colleagues from another department had to deal with a certain problem for more than a year. People had heard of this problem but no action was taken. They came to us for advice and help. I told them to ask for help from us using the Protocol mechanism. This way upper management would be informed. If we acted without their knowledge and something went wrong, it would backfire on us (not even them). They indeed filed such a request in April, requesting help from upper management (who if actually cared to solve the problem would delegate the issue to us).
Out of the goodness of his heart, and despite my pointing out the law of unintended consequences, a colleague decided to help them (we argued heavily on the matter but he decided to go through). The very first time the problem re-emerged, they looked for him. As luck would have it he was on leave. Like I wrote in the post, if you are not careful enough the process will have a name hard-coded. This is not a solution; it is two problems.
The problem manifested itself more heavily a second time in July. Upper management (verbally, not written) decided that employees from both departments were to blame. Only then written proof existed that upper management had not acted on the problem since April. Funnily, after document display, instead of trying to delegate fault, management asked the magic question: “What can be done?” The need for a process creation was justified. Yes the new process may not be available until the end of September, but I heavily doubt that there would be any movement to this direction if there was no written documentation of the need and we had adopted the fireman approach that out of the goodness of his heart our colleague had tried to implement (which was not thought out well enough since it lacked scaling, was undocumented and was not automated).
Yes it is hard to demand process creation in such environments. But as Fowler and Christakis say “Be the change you want in your network”.