Ah the feedback loop …

And modelers rarely if ever consider the feedback loop and the ramifications of their predatory models on our culture.

This does not happen with modelers only. It happens with everyone that has a “good idea” and rushes it forward without really thinking whether it will work or not, and how long afterwards we are going to see how what was introduced interferes with what was already there, fine tuned and working. And that is why I read about cybernetics; not because it is less obscure, or more useful than people think, but because it always reminds me of the feedback loop.

investigators are part of the system
investigators are part of the system

Homeostasis from a sysadmin perspective.

When times are tough …

“When times are tough IT gets beaten hard” –Rolf von Roessing

When times are hard IT is the first to take the heat. Budget cuts, “rightsizing” (which always equals downsizing) and a mandate to do more with less is in order. And let us not forget outsourcing which of course costs less, provided there is still someone there, the key person, to inform the outsourcer and interface with the rest of the organization. Most likely that person jumped away upon seeing the iceberg and not after the hit.

So what do we get after this? We get a lower budget for the next year (hence a success!), poorly documented systems for which there is no one around to ask details about (and guess when will you need this tiny bit of missing detail) and systems that must continue to operate with large portions of them unpatched, unmaintained and halted not following the normal upgrade path of their cycle. This operations nightmare can easily become a security one too.

And what about training your staff? Training is spending and thus cut. After all, staff is supposed to freely dig the Net and find out appropriate answers. For that is the best it will get, answers, not training. No one will stop staff being trained on their personal time and budget, but not on company time. “What if we train them and they leave?”. Is that sensible? How about not training them and see how far they can go.

Management avoids errors of commission by making errors of omission that have hidden costs which appear further in time, in most unexpected circumstances and of course at a time when the responsible one for the chain of events has left the helm. But then again management stays less time in office than those who foresee such errors and easily silences them in an uneven power game.

All that is left are the system administrators, developers and security guys still on board trying to clean up the mess. Sometimes you have to spend your way up, but almost always management interprets this like a supernova: They keep absorbing amounts of energy (budget) trying to keep doing their thing avoiding evolution (organized abandonment) like dinosaurs. I used to dislike Rand’s advice about when to jump ship, but he is certainly correct.

Pro Website Development and Operations

I read about “Pro Website Development and Operations” at Tom Limoncelli’s site, so I immediately marked it on my list of books to buy. A few days later Apress held a $10 ebook sale on every title they had, so I bought it. The good news first: The epub version of the book renders nicely on my BeBook Mini.

I have not written a book yet, but I know one when I read one. And this is not a book. It is a series of good long blog posts expanded to fill the size of a book (124 pages). It was not very well proof read so that it has many grammatical and syntactic errors and some others like “you might have several hundred servers in the subnet 10.10.20.0/24,”. You can have a couple hundred servers on a /24 but not three hundred so it is not some, sorry. A trivial mistake, but indicating of things that can annoy you in the book.

The word engineering and its derivatives is overused. There’s a reason that the intended audience of the book are DevOps and not engineers, or software engineers (even though there exist people who can carry all three hats). And that is that they are not. There is a great difference between engineering a solution and calling everyone on board an engineer. That is unless what you build has a direct impact on human lives (or loss of them) or is something that when failing can cause a national economy to go under or a disaster of a similar magnitude.

An interesting thing about the book is that it talks a lot about the significance of measurements in order to understand a site’s usage patterns. However there is not a single formula or methodology mentioned which the read can use in order to measure things! It is more along the lines of “you need to measure stuff because it is important” but nothing about how to measure or how to lay out a plan for a measurement infrastructure. Because forecasting performance is a must in website development and operations, I was expecting something like “Forecasting Oracle Performance“. I was also expecting hints on how to size a new server. Of course I will size a new server carefully, but I bought your book not to read generalizations, but how you actually do it. Again no formula (if you want to see some interesting mathematics on the subject, see “Mathematical Server Sizing“). We need to model stuff, so where is how I build and test a model?

Another thing I take issue with is the special projects team that the author advocates. The author is right in advising rotating roles between members of the special projects team in order to diffuse knowledge among them, but I believe that he has managed to be a member of special project teams only. Otherwise he would have described the impact on the morale of the operators who are not members of the special team that builds exciting new projects. Projects that it that get the budget, get to use new technologies and hardware to experiment on, while the rest must work on (a restricted) budget to maintain a (legacy) system that already brings money on the table. So in fact you not only have to rotate people among roles in the special projects team, you have to rotate them in and out of the team too. This also brings the advantage of avoiding the build up of IT silos or other small dominions with a single point (the operator) of failure.

Is there anything good in this book? Taking good care of your people and their health is one. Making it sure that they get proper sleep, even before launching is important. Not only for the health of the workers, but for the health of the company (and its culture) too.

The other thing I really enjoyed in the book, was the interviews the author did with Tom Limoncelli and with Santiago Suarez Ordoñez of Selenium fame.

In its effort to be technology agnostic so as to stand the test of time, the book suffers from generalizations and is disconnected from practice. Wait for the second edition.

The sysadmin oxymoron

/* This has been gathering dust for quite some time now */

The cost of communication waste” made me think of the standard oxymoron in our line of business:

“We are forced to kill our children to make our point” says a fellow sysadmin. This is the case where, although the system administrator has recommended an update, upgrade, improvement etc. this is denied thus forcing the system to degrade slowly (or fast sometimes) no matter the effort put against decay by the administrator. Who will finally be forced to let the system die proving the point along the way. But this is no victory. System Adminitrators derive no joy by being right when disaster strikes; they want to have resources for their systems not to fail or at least abandon them properly.

As system administrators we build systems that are supposed to support certain operations for our employer. We build stuff, automate processes and generally walk a path, that the untrained eye thinks it leads to eliminating the necessity of our services. In essence we build our own obsolescence. When things do not work, users (sometimes rightly) believe that we do a lousy work. On the other hand when we sit idle in our offices, we are not needed and may sometimes even be considered a burden, a cost center. Why would one pay a system administrator, if he seems idle all the time? The contradiction is clear: When the systems do not work, it is because we do a lousy job. When the systems work, our services are supposedly not needed.

Fat behaviors rule the IT workplace (IT Gestapo, toxic meetings, pointless report generation) and BOFHs are equally guilty of this. This is the balance we have to keep up with daily, but when running systems with a lot of responsibility but without authority, we have to find ways to make management not only listen, but also act. Because management takes a bet: Things won’t break while they are in office. Since we have to either leave or live with that, we either take the same bet or we make damn sure that our technical points are permanently recorded.

Lean vs Fat behaviours and the Public Sector

Lean behaviors” is a fantastic paper written by Bob Emiliani. If I was to use a highlight marker, I could easily paint the whole paper.

“[an organization] must possess an ability to change how it thinks, which requires a culture characterized by trust, shared responsibility and openness to experimentation without fear of failure“.

“Lean behaviors are defined simply as behaviors that add or create value. It is the minimization of waste associated with arbitrary or contradictory thoughts that leads to defensive behavior; ineffective relationships, poor co-operation, and negative attitudes. […] [“fat” behaviors] include the display of irrational and confusing information that results in delays or work stoppages, or the articulation of unsubstantiable subjective thoughts and opinions. Fat behaviors are recognizable as lots of talk where nothing has actually been said, or indirect words whose meanings are subject to variable interpretations”.

What an accurate description of the Public Sector!

[The cost of communication waste]

Happy Sysadmin Day today!

Yes! It is this time of the year again. Sysadmin Day is today. So off your desk and walk towards that guy who is taking care of your network and machines and is required to perform magic when your PC is damaged, the network is not responding and your work is lost. Give him a hug, say “Thank you“, buy him coffee, a book, do a small gesture of appreciation, for if you reflect back in the past 364 days, you surely had a chance to thank them and you did not.

When treated properly at least once a year, sysadmins do not bite!

Definition of A System

“A system is a construct or collection of different elements that together produce results not obtainable by the elements alone. The elements, or parts, can include people, hardware, software, facilities, policies, and documents; that is, all things required to produce systems-level results. The results include system level qualities, properties, characteristics, functions, behavior and performance. The value added by the system as a whole, beyond that contributed independently by the parts, is primarily created by the relationship among the parts; that is, how they are interconnected.”

[source]

Two types of leadership

I was trying to find a particular quote on “responsibility without authority” (and still am) and thanks to Google Books I bumped into this:

Though he would probably have preferred another word to describe it, “purposing” was something like what James M. Burns had had in mind a decade earlier. Leadership is nothing, he wrote in his insightful and influential 1978 study of the subject, “if [it is] not linked to collective purpose.” Burns agreed that a crisis had arrived. People in power were too often mediocre or irresponsible. But he thought this crisis was fundamentally an intellectual one: We simply do not know enough (indeed, we know very little) about leadership. He argued that leadership comes basically in two forms: transactional and transforming. Transactional leadership, in which the relationship between leaders and followers is based on an exchange of favors for support, had become the dominant mode. The transforming variety is “a relationship of mutual stimulation and elevation that converts followers into leaders and may convert leaders into moral agents”. To be a transforming leader is to recognize the relationship with followers and their needs and goals. That relationship and the character of the leader were prime ingredients of Burns’s theory of effective leadership.

Emphasis and links added by me. Good stuff to contemplate on until the May 6 elections…

Update: I use to say that one of the greatest faults in the Greek Public Sector is the existence of parallel hierarchies. Which means that there exists the documented organizational hierarchy and there exist also conditions where a subordinate, because of being close to people “with power” (meaning a political ally, a party member or even an elected member of the Parliament), is ranked higher than his direct supervisor. How is one supposed to lead someone who he cannot order to perform any function at all? Transactional leadership at its best.

The Cybernetic State

“The Cybernetic State” is a book written by Javier Livas and is available as PDF on request from the author. From the preface:

The emergence of a cybernetic State is now a real possibility, and most likely inevitable in the near future. This book sketches this information age organization and the cybernetic management principles on which it is based. As we shall see, many of its features are already present in embrionary form in the modern democratic State.

The description of the cybernetic State relies on the Viable System Model (VSM) developed by professor Stafford Beer and explained in several of his books. This model originates from control theory and the cybernetics of the human nervous system, and has been adopted and validated by management science. In this book the VSM is used to show the nature of the State.

The enormous explanatory power of this cybernetic map will show that Economics, Law, and Political Science, which have mostly been studied separately, actually refer to three different aspects of the same phenomena, namely the State. In this sense, the book attempts a synthesis of ideas that were born disconnected and remained so for a long time. Helpful insights about the evolution of economic, legal and political theory are a byproduct.

[via CYBCOM]