Fighting the automation paradox || Deployment θα κάνουμε φωνάζοντας "αέρα"
Αυτό είναι το τρίτο Παγκόσμιο Κύπελλο που βλέπει αυτό το blog (μια και ξεκίνησε κάπου στο 2006). Το 2006 είχα γράψει κάτι λίγα (βοήθησε και ο Ζιντάν). Το 2010 δεν θυμάμαι (και βαριέμαι να ψάξω) εάν έγραψα κάτι (κάτι έγραψα). Τώρα πάλι δεν ξέρω.
Θα ήθελα να δω τελικό Βραζιλία – Ουρουγουάη, αλλά νομίζω δεν βγαίνουν οι διασταυρώσεις για κάτι τέτοιο.
We’ve come a long way dear blog and we’ve changed.
“When you come to a fork in the road, take it.” —Yogi Berra
Once during a cataclysmic flood there was this priest that the rescue teams tried to save. “Children he said, go on and save other people in need. I am at the hands of God. God will save me.”. This happened between the priest and the rescue teams a number of times, until eventually the priest got drowned. When before the Maker he asked “How is it possible that you left me to drown, me who I lead such a devoted life?” and the Maker replied “And who do you think sent all those rescue teams towards you?”.
One of the hardest lessons to learn in your career is to let it go. But the hardest, especially if you think high of organisational loyalty is to let yourself go.
So I am taking my sense of duty and taking the fork.
– Are you not afraid? Of course I am.
Good friend Martijn Grooten (twitter, blog) and chief editor at the Virus Bulletin was at Athens for a day or so yesterday and was kind enough to make a presentation about Dual EC DRBG and the possible involvement of the NSA on it. The event was hosted at the Hackerspace. It was a scientifically balanced presentation with a cool explanation of how elliptic curves are used in such cases and what kind of risks this entails.
Pizzas after the presentation were curtesy of the Greek OWASP Chapter.
– Έχουμε σκεφτεί και νέους τρόπους ελέγχου, είπε ο Η. Νικολακόπουλος πριν λίγο.
Άκου να δεις όμως τι συμβαίνει μάστορα: Όταν η πλειονότητα όσων ρωτάς δεν πιστεύει στην αξία των exit polls και αντίθετα θέλει πάρα πολύ να αποτύχουν, όσους ελέγχους και να βάλεις, έξω θα πέσεις. Κοντά θα πέσεις από σύμπτωση.
Και ξέρεις γιατί συμβαίνει αυτό μάστορα; Γιατί ποτέ δεν εξήγησες σε αυτόν που ρωτάς ποια αξία έχει για αυτόν το να σου απαντήσει ειλικρινά. Την αξία που έχει για εσένα (“Εδώ το μάθατε πρώτοι”) την ξέρουμε όλοι. Αλλά επειδή το exit poll δεν γίνεται για την επιστήμη αλλά για να μπορέσεις να οργανώσεις τις επόμενες τηλεοπτικές σου ώρες, θα έχεις garbage in, garbage out. Γιατί έτσι βγαίνει και ένα τηλεοπτικό προϊόν με ανατροπές στους πανηγυρισμούς στο οποίο αρέσκεται το φιλοθεάμον κοινό.
Ansible offers the convenience of running scripts on remote servers. But as the documentation notes:
It is usually preferable to write Ansible modules than pushing scripts. Convert your script to an Ansible module for bonus points!
There is a reason for this. Usually you have ansible run a script on your behalf when what you want to do is not achievable via a module or some combination of modules in a playbook. In extreme circumstances you will need to run a script via ansible when the receiving computer has no Python installed.
But there is a problem with running scripts this way: They are opaque.
A playbook that is applied to your machines is actually a model of that part of the machines that you want to manage. And ansible is your sensor that deals with the situation when things go sour.
It is very easy to write a script that does one thing well to one machine and does not check for failure. Now apply this to 100 or 500 machines that are similar, yet have some subtle differences between them. Can you imagine what a rewrite your script needs in order to account for all corner cases? And if you make it bullet-proof, congratulations! You’re half-way through to making your own incompatible version of ansible.
Having said that, I am guilty of running scripts instead of describing work to be done in a playbook. This mostly involves stuff that needs to be executed from a login shell (hello rvm!) which means the script begins with #!/bin/bash. However, in order to exercise better control in such situations I am not running more than one command plus checks for return codes in every script. This breaks the script down in many smaller ones, but allows me a better view when something goes wrong. Because my playbooks instead of having one script directive, they have 5 or six in a row.
You may have not described an accurate model of what you want to do using a playbook’s markup, but at least the name: directive for every single task is accurate enough to let you know what is executing, rather than having it issue a larger script where you wait whether it succeeded or not, and if not try to find out from which point exactly to roll back (if rolling back is possible).
So the new rule is:
When pushing a script through ansible, it should execute one command only plus any checks needed for return status.
Nothing coherent today, just three excerpts from an article, an interview and a book that are separated years apart, yet I find them somehow connected in my mind. From “Electrical Engineering — A Diminishing Role?“:
“Projecting the current trends, future computers will consist of a single chip. No one will have the foggiest idea what is on that chip. Somewhere in the basement of Intel or its successor will be a huge computer file with the listing of that chip. The last electrical engineer will sit beside the file, handcuffed to the disk drive like a scene out of “Ben Hur.” That engineer will be extremely well paid, and his or her every demand will be immediately satisfied. That engineer will be the last keeper of the secret of the universe: E = IR.”
Ever since I first read it, I always thought it was talking about The Last System Administrator.
The next piece comes from an interview that Raspberry Pi creator, Eben Upton gave to the IEEE Techwise Conversations podcast:
“I think we’ve had a reduction from, say, if you think about 1995, which was when I went to college, you could typically rely on an undergraduate having done a substantial amount of real programming, often quite a deep level of technical work on one or more platforms. Many of us could program in one or more assembly languages. And yeah, within 10 years of that point, we were getting to a point where your average applicant was maybe somebody who’d done, as you say, a little bit of Web design, maybe a little bit of Web programming—you know, we saw quite a bit of people who‘d maybe done some PHP but not that kind of deep technical understanding of how machines work.”
And the last piece comes from the “Flash Boys“:
“Russians had a reputation for being the best programmers on Wall Street [… because in Russia … ] they had been forced to learn to program computers without the luxury of endless computer time.”
Stuff to think about now that your data center has been reduced to a tab in your browser.
Every now and then I like finding links between Cybernetics (or Systems Dynamics, or Systems Thinking, pick your favorite variation) and System Administration. I am not the only one in this. For example Matt Simmons has written about how System Administrators act as homeostasis mechanisms for the systems they manage. And minutes before this slide came up my way:
Rule #4: Monitoring systems need to be more available and scalable than the systems being monitored
which reminded me of a theorem and a law as applied in the monitoring systems domain. The Good Regulator Theorem states that every good regulator of a system must be a model of that system. You provide the monitoring system with a model of what you need to monitor in its appropriate DSL or clickware. The more precise the model, the better the monitor.
The rule in the slide ties closely with the law of requisite variety where the variety in the control system must be equal to or larger than the variety of the perturbations in order to achieve control. Think about it: At least the downtime of your monitoring system needs to be significantly less than that of the system monitored. Otherwise what exactly are you seeing? Think of Nyqvist-Shannon sampling here. Or as John Gall has put it in The Systems Bible, a system is no better than its sensory organs.
Is it practical to make these observations? For everyday job not really, but when I find such connections between “obscure” theory (obscure for admins) and system administration, I always smile :)
PS: @adrianco during the discussion left another piece of advice:
on a later slide I said best to use two independent monitoring systems then they can watch each other.
“All of a sudden the market is all about algos and routers. It’s hard to figure this stuff out. There’s no book you can read.”
I first heard about high frequency trading more than 10 years ago from a bright person who occupied the desk next to mine at the lab. I thought it was interesting but I could not foresee what it would come to mean today.
Flash Boys is a book by Michael Lewis that deals with the implications of HFT and dark pools to the stock market. It chronicles the journey of Brad Katsuyama from when he first notices that the market view on his computer screen is an illusion, his effort to understand the mechanisms that rule the market, building a team for the task and ultimately quitting his well paid job in order to create a fair marketplace for the investors. That marketplace is the IEX. I would call the IEX a white pool though.
I like the book for a number reasons: First it is about a good regulator (the cybernetic term). Katsuyama wants to understand something so complicated and complex that no one else really understands. He needs to create a market view that is explainable to investors to the microsecond and according to the book this is done.
The story behind team formation is interesting too. Every member in the team has something in their life worth reading about and they offer value to the team formed. I found lessons in team formation there.
There exists this interesting term called the Regulatorium: a complex system of rules and requirements where each rule is necessary because of another, related rule. This leads to collusion between regulators and those regulated preventing change. And the Flash Boys wanted to change the system.
The moral values of Katsuyama appeal to me. A potential investor asked: “Why does a person take the harder path? It’s a different situation from what you typically see. If it works, he will make money. But he’ll make less” than if he had stayed at RBC.
The technical problems they faced I understood and I loved some of the solutions (“coil the fiber”). I could actually follow this fast paced, well-written chronicle with just a bit of googling for a few financial terms (and I have never dealt with a stock market).
I finished the book with a “damn, I want to work with these guys!”. So Brad, if you ever read this and in need of a remote worker in a different timezone, ping me :)
I bought myself a present yesterday: A Chromecast. Why you may ask? Mostly because I wanted something to send Youtube videos to the TV for the kids. I was doing it with OpenELEC on the Raspberry Pi, but it was not the most straight forward thing for the children.
Setting up the device was not straight forward, but this was not Chromecast’s fault. I had to fiddle with the WiFi channel of my home router (I switched from auto selection to some specific channel) in order to make it see the home network. After that, the device connected to the Net, upgraded itself and worked like a charm.
Things I tested:
Tab and full screen casting has about a 1.5 second delay between the laptop and the TV screen, but once you get used to it, it does not really bother you.
After one day of using it I believe that Chromecast is great both the idea and its execution. But I find it overkill for Greece where you’d mostly stream Youtube and use no other applications. Granted it is cheaper than something like Airtame, but there may be cheaper options. What sold me, was its ability to work with Chrome in order to share your desktop to the screen. Sort of a Hangout screen sharing with your TV.
Now let’s see whether edX streams through Chromecast just like Coursera does.
As I was watching people passing to one another the Easter flame, I could not help but wonder:
We know that the wave’s speed is about 12m/s. So what is the speed of passing the flame?
Network scientists would also be interested in other stuff, like who passes the flame to who, or to how many and the like.
Happy Easter everyone!
Managing organized complexity 