Are all the servers running the latest version? Ansible to the rescue

After a certain size of servers, it is impossible to remember whether they are all current or not, or even check a documentation wiki page to find out about. So how can one use ansible to find out the answer? The setup module enters the room. Assuming an all Debian installation one could run:

ansible debian-machines -m setup --tree /tmp/invetory
cd /tmp/inventory
grep ansible_distribution_version * | grep -v 7\.2

This will list Debian machines not running 7.2 (Wheezy). You can build more complex versions of the above to match your infrastructure.

PS: Many thanks to @laserllama and @jpmens.

Run Lola, run

“Let me tell you the story of a company who’s on the verge of closing because of n-tier complexity, application server requirements and all that mambo jumbo” said a friend.

That company’s current client is a major public service institution. That institution has a set of complex policies designed, oh, by consultants whose employing firm of course was heavily paid to customize current “best practices” to secure the operating environment and making it use all the buzzwords that run around for it had to be modern. So when said company tried to deliver a software that it had a contract on, it was impossible to debug for they could not have any kind of access on the deployment systems. Which were run, not by the customer but, oh, by another consulting firm who was obliged to follow the rules set by the first one.

The governance of the above scheme looks good on paper, doesn’t it? At least I cannot deny it is a job creator for the consulting firms at the expense of those who want to do actual work.

Which brings me to the elitist question that I am going to fire up the next time I am lectured about Enterprise Architectures: “Have you personally implemented such a system? You, not someone you directed, you! Show me how, NOW!”. I’ve grown tired of people offering their paid opinion on IT systems that will improve anything when in fact the only system they’ve done is restoring their laptop’s Windows installation.

I’ve grown tired of people who prove the laws of Systemantics right with their ambitious, unworkable designs, namely:

A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system.

where in fact we know that since sometimes systems work, this is because:

A complex system that works is invariably found to have evolved from a simple system that works.

But I guess in IT we are big fans of Rube Goldberg machines.

“Hack like it’s the last day of your life”

“Hack like it’s the last day of your life, cuz one day it will be.”

I had a neighbor who studied Movie direction. He told me that their final project was a movie that they would have complete responsibility on it, from the script to the final cut. They were advised to work on it like it would be their last movie, even though it was their first. Their teacher wanted them to do so for two reasons: Because for most of them that would be their last movie and he wanted them to put their best. And also because if this was not their last movie, they would never know which one would be, but they still had to put their best on the current project.

My neighbor had a career in the Health sector but he was always proud of his movie, even though very few people actually saw it. He had done his best.

PS: Just like Sid (whose death prompted Wim Remes‘s tweet), my neighbor is also dead.

“How social networks work”

This popped up in my twitter stream:

How social networks work: Marketing bozos decide to game the system until it collapses. Then they look for a new place to piss. The End

Now what did that remind me of? A comment about marketing droids that I had left somewhere years ago. It took me a while to locate it, but here it is. The blog’s author in his conclusion was wondering:

It’s amazing to me that some people are so blind to that outcome. A savvy marketer ought to already know that it’s not all that smart to burn up the medium in a way that arrests your future ability to make money from it?

To which I responded:

They are not blind. They simply work within their time-frame of maintaining their job in email marketing. How much is this going to be? Three, Five years? Then they will switch subject and will not care for the ruins left behind. People in marketing and management are always that “blind” because they care more about their bonuses than the lifetime of the company they work for. As for the demise of their previous company, it is never their fault, right?

* Sometimes you have to copy comments that you leave elsewhere back somewhere where you exercise more control.

OpenVPN: “The data area passed to a system call is too small”

On a Windows 7 client I was getting the following error message after the VPN connection was completed and everything looked like normal. While the connection was completed (and the “lights” were green) the user could not do anything on the net. The log message on the client was repeating:

The data area passed to a system call is too small

In such cases it may help to check whether both sides have (or have not) comp-lzo set and set at the same value. In my case this was solved by setting:

comp-lzo no

at both the server and the client(s).

100ms

Let’s take the following diagram for granted:

NSA infiltrates links to Yahoo, Google data centers worldwide, Snowden documents say
NSA infiltrates links to Yahoo, Google data centers worldwide, Snowden documents say

Now why would that happen? Why would they choose to have unencrypted traffic through their network? And then something I read struck me:

“For an organization with a large-scale website doing business via advertising or e-commerce, or whose primary source of revenue is via interaction with a website, one of the most notable concerns is web page load times. A well-known metric that supports this comes from Greg Linden of Amazon.com, who posited that an increase in 100 ms of load time caused a 1 percent drop in sales on Amazon.com1.”

At such scales all milliseconds translate to money I guess.

[1] – “Make Data Useful” [PPT slides]

Thank you Packt Publishing

I was going through my copy of “Nginx HTTP Server, 2nd edition” when I located a typographical error regarding WordPress installations with nginx. I used the errata submission form provided by the publisher to inform them. A few days later, they replied confirming the error and they offered me a complimentary copy of any of their titles. And so I got “Haskell Financial Data Modeling and Predictive Analytics” :)

Thank you Packt Publishing!

Ansible, git and Rails

I think I wrote the following playbook a few months ago, when I was halfway through watching the 2 hour introduction (now replaced with this):

---
- name: automatic deploy new version
  hosts: ruby-production
  user: root
  sudo: yes
  tasks:

  - name: stop monit
    command: service monit stop

  - name: checkout correct version
    sudo_user: projectuser
    command: chdir=/workspace/project git checkout release-1.0

  - name: grab latest sources for that release
    sudo_user: projectuser
    command: chdir=/workspace/project git pull origin release-1.0

  - name: run db:migrate
    sudo_user: projectuser
    sudo: no
    script: migrate.sh

  - name: restart apache
    command: service httpd restart

  - name: start monit
    command: service monit start

What the above playbook does is simple:

– Stop monit to avoid any automatic restarts when you’re half way through updating stuff that it monitors.

– It makes sure that you grab the latest updates from the correct branch

– It runs a script on all servers that takes care of any bundle install and db:migrate stuff.

– It restarts apache. Yes, a touch tmp/restart.txt should do the trick but sometimes it does not.

– It restarts monit.

There is plenty of room for improvement here, for example using the git module instead of running an explicit command and even making use of roles as the project expands and becomes more demanding and of course get rid of the script in favor of a more ansible specific play.

So why post it now? Basically at the request of @nanobeep and as a means of self-pressure to improve the playbook. Maybe I should promise this to someone?

So there, nothing complex or elaborate. BTW, here is a similar way to do this with git and Capistrano that I bumped into.