Your first steps installing Graylog

A new colleague needed some help to setup a Graylog installation. He had never done this before, so he asked for assistance. What follows is a rehash of an email I sent him on how to proceed and build knowledge on the subject:

So initially I had zero knowledge of Graylog. What I did to accustom myself with it was to download an OVA file with a prepared virtual machine and run it via VMware Fusion. The same VM can also be imported to VirtualBox and even to AWS, although they also provide ready AMIs for deployment in AWS. Links:

– OVA image

– AWS AMIs

Keep in mind that this is a full installation of what Graylog needs to work with and it also comes with a handly little script named “graylog-ctl” that manipulates a lot of configuration. The big catch is that graylog-ctl is not part of any standard Graylog deployment. It only comes with the OVA and the AMI images.

So after I had some fun with it on a VM on my workstation, reading the documentation and testing stuff, I had an initial deployment of the AMI image in AWS. But this is not an installation that can scale. Which brings us to the next steps:

For Graylog to work you need to provide it with a MongoDB and an ElasticSearch database. It is your choice whether these will be clustered for high availability or not, whether they will run in the same machine or not. You control the complete architecture. So in my case I made the following decisions:
I am running a MongoDB replica set using three VMs. This is a standard setup as it is described in the MongoDB online documentation. Since it is not password protected, it only accepts connections from the Graylog instance. I used AWS security groups for that.
I am using an ElasticSearch cluster with three VMs where the nodes are both data and masters. If you can, use 7 nodes, three masters (lower machines since they do not run queries and do not index any data) and four data nodes (higher end machines). Again, since this is not password protected, I used AWS security groups to allow access only from the Graylog instance.
I am running a single Graylog instance on a separate VM. Currently it only listens for syslog stuff. When the need arises, I will add a two more nodes to increase the availability. I think I changed as many as four or five lines in the main configuration file. Graylog uses MongoDB to store its configuration, which includes anything you configure via the web interface.
Pay extra attention to the versions of ElasticSearch and MongoDB that your Graylog version requires. Use exactly what is mentioned in the documentation. For example in my case I am not running ES 6.x but the latest 5.x.

Now it is time to up your game. Once you see that your installation is working you have to decide whether to password protect access to MongoDB and ElasticSearch and whether to encrypt traffic between all those instances or not. I say give it a go.

I’ve not even touched issues like database management for Mongo and Elastic, backing them up, restoring, deleting indices, etc because this is post from zero to your first week testing Graylog. There is plenty of stuff out there to take you to the next level, once you get used to the complexity of the software involved.

Should you need any more help, ping me anytime.

terraform, route53 and lots of records

At work we try to manage as much as we can with terraform. This also includes Route53 for zones and records. In a certain situation we had about 14 zones and 1476 records managed in a single state file.

As it happened I needed a zone recreated (but not erased) and this affected about 409 records. Well deleting them with terraform apply took ages. To the point that the temporary STS token expired and botched the process. So after a little facepalming, I decided to cleanup the zone from the AWS console and then issue a batch of terraform state rm to reconcile the state. Happily, after that, apply took its time (but reasonably) and all was well.

I am thinking that next time I am faced with such a situation, to lock the state file in Dynamo, copy it over from S3, manipulate it locally, unlock and run a plan to see how it all plays out. Or, wherever I can, use a state per zone instead of a state file encompassing a set of zones.

Rule 110 in Haskell

Because WordPress.com does not always render MarkDown properly, you may need to read a copy of this post here.

One of the things I learned by reading AIM 239 is the Game of Life and Cellular Automata. One particular kind of one dimensional cellular automata, Rule 110 popped by my twitter stream the other day, so I thought I could try and code it with the minimal Haskell subset that I can handle.

Rule 110 is special because it is proven to be able to simulate a Turing machine. Head over to its Wikipedia page if you want to learn more about the proof and the interesting story around it.

Rule 110 starts with a string of zeros and ones and a transition table that decides the next state of the automaton. If you put each line of the strings after the other, interesting patterns can emerge. Let’s see the transition state:

Current pattern	111	110	101	100	011	010	001	000
New state for center cell	0	1	1	0	1	1	1	0

If you look closely, you can use a list of eight digits and its index in order to encode the above state transitions:

rule110 = [
  0, -- ((0,0,0), 0)
  1, -- ((0,0,1), 1)
  1, -- ((0,1,0), 1)
  1, -- ((0,1,1), 1)
  0, -- ((1,0,0), 0)
  1, -- ((1,0,1), 1)
  1, -- ((1,1,0), 1)
  0  -- ((1,1,1), 0)
  ] :: [Int]

But what about the transitions of the leftmost and rightmost digit you might think. Let’s assume that their missing neighbor is zero. Therefore, given an initial state and a rule that governs the transitions, we may calculate the next state with:

nextState :: [Int] -> [Int] -> [Int]
nextState state rule =
  [ rule !! x |
    let t = [0] ++ state ++ [0],
    i <- [1..(length(t)-2)],
    let x = (t !! (i-1)) * 4 + (t !! i) * 2 + (t !! (i+1))
  ]

-- construct an infinite sequence of next states
sequenceState :: [Int] -> [Int] -> [[Int]]
sequenceState state rule =
  [state] ++ sequenceState (nextState state rule) rule

Example:

*Main> state = [0,1,1,0]
*Main> nextState state rule110
[1,1,1,0]

One of the most interesting patterns occurs when we begin with the right most digit being 1 and all the rest being zeros:

*Main> state = [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1] :: [Int]
*Main> x = take 30 $ sequenceState state rule110
*Main> showState x
                             *
                            **
                           ***
                          ** *
                         *****
                        **   *
                       ***  **
                      ** * ***
                     ******* *
                    **     ***
                   ***    ** *
                  ** *   *****
                 *****  **   *
                **   * ***  **
               ***  **** * ***
              ** * **  ***** *
             ******** **   ***
            **      ****  ** *
           ***     **  * *****
          ** *    *** ****   *
         *****   ** ***  *  **
        **   *  ***** * ** ***
       ***  ** **   ******** *
      ** * ******  **      ***
     *******    * ***     ** *
    **     *   **** *    *****
   ***    **  **  ***   **   *
  ** *   *** *** ** *  ***  **
 *****  ** *** ****** ** * ***
**   * ***** ***    ******** *
*Main>

The output was somehow pretty printed:

showState [] = return ()
showState state = do
  -- putStrLn $ show (state !! 0)
  putStrLn $ [ c | d <- (state !! 0), let c = if d == 0 then ' ' else '*' ]
  showState $ tail state

I wish I can find time and play more with cellular automata. I kind of find a day every five years or so.

Update: Here is a pattern using Rule 90:

                             *
                            *
                           * *
                          *
                         * *
                        *   *
                       * * * *
                      *
                     * *
                    *   *
                   * * * *
                  *       *
                 * *     * *
                *   *   *   *
               * * * * * * * *
              *
             * *
            *   *
           * * * *
          *       *
         * *     * *
        *   *   *   *
       * * * * * * * *
      *               *
     * *             * *
    *   *           *   *
   * * * *         * * * *
  *       *       *       *
 * *     * *     * *     * *
*   *   *   *   *   *   *   *

A Collatz sequence in Haskell

WordPress.com does not always render Markadown properly, so a copy of this post resides here.

I am coninuing my adventure in Haskell. In order to make it a bit more fun, I decided to code a simple yet very intriguing problem, I first heard of when I read AI Memo 239: The Collatz conjecture.

Item 133. The Collatz conjecture in AIM 239

Construct a sequence of integers where given an arbitrary interger the value of the next is:
* If the number is even, divide it by two.
* If the number is odd, triple it and add one.

This can easily be coded in Haskell as follows:

collatz :: Int -> Int
collatz 1 = 1
collatz n =
  if (even n)
    then (n `div` 2)
    else (3 * n + 1)

But how can one obtain a sequence of numbers from this? A very clever solution is here where the author implements a variation of takeWhile which includes also the first list item that fails the test the first time. So my question became, can it be done in another way? Yes it can:

collatzSequence :: Int -> [Int]
collatzSequence n =
  if n == 1
    then [1]
    else [n] ++ collatzSequence (collatz n)

Let's see some test runs:

*Main> collatzSequence 5
[5,16,8,4,2,1]
*Main> collatzSequence 50
[50,25,76,38,19,58,29,88,44,22,11,34,17,52,26,13,40,20,10,5,16,8,4,2,1]
*Main> collatzSequence 500
[500,250,125,376,188,94,47,142,71,214,107,322,161,484,242,121,364,182,91,274,137,412,206,103,310,155,466,233,700,350,175,526,263,790,395,1186,593,1780,890,445,1336,668,334,167,502,251,754,377,1132,566,283,850,425,1276,638,319,958,479,1438,719,2158,1079,3238,1619,4858,2429,7288,3644,1822,911,2734,1367,4102,2051,6154,3077,9232,4616,2308,1154,577,1732,866,433,1300,650,325,976,488,244,122,61,184,92,46,23,70,35,106,53,160,80,40,20,10,5,16,8,4,2,1]
*Main> collatzSequence 512
[512,256,128,64,32,16,8,4,2,1]
*Main> collatzSequence 513
[513,1540,770,385,1156,578,289,868,434,217,652,326,163,490,245,736,368,184,92,46,23,70,35,106,53,160,80,40,20,10,5,16,8,4,2,1]
*Main>

You may have observed we only run it on positive integers. When we run it with negative integers, there are a few more cycles that we need to take into account. Here is the updated sequence function, written with guards:

collatzSequence :: Int -> [Int]
collatzSequence n
  | n == 1 = [1]
  | n == (-2) = [(-2)]
  | n == (-5) = [(-5)]
  | n == (-17) = [(-17)]
  | otherwise = [n] ++ collatzSequence (collatz n)

Test:

*Main> collatzSequence (-50)
[-50,-25,-74,-37,-110,-55,-164,-82,-41,-122,-61,-182,-91,-272,-136,-68,-34,-17]

Now if there was also a way to prove the conjecture…

Please note that in Haskell the unary minus is a function and not an operator, hence, you need to parenthesize. This also works:

*Main> collatzSequence $ -50
[-50,-25,-74,-37,-110,-55,-164,-82,-41,-122,-61,-182,-91,-272,-136,-68,-34,-17]

Update: A friend posted me his own elegant version of the Collatz sequence:

collatz :: Int -> [Int]
collatz 1 = [1]
collatz n
    | even n =  n : collatz (n `div` 2)
    | odd n  =  n : collatz (n * 3 + 1)

main = do
  putStrLn $ show $ collatz 1
  putStrLn $ show $ collatz 6
  putStrLn $ show $ collatz 23

pi day today

I was looking for some trivia to write for today since it is Pi Day. Like for example, how to calculate the n-th digit of Pi (in base16). Or how to locate your birthdate in a π sequence. Here is an example for Georgios Papanikolaou and one for Albert Einstein. I found a book on category theory. Or how from searching how to calculate π digits you end up through random clicking to the PSLQ Algorithm (I do not even remember how now). Interesting stuff one rarely gets the chance to get exposed to. Or why tau trumps pi. But I really lack the energy for a proper post today. So it is just links and a piece of art:

6 répartitions aléatoires de 4 carrés noirs et blancs d'après les chiffres pairs et impairs du nombre Pi — François Morellet, 6 répartitions aléatoires de 4 carrés noirs et blancs d’après les chiffres pairs et impairs du nombre Pi, 1958

a newbie does list comprehensions

Formatting this post in WordPress.com was a great pain. It does not render correctly on some browser / device combinations, despite my rewrite efforts. So a Markdown copy of this post can be found as a gist here.

The year is 1998 and @mtheofy then at Glasgow tells me about a relatively new (then) language called Haskell. I’m intrigued but do not do much. A few years later I buy The Haskell School of Expression since The Craft of Functional Programming did not seem enough to motivate me. Time passes and around 2007 I try yet another start. Nothing. I promised my self yet another restart for a 2017 new year’s resolution. Still nothing. So when the current employer offered Haskell classes I could not say no. Armed with the weekly classes and a Safari Learning Path I am trying to correct this. And I am having some fun with list comprehensions. Because as a friend says, if it makes you feel good, go.

So how do you write an infinite list? Let’s say you want list x to include all numbers from 0 to infinity. stack ghci is my friend. Others might try repl.it:

x = [ n | n <- [0..]]

Now you can have the first 20 items of x:

Prelude> x = [ n | n <- [0..]]
Prelude> take 20 x
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
Prelude>

So next I wanted to make an infinite list of the same character. Enter the underscore variable:

Prelude> x = [ 'a' | _ <- [0..]]
Prelude> take 20 x
"aaaaaaaaaaaaaaaaaaaa"
Prelude>

OK, so now let’s try to cycle infinitely characters from a string. I end up with:

Prelude> x = [ c | i  take 20 x
"abcdabcdabcdabcdabcd"
Prelude>

I am kind of unsure why the let statements are needed since I am ~10 days into typing stuff and posted my creation to twitter. What my expression says is that x is comprised of characters from string “abcd”, where given a sequence of numbers, each time a character is chosen based on the sequence number modulo 4. Strings are lists of characters in Haskell and list indexing starts from zero. Helpful comments come my way. Like the obvious cycle (there is a cycle function? Yes ):

Prelude> take 20 (cycle "abcd")
"abcdabcdabcdabcdabcd"
Prelude> take 20 $ cycle "abcd"
"abcdabcdabcdabcdabcd"
Prelude>

Is not the dollar operator nice to get rid of parentheses? Here is another suggestion about cycling a string:

Prelude> x = [ "abcd" !! (i `mod` 4) | i  take 20 x
"abcdabcdabcdabcdabcd"
Prelude>

This one is more concise and does the same thing, always picking a character from "abcd". If the infix notation for mod confuses you, you can:

Prelude> x = [ "abcd" !! (mod i 4) | i  take 20 x
"abcdabcdabcdabcdabcd"
Prelude>

But the Internet does not stop there. It comes back with more helpful suggestions:

Welcome! A little feedback then if I may: the !! operator should be used VERY cautiously it is not typesafe and lists are not random access anyway. Opt for a function returning Maybe x and for a random access datastructure (strings are by default lists).

Which made me think: How about an infinite string randomly chosen from “abcd”?

$ stack install random
$ stack ghci
:
Prelude> import System.Random
Prelude System.Random> g <- newStdGen 
Prelude System.Random> x = [ "abcd" !! i | i <- randomRs (0,3) g ]
Prelude System.Random> take 10 x
"bcbbddcdab"
Prelude System.Random>

If you want a sequence with a different order, you need to reinitialise both g and x. I will figure out a better way some other time when …I have time.

Adventures with Maybe maybe in another post.

Formatting this post in WordPress.com was a great pain.

DOW ↓

“The stock was down 86 cents over the day. That means Bill lost $70 million today, whereas I only lost fuck all. But guess who’ll sleep better?”

Microserfs

[In-Reply-To:]

sleep

These days I am reading Autonomous in which humans and autonomous robots coexist (there are also indentured humans and robots) and at one point a remark is made about a robotic research assistant that does not need to sleep, implied as an advantage.

This is not something new. In Beggars in Spain (the first book) there are also genetically engineered people who do not sleep with this being a competitive advantage and causing friction among sleepers and non-sleepers.

And then I remember Marissa Mayer talking about being “strategic about when you sleep, when you shower, and how often you go to the bathroom”.

I think Age of Em deals with this in a better way having Ems slow down their CPUs (making it a matter of energy cost, but I have not finished that book, so I am not sure how this idea gets developed).

Which is why I want to leave you with the 6-2-1 rule that popped into my twitter stream: 6 hours of sleep, 2 meals per day, 1 shower. Analytics from my Nokia Go say that I am not really following this.

Happy New Year.

resolutions

Last year I promised myself that I would revisit Haskell. Well I did not, so I did not escape the new year’s resolutions cliche. It was an interesting year though, considering that I left my country, worked for Intel, resigned and returned back to Greece and to my previous work.

So for this year I will promise myself something simpler, as a continuation of things I still do in 2017: simply improve my Go-fu. And yes, I also tried to learn Go and miserably failed. Let’s see about that too.

Locations of Ancient Woolworths Stores follow Precise Geometrical Pattern

It has been a while since I last blogged, so here is something funny that I learned today while listening to Relatively Prime:

Matt Parker took it upon himself to debunk junk science related to precise geometry choices by Ancient People. The particular junk piece that triggered his analysis was the assertion of a prehistoric navigation system. However, you may not be familiar with it, but you may have heard terms like “holy geometry” and the like to prove that some Ancient tribe either held some technology now long gone or was in contact with some alien race. But do they really stand?

So Matt, took it upon himself to analyse the locations of the Woolworths in order to figure out whether they had some “outer” help in choosing their store locations:

The results revealed an exact and precise geometric placement of the Woolworths locations. Three stores around Birmingham formed an exact equilateral triangle (Wolverhampton, Lichfield and Birmingham stores) and if the base of the triangle is extended, it forms a 173.8 mile line linking the Conwy and Luton stores. Despite the 173.8 mile distance involved, the Conway Woolworths store is only 40 feet off the exact line and the Luton site is within 30 feet. All four stores align with an accuracy of 0.05%

So there, proof! Aliens helped them :) Plus, sometimes you need to remind people that 3 points always make a triangle, no coincidence there.

You can read the whole analysis here thanks to the Internet Archive. Given enough data, if you’re looking for a pattern, you’re going to find it I guess.