Canceling all Jenkins jobs in queue

This is a continuation from a previous post where I showed how to disable all configured jobs in a Jenkins server (when for example launching a copy for test purposes). To this end, it may be the case that you have placed your Jenkins controller in quiet mode to have some ease of mind examining what goes on with your queue, or you simply want to cleanup the queue and have the system start with no jobs submitted. Whatever the reason, if you need to erase all of your Jenkins queue, python-jenkins and a few lines come to your assistance:

import jenkins

server = jenkins.Jenkins('http://127.0.0.1:8080/',
        timeout=3600,
        username=USERNAME,
        password=PASSWORD)

queue_info = server.get_queue_info()
for i in range(len(queue_info)):
    print(queue_info[i]['id'])
    server.cancel_queue(queue_info[i]['id'])

RUN –mount=type=ssh is not always easy

Let’s take a very barebones Jenkinsfile and use it to build a docker image that clones something from GitHub (and possibly does other stuff next):

pipeline {
  agent any

  environment {
    DOCKER_BUILDKIT=1
  }

  stages {
    stage('200ok') {
      steps {
        sshagent(["readonly-ssh-key-here"]) {
          script {
            sh 'docker build --ssh default -t adamo/200ok .'
          }
        }
      }
    }
  }
}

We are using the SSH Agent Plugin in order to allow a clone that happens in the Dockerfile:

# syntax=docker/dockerfile:experimental
FROM bitnami/git
RUN mkdir /root/.ssh &amp;&amp; ssh-keyscan github.com &gt;&gt; /root/.ssh/known_hosts
RUN --mount=type=ssh git clone git@github.com:a-yiorgos/200ok.git

This builds fine. But what if you need this to be some "rootless" container?

# syntax=docker/dockerfile:experimental
FROM bitnami/git
USER bitnami
WORKDIR /home/bitnami
RUN mkdir /home/bitnami/.ssh &amp;&amp; ssh-keyscan github.com &gt;&gt; /home/bitnami/.ssh/known_hosts
RUN --mount=type=ssh git clone git@github.com:a-yiorgos/200ok.git

This will fail with something like:

#14 [7/7] RUN --mount=type=ssh git clone git@github.com:a-yiorgos/200ok.git
#14       digest: sha256:fb15ac6ca5703d056c7f9bf7dd61bf7ff70b32dea87acbb011e91152b4c78ad4
#14         name: "[7/7] RUN --mount=type=ssh git clone git@github.com:a-yiorgos/200ok.git"
#14      started: 2021-12-17 12:00:22.859388318 +0000 UTC
#14 0.572 fatal: destination path '200ok' already exists and is not an empty directory.
#14    completed: 2021-12-17 12:00:23.508950696 +0000 UTC
#14     duration: 649.562378ms
#14        error: "executor failed running [/bin/sh -c git clone git@github.com:a-yiorgos/200ok.git]: exit code: 128"

rpc error: code = Unknown desc = executor failed running [/bin/sh -c git clone git@github.com:a-yiorgos/200ok.git]: exit code: 128

Why is that? Is not the SSH agent forwarding working? Well, kind of. Let’s add a couple of commands in the Dockerfile to see what might be the issue:

# syntax=docker/dockerfile:experimental
FROM bitnami/git
USER bitnami
WORKDIR /home/bitnami
RUN mkdir /home/bitnami/.ssh &amp;&amp; ssh-keyscan github.com &gt;&gt; /home/bitnami/.ssh/known_hosts
RUN --mount=type=ssh env
RUN --mount=type=ssh ls -l ${SSH_AUTH_SOCK}
RUN --mount=type=ssh git clone git@github.com:a-yiorgos/200ok.git

Then the build output gives us:

:
#13 [6/7] RUN --mount=type=ssh ls -l ${SSH_AUTH_SOCK}
#13       digest: sha256:ce8fcd7187eb813c16d84c13f8d318d21ac90945415b647aef9c753d0112a8a7
#13         name: "[6/7] RUN --mount=type=ssh ls -l ${SSH_AUTH_SOCK}"
#13      started: 2021-12-17 12:00:22.460172872 +0000 UTC
#13 0.320 srw------- 1 root root 0 Dec 17 12:00 /run/buildkit/ssh_agent.0
#13    completed: 2021-12-17 12:00:22.856049431 +0000 UTC
#13     duration: 395.876559ms
:

and subsequently fails to clone. This happens because the socket file /run/buildkit/ssh_agent.0 for the SSH agent forwarding is not accessible by user bitnami and thus no ssh identity is available to it.

I do not know whether it is possible to make use of RUN --mount=type=ssh in combination with USER where the user is not root. Please leave a comment if you know whether/how this can be accomplished.

So on which Jenkins system am I running on?

It is often the case that you run a staging / test Jenkins server that has identically configured jobs as the production one. In such cases you want your pipeline to be able to distinguish in which system it runs on.

One way to do so it by checking the value of the BUILD_URL environment variable. However, this is not very helpful when you’re running the master inside a container, in which case you get back the container hostname in response.

There are also a number of solutions in StackOverflow you can look at, but you may opt to utilise the fact that you can add labels to each master accordingly and then query the master for the value of the labels it carries. Our solution depends on the httpRequest plugin in order to query the master.

import groovy.json.JsonSlurper

def get_jenkins_master_labels() {
    def response = httpRequest httpMode: 'GET', url: "http://127.0.0.1:8080/computer/(master)/api/json"
    def j = new JsonSlurper().parseText(response.content)
    return j.assignedLabels.name
}

def MASTER_NODE = get_jenkins_master_labels()

pipeline {
    agent {
        label 'docker'
    }
    stages {
        stage("test") {
            steps {
                println MASTER_NODE
            }
        }
    }
}

The trick here is that the part outside of the pipeline { ... } block runs directly on the master, so we can go ahead and call http://127.0.0.1:8080/computer/(master)/api/json to figure out stuff. get_jenkins_master_labels() queries the master and returns a list of all the labels assigned to the master (or a single string, master if no other labels are assigned to it). By checking the values of the list, one can infer in which Jenkins environment they are running on and continue from there.

What does the file $JENKINS_HOME/.owner do?

I have four books that on Jenkins and have read numerous posts on the Net that discuss weird Jenkins details and internals (more than I ever wished to know about), but none that explains what the file $JENKINS_HOME/.owner does (even though they include listings like this ). I found out about it recently because I was greeted by the message:

Jenkins detected that you appear to be running more than one instance of Jenkins
that share the same home directory. This greatly confuses Jenkins and you will
likely experience strange behaviours, so please correct the situation.

This Jenkins:  1232342241 contextPath="" at 2288@ip-172.31.0.10
Other Jenkins: 863352860 contextPath="" at 1994@ip-172.31.0.14

[Ignore this problem and keep using Jenkins anyway]

Indeed it appears that Jenkins, after initialisation, does run a test to check whether another process already runs from the same directory. When the check is run, it creates the file $JENKINS_HOME/.owner, The .owner part of the name is hardcoded.

Even more interesting is the fact, that in order to avoid having the two processes write information on .owner at the same time, randomises when the process is going to write on the file, so even if both processes start at the same time, chances that their writes coincide are slim.

What does it write in this file, you ask? There you go. When was this feature added? 2008/01/31. The mechanism is documented in the comments of the code:

The mechanism is simple. This class occasionally updates a known file inside the hudson home directory, and whenever it does so, it monitors the timestamp of the file to make sure no one else is updating this file. In this way, while we cannot detect the problem right away, within a reasonable time frame we can detect the collision.

You may want to keep that in mind, especially in cases when you’re greeted by the above message but know for a fact that a second process is not running. Some abrupt ending of the previous process occurred and you did not take notice. Or indeed a second process is messing with your CI

Mass disabling all Jenkins jobs

There are times that you need to disable all jobs on a Jenkins server. Especially when you’ve made a backup copy for testing or other purposes. You do not want jobs to start executing from that second server before you’re ready. Sure you can start Jenkins in quiet mode but sometime you have to exit it and scheduled jobs will start running. What can you do?

Well, there are plenty of pages that show Groovy code that allows you to stop jobs, and there are even suggestions to locate and change every config.xml file by running something like sed -i 's/disabled>false/disabled>true/' config.xml on each of them. Or even better use the Configuration Slicing plugin. Firstly, you may feel uneasy to mass change all config.xml file from a process external to Jenkins. Secondly, the Configuration Slicing plugin does not give you a "select all option" nor does it handle Multibranch Pipeline jobs. Thirdly, the Groovy scripts I’ve found shared by others online, also do not handle Pipelines and Multibranch Pipelines. If you’re based on Multibranch Pipelines, you’re kind of stuck then. Or you have to go and manually disable each one of them.

Thankfully there’s a solution using Jenkins’s REST API and python-jenkins. An example follows:

import jenkins

server = jenkins.Jenkins('http://127.0.0.1:8080/',
        timeout=3600,
        username=USERNAME,
        password=PASSWORD)


all_jobs = server.get_all_jobs()
for j in range(len(all_jobs)):
    try:
        server.disable_job(all_jobs[j]['fullname'])
    except Exception as e:
        print(all_jobs[j]['fullname'])

I hope it helps you out maintaining your Jenkins.