Category Archives: UNIX & Linux

Letsencrypt und DNS Round-Robin

Ich weiß jetzt wieder, wie ich Letsencrypt auf DNS Round-Robin mit zwei Servern aufgesetzt hatte und finde es auch mit einigen Jahren Abstand gar nicht mal so schrecklich.

Die Server kennen sich auf Port 80 unter ihrem Namen im Round-Robin und unter einem eigenen Rollennamen:


mod_rewrite auf Port 80 fängt zunächst mal alles ab, was nicht nach Letsencrypt aussieht und leitet nach HTTPS weiter; hier auf


RewriteEngine On

RewriteCond %{REQUEST_URI} !^/\.well\-known/acme\-challenge/
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

Was jetzt noch übrig bleibt, wird daraufhin geprüft, ob a) die angeforderte Challenge im Dateisystem nicht vorhanden ist und b) der Request an den Namen aus dem Round-Robin gestellt wurde (um endlose Proxy-Rekursionen zu verhindern). Falls die Challenge lokal nicht existiert, wird der Request per Reverse Proxy an einen nachgelagerten Server gestellt:


RewriteEngine On

RewriteCond %{REQUEST_URI} !^/\.well\-known/acme\-challenge/
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

RewriteCond %{HTTP_HOST} ""
RewriteRule ^/(.*)${REQUEST_URI} [P,QSA,L]

Mit 2 Servern ist klar, wo die Challenge zu holen ist. Bei 3 und mehr Servern würde ich eher nicht riskieren, in multiple Proxy-Weiterleitungen zu gehen, sondern mod_rewrite über RewriteMap aktiv ermitteln lassen, auf welchem der anderen Server die Challenge denn tatsächlich liegt.

Too good to #0005

Prioritizing own NTP servers for systemd-timesyncd

# cat /etc/systemd/timesyncd.conf.d/timesyncd-corporate.conf 

Test if text is empty (even if it does contain a linebreak)

Good job by:

if ! grep -q '[^[:space:]]' "${file}"; then echo "Text is empty"; fi

Watch for changes in the Gnome registry

(To reproduce manual changes, for configuration management.)

dconf watch / # (Sorry, thats all)

Trigger Debian/Ubuntu unattended-upgrade

(For testing configuration changes)

rm /var/lib/apt/periodic/*
systemctl start apt-daily.service 
systemctl start apt-daily-upgrade.service 

And remember:

tail -f /var/log/unattended-upgrades/*log

Too good to #0004

systemd, the good parts: monotonic timers

When systemd makes you suffer because “run job every 10 minutes” is infinitely harder to specify than in crontab, remember there are monotonic timers in systemd that aren’t derived from wallclock time, or “Calendar Events” as they call it.

Run timer once 60 seconds after system startup, then every 10 minutes after the job finished:

# /etc/systemd/system/demo.timer
Description=demo monotonic timer


[Install]    # When in a system session
# # When in a user session (~/.config/systemd, systemctl --user etc.)

Python: Use tabulate to format output in columns

#!/usr/bin/env python3
from tabulate import tabulate

data = [
	[ 'foo', 'bar', 'baz' ],
	[ 'spam', 'eggs', 'bacon' ]

headers = ['Eine', '2', 'Whatever']

print(tabulate(data, headers=headers, tablefmt='simple'))


$ ./ 
Eine    2     Whatever
------  ----  ----------
foo     bar   baz
spam    eggs  bacon

virsh/libvirt, automate key presses

(* Updated to sleep 5 seconds after each keypress.)

for key in R E I S U B; do virsh send-key "${domain}" KEY_LEFTALT KEY_SYSRQ KEY_${key}; sleep 5; done

Too good to #0003

Linux uptime in seconds, once and for all

awk '{printf "%i\n", $1}' /proc/uptime

“Sudo on demand”, re-exec shell script with sudo if not running as root

#!/usr/bin/env bash
printf "Running as: %s\n" "$(id -un)"
[[ ${EUID} -ne 0 ]] && printf "Re-executing with sudo.\n" && exec sudo "${0}"

See also TGT0006, this is just as useful for downgrading privileges on the fly.

“When was the last time apt-get on that Debian/Ubuntu machine installed package upgrades?”

  • Reliably answering this is a lot harder than it looks, subject of countless discussions and really does need to parse /var/log/apt/history.log, which is painful.
  • The script below maintains a file /var/log/apt/lastupgrade with the last upgrade’s time stamp, for further processing.
  • Does NOT track invocations of apt-get upgrade that did not lead to package upgrades.
  • Does NOT look behind logfile rotations, which should not be a problem because it’s closely hooked to dpkg.



while IFS=: read -r key value
	if [[ "${key}" == 'Start-Date' ]]
	elif [[ "${key}" == 'Upgrade' ]]
	elif [[ "${key}" == 'End-Date' ]]
		if [[ ${upgraded} -eq 1 ]]
			printf -v lastupgrade "%s" "${value}"
done < /var/log/apt/history.log

if [[ -v lastupgrade ]]
	tee /var/log/apt/lastupgrade <<-Here
	# Timestamp of last upgrade: ${lastupgrade}
	touch -d "${lastupgrade}" /var/log/apt/lastupgrade


DPkg::Post-Invoke {"/usr/bin/systemd-run --on-active=60 /usr/sbin/apt-lastupgrade || /bin/true"};

Path of running shell script, dirname for locating config files, includes etc.

me_path="$(readlink -f "${0}")"
me_dir="$(dirname "${me_path}")"
me_base="$(basename "${me_path}")"

Too good to #0002

Show and update the comment in an SSH private key

ssh-keygen -l -f <keyfile> # show
ssh-keygen -c -f <keyfile> # change interactively
ssh-keygen -c -C <newcomment> -f <keyfile> # change and provide new

ssh-agent in Gitlab CI/CD

  • Define SSH_KEY as a “file” in Gitlab CI/CD variables and SSH_PASSPHRASE as a regular variable
  • If libcrypto errors on execution, make sure SSH_KEY has an additional line ending at the end
  - DEBIAN_FRONTEND=noninteractive apt-get update -y
  - DEBIAN_FRONTEND=noninteractive apt-get install -y openssh-client

  stage: deploy
    - chmod 600 "$SSH_KEY"
    - eval $(ssh-agent)
    - ssh-add "$SSH_KEY" <<< "$SSH_PASSPHRASE"
    - ssh-add -l

Urlwatch, but with a local pop-up window on Linux, no additional infrastructure

Systemd user service:

# ~/.config/systemd/user/urlwatch-to-zenity.service
Description=urlwatch to zenity (service)

ExecCondition=sh -c 'nmcli net co | grep full'
ExecCondition=sh -c 'urlwatch > %h/.cache/urlwatch.out'
ExecCondition=sh -c 'test -s %h/.cache/urlwatch.out'
ExecStart=sh -c 'zenity --title Urlwatch --text-info --width=600 --height=400 --filename=%h/.cache/urlwatch.out'


Systemd timer:

# ~/.config/systemd/user/urlwatch-to-zenity.timer
Description=urlwatch to zenity (timer)



cookiecount – Load a page and show the cookies it sets

$ ./cookiecount
0 cookies received.

ps1_anon.bash – anonymize bash prompt, for screenshots and pastes

# Anonymize bash prompt for screenshots and pastes
# Source from or add to ~/.bashrc
function ps1_anon (){
	if [[ -v SAVED_PS1 ]] 
		unset SAVED_PS1 
		PS1='\$ '


[martin@idefix ~/devel/cookiecount(main=)]$ ps1_anon
$ ./cookiecount --json | jq .cookiecount
$ ps1_anon
[martin@idefix ~/devel/cookiecount(main=)]$

Too good to #0001

Set X11 keyboard layout manually and temporarily (e.g. in i3 if I need to test exotic window managers for my users)

setxkbmap -layout us,us -variant altgr-intl -option caps:none

Speaking of which, set keyboard layout permanently in Gnome

gsettings set org.gnome.desktop.input-sources sources "[('xkb', 'us+altgr-intl')]"

Check if Gnome Screensaver is active (I use this for automated time tracking)

busctl --user call org.gnome.ScreenSaver /org/gnome/ScreenSaver org.gnome.ScreenSaver GetActive

Urlwatch for a new package version in an APT repository

After a long wait, I was finally able to find another use for my awful Perl one-liner from 2019 that pivots a Debian Packages.gz into a space-separated table!

name: "Pop!OS Firefox package"
command: curl -s |
  gzip -dc | perl -ane 'if($F[0]=~/^Package:/){$p=$F[1]};if($F[0]=~/^Version:/){$v=$F[1];print"$p $v\n";}' |
  grep '^firefox '
  - grep: '^[+-][^+-]'

IPv6 Privacy Stable Addressing Roundup

“Okay, let’s see whether we can reach your Macbook externally via IPv6. What’s the address?”
Sure, let’s have a look.

$ ifconfig
 inet6 2a03:2260:a:b:8aa:22bf:7190:ef36 prefixlen 64 autoconf secured
 inet6 2a03:2260:a:b:b962:5127:c7ec:d2df prefixlen 64 autoconf temporary

Everybody knows that one of these is a random IP address according to RFC 4941: Privacy Extensions for Stateless Address Autoconfiguration in IPv6 that changes once in a while so external observers (e.g. accessed web servers) can’t keep track of my hardware’s ethernet MAC address. This is the one we do NOT want if we want to access the Macbook from the internet. We want the stable one, the one that has the MAC address encoded within, the one with the ff:fe in the middle, as we all learned in IPv6 101.
It turns out, all of my devices that configure themselves via SLAAC, namely a Macbook, an iPhone, an iPad, a Linux laptop and a Windows 10 workstation, don’t have ff:fe addresses. Damn, I have SAGE status at, I must figure out what’s going on here!
After a bit of research totally from scratch with the most adventurous search terms, it turns out that these ff:fe, or more professionally, EUI-64 addresses, have become a lot less common than 90% of IPv6 how-to documents and privacy sceptics want us to believe. On most platforms, they have been replaced by Cryptographically Generated Addresses (CGAs), as described in RFC 3972. The RFC is a close relative to RFC 3971, which describes a Secure Neighbor Discovery Protocol (SeND). Together, they describe a cryptographically secure, PKI-based method of IPv6 address generation. However, as of this writing, only a PKI-less stub implementation of RFC 3972 seems to have become commonplace.
Those CGAs, or as some platforms seem to call them, Privacy Stable Addresses, are generated once during the first startup of the system. The address itself, or the seed used to randomize it, may be (and usually is) persistently stored on the system, so the system will come up every time with this same IPv6 address instead of one following the well-known ff:fe pattern.
To stick with the excerpt from my macOS ifconfig output above, the address marked temporary is a Privacy Extension address (RFC 4941), while the one marked secure is the CGA (RFC 3972).
It’s surprisingly hard to come up with discussions on the web where those two types aren’t constantly confused, used synonymously, or treated like ones that both need to be exterminated, no matter the cost. This mailing list thread actually is one of the most useful resources on them.
This blog post is a decent analysis on the behaviour on macOS, although it’s advised to ignore the comments.
This one about the situation on Windows suffers from a bit of confusion, but is where I found a few helpful Windows commands.
The nicest resource about the situation on Linux is this german Ubuntuwiki entry, which, given a bit of creativity, may provide a few hints also to non-german speakers.
So, how to configure this?

  • macOS
    • The related sysctl is net.inet6.send.opmode.
    • Default is 1 (on).
    • Note how this is the only one that refers to SeND in its name.
  • Windows
    • netsh interface ipv6 set global randomizeidentifiers=enabled store=persistent
    • netsh interface ipv6 set global randomizeidentifiers=disabled store=persistent
    • Default seems to be enabled.
    • Use store=active and marvel at how Windows instantly(!) replaces the address.
  • Linux
    • It’s complicated.
    • NetworkManager defaults to using addr-gen-mode=stable-privacy in the [ipv6] section of /etc/NetworkManager/system-connections/<Connection>.
    • The kernel itself generates a CGA if addrgenmode for the interface is set to none and /proc/sys/net/ipv6/conf/<interface>/stable_secret gets written to.
    • NetworkManager and/or systemd-networkd take care of this. I have no actual idea.
    • In manual configuration, CGA can be configured by using ip link set addrgenmode none dev <interface> and writing the stable_secret in a pre-up action. (See the Ubuntu page linked above for an example.)
  • FreeBSD
    • FreeBSD has no support for CGAs, other than a user-space implementation through the package “send”, which I had no success configuring.

So far, I haven’t been able to tell where macOS, Windows and NetworkManager persistently store their seeds for CGA generation. But the next time someone goes looking for an ff:fe address, I’ll know why it can’t be found.

Debian /boot old kernel images

So I was looking at yet another failed apt-get upgrade because /boot was full.
After my initial whining on Twitter, I immediately received a hint towards /etc/apt/apt.conf.d/01autoremove-kernels, which gets generated from /etc/kernel/postinst.d/apt-auto-removal after the installation of new kernel images. The file contains a list of kernels that the package manager considers vital at this time. In theory, all kernels not covered by this list should be able to be autoremoved by running apt-get autoremove.
However it turns out that apt-get autoremove would not remove any kernels at all, at least not on this system. After a bit of peeking around on Stackexchange, it turns out that this still somewhat newish concept seems to be ridden by a few bugs, especially concerning kernels that are (Wrongfully? Rightfully? I just don’t know.) marked as manually-installed in the APT database: “Why doesn’t apt-get autoremove remove my old kernels?”
The solution, as suggested by an answer to the linked question, is to mark all kernel packages as autoinstalled before running apt-get autoremove:

apt-mark showmanual |
 grep -E "^linux-([[:alpha:]]+-)+[[:digit:].]+-[^-]+(|-.+)$" |
 xargs -n 1 apt-mark auto

I’m not an APT expert, but I’m posting this because the post-install hook that prevents the current kernel from being autoremoved makes the procedure appear “safe enough”. As always, reader discretion is advised. And there’s also the hope that it will get sorted out fully in the future.

How expiration dates in the shadow file really work

tl;dr: Accounts expire as soon as UTC reaches the expiration date.
In today‘s installment of my classic shame-inducing series “UNIX basics for UNIX professionals”, I want to talk about account (and password) expiration in /etc/shadow on Linux.
The expiration time is specified as days since january 1st, 1970. In the case of account expiration, the according value can be found in the second to last field in /etc/shadow.
Account expiration can be configured using the option „-E“ to the „chage“ tool. In this case, I want the user „games“, which I‘ll be using for demonstration purposes, to expire on the 31st of december, 2017:

# chage -E 2017-12-31 games

Using the „-l“ option, I can now list the expiration date of the user:

# chage -l games
Account expires : Dec 31, 2017

The first thing to be taken away here is that, as I can only use a number of days, I can not let a user expire at any given time of day. In /etc/shadow, I have now:

# getent shadow | awk -F: '/^games:/{print $8}'

This of course can to be converted to a readable date:

# date --date='1970-01-01 00:00:00 UTC 17531 days'
Sun Dec 31 01:00:00 CET 2017

So, will the account still be usable on december 31st? Let‘s change it‘s expiration to today (the 7th of July, 2017) to see what happens:

# date
Fri Jul 7 12:58:32 CEST 2017
# chage -E today games
# chage -l games
Account expires : Jul 07, 2017
# su - games
Your account has expired; please contact your system administrator

I’m now only left with the question whether this expiration day is aligned on UTC or local time.

# getent shadow | awk -F: '/^games:/{print $8}'
# date --date='1970-01-01 00:00:00 UTC 17354 days'
Fri Jul 7 02:00:00 CEST 2017

I‘ll stop my NTP daemon, manually set the date to 00:30 today and see if the games user has already expired:

# date --set 00:30:00
Fri Jul 7 00:30:00 CEST 2017
# su - games
This account is currently not available.

This is the output from /usr/sbin/nologin, meaning that the account is not expired yet, so I know for sure that the expiration date is not according to local time but to UTC.
Let‘s move closer to our expected threshold:

# date --set 01:30:00
Fri Jul 7 01:30:00 CEST 2017
# su - games
This account is currently not available.

Still not expired. And after 02:00:

# date --set 02:30:00
Fri Jul 7 02:30:00 CEST 2017
# su - games
Your account has expired; please contact your system administrator

So, in order to tell from a script whether an account has expired, I simply need to get the number of days since 1970-01-01. If this number is greater or equal to the value in /etc/shadow, the user has expired.

DAYSSINCE=$(( $(date +%s) / 86400 )) # This is days till now as per UTC.
EXPIREDAY=$(getent shadow | awk -F: '/^games:/{print $8}')
if [[ $DAYSSINCE -ge $EXPIREDAY ]] # Greater or equal

One last thought: We’ve looked at a time zone with a small offset from UTC. What about timezones with larger offsets, in the other direction?

  • If we move the timezone to the east, further into the positive from UTC, it will behave the same as here in CEST and the account will expire sometime during the specified day, when UTC hits the same date.
  • If we move the timezone far to the west, like e.g. PST, and an absolute date is given to “chage -E“, the account will probably expire early, the day before scheduled expiration. I was not able to find anything useful on the web and even my oldest UNIX books from the 1990s mention password expiration only casually, without any detail. Active use of password expiration based on /etc/shadow seems to be uncommon. The code that seems to do the checking is here and it does not appear to care about time zones at all.
  • Any comments that clarify the behaviour in negative offsets from UTC will be appreciated.

SSH firewall bypass roundup

So my SSH workflow has reached a turning point, where I’m going to clean up my ~/.ssh/config. Some entries had been used to leverage corporate firewall and proxy setups for accessing external SSH servers from internal networks. These are being archived here for the inevitable future reference.
I never use “trivial” chained SSH commands, but always want to bring up a ProxyCommand, so I have a transparent SSH session for full port, X11, dynamic and agent forwarding support.
ProxyCommand lines have been broken up for readability, but I don’t think this is supported in ~/.ssh/config and they will need to be joined again to work.
Scenario 1: The client has access to a server in a DMZ
The client has access to a server in an internet DMZ, which in turn can access the external server on the internet. Most Linux servers nowadays have Netcat installed, so this fairly trivial constellation works 95.4% of the time.

# ~/.ssh/config
Host host.external
ServerAliveInterval 10
ProxyCommand ssh host.dmz /usr/bin/nc -w 60 host.external 22

Scenario 2: As scenario 1, but the server in the DMZ doesn’t have Netcat
It may not have Netcat, but it surely has an ssh client, which we use to run an instance of sshd in inetd mode on the destination server. This will be our ProxyCommand.

# ~/.ssh/config
Host host.external
ServerAliveInterval 10
ProxyCommand ssh -A host.dmz ssh host.external /usr/sbin/sshd -i

Scenario 2½: Modern version of the Netcat scenario (Update)
Since OpenSSH 5.4, the ssh client has it’s own way of reproducing the Netcat behavior from scenario 1:

# ~/.ssh/config
Host host.external
ServerAliveInterval 10
ProxyCommand ssh -W host.external:22 host.dmz

Scenario 3: The client has access to a proxy server
The client has access to a proxy server, through which it will connect to an external SSH service running on Port 443 (because no proxy will usually allow connecting to port 22).

# ~/.ssh/config
Host host.external
ServerAliveInterval 10
ProxyCommand /usr/local/bin/corkscrew
   proxy.server 3128
   host.external 443
# ~/.corkscrew/authfile

(Omit the authfile part, if the proxy does not require authentication.)
Scenario 4: The client has access to a very restrictive proxy server
This proxy server has authentication, knows it all, intercepts SSL sessions and checks for a minimum client version.

# ~/.ssh/config
Host host.external
ServerAliveInterval 10
ProxyCommand /usr/local/bin/proxytunnel
   -p proxy.server:3128
   -F ~/.proxytunnel.auth
   -r host.external:80
   -H "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0\nContent-Length: 0\nPragma: no-cache"
# ~/.proxytunnel.auth

What happens here:

  1. host.external has an apache web server running with forward proxying enabled.
  2. proxytunnel connects to the proxy specified with -r, via the corporate proxy specified with -p and uses it to connect to, on the forward-proxying apache.
  3. It sends a hand-crafted request header to the intrusive proxy, which mimics the expected client version.
  4. Mind you that although the connection is to a non-SSL service, it still is secure, because encryption is being brought in by SSH.
  5. What we have here is a hand-crafted exploit against the know-it-all proxy’s configuration. Your mileage may vary.

Super sensible discretion regarding the security of your internal network is advised. Don’t fuck up, don’t use this to bring in anything that will spoil the fun. Bypass all teh firewalls responsibly.