Tag Archives: shell

Failsafe curl

Nothing serious, just a few notes I like to share with friends and colleagues who, like me, script around curl.

curl -f / --fail

I try to use --fail whenever I can, because why would I want to exit zero on server errors?

$ curl -L https://download.grml.org/grml64-small_2024.02.iso.NO
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache/2.4.41 (Ubuntu) Server at ftp.fau.de Port 443</address>
</body></html>
$ echo $?
0
$ curl -f -L https://download.grml.org/grml64-small_2024.02.iso.NO
curl: (22) The requested URL returned error: 404
$ echo $?
22

curl --fail-with-body

I have a CI/CD situation where curl calls a webhook and it’s incredibly useful to see its error message in case of failure.

$ curl --fail https://binblog.de/xmlrpc.php
curl: (22) The requested URL returned error: 405
$ curl --fail-with-body https://binblog.de/xmlrpc.php
curl: (22) The requested URL returned error: 405
XML-RPC server accepts POST requests only.

set -o pipefail

When curl‘s output gets piped to any other command, I try to remember to set -o pipefail along with curl --fail so if curl fails, the pipe exits non-zero.

#!/usr/bin/env bash

url='https://download.grml.org/grml64-small_2024.02.iso.NONO'

if curl -s -f -L "${url}" | sha256sum
then
        echo "Success."
else
        echo "Failure."
fi

set -o pipefail

if curl -s -f -L "${url}" | sha256sum
then
        echo "Success."
else
        echo "Failure."
fi

curl --connect-timeout

Useful to get quicker response in scripts instead of waiting for the system’s default timeouts.

curl -w / --write-out

This may be over the top most of the time, but I have one situation that requires extremely detailed error handling. (The reason being a bit of a foul split DNS situation in the environment, long story.) This is where I use --write-out to analyze the server response.

curl_http_status="$(curl -o "${tmpfile}" --write-out '%{http_code}\n' "${url}")"
curl_exit_status=$?

Update: curl versions from 8.3.0 allow writing out to files.

curl -o "${tmpfile}" --write-out '%output{http_status.txt}%{http_code}' "${url}"
curl_exit_status=$?
curl_http_status="$(< http_status.txt)"

curl -n / --netrc / [ --netrc-file ]

Username:password authentication is a thing, no matter how much it’s discouraged. Here’s how to at least hide username and password from the process list.

$ chmod 600 ~/.netrc
$ cat ~/.netrc
machine binblog.de
login foo
password bar
$ curl -v -o /dev/null -n https://binblog.de
...
* Server auth using Basic with user 'foo'
...

To use any other file instead of ~/.netrc, use --netrc-file instead.

Too good to #0005

Prioritizing own NTP servers for systemd-timesyncd

# cat /etc/systemd/timesyncd.conf.d/timesyncd-corporate.conf 
[Time]
NTP=ntp1.example.com ntp2.example.com ntp.ubuntu.com

Test if text is empty (even if it does contain a linebreak)

Good job by: https://unix.stackexchange.com/q/386499/2028

if ! grep -q '[^[:space:]]' "${file}"; then echo "Text is empty"; fi

Watch for changes in the Gnome registry

(To reproduce manual changes, for configuration management.)

dconf watch / # (Sorry, thats all)

Trigger Debian/Ubuntu unattended-upgrade

(For testing configuration changes)

rm /var/lib/apt/periodic/*
systemctl start apt-daily.service 
systemctl start apt-daily-upgrade.service 

And remember:

tail -f /var/log/unattended-upgrades/*log

Too good to #0003

Linux uptime in seconds, once and for all

awk '{printf "%i\n", $1}' /proc/uptime

“Sudo on demand”, re-exec shell script with sudo if not running as root

#!/usr/bin/env bash
printf "Running as: %s\n" "$(id -un)"
[[ ${EUID} -ne 0 ]] && printf "Re-executing with sudo.\n" && exec sudo "${0}"

See also TGT0006, this is just as useful for downgrading privileges on the fly.


“When was the last time apt-get on that Debian/Ubuntu machine installed package upgrades?”

  • Reliably answering this is a lot harder than it looks, subject of countless discussions and really does need to parse /var/log/apt/history.log, which is painful.
  • The script below maintains a file /var/log/apt/lastupgrade with the last upgrade’s time stamp, for further processing.
  • Does NOT track invocations of apt-get upgrade that did not lead to package upgrades.
  • Does NOT look behind logfile rotations, which should not be a problem because it’s closely hooked to dpkg.

/usr/sbin/apt-lastupgrade:

#!/bin/bash

while IFS=: read -r key value
do
	if [[ "${key}" == 'Start-Date' ]]
	then
		upgraded=0
	elif [[ "${key}" == 'Upgrade' ]]
	then
		upgraded=1
	elif [[ "${key}" == 'End-Date' ]]
	then
		if [[ ${upgraded} -eq 1 ]]
		then
			printf -v lastupgrade "%s" "${value}"
		fi
		upgraded=0
	fi
done < /var/log/apt/history.log

if [[ -v lastupgrade ]]
then
	tee /var/log/apt/lastupgrade <<-Here
	# Timestamp of last upgrade: ${lastupgrade}
	Here
	touch -d "${lastupgrade}" /var/log/apt/lastupgrade
fi

/etc/apt/apt.conf.d/90lastupgrade:

DPkg::Post-Invoke {"/usr/bin/systemd-run --on-active=60 /usr/sbin/apt-lastupgrade || /bin/true"};

Path of running shell script, dirname for locating config files, includes etc.

me_path="$(readlink -f "${0}")"
me_dir="$(dirname "${me_path}")"
me_base="$(basename "${me_path}")"

Too good to #0002

Show and update the comment in an SSH private key

ssh-keygen -l -f <keyfile> # show
ssh-keygen -c -f <keyfile> # change interactively
ssh-keygen -c -C <newcomment> -f <keyfile> # change and provide new

ssh-agent in Gitlab CI/CD

  • Define SSH_KEY as a “file” in Gitlab CI/CD variables and SSH_PASSPHRASE as a regular variable
  • If libcrypto errors on execution, make sure SSH_KEY has an additional line ending at the end
before_script:
  - DEBIAN_FRONTEND=noninteractive apt-get update -y
  - DEBIAN_FRONTEND=noninteractive apt-get install -y openssh-client

sftp_to_somewhere:
  stage: deploy
  script:
    - chmod 600 "$SSH_KEY"
    - eval $(ssh-agent)
    - ssh-add "$SSH_KEY" <<< "$SSH_PASSPHRASE"
    - ssh-add -l

Urlwatch, but with a local pop-up window on Linux, no additional infrastructure

Systemd user service:

# ~/.config/systemd/user/urlwatch-to-zenity.service
[Unit]
Description=urlwatch to zenity (service)

[Service]
Type=oneshot
ExecCondition=sh -c 'nmcli net co | grep full'
ExecCondition=sh -c 'urlwatch > %h/.cache/urlwatch.out'
ExecCondition=sh -c 'test -s %h/.cache/urlwatch.out'
ExecStart=sh -c 'zenity --title Urlwatch --text-info --width=600 --height=400 --filename=%h/.cache/urlwatch.out'

[Install]
WantedBy=default.target

Systemd timer:

# ~/.config/systemd/user/urlwatch-to-zenity.timer
[Unit]
Description=urlwatch to zenity (timer)

[Timer]
OnStartupSec=1h
OnUnitInactiveSec=2h

[Install]
WantedBy=default.target

cookiecount – Load a page and show the cookies it sets

$ ./cookiecount https://example.com
0 cookies received.

ps1_anon.bash – anonymize bash prompt, for screenshots and pastes

# Anonymize bash prompt for screenshots and pastes
# Source from or add to ~/.bashrc
function ps1_anon (){
	if [[ -v SAVED_PS1 ]] 
	then
		PS1="${SAVED_PS1}" 
		unset SAVED_PS1 
	else
		SAVED_PS1="${PS1}" 
		PS1='\$ '
	fi
}

Example:

[martin@idefix ~/devel/cookiecount(main=)]$ ps1_anon
$ ./cookiecount https://reddit.com --json | jq .cookiecount
7
$ ps1_anon
[martin@idefix ~/devel/cookiecount(main=)]$

Too good to #0001

Set X11 keyboard layout manually and temporarily (e.g. in i3 if I need to test exotic window managers for my users)

setxkbmap -layout us,us -variant altgr-intl -option caps:none

Speaking of which, set keyboard layout permanently in Gnome

gsettings set org.gnome.desktop.input-sources sources "[('xkb', 'us+altgr-intl')]"

Check if Gnome Screensaver is active (I use this for automated time tracking)

busctl --user call org.gnome.ScreenSaver /org/gnome/ScreenSaver org.gnome.ScreenSaver GetActive


Urlwatch for a new package version in an APT repository

After a long wait, I was finally able to find another use for my awful Perl one-liner from 2019 that pivots a Debian Packages.gz into a space-separated table!

---
name: "Pop!OS Firefox package"
command: curl -s http://apt.pop-os.org/release/dists/jammy/main/binary-amd64/Packages.gz |
  gzip -dc | perl -ane 'if($F[0]=~/^Package:/){$p=$F[1]};if($F[0]=~/^Version:/){$v=$F[1];print"$p $v\n";}' |
  grep '^firefox '
diff_filter:
  - grep: '^[+-][^+-]'
---