Nothing serious, just a few notes I like to share with friends and colleagues who, like me, script around curl.
curl -f / --fail
I try to use --fail whenever I can, because why would I want to exit zero on server errors?
$ curl -L https://download.grml.org/grml64-small_2024.02.iso.NO
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache/2.4.41 (Ubuntu) Server at ftp.fau.de Port 443</address>
</body></html>
$ echo $?
0
$ curl -f -L https://download.grml.org/grml64-small_2024.02.iso.NO
curl: (22) The requested URL returned error: 404
$ echo $?
22
curl --fail-with-body
I have a CI/CD situation where curl calls a webhook and it’s incredibly useful to see its error message in case of failure.
$ curl --fail https://binblog.de/xmlrpc.php
curl: (22) The requested URL returned error: 405
$ curl --fail-with-body https://binblog.de/xmlrpc.php
curl: (22) The requested URL returned error: 405
XML-RPC server accepts POST requests only.
set -o pipefail
When curl‘s output gets piped to any other command, I try to remember to set -o pipefail along with curl --fail so if curl fails, the pipe exits non-zero.
#!/usr/bin/env bash
url='https://download.grml.org/grml64-small_2024.02.iso.NONO'
if curl -s -f -L "${url}" | sha256sum
then
echo "Success."
else
echo "Failure."
fi
set -o pipefail
if curl -s -f -L "${url}" | sha256sum
then
echo "Success."
else
echo "Failure."
fi
curl --connect-timeout
Useful to get quicker response in scripts instead of waiting for the system’s default timeouts.
curl -w / --write-out
This may be over the top most of the time, but I have one situation that requires extremely detailed error handling. (The reason being a bit of a foul split DNS situation in the environment, long story.) This is where I use --write-out to analyze the server response.
curl_http_status="$(curl -o "${tmpfile}" --write-out '%{http_code}\n' "${url}")"
curl_exit_status=$?
Update: curl versions from 8.3.0 allow writing out to files.
curl -o "${tmpfile}" --write-out '%output{http_status.txt}%{http_code}' "${url}"
curl_exit_status=$?
curl_http_status="$(< http_status.txt)"
curl -n / --netrc / [ --netrc-file ]
Username:password authentication is a thing, no matter how much it’s discouraged. Here’s how to at least hide username and password from the process list.
$ chmod 600 ~/.netrc
$ cat ~/.netrc
machine binblog.de
login foo
password bar
$ curl -v -o /dev/null -n https://binblog.de
...
* Server auth using Basic with user 'foo'
...
To use any other file instead of ~/.netrc, use --netrc-file instead.