So the requirement over here is, “use SHA512 for /etc/shadow, but with 640000 rounds instead of the default 5000, to slow down brute force attacks”. (Not sure why exactly 640000 though.)

Let’s confirm that this slows down brute force attacks. First create one pure 5000-round hash file, and one pure 640000-round hash file. Note how 640000 rounds hashing takes much longer at this stage already:

$ openssl rand -hex 2 | (time mkpasswd --method=sha512crypt --stdin) | tee shadow-sha512
$6$ZcZ6RoMB5pSad9Ca$alLttTrpP1BezuOued3JrVgv/0tq7mkI5jypP4cZ/smgWF30HuLmtAl.DExd23j3xPLCWc6zWF4eLNLGKLr77.

real    0m0.006s <--
user    0m0.000s
sys     0m0.003s

$ openssl rand -hex 2 | (time mkpasswd --rounds=640000 --method=sha512crypt --stdin) | tee shadow-sha512-640000rounds
$6$rounds=640000$ZBpVIbg3SKT.KerX$hTLaX/OVOWQol5UeVMq2pO1EI2L4nG4WWOIXPhmujq7EqxohLu/dQn3f.TSE8upaPmw/5y1nHrA24Kx2OfCzE/

real    0m0.284s <--
user    0m0.281s
sys     0m0.000s

In hashcat‘s nomenclature, SHA512 with its $6$ prefix is hash type 1800:

1800 | sha512crypt $6$, SHA512 (Unix)

Start cracking the 5000-round hash. --attack-mode 3 means “brute force”:

$ hashcat --status --attack-mode 3 --hash-type 1800 --increment shadow-sha512

The hash rate on this system’s GPU turns out to be about 90000 hashes per second, and finding the 4-character password generated by openssl rand -hex 2 succeeds in about 30 seconds.

Speed.#1.........:    90438 H/s (4.43ms) @ Accel:64 Loops:512 Thr:64 Vec:1

On to the 640000-rounds hash:

$ hashcat --status --attack-mode 3 --hash-type 1800 --increment shadow-sha512-640000rounds

After a very long time grinding the really short password increments, which it obviously isn’t optimized for, hashcat eventually ramps up to around 500 hashes per second.

I stopped the attempt after an hour when the system was approaching 50 degrees on the outer case.