I upgraded Nerd Fonts to v3.0.1 in nixpkgs and in the process switched it to the newly provided tar.xz archives. After this change the derivation now builds almost twice as fast as it did before!
NerdFonts 3.0.0
real 12m15.649s
user 0m2.494s
sys 0m4.370s
NerdFonts 3.0.1
real 5m29.933s
user 0m2.209s
sys 0m4.043s
#NixOS