Rehoming a 26,000-file Reference library

by

I’m a true pack rat, I store everything just in case so I had a massive Calibre library living on a network drive kept “freezing” during library checks. The fix was to move it onto a local SSD and back it up to the NAS instead. Along the way: an rsync gotcha, a format-cleanup script that nearly ate the whole library, a recovery that worked only because of a good habit, and a surprisingly deep macOS permissions rabbit hole to automate it all. Here’s the full story, with the commands that mattered.

I. “Frozen” wasn’t frozen

1. The symptom

Calibre showed as “Application Not Responding,” yet Activity Monitor showed the CPU slowly ticking. That combination is the tell: the app isn’t dead, it’s busy.

  • The Force Quit / “frozen” label only measures the UI thread answering the OS’s pings.
  • A library check on a big library hogs the work thread, starving the UI, so macOS paints it red.
  • The slowly moving CPU (and disk) activity means it’s genuinely progressing, buy way too slowly. The fix for a real hang is different from the fix for a busy app.

2. The real cause

The library lived on a Synology NAS, and the check had to deal with ~37,000 files and folders over the network. Each lookup is a separate SMB round-trip. Trivial locally, brutal in aggregate.

II. The architecture fix

1. Go local, back up to the NAS

DECISION

Keep the working library on a directly-attached SSD; use the NAS as a backup target. Metadata-heavy operations (library checks, scans, traversals) are exactly what suffers on a network share and benefits most from going local.

  • Local stat calls are microseconds; the same calls over SMB are milliseconds, roughly 1000× the latency.
  • The only unavoidably network-bound step is moving actual bytes, i.e. the backup, which happens quietly in the background.

III. Copying the library (and the exFAT trap)

1. rsync, not drag-and-drop

rsync skips what’s already there and resumes cleanly if interrupted. It’s picky about trailing slashes: a slash on the source copies its contents, no slash nests the folder. 

rsync -av --ignore-existing "/Volumes/Media/Library/" "/Volumes/Dock SSD/library/"

2. “exited with status 23”

GOTCHA

Exit 23 = partial transfer. The culprit: the external SSD was exFAT, which has no concept of Unix permissions or ownership. The -a flag tries to preserve exactly those, so rsync copies the file then errors setting its attributes.

The fix is to stop trying to preserve what the filesystem can’t hold:

rsync -rtv --no-perms --no-owner --no-group \   "/Volumes/Media/Library/" "/Volumes/Dock SSD/library/"

IV. The EPUB cleanup that nearly ate the library

1. The goal

Shrink the library by keeping one format (EPUB), converting books that lacked it and stripping the redundant MOBI/AZW3/etc. A script did this via calibredb so the database stayed in sync, never deleting a book’s only format.

2. What went wrong

INCIDENT

Most books suddenly showed as empty. The chain of failure:

  • Some MOBI→EPUB conversions “succeeded” (exit 0) but produced empty/broken EPUBs.
  • The strip phase checked only that an EPUB existed, not that it was any good.
  • So it deleted the real MOBI source, leaving only the dud EPUB. Books looked gone.

The dry-run looked good because a dry-run neither converts nor deletes, so it couldn’t reveal that the conversions would be duds.

“Verify a conversion is actually valid before you delete the thing it was converted from.”The lesson, in one line

V. Recovery, and the habit that saved it

1. The original was untouched

SAVED

The script only ever ran against the SSD copy. The original library on the NAS was never written to, so it was a complete, pristine backup. Recovery was a single reverse mirror:

rsync -rtv --delete "/Volumes/Media/Library/" "/Volumes/Dock SSD/library/"

Net books lost: zero.

2. Takeaways from the this

  • Keep a pristine original you never write to until you’re certain. That one habit turned a disaster into a shrug and a loss of time.
  • For library-structural changes, prefer Calibre’s own tools (they keep metadata.db in sync) over scripts that delete files directly.
  • A dry-run only tests the steps it can safely simulate. It can’t foresee bad output.
  • Calibre cannot cope with directories being dumped into it’s tree by finder.

In the end, the format cleanup and duplicate sorting were redone with Calibre’s built-in tools, with the library rebuilt fresh.

VI. A safe, automated backup

1. Design for “can’t make it worse”

SCRIPT

The automated backup is additive (no --delete), guards against unmounted drives, excludes Calibre’s trash, and notifies on failure. Worst case is “backup missed the newest change,” never “backup got wiped.”

#!/bin/bash
set -uo pipefail
SRC="/Volumes/Dock SSD/Reference"
DEST="/Volumes/Media/Library-backup"
LOG="$HOME/Library/Logs/calibre-backup.log"
NOTIFY_SUCCESS=1
ts()     { date '+%Y-%m-%d %H:%M:%S'; }
notify() { /usr/bin/osascript -e "display notification \"$2\" with title \"$1\" sound name \"$3\"" >/dev/null 2>&1; }

# Guards: drive/NAS not mounted = quiet skip, not a failure
[ -f "$SRC/metadata.db" ] || { echo "$(ts) SKIP: source not mounted" >> "$LOG"; exit 0; }
[ -d "/Volumes/Media" ]   || { echo "$(ts) SKIP: NAS not mounted"    >> "$LOG"; exit 0; }
mkdir -p "$DEST"

echo "$(ts) START" >> "$LOG"
rsync -rt --exclude='/.caltrash/' --exclude='.DS_Store' --exclude='._*' \
  "$SRC/" "$DEST/" >> "$LOG" 2>&1
rc=$?
echo "$(ts) DONE (exit $rc)" >> "$LOG"

if [ "$rc" -ne 0 ]; then
  notify "Calibre backup FAILED" "rsync exit $rc" "Basso"
elif [ "$NOTIFY_SUCCESS" -eq 1 ]; then
  notify "Calibre backup OK" "Library mirrored to NAS" "Glass"
fi

2. Scheduling it

A user launchd LaunchAgent runs it on a schedule. If the Mac is asleep at the scheduled time, launchd runs the job on wake. The guards make a missing drive a no-op.

VII. The macOS permissions rabbit hole

1. Works in Terminal, fails in launchd

The script ran fine by hand but failed under launchd with “Operation not permitted” on both the SSD and the NAS. A Terminal run inherits Terminal’s privacy permissions; a launchd job gets none. macOS now gates removable and network volumes.

2. Two dead ends

  • Copying /bin/bash to a private path to give it its own permission identity: fails, because copying strips Apple’s code signature and macOS won’t run an unsigned system binary in the background.
  • An Automator “Application” wrapper: the bundle wouldn’t launch (“not supported on this Mac”), a known Automator export quirk.

3. What actually worked

FIXED

What the Internet suggested was Granting /bin/bash Full Disk Access. Broad, yes, but on a single-user Mac the real-world risk is modest, and it’s the one lever that reliably covers both the removable and network volumes a launchd job needs. How ever I don’t like taking that risk.

The solution I settled on was creating a signed AppleScript app wrapper via Script Editor, verified by double-clicking before wiring it into launchd.

VIII. Where it landed

The final setup

DONE

  • Fast local library on the Dock SSD. No more frozen library checks.
  • Deduplicated and standardised using Calibre’s own tools.
  • Automated, permissioned, self-notifying nightly backup to the NAS, additive and guarded.
  • A pristine original kept untouched throughout, the single reason the scary moment cost nothing.

Leave a comment