If $xdg_chache_home is empty, this is not a valid fish expression:
[ \( -z \) -o \( ! -d \) ]
and results into an error.
While at it, also use $XDG_CACHE_HOME if that directory does not exist.
This seems better than falling back to $HOME/.cache, which the user has
explicitly overridden via $XDG_CACHE_HOME.