Try to fix perf regression in rustdoc after hir attributes Slight performance regression introduced in #131808 r? `@jieyouxu`