Got this error. lsof gave nothing. No user was using it, was not in mounts.
Solution was:
echo 1 > /proc/sys/vm/drop_caches
echo 2 > /proc/sys/vm/drop_caches
echo 3 > /proc/sys/vm/drop_caches
The few 100 GB of ram cache was written to disk and lvremove worked again.