Got this error. lsof gave nothing. No user was using it, was not in mounts.

Solution was:

echo 1 > /proc/sys/vm/drop_caches
echo 2 > /proc/sys/vm/drop_caches
echo 3 > /proc/sys/vm/drop_caches

The few 100 GB of ram cache was written to disk and lvremove worked again.