xfs: fixes for online shrink

This is a single fix to the per-AG reservation code to make it so that
shrink can abort successfully if shrinking would leave too little space
in the AG to handle metadata btree expansion.