xfs: fix serious bugs in rmap key comparison

Last week I found some time to spend auditing the effectiveness of the
online repair prototype code, and discovered a serious bug in the rmap
code.  Each btree type provides four comparison predicates: one for
comparing a key against the current record; one for comparing two
arbitrary record keys, and one each for checking if a btree block's
records or keys are in the correct order.

Unfortunately, I encoded a major thinko into those last three functions.
The XFS_RMAP_OFF macro masks off the three namespace bits before we
perform a comparison, which means that key comparisons do not notice
differences between the unwritten, bmbt, or attr fork status.  On a
consistent filesystem this is not an issue because there can only ever
be overlapping rmap records for written inode data fork extents, which
is why we've not yet seen any problems in the field.

Fortunately, the last two functions are used by debugging asserts and
online scrub to check the contents of a btree block, so the severity of
the flaw there is not high.

Unfortunately, the flaw in _diff_two_keys is more severe, because it is
used for query_range requests.  Ranged queries are used by the regular
rmap handling code when reflink is enabled; and it is used in the rmap
btree validation routines of both xfs_scrub and xfs_repair.  As I
mentioned above, the flaw should not manifest on a *consistent*
filesystem, but for fuzzed (or corrupt) filesystems, this seriously
impacts our ability to detect problems.

The first two patches in this series fix two places where we pass the
wrong flags arguments to the rmap query functions (which didn't
previously cause lookup failures due to the broken code) and the third
patch fixes the comparison functions.