Security hardening for bcachefs:

PaX and grsecurity:
Enable as many hardening features as you can. If the kernel fails to
build, or if bcachefs causes problems at runtime, the kernel log will
provide information on what caused the issues. The PaX size overflow
plugin may require some additional attention, but that's explained below.
Using grsecurity/PaX for finding bugs is an incredibly underrated but very
effective technique. Any bug which is found in this way is a real bug, not
something specific to these security patches. The latest version of
grsecurity is always on https://grsecurity.net/download.php#test. Verify
the GPG signature for the patch!

PaX RAP:
RAP (Reuse Attack Protector) implements Control Flow Integrity, which
defeats all sorts of code reuse attacks, most notably ROP. RAP is special
because it is both forward and backward-edge CFI, and is fine-grained. The
The public version is hash-based and deterministic. It can detect bugs
because, in order to function, it requires all function pointer prototypes
and function pointer calls to have the exact same signature. Even if the
function pointer has an equivalent signature, it will not be compatible
with RAP unless it is exactly identical. For example, a the prototype
"void foo(struct bar)" may accept "void *baz = (struct bar)val" as the
source of an argument, but it would make RAP very unhappy, and would
trigger a kernel panic when the function is called with that (void *)
argument. Information is available at https://grsecurity.net/rap_faq.php.

PaX Size Overflow:
See https://forums.grsecurity.net/viewtopic.php?f=7&t=3043 for information
on its function. It requires a size overflow hash table which is generated
for code in the upstream kernel, and the program used to generate it isn't
public, so you'll need to understand how it works and instrument it. But
once you get it correctly working, you will be able to detect a massive
subset of size overflows. This is especially important for something like
a filesystem which is going to have a lot of these issues. Due to an
internal naming conflict, you'll have to disable signed integer overflow
instrumentation in UBSan.

UBSan:
This catches a subset of undefined behavior in code. It excludes a few
overly aggressive checks or checks which people often intentionally abuse
by default. Patch scripts/Makefile.ubsan to make it more aggressive by
replacing its entire contents with the following:

ifdef CONFIG_UBSAN
  CFLAGS_UBSAN += $(call cc-option, -fsanitize=undefined)
  CFLAGS_UBSAN += $(call cc-option, -fsanitize=float-divide-by-zero)
  CFLAGS_UBSAN += $(call cc-option, -fsanitize=float-cast-overflow
ifdef CONFIG_PAX_SIZE_OVERFLOW
  CFLAGS_UBSAN += $(call cc-option, -fno-sanitize=signed-integer-overflow)
endif
  CFLAGS_UBSAN += $(call cc-option, -Wno-maybe-uninitialized)
endif

Then set CONFIG_UBSAN=y and CONFIG_UBSAN_SANITIZE_ALL=n in the config, and
add UBSAN_SANITIZE := y to the top bcachefs Makefile. Information about
using UBSan is at https://kernel.org/doc/html/latest/dev-tools/ubsan.html.
It's extremely useful because it provides precise information about the
exact type of overflow, what caused it, as well as the location. A neat
feature of UBSan is that it can be used for security instead of just for
debugging by using -fsanitize-undefined-trap-on-error, which traps to the
ud2 CPU instruction, which, on x86 at least, triggers a fatal BUG().

If for whatever reason the modified Makefile.ubsan doesn't work (due to a
lack of traps for the extra sanitizers), revert it to its previous state,
and only add the CONFIG_PAX_SIZE_OVERFLOW ifdef back.

KASan:
The kernel version of ASan. I'm sure you've been debugging with this, but
just in case: https://kernel.org/doc/html/latest/dev-tools/kasan.html. It
is not designed for hardening (unlike UBSan, which can, when configured
properly), but it is great for detecting generic memory problems. Use GCC
5.0 or later to get the best results.

SLUB debugging:
A simple bug useful way to debug memory allocations. You should use SLUB
as your slab allocator, since it is not only the most secure, but supports
extensive debugging. Boot with slub_debug=FZP to enable sanity checks (F),
redzoning (Z), and poisoning (P). More documentation is available at
https://www.kernel.org/doc/Documentation/vm/slub.txt.

Dynamic code analysis:
It's very important to fuzz all bcachefs-related syscalls. This includes
IOCTLs, as well as I/O syscalls like fallocate(), truncate(), open(), etc.
The best syscall fuzzer currently is syzkaller, an instrumented fuzzer. It
provides the best results when the above steps are used to make the kernel
hyper-sensitive to bugs. An even more effective method is to plug the
filesystem into AFL. This requires the KCOV framework. Oracle has written
a fuzzer, and has finally released the source code on Github, along with
example usage: https://github.com/oracle/kernel-fuzzing/.

Fuzzing may be especially effective when used with KTSan, an experimental
data race detector made by Google: https://github.com/google/ktsan.

If you provide a virtual machine image with a fuzzer pre-configured, or
one requiring minimal configuration, I can deploy it on about 100 logical
cores (48 being on a high-end Xeon system), likely for as long as bugs are
being found and are being attended to.

Static code analysis:
Checking code for mistakes with static code analysis tools helps find bugs
which may not be discovered by fuzzers. It's also faster, and provides
pointers to where code needs to be improved, even if it doesn't currently
trigger a bug, because it may lead to one in the future. A big list is at
https://www.dwheeler.com/essays/static-analysis-tools.html. There's also
scripts in the kernel directory for coconelle, and, of course, GCC itself,
when providing its most aggressive warnings.