summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorHunter Shaffer <huntershaffer182456@gmail.com>2023-09-27 21:28:29 -0400
committerHunter Shaffer <huntershaffer182456@gmail.com>2023-09-27 21:58:23 -0400
commit321e0f3ab746decd9d4fb9ad0d82aa2b83d1ef9e (patch)
treebfd0dc3208971c2d2f138f6d63bfa32cd4c980c3
parent2a71110142c7a91b7c8de1671158bc231aa6d995 (diff)
Added a blog section w/ writeup on members_v2
Signed-off-by: Hunter Shaffer <huntershaffer182456@gmail.com>
-rw-r--r--News-full.mdwn1
-rw-r--r--index.mdwn5
-rw-r--r--news/members-v2.mdwn58
3 files changed, 64 insertions, 0 deletions
diff --git a/News-full.mdwn b/News-full.mdwn
new file mode 100644
index 0000000..161f40b
--- /dev/null
+++ b/News-full.mdwn
@@ -0,0 +1 @@
+[[!inline pages="news/*" archive="yes"]]
diff --git a/index.mdwn b/index.mdwn
index fba1af7..025feb5 100644
--- a/index.mdwn
+++ b/index.mdwn
@@ -133,3 +133,8 @@ linux-bcachefs@vger.kernel.org.
Bug trackers: [[bcachefs|https://github.com/koverstreet/bcachefs/issues]],
[[bcachefs-tools|https://github.com/koverstreet/bcachefs-tools/issues]]
+
+## News
+
+Blog: [[!inline pages="news/*"]]
+Archive: [[Archive|News-full.mdwn]]
diff --git a/news/members-v2.mdwn b/news/members-v2.mdwn
new file mode 100644
index 0000000..c87c9ce
--- /dev/null
+++ b/news/members-v2.mdwn
@@ -0,0 +1,58 @@
+# Members v2, configurationless tiering
+A feature request we've had is configurationless tiering, smart tiering of
+member devices in a filesystem based on performance. This feature will allow
+easy and simple tiering of devices within a filesystem based on the
+performance of the device. The effect of this is that it will allow data that
+is commonly accessed, hot data, stored on the faster drives while data that is
+not used as often, cold data, will be stored on slower drives. This will
+increase filesystem performance greatly.
+
+## Background: extensible metadata
+Extensible metadata means having to ability to add new fields to metadata
+while still being compatible with older versions of the filesystem. To achieve
+this structs cannot be of a fixed size and need to be able to add new fields
+using a bounds check or filling in wth zeroes when reading data from a
+previous version.
+
+## Non extensible members
+Early in the process of implementing the tiering feature we ran into an
+interesting problem. Members within the superblock were not large enough to
+properly store this data. While we could have just resized the member, this
+would have caused further issues regarding compatability. Instead we opted to
+implement resizable members, members v2 if you will. The effect of adding
+resizable members allows us to add new fields to the members while still
+ensuring backwards compatability.
+
+## Members v2
+The superblock of a filesystem is the start of that filesystem and requires
+extensible fields which contain important data such as a list of member
+devices. A suberblock needs extensible fields in cases such as a new device
+being added to the filesystem, in which case the members field needs to be
+extended.
+
+In the case of members v1, the members array itself was extensible, the
+members themselves were a fixed size. Due to their fixed size it was quite easy
+to index and retrieve members from the list. However, when members can be
+dynamically resizable it is not that easy. The location of each member can not
+be known before runtime and therefore has to be found and accessed manually within the
+array of members. This was at times a complicated process for me to implement
+but will make future expansions of the members much simpler.
+
+## Configurationless tiering
+Configurationless tiering is a feature that has been commonly requested.
+Instead the user specifying foreground and background targets, foreground
+allocations will go to the fastest device(s) and cold data will be moved to the
+slower device(s) in the background. To implement this the filesystem will
+require some idea of device performance which needs to be stored in the
+superblock.
+
+## Storing device performance
+The devices within the filesytem will now store IOPS measurements for randread,
+randwrite, seq-read, and seq-write. In the future the new IOPS field can also be
+useful in other features such as monitoring device health.
+
+## Addendum: Cap'n proto
+Some of the ideas in bcachefs about how to handle metadata were inspired by
+[[Cap'n Proto|https://capnproto.org/]], which is highly recommended reading -
+it's a library that does everything we have to do by hand in C, exactly the
+way we want it.