diff options
author | Hunter Shaffer <huntershaffer182456@gmail.com> | 2023-09-27 21:28:29 -0400 |
---|---|---|
committer | Hunter Shaffer <huntershaffer182456@gmail.com> | 2023-09-27 21:58:23 -0400 |
commit | 321e0f3ab746decd9d4fb9ad0d82aa2b83d1ef9e (patch) | |
tree | bfd0dc3208971c2d2f138f6d63bfa32cd4c980c3 | |
parent | 2a71110142c7a91b7c8de1671158bc231aa6d995 (diff) |
Added a blog section w/ writeup on members_v2
Signed-off-by: Hunter Shaffer <huntershaffer182456@gmail.com>
-rw-r--r-- | News-full.mdwn | 1 | ||||
-rw-r--r-- | index.mdwn | 5 | ||||
-rw-r--r-- | news/members-v2.mdwn | 58 |
3 files changed, 64 insertions, 0 deletions
diff --git a/News-full.mdwn b/News-full.mdwn new file mode 100644 index 0000000..161f40b --- /dev/null +++ b/News-full.mdwn @@ -0,0 +1 @@ +[[!inline pages="news/*" archive="yes"]] @@ -133,3 +133,8 @@ linux-bcachefs@vger.kernel.org. Bug trackers: [[bcachefs|https://github.com/koverstreet/bcachefs/issues]], [[bcachefs-tools|https://github.com/koverstreet/bcachefs-tools/issues]] + +## News + +Blog: [[!inline pages="news/*"]] +Archive: [[Archive|News-full.mdwn]] diff --git a/news/members-v2.mdwn b/news/members-v2.mdwn new file mode 100644 index 0000000..c87c9ce --- /dev/null +++ b/news/members-v2.mdwn @@ -0,0 +1,58 @@ +# Members v2, configurationless tiering +A feature request we've had is configurationless tiering, smart tiering of +member devices in a filesystem based on performance. This feature will allow +easy and simple tiering of devices within a filesystem based on the +performance of the device. The effect of this is that it will allow data that +is commonly accessed, hot data, stored on the faster drives while data that is +not used as often, cold data, will be stored on slower drives. This will +increase filesystem performance greatly. + +## Background: extensible metadata +Extensible metadata means having to ability to add new fields to metadata +while still being compatible with older versions of the filesystem. To achieve +this structs cannot be of a fixed size and need to be able to add new fields +using a bounds check or filling in wth zeroes when reading data from a +previous version. + +## Non extensible members +Early in the process of implementing the tiering feature we ran into an +interesting problem. Members within the superblock were not large enough to +properly store this data. While we could have just resized the member, this +would have caused further issues regarding compatability. Instead we opted to +implement resizable members, members v2 if you will. The effect of adding +resizable members allows us to add new fields to the members while still +ensuring backwards compatability. + +## Members v2 +The superblock of a filesystem is the start of that filesystem and requires +extensible fields which contain important data such as a list of member +devices. A suberblock needs extensible fields in cases such as a new device +being added to the filesystem, in which case the members field needs to be +extended. + +In the case of members v1, the members array itself was extensible, the +members themselves were a fixed size. Due to their fixed size it was quite easy +to index and retrieve members from the list. However, when members can be +dynamically resizable it is not that easy. The location of each member can not +be known before runtime and therefore has to be found and accessed manually within the +array of members. This was at times a complicated process for me to implement +but will make future expansions of the members much simpler. + +## Configurationless tiering +Configurationless tiering is a feature that has been commonly requested. +Instead the user specifying foreground and background targets, foreground +allocations will go to the fastest device(s) and cold data will be moved to the +slower device(s) in the background. To implement this the filesystem will +require some idea of device performance which needs to be stored in the +superblock. + +## Storing device performance +The devices within the filesytem will now store IOPS measurements for randread, +randwrite, seq-read, and seq-write. In the future the new IOPS field can also be +useful in other features such as monitoring device health. + +## Addendum: Cap'n proto +Some of the ideas in bcachefs about how to handle metadata were inspired by +[[Cap'n Proto|https://capnproto.org/]], which is highly recommended reading - +it's a library that does everything we have to do by hand in C, exactly the +way we want it. |