Wrote a device‑mapper “XOR filter” and ext2 got mad at me. Where did I go wrong?

Decrypt after read, encrypt before write. And don’t poison the upper cache.

Essence:
The main mistake is mutating data in map.
For reads, the data aren’t there yet — the pages will be filled later by the lower layer. Do decryption in the clone’s end_io.

For writes, you mustn’t XOR pages owned by the upper FS page cache. Use a bounce bio with new pages: copy → XOR → submit downward.

Pipeline step by step:

1. READ:
Clone the bio (thin clone), set bi_end_io, submit down. In end_io walk the segments, apply XOR, complete the original via bio_endio.

2. WRITE:
Create a bounce bio with its own pages, copy data from the original bio, XOR in the bounce pages, submit the bounce; in its end_io complete the original bio.

3. Routing:
In map do bio_set_dev(...) and adjust the sector.
On reads — submit_bio(clone) and return DM_MAPIO_SUBMITTED.
On writes — also SUBMITTED (since you submit a bounce).

4. API nuances:
For newer kernels look at bio_op(bio)/REQ_OP_WRITE or op_is_write(bio->bi_opf); for older ones — bio_data_dir(bio). Use GFP_NOIO allocations; a bio_set/mempool will help.

Mini‑skeleton (abridged, pseudo‑C):

		

			
				
			
		
		struct xor_c {
    struct block_device *bdev;
    sector_t start;
    u8 key;
    struct bio_set *bs; // pool for bios/pages
};

static void xor_range(u8 *p, unsigned int len, u8 key) {
    for (unsigned int i = 0; i bi_private;
    if (clone->bi_status) {
        orig->bi_status = clone->bi_status;
        bio_endio(orig);
        bio_put(clone);
        return;
    }
    struct bio_vec bvec;
    struct bvec_iter iter;
    bio_for_each_segment(bvec, clone, iter) {
        void *kaddr = kmap_atomic(bvec.bv_page);
        xor_range((u8*)kaddr + bvec.bv_offset, bvec.bv_len, 0x0F);
        kunmap_atomic(kaddr);
    }
    bio_endio(orig);
    bio_put(clone);
}
/* WRITE end_io */
static void xor_write_endio(struct bio *bounce)
{
    struct bio *orig = bounce->bi_private;
    if (bounce->bi_status) orig->bi_status = bounce->bi_status;
    bio_endio(orig);
    bio_put(bounce);
}
static int xor_map(struct dm_target *ti, struct bio *bio)
{
    struct xor_c *c = ti->private;
    bio_set_dev(bio, c->bdev);
    bio->bi_iter.bi_sector += c->start;
    if (bio_data_dir(bio) == READ) {
        struct bio *clone = bio_clone_fast(bio, GFP_NOIO, c->bs);
        clone->bi_private = bio;
        clone->bi_end_io  = xor_read_endio;
        submit_bio(clone);
        return DM_MAPIO_SUBMITTED;
    } else { // WRITE
        struct bio *bounce = bio_alloc_bioset(GFP_NOIO, bio_segments(bio), c->bs);
        bounce->bi_private = bio;
        bounce->bi_end_io  = xor_write_endio;
        bio_set_dev(bounce, c->bdev);
        bounce->bi_iter.bi_sector = bio->bi_iter.bi_sector;
        struct bio_vec src;
        struct bvec_iter siter;
        bio_for_each_segment(src, bio, siter) {
            struct page *pg = alloc_page(GFP_NOIO);
            bio_add_page(bounce, pg, src.bv_len, 0);
            void *dst  = kmap_atomic(pg);
            void *srcp = kmap_atomic(src.bv_page);
            memcpy(dst, (u8*)srcp + src.bv_offset, src.bv_len);
            xor_range(dst, src.bv_len, 0x0F);
            kunmap_atomic(srcp);
            kunmap_atomic(dst);
        }
        submit_bio(bounce);
        return DM_MAPIO_SUBMITTED;
    }
}
	

Why do you have “zeroed” pages?
Because they haven’t been filled yet; by changing them prematurely you break the FS’s expectations and poison the cache. For writes it’s even worse: XOR’ing the FS’s “native” pages leaves encrypted bytes in the cache, and the FS then “reads itself in cipher.”

Bottom line: copy the dm-crypt logic: before the disk — encrypt, after the disk — decrypt. In your case it’s a teaching XOR instead of AES.

How about not reinventing the wheel? dm‑crypt, cryptsetup and friends

Gist:
Everything you’re wiring by hand (BIO clones, end_io, bounce pages) is already battle‑tested in dm-crypt. Yes, XOR isn’t a “cipher” there (and that’s a good thing), but the architecture is the same: encrypt before write, decrypt after read, don’t poison the upper cache.

If the goal is educational, the dm-crypt code is a great reference. If you want practicality, cryptsetup solves the task without kernel hacking. For experiments you can also try FUSE at the file level (easier to debug, but that’s no longer a block filter).

Quick proof‑of‑concept: don’t encrypt metadata (for debugging only!)

Idea:
To separate “broken pipeline” from “wrong layout,” temporarily skip XOR for the first 1024 bytes of the device (that’s where ext2/3/4 keep the superblock). The FS will stop yelling bad superblock — which means the root cause really is the transformation timing, as in A1.

Cons:

• It’s a band‑aid: part of the data remain unencrypted.
• There are backup superblocks and other metadata — you’d need to exclude those too.
• Once you build the correct READ/WRITE pipeline (see A1), throw this hack away.

For a quick test you can leave the first 1024 bytes alone — it’ll make it clearer whether the problem is the pipeline or the layout.

ext2’s superblock is after the first kilobyte. If you encrypt everything blindly, the FS reads something it doesn’t expect.

XOR in map on a read is too early. There’s nothing to XOR yet