12 years later... It doesn't seem likely that I'll revisit dm-cache or bcache any time soon but I do have some notes and thoughts on setting up some infrastructure to replace some services I was paying for and I might make time to comment on that in the future!

SSD Caching 2: One step forward...

Overview

After several days of using the my dm-cache setup (see: Adding an SSD cache to an existing encrypted filesystem with dm-cache) to cache my encrypted /home filesystem I have noticed that very little was being cached. Based on my interpretations of the block statistics for the 3 block devices involved the cache was saving me a huge number of writes to disk. Still, I hadn’t noticed a significant perceptable performance improvement. Caching just /home may not be enough to change my perception of my workstation as being held back by a slow drive. However, the real problem is that I have no baseline to compare with. It’s time to remove the cache and set up some stats collection.

Block device review

/dev/mapper/benwayvg-ehome         luks block device on an LVM partition
/dev/mapper/cehome                 cache device from dmsetup
/dev/mapper/ehome                  opened luks device (an ext4 filesystem)

Save the Stats

The cumulative block stats for the three devices are still illustrative. This is after 4 days of uptime: cat /sys/block/dm-{17,18,19}

device                       rd i/o  rmerges rsectors   rticks   wr i/o  wmerges wsectors   wticks inflight io_ticks  time_in_queue
/dev/mapper/benwayvg-ehome   120223        0  4702124   651290   127889        0 20907992  3195852        0   644128  3847142
/dev/mapper/cehome           133223        0  4518728   595193   578275        0 32206792  5623591        0   896061  6218948
/dev/mapper/ehome            133137        0  4517804   604400   550712        0 32206792 12038764        0   911568 12643178

I think some general observations can be made here. I didn’t notice much of a performance change after adding the cache device but it’s clear it saved a big percentage of writes to disk. The writes to the luks device were 550712, but luks has some overhead it looks like and the writes to the cache device were 578275. However the cache device only wrote to disk 127889 times!

There wasn’t much savings of disk read i/o however, but perhaps some tuning would have changed that. I plan to revisit it all after establish some baseline stats over the next week.

The output of dmsetup status cehome also revealed only 1471 blocks in the cache. Blocks in this case are 256k each, which is only 350 MiB or so. I guess my expectation was more would be cached. The documentation for dm-cache (device-mapper/cache.txt in your kernel documentation directory) explains all fields in the output from dmsetup status and for illustration I’ll show the full output from dmsetup status:

0 209715200 cache 376/524288 19279 877645 583982 265753 0 662 1471 1438 0 2 migration_threshold 204800 4 random_threshold 4 sequential_threshold 2048

Perhaps dm-cache is simply doing it’s job and not caching blocks that would be replaced too quickly. Maybe it’s simply that my /home partition is a good candidate for caching, but simply doesn’t require very much cache to satisfy it’s needs.

Prepare for removal

First I have to umount /home and to do that I had to kill all user processes and any other processes that might have files open on /home. Be sure to save your work and log in as root if you’re attempting something similar.

pkill -u coxa ; umount /home

Now we should close the encrypted Luks device: cryptsetup luksClose ehome

Removing the Cache

Basically I just need to remove the cache, and to do that we need to use the cleaner policy which forces everything back to disk.

[root@benway ~]# dmsetup table cehome
0 209715200 cache 8:17 8:18 253:17 512 1 writeback default 0
[root@benway ~]# dmsetup status cehome
0 209715200 cache 376/524288 19279 877645 583982 265753 0 662 1471 1438 0 2 migration_threshold 204800 4 random_threshold 4 sequential_threshold 2048
[root@benway ~]# dmsetup suspend cehome
[root@benway ~]# dmsetup reload cehome --table '0 209715200 cache 8:17 8:18 253:17 512 1 writeback cleaner 0'
[root@benway ~]# dmsetup resume cehome
[root@benway ~]# dmsetup status cehome
0 209715200 cache 376/524288 19279 877657 583982 265753 0 0 1471 1009 0 2 migration_threshold 204800 0
[root@benway ~]# dmsetup status cehome
0 209715200 cache 376/524288 19279 877657 583982 265753 0 0 1471 530 0 2 migration_threshold 204800 0
[root@benway ~]# dmsetup status cehome
0 209715200 cache 376/524288 19279 877657 583982 265753 0 0 1471 361 0 2 migration_threshold 204800 0
[root@benway ~]# dmsetup status cehome
0 209715200 cache 376/524288 19279 877657 583982 265753 0 0 1471 0 0 2 migration_threshold 204800 0
[root@benway ~]# dmsetup status cehome
0 209715200 cache 376/524288 19279 877657 583982 265753 0 0 1471 0 0 2 migration_threshold 204800 0
[root@benway ~]# dmsetup wait cehome
[root@benway ~]# dmsetup remove cehome

You can see the column showing dirty blocks dropping down to zero after resuming the device with the cleaner policy in place.

Tidying Up

I needed to modify /etc/crypttab to make sure it references the logical volume that we are no longer caching, instead of the cached block device we removed above.

#ehome  /dev/mapper/cehome
ehome   /dev/benwayvg/ehome

I was using a systemd service to create the dm-cache device, we need to make sure that is not started: rm -f /usr/lib/systemd/system/local-fs.target.wants/dmsetup-dm-cache.service /etc/systemd/system/dmsetup-dm-cache.service

Next steps

The next step is to start gathering some detailed statistics about the disk io on my system. For that I am going to try graphite/carbon or collectd. My next post will include some setup notes from that process.

Adding an SSD cache to an existing encrypted filesystem with dm-cache

Overview

I wanted to try SSD caching in Fedora and dm-cache seemed like the logical choice to use on existing data. I had an existing home filesystem encrypted using cryptsetup/LUKS and I wanted to cache the encrypted blocks so as not to leave unencrypted data on the SSD. This meant that I had to insert the dm-cache device between the logical volume and the encrypted filesystem.

My 32GB OCX Vertex was pressed into service as the cache. I primarily followed the excellent write-up here: SSD Caching Using dm-cache Tutorial with changes neccesary for my environment.

Nuts & Bolts

My home filesystem is on is /dev/benwayvg/ehome or /dev/mapper/benwayvg-ehome. /etc/crypttab contained:

ehome   /dev/mapper/benwayvg-ehome

I found it easier to comment out or delete /etc/crypttab, comment out the entry for /home in /etc/fstab and simply reboot to set this up, but you could do it all manually by killing user processes, umounting /home, etc. After logging in as root I partitioned my SSD. I created a 2GB partition (too large I think) for metadata and the rest of the the SSD for cached blocks. I determined the size of the LV containing my home partition using: blockdev --getsz /dev/benwayvg/ehome which gave me a result of 209715200.

The real work is done by dmsetup:

/sbin/dmsetup create cehome --table '0 209715200 cache /dev/disk/by-id/ata-OCZ-VERTEX_FPRTEF1A3C31XA69I1XB-part1 /dev/disk/by-id/ata-OCZ-VERTEX_FPRTEF1A3C31XA69I1XB-part2 /dev/mapper/benwayvg-ehome 512 1 writeback default 0'

The magic above is explained well in the article I linked and I am basically copying it verbatim substituting for my partitions and sizes. I chose to use real partitions and not use the linear target.

The next step is to use cryptsetup to create the device we’ll actually mount: cryptsetup luksOpen /dev/mapper/cehome ehome

This creates /dev/mapper/ehome which can be mounted as a filesystem. All my data was preserved I am now simply accessing it through the cache.

Automatically run dmsetup at boot

I wanted this to all happen automatically. To achieve that I needed to get Fedora to create the cache device, and do so before it runs cryptsetup. To that end I created /etc/systemd/system/dmsetup-dm-cache.service with these contents:

[Unit]
Description=Initialize dm-cache device
DefaultDependencies=no
Conflicts=shutdown.target
After=fedora-storage-init.service
Before=cryptsetup.service

[Service]
ExecStart=/sbin/dmsetup create cehome --table '0 209715200 cache /dev/disk/by-id/ata-OCZ-VERTEX_FPRTEF1A3C31XA69I1XB-part1 /dev/disk/by-id/ata-OCZ-VERTEX_FPRTEF1A3C31XA69I1XB-part2 /dev/mapper/benwayvg-ehome 512 1 writeback default 0'
Type=oneshot
TimeoutSec=0
RemainAfterExit=yes

To be honest, I am not sure how many of the options above are needed, but it works! I also needed to ensure that the “service” gets started by making a link in /usr/lib/systemd/system/local-fs.target.wants to our service above:

ln -s /etc/systemd/system/dmsetup-dm-cache.service /usr/lib/systemd/system/local-fs.target.wants/

The last step to ensure this all works at boot is to update /etc/crypttab:

ehome /dev/mapper/cehome

Device summary

To summarize, all the devices in my setup:

/dev/mapper/benwayvg-ehome         luks block device
/dev/mapper/cehome                 cache device from dmsetup
/dev/mapper/ehome                  opened luks device

More to do…

Now I am looking at tuning and performance. So far, very little is actually getting cached, but I’ll comment on that some other time.

A little snag on my way to Fedora 19

fedup refused to upgrade my Fedora 18 system to Fedora 19. The output from fedup was:

[coxa@thinky ~]$ sudo fedup-cli --network 19
setting up repos...
getting boot images...
.treeinfo                                                                                                                      | 1.1 kB  00:00:00     
setting up update...
Your system is already upgraded!
Finished. Nothing to do.

I had to run a clean operation first:

[coxa@thinky ~]$ sudo fedup-cli --clean
resetting bootloader config
removing boot images
removing downloaded packages
removing miscellaneous files

After that, the download process started!