You spin my head right round, right round
When you go down, when you go down, down
The last few days I built a new NAS (Network Attached Storage) to store my backups and ever-increasing image collection. To reduce the power footprint, I want to spin down the HDDs¹ whenever I don’t use the storage. And, like for cars, this is the case most of the time. To prevent any comments on how this is maybe not so healthy for the HDDs, they are rated for 600.000 load cycles; this is more than enough for the lifetime I expect to see from them.
But let’s come back to the spindown. The first problem after the initial RAID 1 sync was, that the HDDs were still very noticeable because there was an access and therefore a clicking sound every few seconds. Absolutely annoying! Especially because I would like to have this NAS in my living room, which also happens to be my workplace. So I started to debug the issue and tried to disable everything that would access the disk and also created an inotify watcher on the mount path. Nothing helped, and there was no access visible. But on the other side, the clicking stopped the moment I unmounted the disks. Oh, and there was no spinup if I sent the disk to suspend manually.
Strange, but after some more debugging and trying to understand why the hell spindown doesn’t work, I found a hint. I formatted the 16 TB raid device with an ext4 file system. What I didn’t know until then and learned this day was that, in order to speed up the formatting, ext4 is doing something called lazy-init.
This means that the formatting would skip some tasks and the kernel will start to fully initialize the journal inodes and inode table after the first mount. Great! And because 16 TB are big, this took at least 2 days until I couldn’t bear it anymore. Luckily, I had not that much data copied to the new file system until then and could copy it to the system SSD and reformat the disk without
lazy-init. This is possible with the command mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/sda. Surprisingly, this took only a few minutes instead of a few seconds, and after it finished, silence! Beautiful!
But this was not everything I wanted to archive, because even then, the HDDs wouldn’t spin down. So I started again and checked the BIOS, the SATA controller card, the backplane of my case and the stupid cables. Nothing helped, and I couldn’t find anything. Most of the internet was full of people who wanted to disable spindown but the few that wanted to configure it had not that big of a problem doing so. After hours of debugging, I was checking the disks config with the tool openSeaChest provided by Seagate, but still, nothing helped but sending it to a manual spindown.
And then there was a sound, the sound of a HDD spinning down and sending the head to the parking position. The power stats confirmed it; one of the disks was in idle_c state and a few minutes late in standby_z mode. Great, but I had no idea what changed. I checked everything three times, but I couldn’t find any difference between the disk that would spin down and the disk that wouldn’t. Another few hours later, I found that the disk that wasn’t spinning down was the disk that had written 16,4 TB compared to 0,4 TB on the disk that worked. It was the disk that got written the whole other disk on it during the initial RAID 1 sync. At nearly the same time, I found this GitHub issue and this wiki page of the openSeaChest project. This explained that there are some, I’ve found no way to list them, background tasks that the controller of the HDDs performs at some unknown points in time. At this point, I assumed that maybe such a task was the cause of my disk not spinning down. So I decided to wait. And lucky me, after less thant another 24 hours, I came to my living room lying there in silence. The metrics where good and power usage was good too.
I learned a lot on my way to the result that would have come at it’s on, but I hope, if you experience some similar issues, you will stumble over this note and didn’t need to debug all this stuff. Just wait a few days.
¹ Yes they are still the way to go if you want a cheap TB/€ price and a lot of storage.