Category Archives: linux

KeyError: ‘/dev/sda’

At Etsy, we have a nice, clean, streamlined build process. We have a command for setting up RAID, and another for OS installation. OS installation comes with automagic for LDAP, Chef roles, etc.

We came across an odd scenario today when a co-worker was building a box that gave the following error:

Traceback (most recent call first):
File “/usr/lib/anaconda/storage/partitioning.py”, line 1066, in allocatePartitions
disklabel = disklabels[_disk.path]
File “/usr/lib/anaconda/storage/partitioning.py”, line 977, in doPartitioning
allocatePartitions(storage, disks, partitions, free)
File “/usr/lib/anaconda/storage/partitioning.py”, line 274, in doAutoPartition
exclusiveDisks=exclusiveDisks)
File “/usr/lib/anaconda/dispatch.py”, line 210, in moveStep
rc = stepFunc(self.anaconda)
File “/usr/lib/anaconda/dispatch.py”, line 126, in gotoNext
self.moveStep()
File “/usr/lib/anaconda/dispatch.py”, line 233, in currentStep
self.gotoNext()
File “/usr/lib/anaconda/text.py”, line 602, in run
(step, instance) = anaconda.dispatch.currentStep()
File “/usr/bin/anaconda”, line 1131, in <module>
anaconda.intf.run(anaconda)
KeyError: ‘/dev/sda’
It suggests a problem with setting up partitions on /dev/sda, where we would put the boot partition. I knew it seemed familiar but I couldn’t recall the solution, and Google, while usually wonderful, got us to a Red Hat Support article behind a paywall.  A few other results suggested the boot order was incorrect. The OS was thinking the drives were out of order. Being a Dell box, I checked the virtual drive order, which in my experience always has matched the boot order:
Screen Shot 2015-12-29 at 3.02.57 PM.png
After the anaconda failure, I went into another terminal to a prompt and checked /proc/partitions. Sure enough, we started at sdb, not sda. Then it hit me. There were 4 people viewing the console in iDRAC, so what if someone else had mounted a virtual disk and that was /dev/sda? Sure enough:
Screen Shot 2015 12 29 at 3.23.07 PM.png
Deleting the virtual media session, rebooting and starting the OS install again proved out everything worked fine.
The bonus humor here is that this isn’t the first time we’ve run into this. Hopefully after posting this, Google will index this page and point us to the answer a bit quicker next time.
Advertisements

XFS and EXT4 Testing Concluded

I had a few more suggestions thrown out at me before I could wrap this one up.

  • Try disabling the RAID controller read-ahead
  • Try a few custom options to XFS
  • Try RAID-10

First, my final “best” state benchmarks for comparison:

FS  Raid Size Mount Options Transfer/s Requests/s Avg/Request 95%/Request
xfs 6 4T noatime,nodiratime,nobarrier 28.597Mb/sec 1830.24 0.51ms 2.06ms
ext4 6 4T noatime,nodiratime,nobarrier 32.583Mb/sec 2085.33 0.46ms 1.89ms

Disabling the read-ahead was an interesting thought.

FS RAID Size Mount Options Transfer/s Requests/s Avg/Request 95%/Request
xfs 6 4T noatime,nodiratime,nobarrier 28.704Mb/sec 1837.07 0.50ms 2.04ms
ext4 6 4T noatime,nodiratime,nobarrier 32.715Mb/sec 2093.75 0.46ms 1.88ms

It didn’t seem to make any real difference however.


The second suggestion was to use modified XFS options (mkfs.xfs -f -d sunit=128,swidth=$((512*8)),agcount=32 /dev/sdb2).

FS RAID Size Mount Options Transfer/s Requests/s Avg/Request 95%/Request
xfs 6 4T noatime,nodiratime,nobarrier 26.376Mb/sec 1688.07 0.55ms 2.18ms

It’s hard to tell, but it seems these actually degraded performance.


The last test was to switch to RAID-10. This would reduce overall storage capacity to 72TB, but given our requirements, this really shouldn’t cause any problem for the project. RAID-10 should have a significant boost to write performance.

FS RAID Size Mount Options Transfer/s Requests/s Avg/Request 95%/Request
xfs 10 36T noatime,nodiratime,nobarrier 32.808Mb/sec 2099.72 0.46ms 1.80ms
ext4 10 36T noatime,nodiratime,nobarrier 54.112Mb/sec 3463.17 0.28ms 1.11ms

These numbers back up the improvement to write speed, but XFS still lags behind at larger volume sizes.

Since I am had to reconfigure the array, I wanted to try the larger volume size (36T) above and then a smaller size (2T) to try to reproduce my earlier results showing XFS to perform better at lower volume size.

FS RAID Size Mount Options Transfer/s Requests/s Avg/Request 95%/Request
xfs 10 2.2T noatime,nodiratime,nobarrier 60.066Mb/sec 3844.2 0.25ms 1.00ms
ext4 10 2.2T noatime,nodiratime,nobarrier 64.766Mb/sec 4145.01 0.23ms 0.90ms

This was by far the best test results I had seen and has doubled the results from the original async test.


Testing conclusions

  • XFS seems to be very sensitive to partition size
  • In all but one case, EXT4 performed better on the random read-write tests
  • Know your other caveats of both file systems before picking the one for you

More EXT4 vs XFS IO Testing

Following my previous post, I got some excellent feedback in the forms of comments, tweets and other chat. In no particular order:

  • Commenter Tibi noted that ensuring I’m mounting with noatime, nodiratime and nobarrier should all improve performance.
  • Commenter benbradley pointed out a missing flag on some of my sysbench tests which will necessitate re-testing.
  • Former co-worker @preston4tw suggests looking at different IO schedulers. For all tests past, I used deadline which seems to be best, but re-testing with noop could be useful.
  • Fellow DBA @kormoc encouraged me to try many smaller partitions to limit the number of concurrent fsyncs.

There seem to be plenty of options here that should allow me to re-try my testing with a slightly more consistent method. The consistent difference seems to be in the file system, EXT4 vs XFS, with XFS performing at about half the speed of EXT4.

Continue reading

Learning to Deal With Learning

Note: This post originally appeared as a post on my former employers site (inside.godaddy.com), and has since been removed. Reposting here to share the information.

We here at GoDaddy deploy our MySQL database servers with RAID 10 for performance and reliability. Supporting that, we utilize hardware RAID option with Dell branded PERC cards. These cards offer a write back cache to boost write performance. Writes are stored in memory on the RAID controller and then flushed to disk in order to improve performance. This provides a noticeable improvement in writes because from the OS perspective, a write is complete when it hits the cache, not the actual disk. Since data in the cache is volatile, that is, susceptible to power loss, there is also a battery that allows the cache to be preserved in the event of a power loss. This eliminates the possibility of data loss while preserving the speed benefits of a write cache.

Continue reading

Running ElasticSearch, LogStash and Kibana in Docker

As any server farm scales out, it becomes increasingly difficult to Watch All The Things™. I’ve been watching the progress of LogStash+ElasticSearch+Kibana (also known as an ELK stack) for a while and gave it a go this weekend. The trick for me was wanting to run each element inside of a separate Docker container so that I have easily portable elements to scale out with.

A step back. What is Docker? Docker is a container (using LXC) around an application. In short, you install Docker, start a container using a base image (CentOS, Ubuntu, etc.) and then run the container, dropping you into a shell. From here, you configure your application, then save your container. You can stop and start it at any time, relocate it to another server, or generally break it as badly as you want and you’ve done absolutely nothing to your host machine.

ElasticSearch is a data store and search tool for data. It will serve as the place for our logs. LogStash is a log parser. It understands what the source format is and has many output formats (including ElasticSearch). Kibana is a data visualization tool for searching your data store and drawing graphs to help see what’s going on.

Continue reading