Note: This post originally appeared as a post on my former employers site (inside.godaddy.com), and has since been removed. Reposting here to share the information.
We here at GoDaddy deploy our MySQL database servers with RAID 10 for performance and reliability. Supporting that, we utilize hardware RAID option with Dell branded PERC cards. These cards offer a write back cache to boost write performance. Writes are stored in memory on the RAID controller and then flushed to disk in order to improve performance. This provides a noticeable improvement in writes because from the OS perspective, a write is complete when it hits the cache, not the actual disk. Since data in the cache is volatile, that is, susceptible to power loss, there is also a battery that allows the cache to be preserved in the event of a power loss. This eliminates the possibility of data loss while preserving the speed benefits of a write cache.
A real fast list of stuff from the Percona Live 2014 event.
Yahoo’s Performance Analyzer
Yahoo is developing a MySQL performance analyzer that should be released as open source later this year. From the demo, it looks like it pulls in most of the MySQL metrics, shows you a processlist and then lets you drill into a processlist with explain. Will have to keep my eye out for this.
ChatOps with Hubot
GitHub’s Sam Lambert has a set of hubot chatops scripts for MySQL. I was already looking at depoying hubot for the ability to push messages from a remote source into an IRC channel, so this would be a natural fit. He also mentioned using Janky to tie CI with hubot.
As any server farm scales out, it becomes increasingly difficult to Watch All The Things™. I’ve been watching the progress of LogStash+ElasticSearch+Kibana (also known as an ELK stack) for a while and gave it a go this weekend. The trick for me was wanting to run each element inside of a separate Docker container so that I have easily portable elements to scale out with.
A step back. What is Docker? Docker is a container (using LXC) around an application. In short, you install Docker, start a container using a base image (CentOS, Ubuntu, etc.) and then run the container, dropping you into a shell. From here, you configure your application, then save your container. You can stop and start it at any time, relocate it to another server, or generally break it as badly as you want and you’ve done absolutely nothing to your host machine.
ElasticSearch is a data store and search tool for data. It will serve as the place for our logs. LogStash is a log parser. It understands what the source format is and has many output formats (including ElasticSearch). Kibana is a data visualization tool for searching your data store and drawing graphs to help see what’s going on.
Over the past couple of years, several tools have come up to solve some of the long standing problems for developers, DBAs and operations teams. This list is no where near exhaustive and your mileage may vary.
These tools should let you get up and coding, saving your code to a central place.