A real fast list of stuff from the Percona Live 2014 event.
Yahoo’s Performance Analyzer
Yahoo is developing a MySQL performance analyzer that should be released as open source later this year. From the demo, it looks like it pulls in most of the MySQL metrics, shows you a processlist and then lets you drill into a processlist with explain. Will have to keep my eye out for this.
ChatOps with Hubot
GitHub’s Sam Lambert has a set of hubot chatops scripts for MySQL. I was already looking at depoying hubot for the ability to push messages from a remote source into an IRC channel, so this would be a natural fit. He also mentioned using Janky to tie CI with hubot.
DevOps at Outbrain
Shlomi Noach talked about how they tackle DevOps at Outbrain (slides) and I was thoroughly impressed. They use onering for inventory management that has a RESTful API. It makes it very easy to search for servers matching a specific criteria: what servers are running httpd or what servers have a RAID controller. I think this would be really nice to integrate with Ansible.
He also talked about how they used Anemometer from Box to forward slow query logs via LogStash to a processing server running pt-query-digest. They also do a similar task to capture binlog events (actually, relay logs on a slave) for DML statements which they dubbed Anemomaster. All this data also gets sent into Graphite. For monitoring, they only monitor graphite, rather than monitoring each server individually.
They tie this in with an audit_login plugin for MySQL that tracks each connection and ships it off to LogStash/Kibana and also a text file save for easy grepping.
Next, he talked about Propagator, which is their own solution to continuous deployment at the schema level. The devs use the tool to submit their ALTER and it requires a DBA to review it before it gets pushed out to production.
Last, he explained how they use Zookeeper to provide a callback method for throttling back applications when databases become saturated with things like INSERTS. If the application can be aware that we have slave lag, they can impose limitations on frequency of statements or shift away from using replagged slaves with no action from the DBA.
Sunny Gleason had a nice talk on using Zookeeper as a tool to help achieve fast-failover in milliseconds. Looks like something you could build around since it allows you to do callbacks to the application level essentially informing them of the change in configuration.