Chatops (Operations automation with a bot)

About

Chatops:

• The important phases of development cycle are brought in one place, in one room, a chat room through a bot.
• Automating common operations task with a bot.
• Breaking communication wall between different teams.

The bot is the primary control for the whole infrastructure.

The chat becomes a console

Articles Related

Chat

• Campfire is a web-based group chat tool that lets you set up password-protected chat rooms in just seconds. Invite a client, colleague, or vendor to chat, collaborate, and make decisions. Link to a room on your intranet for internal communications.

Bot

Hubot

Ryan wrote Hubot at Github.

rolling a new version

git checkout -b git-gh13
# bump version number
git commit -am 'Bump git version'
git push origin git-gh13
# create pull request

Change as passed CI (ie Build was succesful)

• Deploy

hubot deploy puppet to production

• noop means simulate what happens on that branch git-gh13 against the file server fs1, noop mode simulates the changes

hubot puppet noop git-gh13 fs1 

hubot puppet force git-gh13 fs1

• Testing: Graph with graphite and collectd

hubot graph me -1h @collectd.load(fs1)

• The pull request is good, we want to push it in production (everywhere)

#merge pull request

• To ensure that the master is green

ci runs
# Build 1087 (eragaf66&&) of puppet/master was successful (399s)

• When the CI passes, the code is deploy in production

auto deploy

• Now a new version of git is on master. It has been shipped everywhere.

# Puppet shipped it on a interval but you can push it 
hubot puppet force production fs

• Graph of everything (of all file servers)

hubot graph me -1h @collectd.load(fs*)

• Logs to make sure that we didn't break anything.

hubot log me hooks github/github

Others

• CI status of a random branch

hubot ci status github/smoke-perf

• Roll it out to one file server

hubot deploy github/smoke-perf to prod/fe1

• Verifying things are return to normal

hubot log me smoke fe1

Nagios

hubot nagios check fs3b/syslog
hubot nagios status fs3b/syslog
hubot nagios downtime fs3b/syslog 90
hubot nagios mute fs3b/syslog
hubot nagios unmute fs3b/syslog

Example

https://response.pagerduty.com/resources/chatops/

Documentation / Reference

• Chatops at Github
• Hubot in Github
• https://www.reddit.com/r/chatops/
• Lazlo Bot for Slack in Go