764 views
# Loomio devops bus test I've developed this survey for dev team members to run through, in an in person setting. It serves two purposes: 1. It ensures that we have accurate, accessible documentation about how our Loomio.org systems work. 2. It ensures that dev team members have access (eg: ssh access) and basic knowledge of how to service the system if required. ## Accessing core services Check that you can login to the following services: - github _ensure that you can access `deploy-loomio-org` repo_ - digital ocean _you should have your own account for this._ - cloud flare _we have a shared login, kept in secrets.txt_ - all the servers in our digitalocean team account. _ssh into each of the servers in the digitalocean account with `ssh root@hostname` - heroku _use your own login to access loomio-production, loomio-clone, loomio-test via heroku_ - new relic _login to new relic and check performance and server health_ - errbit.loomio.io _check you can login to errbit and that you recieve application exception emails_ - iwantmyname.com _shared account details stored in secrets.txt_ ### Pop quiz: - Where is a list of all the servers available? - Where is the DNS hosted? - Where are our domain names renewed at? ## Driving test ### Please check what docker containers are running on reply.loomioclone.org ``` docker ps ``` ### How do you read the logs of a container? ``` cd deploy-loomio-org/loomio.org/reply docker logs -t -f --since="20m" haraka_smtp_1 ``` ### If a service has stopped, what should you check? - Does the dns resolve? - Is the server running? - does it respond to ping? - Is the container on the server running? - How much disk space is on the server? - What do the logs of the container say? ### Can you restart a service? What do each of these commands do? ``` docker-compose down docker-compose stop docker-compose start docker-compose up -d ``` ### Do you know when the DNS expires? _all dns records are in namecheap with auto renewal enabled_ ### or when our SSL certificates expire? ### What are our main TLD's and what do we use them for? ### If an application exception happens, do you get notified? ### if the error rate of loomio.org is over 5% do you get notified? ### If a service stops (such as reply.loomio.org) do you get notified.? ### If you make any changes to any service, or login to loomio-production and run some commands, what do you need to do? Add an entry to the notes directory of deploy-loomio-org detailing what the incident was, and what commands you ran (use the history command) to resolve it. ### How to setup a new container for a service? Not a super good answer for this yet. checklist - backups, publickeys, swapfile, new relic monitoring ### how do you give a new person access to the existing systems? add them to digitalocean team, github, heroku, errbit and publickeys push to all servers. ``` scp publickeys.txt root@errbit.loomio.org:.ssh/authorized_keys ```