Shaddock

Shaddock is a container based orchestration and deployment tool that rely on a definition model to describe and manage a distributed multi-service infrastructure.

Documentation: http://shaddock.epheo.eu

Sources: https://github.com/epheo/shaddock

Shaddock is distributed under the terms of the Apache License, Version 2.0. The full terms and conditions of this license are detailed in the LICENSE file.

Installation

pip install shaddock

Or, if you have virtualenvwrapper installed:

mkvirtualenv shaddock
pip install shaddock

Installing it from sources:

git clone https://github.com/epheo/shaddock
cd shaddock && sudo pip install .

Using an existing yaml definition model:

git clone https://github.com/epheo/shaddock-openstack

Play with it!

shaddock -f openstack-deployer.yml

(shaddock) ps
(shaddock) build
(shaddock) start

(shaddock) rm mysql001

The Shaddock Definiton Model

Your model consist in a set of files that describe your general architecture, It can be a simple Yaml few lines, a more complete Jinja2 templated multi files hierarchy or an integrated Python module that you interface with the rest of your infratsructure.

Defining a service

The following examples resume how you can describe a service in a cluster:

You will need at least the following options:

clusters: 
  - name: simple-service
    images: images/testdir
    services:
      - name: simpleservice
        image: testuser/arch_base:latest

The following possibilities are currently implemented:

clusters: 
  - name: complete-service
    images: images/testdir
    hosts: !include site01/hosts_dc01.yml
    vars: !include site01/test-variables01.yml
    services:
      - name: test900
        image: testuser/arch_base:latest
        priority: 900
        host: node001-socket
        ports:
          '4321/tcp': ('0.0.0.0', 4322)
          '4325/tcp': ('0.0.0.0', 4325)
          '4326/tcp': 4327
          '4329/tcp': 4329
        volumes: 
          - '/home/user1/:/mnt/vol2:rw'}
          -  '/var/www:/mnt/vol1:ro'} 
        environment:
          VAR001: '{{ var_001 }}'
        depends-on:
          - {host: 127.0.0.1, type: http, get: '/', code: 200, useproxy: False}
        command: "echo $VAR001"

Including other files

Shaddock allow you to split your definition model into multiple yaml files using the !include notation.

This, conbined with the templating allow you to design very complex architectures without repetitions or loosing in readabality of your model.

clusters: 

  - name: include-test
    vars: !include site01/test-variables01.yml
    images: images/testdir
    services:
      - name: test120
        image: testuser/arch_base:latest
        priority: 120
        environment:
          VAR001: '{{ var_001 }}'
        command: "env |grep VAR001"

  - name: command-test
    vars: 
        foo: bar
    images: images/testdir
    services:
      - name: test121
        image: testuser/arch_base:latest
        priority: 121
        environment:
          FOO_BAR: '{{ foo }}'
        command: "env |grep FOO_BAR"

Using the scheduler

The Shaddock scheduler will ensure that all the requirements you provide are matched before starting a new service.

You can check: * A container status * If a port is open (tcp or udp) * The return code of a http GET

You can also specify the number of retry, the time to wait before 2 checks, and if the check should use the system proxy vars or not.

- {name: nova, status: stopped}
- {name: nova, port: 8774, type: tcp}
- {name: nova, port: 8774, state: down, type: tcp}
- {host: google.com, port: 8774, state: down, type: tcp}
- {name: nova, type: http, get: '/v2.0', port: 5000, code: 200}
- {host: google.com, type: http, get: '/v2.0', port: 5000, code: 200}
- {host: 127.0.0.1, type: http, get: '/', code: 200, useproxy: False }
- {name: nova, sleep: 20} # defaults to 10
- {name: nova, retry: 10} # defaults to 5

clusters: 
  - name: scheduler-tests
    images: images/testdir
    services:
      - name: test500
        image: testuser/arch_base:latest
        priority: 500
      
      - name: test501
        image: testuser/arch_base:latest
        priority: 510
        command: 'sleep 100'

      - name: test503
        image: testuser/arch_base:latest
        priority: 520
        depends-on:
          - {name: test500, status: stopped, retry: 10, sleep: 10}
          - {name: test501, status: running}

  - name: scheduler-tcp-tests
    images: images/testdir
    services:     
      - name: test504
        image: testuser/netcat:latest
        priority: 550
        ports: {'1234/tcp': 1234}
        command: ["/bin/bash", "-c", "echo 1| nc -l 1234"]

      - name: test520
        image: testuser/arch_base:latest
        priority: 560
        # depends-on:
        #   - {name: test504, port: 1234}

Managing multiple hosts

Shaddock is able to schedule your services on different hosts accros your datacenter. The only prerequirements for a host to be part of a Shaddock cluster is toi have the Docker API installed and listening on a port. You can then configure your hosts in your cluster defintion.

- name: node001-socket
  api:
    base_url: unix://var/run/docker.sock
    version: 1.30
    timeout: 50

- name: node002-tcp
  url: tcp://127.0.0.1:2376
  verion: 1.30

- name: node003-tls
  url: tcp://127.0.0.1:2376
  tls: False
  cert_path: None
  key_path: None
  cacert_path: None
  tls_verify: False

clusters: 
  - name: host-cluster1
    images: images/testdir/
    hosts: !include site01/hosts_dc01.yml
    services:
      - name: service202
        image: testuser/arch_base:latest
        priority: 210

      - name: service203
        image: testuser/arch_base:latest
        host: node001-socket
        priority: 211

  - name: host-cluster2
    images: images/testdir/
    hosts: !include site01/hosts_dc02.yml
    services:
      - name: service204
        image: testuser/arch_base:latest
        host: node002-tcp
        priority: 212

      - name: service205
        image: testuser/arch_base:latest
        host: node003-tls
        priority: 213

Using the templating functionalities

The model definition variables {{ your_var }} are templated using Jinja2 before being interpreted by Shaddock. You can define any variables value in the vars: section of a cluster definiton.

refs: http://jinja.pocoo.org/

clusters: 
  - name: jinja2-tests01
    vars: !include site01/test-variables01.yml
    images: images/testdir
    services: |
      {% for service in servicelist %}
        - name: clu1-{{service}}
          image: testuser/arch_base:latest
          priority: 401
          environment:
            VAR001: {{var_001}}
          command: env
      {% endfor %}

  - name: jinja2-tests02
    vars: !include 'site01/test-variables01.yml'
    images: images/testdir
    services: |
      {% for service in largeservicelist %}
        - name: clu2-{{service}}
          image: testuser/arch_base:latest
          priority: 401
          environment:
            VAR001: {{var_001}}
          command: env
      {% endfor %}

Usage

Using the command line interface

usage: shdk [--version] [-v] [--log-file LOG_FILE] [-q] [-h] [--debug]
            [-f SHDK_MODEL] [-d SHDK_IMGDIR] [-c SHDK_CLUSTER]
            [--docker-version DOCKER_VERSION] [-i DOCKER_URL] [--boot2docker]
            [--tls] [--tlscert DOCKER_CERT_PATH] [--tlskey DOCKER_KEY_PATH]
            [--tlsverify DOCKER_TLS_VERIFY] [--tlscacert DOCKER_CACERT_PATH]

Shaddock shell:

optional arguments:
  --version             Show Shaddock's version number and exit.
  -v, --verbose         Increase verbosity of output. Can be repeated.
  --log-file LOG_FILE   Specify a file to log output. Disabled by default.
  -q, --quiet           Suppress output except warnings and errors.
  -h, --help            Show this help message and exit.
  --debug               Show tracebacks on errors.
  -f SHDK_MODEL, --file SHDK_MODEL
                        Template file to use.
  -d SHDK_IMGDIR, --images-dir SHDK_IMGDIR
                        Directory to build Docker images from.
  -c SHDK_CLUSTER, --cluster SHDK_CLUSTER
                        The cluster to use (No value is all by default).
  --docker-version DOCKER_VERSION
                        Docker API version number
  -i DOCKER_URL, --url DOCKER_URL
                        Force a specific host url API.
  --boot2docker         Use Boot2Docker TLS conf.You should first:"eval $(sudo
                        docker-machine env machine_name)"
  --tls                 Use TLS; implied by tls-verify flags.
  --tlscert DOCKER_CERT_PATH
                        Path to TLS certificate file.
  --tlskey DOCKER_KEY_PATH
                        Path to TLS key file.
  --tlsverify DOCKER_TLS_VERIFY
                        Use TLS and verify the remote.
  --tlscacert DOCKER_CACERT_PATH
                        Trust only remotes providing a certificate signed by
                        theCA given here.

Commands:

build          Build a service
complete       print bash completion command
create         Create a new container
help           print detailed help for another command
logs           Display the logs of a container
ps             Show a list of Containers.
pull           Pull a container from the Docker Repository
restart        Restart a container
rm             Remove a container
show           Show details about a container
start          Start a new container
stop           Stop a container

Using shaddock in a project

import shaddock

Examples

Shaddock includes many tests to demonstrate the use of Shaddock. All of the tests are in the ‘tests’ folder. It can be used as a starting point for new projects. The tests also serve as SHDK docs.

Please look in the tests folder for examples of how to define yaml files for use with SHDK. Most of the test examples are snippets of yaml, for a more complete example look at the tests in the ‘tests/model’ folder.

OpenStack

https://github.com/epheo/shaddock-openstack.git is a yml definition model example you can use with Shaddock to build, deploy and manage the lifecycle of an OpenStack platform from the upstream git sources.

Alternative configuration and other systems

Docker Machine and OS X support

Please use the --boot2docker option and Shaddock will source the environement variables set by Docker machine.

In order to set those variables you may want to $(sudo docker-machine env machine_name)" first.

Run the shaddock shell from a container

docker run --rm -i -v shaddock/tests/model/:/model \
    --env DOCKER_HOST="https://<your_host>:2376" \
    --env TEMPLATE_FILE=/model/service-tests.yml \
    -t shaddock/shaddock

Seting up the Docker API to listen on tcp:

https://docs.docker.com/reference/api/docker_remote_api/

Contributing

Please use Github for any isse reporting or pull request.

Running tests

You can run the tests using tox:

pip install -r test-requirements.txt
tox

Automated Testing

The project have it’s own CI system and use Jenkins to test and merge automaticaly.

You can contact us on freenode in the #shaddock channel.

shaddock style commandments

Step 0

• Step 1: Read pep8
• Step 2: Read pep8 again
• Step 3: Read on

Read the OpenStack Style Commandments http://docs.openstack.org/developer/hacking/

Informations

License

Shaddock is distributed under the terms of the Apache License, Version 2.0. The full terms and conditions of this license are detailed in the LICENSE file.

References

Docker-py API Documentation: http://docker-py.readthedocs.org/

OpenStack Official Documentation: http://docs.openstack.org/