Moving my websites to Docker

Quick summary

I have 2 websites hosted on a Linode machine… I want those website to be deployed as 2 Docker containers on a DigitalOcean Droplet.
Both sites generate very low traffic. Both serve static pages (the first one via sailsjs, the second one with nginx).

Sections of this article:

  • My Docker knowledge
  • The Context
  • Containerize the sailsjs application
  • Containerize the static web site
  • Add a reverse proxy container
  • Use Docker Compose to link the 3 containers
  • Use Docker Machine to create a DigitalOcean Docker Host
  • Change DNS
  • Conclusion

My Docker knowledge

I’ve been following a lot of the buzz surrounding Docker since more than one year and this is an ecosystem I enjoy learning. I’ve done some tests based on tutorials and articles I found on the web, read a couple of books and started to setup some containers in the startup I’m currently working at. I’m glad I can work on this at work as I’m sure our development team can gain a lot from this.

Stuff I have tested (or I’m currently testing):

  • docker-compose
  • docker-machine
  • docker-swarm
  • docker hub
  • running containers on mesos/marathon
  • CoreOS
  • Kubernetes

Not long ago I remember having a couple of web sites sitting around and realized that could be fun (yes, fun :) ) to see what was needed to move then to Docker containers.

Context

I’ve developed 2 websites a couple of years ago:

  • cafechinois.fr dedicated to meetings between Chinese and French for language exchange. This site is made with sailsjs (a great framework to develop nodejs applications, if you don’t know it, just check it out
  • heden-cams.com dedicated to backup the iOS application of the same name (this app is not well noted in the states (do not really know why, but it has some quite good comments in France). This site is purely static using standard html and twitter bootstrap. Pages are served over nginx

Both sites are hosted on the same Linode machine (20$/month). On this machine is installed nginx with a reverseproxy configuration so that:

  • http requests towards cafechinois.fr are proxied to the cafechinois upstream
  • http requests towards heden-cams.com are served from the appropriate filesystem’s folder

Configuration for cafechinois.fr:


server {
    listen 80;
    server_name cafechinois.fr www.cafechinois.fr;
    location / {
        proxy_pass http://localhost:1337/;
    }
}

Configuration for heden-cams.com:


server {
    listen 80;
    server_name heden-cams.com www.heden-cams.com;
    location / {
        root /var/heden-cams;
        index index.html;
    }
}

The purposes of this articles is to detail the steps I follow to containerize those sites and deploy them on a digitalocean Droplet.

Containerize the sailsjs application

For the sails application, I’ve started to build a iojs image and shared it on Docker Hub. Several images already existed but I wanted to do mine to see how simple it was.

I’ve started to create the following Docker file


FROM ubuntu:14.04.2
MAINTAINER Luc Juggery
ENV REFRESHED_AT 20150712T190000

# update package sources
RUN apt-get -y update

# Packages needed for compilation
RUN apt-get -y install curl build-essential libssl-dev git git-core python

# Install iojs 2.3.4 through nave
RUN curl https://raw.githubusercontent.com/isaacs/nave/master/nave.sh > /tmp/nave.sh
RUN chmod +x /tmp/nave.sh
RUN /tmp/nave.sh usemain 2.3.4

This one is quite simple, it starts with the ubuntu trusty images and install iojs (last version 2.3.4 at the writing date) through the great nodejs / iojs virtual environment Nave.
This image is not aimed to be ran, but mainly to be used as a based image for nodejs application (sailsjs application in our case).

The image is then created with:

docker build -t lucj/iojs .

And pushed to the lucj/iojs docker’s hub repository with:

docker push lucj/iojs

At the root of my sailsjs application, I’ve created the following Dockerfile:


FROM lucj/iojs
MAINTAINER Luc Juggery
ENV LAST_UPDATED 20150713T180000

# Get current version of code
ADD . /app

# Use /app working directory
WORKDIR /app

# Application will run on port 8000
ENV PORT 8000
EXPOSE 8000

# Build dependencies
RUN npm install

# Run sails application
CMD ["npm", "start"]

This file describe how the image will be created.

  • based on the iojs image created above
  • the application code (sailsjs application folders) is copied to the /app folder within the image
  • the dependencies are installed
  • Defines port 8000 has the exposed port of the container (for documentation purposes)
  • Defines, through CMD, the command that will be executed when running the image. In this case it is ran with « npm start » (which translates to « node cafe.js » in the package.json)

The image can be created with the following command:

docker build -t lucj/cafechinois .

Let’s run it and see if it runs ok:

docker run -p 8000:8000 lucj/cafechinois

Being on a Mac I use Boot2Docker as the Docker host. When installing boot2docker, a lightweight VM is created and provisioned with the Docker daemon. A Docker client is also installed on the Mac and communicate with the VM. The IP of my Docker host is thus the IP of the boot2docker VM


    $ boot2docker ip
    192.168.59.103

I thus need to check the following URL to see if running this new image is working fine:



http://192.168.59.103:8000

Ok, this is fine, the sails application is ok. The image can now be pushed to a Docker Hub private repository

docker push lucj/cafechinois

The image will now stand in docker’s hub and wait to be deployed.

Containerize the static web site

The static site is simpler to containerize (do not know if this work exists though…) as it’s only a couple of folder (css, images) and an index.html file.

At the root of my folder I’ve created the following docker file:


FROM ubuntu:14.04.2
MAINTAINER Luc Juggery
ENV LAST_UPDATED 20150713T180000

# Install python
RUN apt-get install python -y

# Get current version of code
ADD . /app

# Use /app working directory
WORKDIR /app

# Application will run on port 8000 (default python's SimpleHTTPServer port)
EXPOSE 8000

# Run application
CMD ["python", "-m", "SimpleHTTPServer"]

This Dockerfile defines the following steps:

  • Start from a fresh ubuntu trusty image
  • Install python
  • Copy the application code in the /app folder
  • Defines port 8000 has the exposed port of the container (for documentation purposes)
  • Define, through CMD, the command that will be executed when running the image. In this case of a static web site, I use the super simple python’s SimpleHTTPServer that serves all the files in the current folder (it uses port 8000 by default).

    Let’s create the image:

    docker build -t lucj/hedencams .

    and run it:

    docker run -p 8000:8000 lucj/hedencams

    Ok, this is fine, the newly created image can now be pushed to its docker’s hub private repo:

    
        docker push lucj/hedencams
    

    and wait there to be used.

    Add a reverse proxy container

    I now have my two images waiting to be deployed in a Docker host. As I plan to use 2 sites on the same machine, I need a way to have an entry point (on the Docker host) that will proxy the HTTP request towards the right domain.

    For this reason, I’ve created a simple reverse proxy image using nginx. The Dockerfile is the following one:

    
    FROM ubuntu:14.04.2
    MAINTAINER Luc Juggery
    ENV REFRESHED_AT 20150712T190000
    
    # Update package sources
    RUN apt-get -y update
    
    # Install nginx
    RUN apt-get install nginx -y
    
    # Overwrite configuration file
    COPY nginx.conf /etc/nginx/nginx.conf
    
    # Delete default site file
    CMD rm /etc/nginx/sites-enabled/default
    
    # Copy site definitions in sites-enabled
    COPY heden-cams.com /etc/nginx/sites-enabled/heden-cams.com
    COPY cafechinois.fr /etc/nginx/sites-enabled/cafechinois.fr
    
    # Run nginx in the foreground
    CMD ["nginx"]
    

    Basically, this one:

    • use the ubuntu/trusty base image
    • install nginx
    • copy the nginx configuration file of each site
    • start nginx (« daemon off; » is specified in the main nginx configuration file)

    The configuration file of cafechinois.fr:

    
    upstream cafechinois {
        server cafechinois:8000;
    }
    
    server {
        listen 80;
        server_name cafechinois.fr;
    
        location / {
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass   http://cafechinois;
        }
    }
    

    The configuration file of hedencams:

    
    upstream heden {
        server hedencams:8000;
    }
    
    server {
        listen 80;
        server_name heden-cams.com;
    
        location / {
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass   http://heden;
        }
    }
    

    In those 2 configuration files, note the server part of the upstream, this hostname will be detailed in the Docker Compose section below.

    Same as above, I create the image and push it to its private repo:

    
    docker build -t lucj/reverseproxy .
    docker push lucj/reverseproxy
    

    The test of this new image will be done in the next step with Docker Compose

    Use Docker Compose to link the 3 containers

    Docker Compose is a great tool of the Docker ecosystem that enables to link dependent containers. In the present case, I need the reverseproxy container to be dependent of the cafechinois and hedencams ones. The reason is simple: when I run the reverse proxy I need it to know where to find (ip + port) the 2 containers (one for each site).

    One of the great functionality of Docker Compose is the way containers are linked:

    • environment variables are set with the ip/port of the linked containers
    • /etc/hosts is updated with the hostname of the linked containers

    I have set up the following docker-compose file:

    cafechinois:
      image: lucj/cafechinois
    hedencams:
      image: lucj/hedencams
    reverseproxy:
      image: lucj/reverseproxy
      ports:
        - "80:80"
      links:
        - cafechinois:cafechinois
        - hedencams:hedencams
    

    Doing so, I know that the /etc/hosts of the reverse proxy will be updated with the following hostnames:

    • cafechinois
    • hedencams

    Note: those hostname are the one used in the nginx configuration of each site (see above). Doing so, I do not need to hack nginx around so it takes into account the hostname in the upstream section.

    Once the docker-compose.yml file is ok, it can be tested on any docker host:

    
       docker-compose up
    

    In this case, the DNS part is missing, so I’ve updated the /etc/hosts of my machine to match the domain name with the ip of the Docker host.

    For test purposes I’ve then updated the /etc/hosts (of my Mac) with

    
        192.168.59.103 cafechinois.fr
        192.168.59.103 heden-cams.com
    

    As the application (the one defined in docker-compose.yml) is running I can now access both sites from my browser.

    Both site can now be deploy on a « production » Docker machine and setup with the correct DNS entries.

    Use docker-machine to create a digitalocean droplet

    Now that the application is defined with Docker Compose, it’s time to set up a Docker host to deploy it on. One of the easiest way is to use Docker Machine. Basically, Docker Machine enables to turn your laptop, a VPS of private cloud provider (GCE, DigitalOcean, AWS, …), a bare metal server, … into a Docker host (= a host with the Docker deamon running).

    I’ve chosen to use the DigitalOcean driver and create a Docker host running on a Droplet. The only thing needed for this is a DigitalOcean access token so Docker Machine use the API to create and provision the Droplet. This access token can easily be created within your DigitalOcean account (if you do not have one yet, do not hesitate to use my referal link link ).

    Once the access token is created, it can be used in the following command which will create the Droplet, and install the last version of Docker daemon on it.

    
        docker-machine create \
        --driver digitalocean \
        --digitalocean-access-token MY_ACCESS_TOKEN \
        luc-www
    

    Note: by default, it’s created as a 512M / 1CPU Droplet located in a NY datacenter. Some additional DigitalOcean option can be provided to Docker Machine to set the amount of RAM, the number of CPU, the location of the data center where this Droplet will be deployed, …

    Now the Docker host is created, we need to set the current shell in the context of this new Docker host:

    
        eval "$(docker-machine env luc-www)"
    

    Each Docker command that will be ran from now on will be executed on the newly created Droplet. Neat, isn’t it ?

    The command that is interesting here is the Docker Compose command that enables to deploy the application as 3 dependents containers.

    
        docker-compose up -d
    

    The application is now available on the new host. It’s not visible from the outside as the dns are not updated so cafechinois.fr and heden-cams.com still point towards the Linode server.

    Change DNS

    From the web interface of the registrar used to create some domain names, I just needed to replace the dns servers with the DigitalOcean ones:

    
        ns1.digitalocean.com
        ns2.digitalocean.com
        ns3.digitalocean.com
    

    And then create a DNS entry for each domain name in the DigitalOcean DNS menu. The entry that needs to be created is a A record mapping the domain name with the IP of the Droplet.

    Conclusion

    This exercice is not a complicated thing but it enables to see several interesting components of the Docker ecosystem. I really like the Docker Compose, Docker Machine products. Some work is currently beeing done to enable to deploy a Docker Compose application across several Docker hosts. Can’t wait to see that included in the next version.

    Please let me know what you think.

Books read

2011
Four hours work week
Lean Startup
Scrum en action
The millionaires next door – Thomas J. Stanley, William D. Danko

2012
Getting things done – David Allen
Le Cygne noir – Nassim Nicholas Taleb
L’engrenage, mémoires d’un trader – Jérôme Kerviel

2013
Getting real – 37 Signals
Rework – 37 Signals
Who moved my cheese
Delivering Hapiness – Tony Hsieh
Le marketing expliqué à ma mère
Le management lean – Michael Ballé
Les pratiques du Lean management dans l’IT – Régis Médina
L’art du Lean Software Development
The Phoenix project
The Toyota Way
The Goal – Elihayu Goldratt
It’s not luck – Elihayu Goldratt

2014
Critical Chain – Elihayu Goldratt
Isn’t it obvious – Elihayu Goldratt
L’art de la guerre – Zun Tzu
Remote – DHH
Click Millionaires – Scott Fox
the personal MBA – Josh Kaufman
Effectuation – Les principes de l’entrepreneuriat pour tous – Philippe Silberzahn
The Lean Year Round Nutrition Guide – Anthony Dexmier
Predictably Irrational – Dan Ariely
L’art d’aller à l’essentiel – Leo Babauta
Rich Dad, Poor Dad – Robert Kiyosaki

In progress
The 7 habits of highly effective people
L’entrepreneur minute

Hubic, des problèmes de jeunesse ?

J’ai essayé plusieurs services de stockage en ligne ces derniers mois:

– dropbox (je m’en sers pour sauvegarder des livres et autres docs mais la version payante est relativement chère: 100Go pour 99$/an)
– sugarsync (j’ai laissé tombé, l’interface est trop compliquée à mon gout)
– wuala (je m’en sers pour les données sécurisées)
– et dernièrement Hubic (OVH) attirant par son prix largement moins élevé que la concurrence: 100Go pour 15.54€/an)

J’ai souhaité essayer Hubic pour le stockage de 40Go de documents (principalement des photos), j’ai donc souscrit à l’offre de 100Go (ce qui est effectivement beaucoup plus intéressant que toutes les offres que j’ai pu voir jusqu’a présent). Installation du client Hubic sur le mac, configuration et hop c’est parti pour l’upload de 40Go (à peu près 15500 fichiers).

1er problème

Au bout de 3 ou 4 jours (l’upload est très long), le serveur hubic a connu des problèmes et mon client n’a pas pu se connecter (erreur 500). J’avais à ce moment la uploadé environ 27Go de données. J’ai relancé le client qui s’est comporté comme si il s’installait pour la première fois et la… erreur de ma part… (mais c’était pas forcément intuitif) j’ai spécifié le même répertoire que j’avais spécifié lors de la première installation => le contenu du serveur (ce qui avait déjà été uploadé) est venu ecraser mon répertoire local: bilan 13Go de fichiers de disparu. Et la j’ai été content d’avoir fait un backup sur un DD externe avant de me lancer.

J’ai donc continué en copiant dans mon répertoire local les 13Go manquant. La synchronisation a continué (1 ou 2 jours) et le client local m’a alors indiqué qu’il n’y avait plus d’activité (comprendre que le repertoire local et celui distant étaient synchronisés).

2ème problème

Mon répertoire local indique environ 40Go (pour 15500 fichiers), celui distant (accessible via l’interface web hubic.com) indique environ 37.5Go. Comment 2 répertoires n’ayant pas la même taille peuvent être synchronisé ? De plus, il n’est pas possible depuis l’interface web de hubic (ou bien c’est moi qui n’est pas trouvé) de connaitre la taille et le nombre de fichiers d’un répertoire en particulier… pas vraiment pratique.

Hier soir j’ajoute un répertoire à mon dossier local. Je laisse la synchronisation (l’upload du nouveau dossier) se faire pendant la nuit… et la…

3ème problème

l’upload a planté cette nuit. Je redémarre le client mais plusieurs répertoires que je n’avais pas touchés semble ne plus être synchronisés (ils l’étaient pourtant la veille au soir) et le client m’indique que des uploads et des downloads sont en cours. Je ne comprends pas pourquoi l’ajout d’un folder en local cause le download de fichiers situés dans d’autres dossiers (qui n’ont d’ailleurs pas été modifiés)…

En ce moment, upload et download dans tous les sens, et j’avoue ne pas comprendre la logique de la chose. Ma confiance en ce service est en train de s’écorner rapidement.

Ce matin, 1€ = 1.37$ => un abonnement d’un an a Dropbox revient donc à environ 72€, presque 5 fois plus que Hubic, mais je suis pas loin de craquer…

Mise à jour

3 jours après ce billet, toujours pas de réponse aux questions que j’ai posées au support. J’ai fait une demande de remboursement pour ce service qui ne m’a vraiment pas donné satisfaction, prix alléchant mais le service n’est pas à la hauteur. J’ai finalement choisit de m’abonner à Dropbox, je paye plus cher (5 fois plus) mais j’ai une plus grande confiance dans le service.

En résumé

Hubic, j’y ai cru… je l’ai testé… mais il m’a deçu !
Aujourd’hui, je ne le recommande pas…

Autour de NodeJS

J’ai découvert NodeJS il y a environ 2 ans (version 0.4.8 à l’époque).
Rapidement j’ai testé cette technologie dans le cadre du développement back-end d’une application iOS (le projet Sharies pour lequel j’ai publié plusieurs billets).

Par la suite, j’ai utilisé NodeJS lors du développement d’une application pour la startup dans laquelle je travaille. Son utilisation m’a donnée une très grande satisfaction et des résultats encourageant !

Certaines des principales spécificités de NodeJS sont:

  • l’utilisation du language JavaScript côté serveur
  • l’unique thread évènementiel
  • les api asynchrones
  • les functions de callback

Ces différentes spécificités permettent de développer des applications très scalables et très performantes.

Aujourd’hui, je continue à utiliser NodeJS (au travail et également pour des projets de développement personnels). Cela me permet, jour après jour, de me familiariser avec l’écosystème de NodeJS, d’acquérir des bonnes pratiques et de découvrir certains des modules parmis les plus utilisés.

Avec Thomas Castelly, un développeur très talentueux, avec qui j’ai eu l’occasion de travailler il y a quelques années, nous avons décidé de rassembler nos connaissances de NodeJS dans un ouvrage Autour de NodeJS. Cet ouvrage d’adresse à toute personne intéressée (voire intriguée) par cette technologie dont beaucoup parlent.

couverture-nodejs

Cet ouvrage est découpé en 3 parties: la première détaille ce qui fait la spécificité de NodeJS, la seconde partie présente certains des modules les plus utilisés dans l’écosystème de NodeJS, enfin la troisième partie utilise les connaissances acquises précédemment pour réaliser une API pas à pas.

Autour de NodeJS permet au lecteur de s’immerger progressivement dans l’écosystème de NodeJS et, nous l’espérons, d’en extraire des connaissances qui lui donneront envie de continuer à étudier cette technologie passionnante.

Gérer ses tâches quotidiennes

Il y a quelques mois je me suis demandé si le multitasking, le fait de passer souvent d’une tâche à une autre, est vraiment efficace. Cela peut effectivement donner l’impression d’être plus productif mais beaucoup d’études montrent que ce n’est pas le cas.

En faisant quelques recherches sur le sujet, je suis tombé sur la méthodologie GTD de David Allen.

Lire la suite

try, fail, learn… and try again

I’ve always had scores of potential projects in the head… (mainly web or mobile applications but not only)
Quite often when I had an idea I dove almost directly into the implementation (using the hot frameworks of the moment: rails, nodeJS, backbone, …), without analyzing the viability of the results. Is that a good solution ? Well, it does not seem so…

Lire la suite

Handling several version of node.js with nave

A couple of months ago I decided to give nodeJS a try and I honestly really enjoyed it since then.

The first version I used was 0.4.8. In my application I used several modules:

  • expressjs (you can make an API server in no time with this guy)
  • node-proxy (very usefull to make cross-domain AJAX request)
  • cluster (to use the power of a multi-core CPU)
  • … plus a couple of others

Lire la suite