Traefik 2.0 upgrade with Docker Compose

So apparently I wasn’t the only one who woke up one day to find everything offline because Traefik 2 had been released with breaking changes because I was running Watchtower and using the Traefik:latest tag.

I was also not the only one to quickly try and fix the issue, but Traefik 2 was quite a significant change, and the issue was not going to solved in a few mins. It was rollback time for many people, including myself, particularly with the lack of decent documentation available at the time.

Anyway, yesterday I was quite sick and at home, and decided to attempt this upgrade. I mean, how hard can it be? Not that hard to be honest, I was done in about 3hrs.

Here’s what you need to do in 2 simple steps

  1. Run the traefik-migration-tool to convert your Letsencrypt ACME config and your static config to 2.0 format.
  2. Update your docker compose file based on the new Traefik architecture (routers, middleware and services)

I won’t go over step 1, as this is very simple and the tool provides enough help – short version is you just run the tool against the acme.json file and traefik.toml file to get new v2 compatible versions. I had some warnings about SSL redirects, which were previous defined in the entrypoint but are now handled by middleware.

Step 2, however, is where things get a little tricker due to the new architecture of Traefik. There is a v1 to v2 migration guide, which helped a lot, and a blog post on version 2, but there were still things that were not well explained in the guide, which is what I’d like to highlight here.

New Traefik Architecture

First, lets look at a couple of examples, all of which can be found either in the current version, or previous versions, of my docker-compose.yml on Github.

Traefik Image

This is the labels section from the Traefik docker container (nothing else changed).

    labels:
      - traefik.enable=true
      - traefik.http.routers.traefik.rule=Host(`traefik.${DOMAINNAME}`)
      - traefik.http.routers.traefik.entrypoints=http
      - traefik.http.routers.traefik-secure.rule=Host(`traefik.${DOMAINNAME}`)
      - traefik.http.routers.traefik-secure.entrypoints=https
      - traefik.http.routers.traefik-secure.tls=true
      - traefik.http.routers.traefik-secure.middlewares=auth
      - [email protected]
      - traefik.http.services.traefik.loadbalancer.server.port=8080
      - traefik.http.middlewares.sslredirect.redirectscheme.scheme=https
      - traefik.http.middlewares.auth.basicauth.users=${HTTP_USERNAME}:${HTTP_PASSWORD}

Let’s break this down section by section.

    labels:
      - traefik.enable=true
      - traefik.http.routers.traefik.rule=Host(`traefik.${DOMAINNAME}`)
      - traefik.http.routers.traefik.entrypoints=http
      - traefik.http.routers.traefik.middlewares=sslredirect
      - traefik.http.routers.traefik-secure.rule=Host(`traefik.${DOMAINNAME}`)
      - traefik.http.routers.traefik-secure.entrypoints=https
      - traefik.http.routers.traefik-secure.tls=true

Here, I am enabling traefik to route traffic, because I have “exposedByDefault = false” set in my traefik.toml. I am then defining two routers, with a rule to match by hostname, their entry points, and enabling TLS on the secure one. I am also telling the non-secure one to send traffic to my sslredirect middleware (defined further down).

      - traefik.http.routers.traefik-secure.middlewares=auth
      - [email protected]
      - traefik.http.services.traefik.loadbalancer.server.port=8080

Here, I am telling the secure router to pass traffic through an authentication middleware (defined below), and to send traffic to Traefik’s own API (expose the Web UI). The last line is how you now define the port to send traffic to, which stumped me for a while.

      - traefik.http.middlewares.sslredirect.redirectscheme.scheme=https
      - traefik.http.middlewares.auth.basicauth.users=${HTTP_USERNAME}:${HTTP_PASSWORD}

Lastly, we are actually defining two different middleware, which can be reused on other containers: one for redirecting non-SSL traffic, named “sslredirect” by giving it a scheme of https, and the second “auth” middleware defines an authentication middleware .

Now, we have Traefik up and running with an accessible Web UI. Now all we need to do now is set Traefik up to pass traffic through to other containers you may have running.

I’ll use Transmission as an example of one container I have running, which Traefik passes HTTPS traffic to.

  transmission:
    image: linuxserver/transmission
    container_name: transmission
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=${TZ}
      - TRANSMISSION_WEB_HOME=/transmission-web-control/
    volumes:
      - ${USERDIR}/docker/transmission:/config
      - ${USERDIR}/TransmissionDL:/downloads
    ports:
      - 55700:55700
      - 9091:9091
      - 55700:55700/udp
    restart: unless-stopped
    networks:
      - traefik_proxy
      - default
    labels:
      - traefik.enable=true
      - traefik.http.routers.transmission.rule=Host(`transmission.${DOMAINNAME}`)
      - traefik.http.routers.transmission.entrypoints=http
      - traefik.http.routers.transmission.middlewares=sslredirect
      - traefik.http.routers.transmission-secure.rule=Host(`transmission.${DOMAINNAME}`)
      - traefik.http.routers.transmission-secure.entrypoints=https
      - traefik.http.routers.transmission-secure.tls=true
      - traefik.http.services.transmission.loadbalancer.server.port=9091
      - traefik.docker.network=traefik_proxy

In this example, the only section that changed between Traefik 1.7 and 2.0 is the labels section. Hopefully, everything is self-explanatory after the above explanation. We are just defining two routers, one with an sslredirect middleware, and setting the port on the router depending on your container.

That’s it, rinse and repeat, or look at my docker-compose.yml for more examples.

Fully automated media centre using Flexget, emby, Trakt and IMDb

Update Apr 24, 2019: Added latest config.
Update Jun 20, 2019: Current flexget config can be found on Github.

Further to my post back in 2014 about creating your own automated media centre, I’ve improved on a few aspects of this.

Improvements made:

  1. Replaced Serviio with Emby. Emby is better than Serviio by a long way, and continuously improved. I have a lifetime subscription.
  2. Enhanced my Flexget setup considerably with new private trackers, variables and IRC.
  3. Replaced e-mail notifications with Pushbullet push notifications to my iPhone.
  4. Added an NVIDIA Shield (not pro version, don’t need the HDD) with Emby Android App Kodi.
  5. Added Filebot to do automatic extraction and renaming of files into the relevant place for emby to consume. I used the Paid version of this software, well worth it.
  6. Added in 2019: Systemd unit replaces Upstart for auto start in current Ubuntu (18.04).

Let’s look at the changes one by one.

Emby

Not too much to say here, other than Emby is awesome. You can add multiple accounts for friends, and content and watch history is synced across devices. I have multiple devices accessing it including Android TV/Kodi, Roku, Samsung Smart TV, iPhone App, and Google Chrome. I’m not going to detail how to set it up as that is well documented, but suffice to say emby is a great product and I’m a lifetime premium subscriber.

In my setup, media is automatically extracted by Filebot (below) after being downloaded, indexed by emby using real-time file system monitoring, and relevant meta data is downloaded automatically.

Notable plugins that I have installed are:

  1. Emby.Kodi Sync Queue – very important, and provides instant updates to Kodi when new media is available and syncs library deltas to Kodi instead of the entire library. I now use the Emby Android App as it provides a better experience imho.
  2. CoverArt – make things look pretty.
  3. Rotten Tomatoes Reviews – self-explanatory.
  4. Trakt – syncs my libary and watch history with Trakt (cos I’m nerdy).

Flexget

Flexget is an amazingly powerful tool, and is the brains of the setup. Since my original post, I’ve cleaned up my config and updated it for more recent versions of Flexget, and refined my configuration to abstract some of the configuration using YAML alises. My current configuration is below:

Read More

Today is Reset the Net Day

Let’s be honest, the NSA, and by extension the United States government, is out of control. Today is Reset the Net day, an attempt to raise awareness about the insidious problem of government mass surveillance on innocent citizens – and to fight back.

Bottom line, if the NSA targets you for surveillance, you’re screwed. This fact has been agreed on by experts in the industry. However, if you’re just the average citizen who expects some basic privacy, you can fight back. How? Start encrypting everything, that’s how.

You can read more about this effort at the Reset the Net site, which contains helpful hints about what you can do. Soon you will also be able to encrypt your phone calls. You can already browse the net securely. Let’s force the NSA to do targeted surveillance, rather than a dragnet.

I have already started. The server this site runs on is running a Tor relay that pumps 150GB of encrypted and anonymous traffic through it, and you can find out more about how to do that yourself, or simply just use Tor for 100% anonymous browsing.

This site is also now runs SSL everywhere with Forward Secrecy and HTTP Strict Transport Security – graded A+ by SSL Labs. Sure, these things are probably not going to stop the NSA targeting you or me, but you can browse this site completely securely and it’s another encrypted connection to cut down on the surveillance dragnet.

How to create your own streaming TV service

Update: This post is out-dated and parts of it have been updated by Fully automated media centre using Flexget, emby, Trakt and IMDb.

I’m not really someone who watches most of the stuff on TV – I watch specific TV shows that interest me, the occasional movie, and I would watch news, documentary and some sports channels if they were available at a reasonable price without all the other crap.

I’ve therefore put my old computer to good use and created my own streaming TV service that shows me the TV shows and movies  want to watch, when I want to watch them, and you can too. Think of it like a PVR over the internet. Basically your own private version of this rumoured streaming service from Rogers with the stuff you want. Also gives me an excuse to go the bar and watch certain sports 🙂

Hardware and software needed

Here is all the stuff you need:

  1. A computer (or certain NAS devices) to store and stream from. You need a sizeable harddrive to store media on. I used two 1TB Western Digital Reds in mirrored RAID configuration (I’m tired of losing data).
  2. A fast wireless router (best to get a 5.8GHz model these days to reduce interference).
  3. Linux server – free. I used a Ubuntu Server 13.1 VM running on a Proxmox VE host, but you could use any Linux distro.
  4. Serviio streaming media server – free or $25 if you need it. I wanted the MediaBrowser web interface so I can play content on my MacBook Air.
  5. Flexget content automation tool – free.
  6. Transmission bittorrent client – free.
  7. Samba  file and print server – free. This is not required, but mounting the files on your laptop is also kind of useful.
  8. An account on Trakt.tv – free. Use to select what TV shows you want, but it’s also a very useful site.
  9. Some time and some knowledge of Linux (or a friend who does) – opportunity cost.

You can also add something like a DLNA-compatible TV to this mix, or an PS3 or Xbox (I found the Serviio Media Browser far preferable for browsing content due to the crap interface/functionality available on the Xbox). Read More