Using Ansible to build a high availablity Sabnzbd usenet downloader

I’m limited to about 40MB/s on downloads on my VPC at Digital Ocean, but I run Sabnzbd for downloading large files from usenet. It doesn’t take long to download at all, but out of curiosity I wanted to see if I could parallelize this and download multiple files at the same. I use Sonarr for searching usenet for freely distributable training videos which then sends them to SABnzbd for downloading. Since Sonarr can send multiple files to sabnzbd which get queued up, I figured I can reduce the queue by downloading them at the same time.

Using Ansible and Terraform (devops automation tools), I can spin up VPC on demand, provision them, configure them as sabnzbd download nodes and then destroy the instances when complete.

The instances all run the same sabnzbd config and the instances use haproxy for round-robin distribution. I will probably change this to Consul, but I just wanted something quick so I used a basic haproxy config.

Terraform builds 4 sabnzbd, 1 haproxy, and 1 ELK instance. It configures a VIP which I point Sonarr to. Here’s the terraform config that builds a Sabnzbd server:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
resource "digitalocean_droplet" "sab1" {
image = "centos-7-x64"
name = "sab1"
tags = ["sab"]
region = "nyc1"
size = "2gb"
private_networking = true
ssh_keys = [
"${var.ssh_fingerprint}"
]
connection {
user = "root"
type = "ssh"
private_key = "${file(var.pvt_key)}"
timeout = "2m"
}
provisioner "remote-exec" {
inline = [
"export PATH=$PATH:/usr/bin",
"sudo yum -y install epel-release"
]
}
}

Here is a terraform play to provision 6 new hosts (1 Elasticsearch, 1 HAproxy and 4 Sabnzbd):

Click here to watch terraform build the instances

Here is the ansible playbook that configuress the sabnzbd instances:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
#!/usr/bin/env ansible-playbook
---
- hosts: all
gather_facts: yes
become: yes

tasks:
- name: TASK | Create usenet group
group:
name: usenet
state: present

- name: TASK | Create usenet user
user:
name: usenet
shell: /bin/bash
groups: usenet
append: yes
password: [ removed ]
update_password: on_create

- name: TASK | Create /apps directory
file:
path: /apps
state: directory
owner: usenet
group: usenet
mode: 0755

- name: TASK | Create /apps/data/.sabnzbd directory
file:
path: /apps/data/.sabnzbd
state: directory
owner: usenet
group: usenet
mode: 0755

- name: TASK | Install EPEL Repo
action: >
yum name=epel-release state=present update_cache=yes

- name: TASK | Install packages
action: >
yum name={{ item }} state=present update_cache=yes
with_items:
- vim
- git
- mlocate
- par2cmdline
- p7zip
- unzip
- tar
- gcc
- python-feedparser
- python-configobj
- python-cheetah
- python-dbus
- python-devel
- screen
- vim
- htop
- iftop
- bind-utils
- tree
- jq
- telnet
- lsof
- tcpdump
- nload

- name: TASK | Install unrar
yum:
name: ftp://rpmfind.net/linux/dag/redhat/el7/en/x86_64/dag/RPMS/unrar-5.0.3-1.el7.rf.x86_64.rpm
state: present

- name: TASK | Install yenc
yum:
name: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/22/Everything/x86_64/os/Packages/p/python-yenc-0.4.0-4.fc22.x86_64.rpm
state: present

- name: TASK | Clone SAB repo
git:
repo: 'https://github.com/sabnzbd/sabnzbd.git'
dest: /apps/sabnzbd
version: master
clone: yes
update: yes

- name: TASK | Change owner of SAB to usenet
file:
path: /apps
state: directory
owner: usenet
group: usenet
mode: 0755

- name: TASK | Install systemd unit file for SAB
template: src="templates/sabnzbd.service.j2" dest="/etc/systemd/system/sabnzbd.service" mode=0644

- name: TASK | Copy SAB config
template: src="templates/sabnzbd_config.ini.j2" dest="/apps/data/.sabnzbd/sabnzbd_config.ini" mode=0644

- name: TASK | Enable and start SAB service
systemd:
name: sabnzbd
enabled: yes
state: started

- name: TASK | Install filebeat
yum:
name: https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.2-x86_64.rpm
state: present
update_cache: yes

- name: TASK | Copy filebeat config file
template: src="templates/filebeat.yml.j2" dest="/etc/filebeat/filebeat.yml" mode=0644

- name: TASK | Enable and start filebeat
systemd:
name: filebeat
enabled: yes
state: restarted

- name: TASK | Start updatedb for locate
shell: "updatedb >/dev/null 2>&1 &"

Technology used:

TerraformAnsibleSabnzbdElasticsearchSonarr

Using Ansible to build a high availablity Sabnzbd usenet downloader

https://chrisbergeron.com/2017/10/08/high_performance_sabnzbd/

Author

Chris Bergeron

Posted on

10-08-2017

Updated on

05-02-2021

Licensed under

Comments