Context

Having started out in the world of Napster & Limewire, I’ve always relied on public sources. It wasn’t until in the early '10s that I lucked into a Gazelle-based tracker that was started by some fellow community members. Unfortunately, I wasn’t paying enough attention when they closed shop and didn’t know how to move elsewhere. Combined with some life circumstances I gave up the pursuit for the time being.

It wasn’t until recently that a friend was kind enough to help me get back and introduced me to current state of automation. Over the course of a few months, I’ve since built up the attached systems. I’ve been having an absolute blast learning and am very impressed with all of the contributions!

After all of the updates due to BF deals, I put together the attached diagram as it was starting to get too complex to keep all of the interactions in my head. 😅

Setup

  • All of the services run in Docker containers.
  • Each container is a separate Compose file managed by Systemd.
  • The system itself is in a VM running on my home server (both Arch, btw).
  • Tailscale is used for remote access to the local network.
  • ProtonVPN is managed by Gluetun and uses a separate network for isolating services.

Questions

  • What am I missing or can be improved?
  • Is there a better way to document?
  • What do you do differently that might be beneficial?

Thoughts

  • I had Calibre set up at one point, but I really don’t like how it tracks files by renaming them. I have been considering trying to automate with the CLI instead, but haven’t gotten around to it yet.
  • I’ve been toying with the idea of creating a file-arr for analyzing disk usage, performing common operations, and exposing a web-based upload/download client so I don’t have to mount the volume everywhere.
  • Similarly, I’m interested in a way to aggregate logs/notifications/metrics. I’m aware of Notifiarr, but would prefer a self-hosted version.
  • I just set up Last FM scrobbling so I don’t have any data yet. I’m hoping to use that for discovery and if possible, playlist syncing or auto-generation.

Notes

  • Diagram was made using D2lang.
  • Some of the connections have been simplified to improve readability / routing.
  • Some services have been redacted out of an abundance of caution.
  • I know VPN with Usenet isn’t necessary, but it’s easier to keep it consistent.

Also, thanks for the recommendations to check out deemix/Deezer. That worked really well! 😀

Edit: HQ version of diagram

  • db0M
    link
    fedilink
    English
    05 months ago

    Very nice. Can you share a docker-compose.yml for others to replicate this? Also your diagram could be a bit higher quality.

    • XyreOP
      link
      fedilink
      English
      0
      edit-2
      5 months ago

      Each service is a separate docker-compose.yml, but they are more-or-less the same as the example configs provided by each service. I did it this way as opposed to a single file to make it easier to add/remove services following this pattern.

      I do have a higher quality version of the diagram, but had to downsize it a lot to get pictrs to accept it…

      • db0M
        link
        fedilink
        English
        05 months ago

        Ah your instance must be limiting the size. lemmy.dbzer0.com allows you to upload anything and just downscales to 1024px max dimention. You can also just host on imgur etc.

    • XyreOP
      link
      fedilink
      English
      05 months ago

      I get what they’re saying and it may be ‘technically correct’, but the issue is more nuanced than that. In my experience, some trackers have strict requirements or restricted auth tokens (e.g. can’t browse & download from different IPs). Proxying may be the solution, but I’d have to look at how it decides what traffic gets routed where.

  • @navigatron@beehaw.org
    link
    fedilink
    English
    05 months ago

    You’re running docker inside a vm? Why?

    The first thing I would do is learn the 5-layer OSI model for networking. (The 7-layer is more common, but wrong). Start thinking of things in terms of services and layers. Make a diagram for each layer (or just the important layers. Layers 3 and up.)

    If you can stomach it, learn network namespaces. It lets you partition services between network stacks without container overhead.

    Using a vm or docker for isolation is perfectly fine, but don’t use both. Either throw docker on your host or put them all in as systemd services on a vm.

    • XyreOP
      link
      fedilink
      English
      05 months ago

      The server itself is running nothing but the hypervisor. I have a few VMs running on it that makes it easy provision isolated environments. Additionally, it’s made it easy to snapshot a VM before performing maintenance in case I need to roll back. The containers provide isolation from the environment itself in the event of a service gone awry.

      Coming from cloud environments where everything is a VM, I’m not sure what issues you’re referring to. The performance penalty is almost non-existent while the benefits are plenty.

      • 1337
        link
        fedilink
        English
        05 months ago

        I recently rebuilt my home server using containers instead of (qemu/KVM) VMs and I notice a performance benefit in some areas. Although I just use systemd-nspawn containers rather than docker as I don’t really see the need to install 3rd party software for a feature already installed on my OS.

        I handle snapshots by using btrfs. Works great

    • XyreOP
      link
      fedilink
      English
      0
      edit-2
      5 months ago

      The problem I’ve found is that the services will query indexers and that not all of the trackers allow you to use multiple IPs. This is where I found it easier to make all outbound requests go through the VPN so I didn’t get in trouble. It’s also why I have the Firefox container set up inside the network with it exposed over the local network as a VNC session. So I can browse the sites while maintaining a single IP.

      I do have qbittorrent set up with a kill switch on the VPN interface managed by Gluetun.

  • @Rizoid@programming.dev
    link
    fedilink
    English
    0
    edit-2
    5 months ago

    Given what you’ve got running I only really recommend, as other have, portainer. It’s made my life so much easier. Edited this since I saw you have homarr and I must’ve missed it the first time.

    • XyreOP
      link
      fedilink
      English
      0
      edit-2
      5 months ago

      For a long time, that was the case. Then the greed nation attacked. Now they’ve reproduced the cable model on the web and more than half of which have terrible clients / infrastructure.

      If I could pay for a single service that operated similar to this setup:

      • Tell it what I’d like to watch while also displaying similar content for discovery.
      • Tracking progress in every show (while not forgetting!).
      • Not losing content I have been watching as it’s now in ‘another castle’.
      • A single place to view all tracked shows rather than loading each service individually.

      I probably would sign up for it as that’s what was so successful for Netflix until all of the studios thought they could do better. And now the consumer has to suffer the consequences.

    • @ElderWendigo@sh.itjust.works
      link
      fedilink
      English
      05 months ago

      Maybe if you’re new to all this and/or have no interest. But if you’ve been tinkering for more than a few years, it’s just a PC version of project car. It’s something you tinker with on the weekends, adding and refining as you go. I would never be able to negotiate multiple streaming services in a unified way to my satisfaction. So it’s not as if I really even have the option of paying for what I actually want from a service.

    • @retro@infosec.pub
      link
      fedilink
      English
      05 months ago

      You can simplify it way down to Kodi + RD and have your own streaming service. This looks more like a hobby though to get every little thing just right.

    • @bobs_monkey@lemm.ee
      link
      fedilink
      English
      05 months ago

      This is taking it above and beyond I’ll agree. I’m still in the old times where I’m manually finding my movie/show/etc and doing all the leg work by hand, only because I haven’t had time to learn all the modern stuff. But things like this are a great resource to get up to speed.

      • XyreOP
        link
        fedilink
        English
        05 months ago

        If you have the time and resources, I highly recommend it. Once it’s all running it becomes mostly a ‘set it and forget it’ situation. You don’t have to remember to scroll through pages of search results to find content. It’ll automatically grab them for you based on your configured quality profile (or upgrade it to better quality). Additionally, you can easily stream it to any devices in our home network (or remote with a VPN).

        You don’t have to do it all at once. Start with a single service you’re interested in and slowly add more over time.

        • @bobs_monkey@lemm.ee
          link
          fedilink
          English
          05 months ago

          Nah I feel ya, I’ve been seeing all the various configurations since I came to Lemmy. I’ve just had a hell of a summer work-wise (6-7 days a week, yay being a small business owner) plus a good amount of travel for weddings and the like, so I just haven’t had the time to sit down with it all. One of my clients has a whole bunch of 2 year old metal and SAS drives they’re giving me in a few weeks once they get underway with operations due to a corporate upgrade, and once I slow down for the winter season I fully intend on diving into this.

          I’ve had a Plex server running on FreeBSD for years, I just haven’t set it up since moving back in June. But I’m getting really tired of all the bullshit from these streaming services, and I’m looking forward to taking back control of my entertainment. I just have to make it palatable to my wife lol.

        • @Appoxo@lemmy.dbzer0.com
          link
          fedilink
          English
          05 months ago

          To me the *arrs are great enough for the wishlist/todo factor alone.
          The download/management feature is just the cherry on top.