Honeypot.net Profile Photo

Honeypot.net

Extracting chaos from order

devops

    ∞

    I read Yann Esposito’s blog post, How I protect my forgejo instance from AI Web Crawlers, and think that’s a great idea. My main concern with the crawlers is that they’re horribly written and behave poorly. My own Forgejo server was getting slammed with about 600,000 crawler requests per day. This little server is where I share tiny personal projects like my Advent of Code solutions. I wouldn’t expect any project there to get more than a handful of queries per day, but suddenly I was serving 10 requests per second. That’s not a lot compared to any popular website, but that’s a lot for this service, on this tiny VPS, on my shoestring budget.

    Worse, the traffic patterns were flat-out abusive. All the content on this site comprises nearly static Git repositories. The scrapers try things like:

    • For every Git commit, fetch the version of every file in the repository at that commit.
    • See git blame for every file at every commit.
    • Attempt to download the archive of each repo at every commit.
    • Run every possible pull request search filter combination.
    • Run every possible issue search filter combination.
    • Fetch each of those URLs at random from some residential IP in Brazil that had not ever accessed my server before.

    My first huge success at cutting through the flurry of bad traffic was with deploying Anubis. You know those anime girl pictures you see before accessing lots of web pages now? Well, those are part of a highly effective bot blocker. There’s a reason you’re seeing more and more of them.

    And this morning, I also adapted Yann’s idea for my server which runs behind Caddy instead of Nginx. I made a file named /etc/caddy/shibboleth like this (but with the cookie name suitably altered to a random local value):

    @needs_cookie {
    not {
        header User-Agent *git/*
    }
    not {
        header User-Agent *git-lfs/*
    }
    not {
        header X-Runner-Uuid *
    }
    not {
        header Cookie *Yogsototh_opens_the_door=1*
    }
}

handle @needs_cookie {
    header Content-Type text/html
    respond 418 {
        body `<script>document.cookie = "Yogsototh_opens_the_door=1; Path=/;"; window.location.reload();</script>`
     }
}

    Note the extra X-Runner-Uuid line that Yann did’t have. This allows my Forgejo Action Runners to connect without going through the cookie handshake.

    Then I added a line to the configurations for services I wanted to protect, like:

    myserver.example.com {
    root * /path/to/files
    ...
    import shibboleth
}

    This way I can easily reuse the snippet for any of those services.

    Thanks for the great idea, Yann!

    ∞

    Mise + fnox + macOS Keychain is a great combo for running Ansible with stored, encrypted secrets.

    In mise.toml:

    [env]
DBSERVER_PW = { value = "{{ exec(command='fnox get DBSERVER') }}", tools = true, redact = true }

    In Ansible’s host vars:

    my_servers:
  hosts:
    dbserver:
      ansible_become_password: "{{ lookup('ansible.builtin.env', 'DBSERVER_PW') }}"

    Now you can run ansible-playbook and friends without hardcoding your sudo passwords anywhere!

    ∞

    I’m using Ansible to manage a small fleet of Raspberry Pis. I’d been using the copy module to set a value in /sys:

    - name: Enable compressed swap now (with zsmalloc)
  become: true
  ansible.builtin.copy:
    content: zsmalloc
    dest: /sys/module/zswap/parameters/zpool
    unsafe_writes: true

    But that always reports that the file changed, even if it already had that value. Today I got the lineinfile module to update the value. This only says the value changed when it actually did:

    - name: Enable compressed swap now (with zsmalloc)
  become: true
  ansible.builtin.lineinfile:
    path: /sys/module/zswap/parameters/zpool
    regexp: "^(?!zsmalloc).*$"
    line: zsmalloc
    unsafe_writes: true

    Since these “files” only have one line, this uses regexp to see if that line already matches the expected value. If so, it moves on. If not, it writes the new value.

    Note: unsafe_writes: true is there because you can’t write arbitrary filenames into /sys and then mv them into place. You have to write directly to the target “file”.

    ∞

    I’ve spent too much of this weekend writing Ansible to make all my Raspberry Pis similar.

    This might say more than I’d wish about my nerd level, and about how many tiny computers I have laying around.

    ∞

    I’ve been using Just for a while as a task runner. It’s similar to Make, but optimized for developer ergonomics with a vastly simpler syntax and a wonderful CLI. I’d also been using Mise for other environment management things, such as installing specific versions of Python and NPM and other tools in a project directory.

    Someone introduced me to Mise’s own newish task runner, and it just might win me over from Just for most things:

    1. Instead of using 2 tools, I can use 1.
    2. Just still feels nicer to me, perhaps because I’m more used to it, but Mise is good enough that I don’t think I’d miss the extra features.
    3. Mise lets you write tasks in separate files, which lets any editor handle them well without having to support justfile syntax, but still shares a CLI with inline tasks.

    I like it.

    ∞

    Forgejo Runner in rootless Podman on Debian

    I wanted to experiment with Forgejo’s Actions as a DIY alternative to GitHub Actions, using a nearby Raspberry Pi as a build server. I also wanted to deviate slightly from their Runner installation process by executing the Runner and rootless Podman as a regular, non-privileged user and without using the system-level systemctl. It was pretty easy once I wrapped my head around it.

    1. Set up the runner user. Since I was using Podman, not Docker, I didn’t have to add it to the docker group. As root:
    root# useradd --create-home forgejo-runner

    This created user number 1001 on my system. Remember that number later when it’s time to configure systemd.

    1. Allow that user to run commands via systemctl without logging in and launching them manually:
    root# loginctl enable-linger forgejo-runner
    1. Use machinectl instead of su to become the forgejo-runner user. Without this, most systemd commands will fail with the Failed to connect to bus: No medium found message. I’m certain there’s a way to get su or sudo to play nicely with dbus but I had more interesting problems to solve today than this.
    root# apt install systemd-container
root# machinectl shell forgejo-runner@
    1. Run podman-system-service as the forgejo-runner user:
    $ systemctl --user enable podman.socket
$ systemctl --user start podman.socket
    1. Run the forgejo-runner program as the forgejo-runner user. I lightly modified the standard forgejo-runner.service file:
    $ cat > .config/systemd/user/forgejo-runner.service <<EOHD
[Unit]
Description=Forgejo Runner
Documentation=https://forgejo.org/docs/latest/admin/actions/
After=podman.socket

[Service]
ExecStart=/usr/local/bin/forgejo-runner daemon
ExecReload=/bin/kill -s HUP $MAINPID
# 1001 is the forgejo-runner user's UID
Environment="DOCKER_HOST=unix:///run/user/1001/podman/podman.sock"

# This user and working directory must already exist
WorkingDirectory=/home/forgejo-runner
Restart=on-failure
TimeoutSec=0
RestartSec=10

[Install]
WantedBy=default.target
EOHD
$ systemctl --user daemon-reload
$ systemctl --user enable forgejo-runner.service
$ systemctl --user start forgejo-runner.service

    I rebooted my RPi to make sure it would start on its own and it did. Yay! Now I can run Forgejo Actions on my little server and everything works as documented.