offgpt.org Enter DIY Licence Blog Privacy Vision
DE IT FR

Here comes a general description of the technical setup of the instance.

Copies of this structure as outlined within the licence are encouraged to build on this setup, but should also try to mutate their versions in order to pile up knowledge, reduce the overall attack surface and promote diversity.

The instance consists of two physical servers: One in North Italy and one in South Germany.

The one in Italy has the latest Proxmox Linux installed. On the Proxmox, three virtual servers are running:

  • qemu kernel virtualized minimized Debian Linux running a wireguard client
  • qemu kernel virtualized, minimized and LUKS encrypted Alpine Linux (6 of them) running a LXC Debian each, running a dockerized Debian Linux each with the open webui interface in it.
  • lxc virtualized Ubuntu Linux running the model on the GPU

    • The Italian server has no open port, and in addition has not even a public IP. It punches a hole through the provider's NAT and connects via wireguard to the Server in South Germany, merging subnets.

    On the Server in South Germany, two virtual Servers are running:

  • dockerized Alpine-slim Linux with the wireguard knot on it
  • dockerized and hardened Alpine-slim Linux with an nginx webserver running as reverse proxy

    • The dynamical dns of the reachable server in south germany is done by deSEC, an association in Berlin.

    The Backups

    The Backups are encrypted (a second time) by the following code:

    openssl enc -aes-256-cbc \
    -md sha512 \
    -pbkdf2 \
    -iter 1000000 \
    -salt \
    -pass file:oi.txt \
    -in /mnt/backup-hdd/dump/vzdump-vmid-date.vma.zst \
    -out /mnt/backup-hdd/dump/vzdump-vmid-date.vma.zst.enc

    before they are rsynced over wireguard and pub key ssh.

    The overall architecture of offgpt can be summarized with the following graphic. The pink keys indicate that the backups get encrypted (another time) before being sent. The forgejo server in singapore only gets non user data related configurations backuped into private repositories:







    Please contact

    luca[ät]offgpt[dot]org

    if you want other shell scripts or the actual configurations.

    The whole setup is open source.
    In the sense that I take all time needed to teach the How To for free (as in free beer).

    As my time is limited, the teaching is limited to the persons that value the licence of redistribution. Which in short has the only purpose to limit and redistribute the number of users.

    Because the main work of this whole setup is and was the very serving of many users. Which is not done with some right commands - or right words to get the big LLM outputs unlocked - but demands time and recurring activity.

    Every teaching of any infrastructure setup has a big beneficial outcome for this very infrastructure. Which in the case of offgpt, being based on GNU and free software licences, is well documented and a complex collective achievement.

    We stand on the shoulders of giants, long live open source.