# Bucket & Code Server IDE: user guide

Bucket & Code Server IDE (QPKG `bucketvs`) keeps source code and files on a QNAP NAS and builds software there.
It brings its own web server, tools and accounts.

This guide describes the package as it runs on the NAS.

## Contents

- [What runs on the NAS](#what-runs-on-the-nas)
- [Installing the package](#installing-the-package)
- [First sign-in](#first-sign-in)
- [Working in the panel](#working-in-the-panel)
- [Accounts](#accounts)
- [The data folder](#the-data-folder)
- [Network](#network)
- [Behind a reverse proxy (Plesk)](#behind-a-reverse-proxy-plesk)
- [Hostnames for the modules](#hostnames-for-the-modules)
- [Publishing on the internet (Cloudflare Tunnel)](#publishing-on-the-internet-cloudflare-tunnel)
- [Modules](#modules)
- [Git](#git)
- [CI runners](#ci-runners)
- [SVN and Bucket](#svn-and-bucket)
- [Editor and AI tools](#editor-and-ai-tools)
- [Toolchains and environments](#toolchains-and-environments)
- [App Builder](#app-builder)
- [Package repository](#package-repository)
- [Kernel workshop](#kernel-workshop)
- [Cron jobs and operations](#cron-jobs-and-operations)
- [Backups](#backups)
- [Notices](#notices)
- [Logs and the support bundle](#logs-and-the-support-bundle)
- [Updates](#updates)
- [The bvs command](#the-bvs-command)
- [Files and folders](#files-and-folders)

## What runs on the NAS

| Part | What it does | Where it listens |
|---|---|---|
| nginx | the only door on the LAN: serves the panel and passes each path to its module | 9950 (HTTP), 9951 (HTTPS) |
| bvsd | starts and watches every other process, keeps the accounts and settings, runs jobs and cron jobs, answers the panel | two unix sockets in `/usr/bin/bucketvs/var/run/` |
| Git (Forgejo) | Git hosting with a web interface | a unix socket in `/usr/bin/bucketvs/var/run/forgejo/`, and 9952 for Git over SSH |
| SVN and Bucket (httpd) | Subversion repositories and WebDAV folders with history | 127.0.0.1:9961, or the next free port when another program holds it |
| Editor (code-server) | VS Code in the browser, one instance per account | a unix socket per account in `/usr/bin/bucketvs/var/run/ide/` |
| CI | the Forgejo Actions runner | no port of its own; nginx listens on 127.0.0.1:9960 for it while the module is on, or on the next free port when another program holds 9960 |

Core (nginx, bvsd, the panel, cron jobs, toolchains) is always on. Git, CI, SVN and Bucket, and the Editor are
modules that an admin switches on and off. On every address of the NAS the modules answer under their paths:
`/git/`, `/svn/`, `/bucket/`, `/ide/`. The panel is at `/`, and this guide at `/guide/`.

While the package runs, its programs are in `/usr/bin/bucketvs/bin`. Everything the package keeps (accounts, settings,
repositories, environments, logs) lives in one folder on a data volume: the [data folder](#the-data-folder).

## Installing the package

The package needs QTS 6.0 or QuTS hero 6.0 or newer, on an x86_64 or ARM 64-bit NAS; there is one package per
architecture (`bucketvs_<version>_x86_64.qpkg`, `bucketvs_<version>_arm_64.qpkg`). It is code-signed: install it
from App Center (a repository that lists it, or **Install Manually** with the downloaded file). App Center checks
the signature; there is no need to allow apps without a valid signature.

The installation creates the service user `bucketvs` (for Forgejo and httpd), the link `/usr/bin/bucketvs` to the
install folder, and starts bvsd. System Logs names the data folder.

### Where the data folder starts

On the first start the data folder is `BucketVS` on the volume the package is installed on, for example
`/share/CACHEDEV1_DATA/BucketVS` (QTS) or `/share/ZFS2_DATA/BucketVS` (QuTS hero). It can be moved to another
volume later from the panel ([Moving the data folder](#moving-the-data-folder)).

The package remembers its data folder in `/etc/config/bucketvs-data-root`, which outlives the package. Removing the
package keeps the data folder and this file, so an installation after a removal starts on the same data and the same
accounts. A start never depends on a folder that is not there. When the data folder is gone (deleted, on a disk that was
replaced, or on a volume that is offline at that moment), the start makes `BucketVS` on the package's own volume, a new
folder unless one of that name is already there, and on a new one the [first sign-in](#first-sign-in) sets the
administrator's password. This holds after a reinstall and for an installed package, at boot too. System Logs has a
warning about it, and `.prev` files keep the old path: `/etc/config/bucketvs-data-root.prev`, and for an installed
package also `etc/data-root.prev` in the install folder. If the old folder was on a volume that was only offline at that
moment, bring the volume online, stop the package, copy that file over `etc/data-root` in the install folder (for
example `/share/CACHEDEV1_DATA/.qpkg/bucketvs/etc/data-root`) and start the package: it runs on the old data again.

### Opening the panel from App Center

**Open** on the package in App Center opens the panel on port 9950 (HTTP) or 9951 (HTTPS). When the ports are
changed in the panel ([Ports](#ports)), the package tells App Center the new ones at its next start.

### Auto launch from an SSH session

While the package's switch in App Center is on, the NAS starts the package at boot. In an SSH session on the NAS, as
admin, the service script sets the same switch:

```sh
/etc/init.d/bucketvs.sh disable   # the next start is refused, at boot as well
/etc/init.d/bucketvs.sh enable    # the package starts at boot again
```

Each command writes the switch (`Enable` in `/etc/config/qpkg.conf`) and nothing else, prints what it set and writes
the same line to System Logs. A running package keeps running after `disable`, and `enable` starts nothing: `stop` and
`start` do that, and `start` refuses while the package is disabled. `status` says whether bvsd runs.

## First sign-in

Open the panel at `https://<NAS address>:9951/`, or `http://<NAS address>:9950/`. The HTTPS address uses a
self-signed certificate at first, so the browser warns until the device trusts it; the
[certificate](#the-certificate) section shows how to compare its fingerprint.

A new installation has no account. Until the first administrator has a password, the panel opens on one screen,
"Set the administrator password", for the account `administrator`. Type a password twice: at least 10 characters
and at most 72 bytes. Saving it creates the account and signs you in.

```mermaid
flowchart TD
    start["The package starts<br/>with no account"] --> screen["Set the administrator password<br/>60 minutes, local network only"]
    screen -->|"saved"| panel["administrator signed in"]
    screen -->|"60 minutes passed"| restart["Restart the package in App Center:<br/>a new 60-minute window"]
    restart --> screen
```

- The screen accepts a password for 60 minutes after the package starts. After that it says the setup window has
  closed; restarting Bucket & Code Server IDE in App Center opens a new 60-minute window.
- It accepts a password only from the local network: from the NAS itself, or from an address in a private range
  (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `169.254.0.0/16`, `fc00::/7`, `fe80::/10`).
- While the screen is open, the QTS System Logs show an information entry for each LAN address of the NAS, such
  as:

  ```text
  bucketvs: no administrator yet: open https://192.168.1.20:9951/ within 60 minutes to set the administrator's password
  ```

- It refuses a request that came through a proxy (one that carries `X-Forwarded-For`, `X-Forwarded-Host`,
  `X-Forwarded-Proto`, `X-Real-IP` or `Forwarded`), even from a proxy on the LAN: open the panel directly from a
  computer in the local network to set the password.

From an SSH session on the NAS, `/usr/bin/bucketvs/bin/bvs user password administrator` creates the account when
there is none yet and prints a random password; the panel then asks for a password of your own at the first
sign-in ([When an admin is locked out](#when-an-admin-is-locked-out)).

After the first sign-in the Overview says that only the core is running. Each module's tile says what the module gives
and has a button that turns it on, such as **Turn on Git**: turn on the ones you need ([Modules](#modules)).

## Working in the panel

The panel works like a code editor: every page opens in a tab, and a command palette reaches the pages and the
actions by name.

- **Activity bar** (the icons at the left). Each icon opens its page and shows its tree in the side bar: the modules,
  repositories and buckets, apps, packages, environments, jobs and backups. A dot on an icon marks a module in
  trouble, a failed build or a failed backup. Clicking the icon of the page in front hides the side bar; Ctrl+B does
  the same.
- **Tabs.** A page opened with a single click is a preview, in italics: the next page opens in its place. Edit the
  page or double-click its tab to keep it. The tabs come back after a reload, for each account in each browser. In a
  window at least 1280 pixels wide, **Move the tab to the other side** (`Ctrl+\`) shows two pages side by side.
- **Objects and their menus.** One click on a row of the side bar, a row of a table or a card opens that object: its
  page comes to the front with the object in a green ring, or with its details open. The address names the object, so
  back, forward and a copied link return to it. A double click keeps the tab. A right click, Shift+F10 or a long press
  on a touch screen opens the object's menu: the ways of opening it (in this tab, in a kept tab, on the other side, in
  a new browser tab), what its page can do with it, and copies of its name and address. A tab's menu closes it, the
  other tabs, the tabs to its right or all of them. Only a control marked ↗ leaves the panel. The **Administration**
  icon opens the System page.
- **Command palette.** Ctrl+Shift+P (or Ctrl+K, or F1) lists the commands your role may run: open a page, build or
  publish an app, run a backup, restart a module, open a shell in an environment. Ctrl+P goes to a page, an app, a
  repository, an environment or a package by a few letters of its name.
- **Bottom panel** (Ctrl+J). **Jobs** shows the running and recent jobs with their progress and log. **Problems**
  lists what the modules report: failed builds, disabled backup targets, modules in trouble, a certificate about to
  expire, and the ELF inspector's findings for the app in front. **Output** follows the job you started last.
- **Moving and resizing.** Drag an edge to change the size of what it borders: the side bar, the bottom panel, two
  pages side by side, or two parts of a page side by side, such as a wizard and its summary or a project's file list
  and the file. A double click on an edge puts it back. Drag a dialog by its title to move it, and by an edge or its
  bottom right corner to resize it; a double click on its title puts it back in the middle. To put the bottom panel
  beside the pages, drag its header to the right edge of the editor or click the button next to its ×; drag it to the
  bottom edge, or click the button again, to put it back. A tab can be dragged to another place in the tab strip or
  onto the other side. The panel keeps each size and place for your account in this browser, and **Reset layout** (in
  the command palette, or under your account's button at the bottom of the activity bar) puts them all back. From the
  keyboard, the arrows move a focused edge (Shift takes bigger steps, Home puts it back), Ctrl+Shift+Left and Right
  move a focused tab, and a dialog has a bar at its top and a grip at its bottom right that the arrows move and resize
  it with. On a phone, dialogs and the bottom panel stay where they are.
- **Status bar.** Module states, running jobs, problems, free space on the data volume, the certificate, the account
  that is signed in, the language (the editor takes it at its next start, English for Swedish and Dutch, and Git follows it unless you pick another language there), the theme (dark, light, as the system, deep blue, graphite or paper: it belongs to your account, and the editor and Git wear it too, unless you pick another theme there) and the version. Each item opens the
  place to look closer.
- **Keyboard.** Ctrl+/ or ? lists the shortcuts. A browser tab keeps Ctrl+W and Ctrl+Tab for itself, so the panel
  closes a tab with Alt+W and switches tabs with Alt+PageDown and Alt+PageUp.
- **On a phone** the first activities are at the bottom, **More** opens the side bar with the rest, and the open
  pages are a list at the top.

## Accounts

The panel has its own accounts. QTS accounts do not sign in to it.

### Roles

| Role | What it may do |
|---|---|
| viewer | reads the panel; the Backups, Storage, Users and Logs pages are for admins. In Forgejo it creates, forks and imports no repository, creates no organization and never pushes; it reads the SVN repositories and buckets it has access to and writes to none |
| member | reads what a viewer reads, and writes: it pushes to the Git repositories it is given write access to, writes to the SVN repositories and buckets it has read and write on, and restores older versions of their files. In Forgejo it creates, forks and imports no repository and creates no organization, and it cannot push to a repository that an account below developer owns or administers, nor, while the CI module is on, to one whose Actions are on. It gets no editor, cron jobs or builds |
| developer | also opens the editor, reads the `bvs use` lines of installed toolchains, creates, changes and runs its own cron jobs, creates and builds apps, publishes packages to the repository (an admin lists them), checks kernel modules, and creates Git repositories whose workflows run on the NAS while the CI module is on |
| admin | everything: accounts, settings, storage, network, modules, toolchains, environments, every cron job, activating apps, loading kernel modules, listing packages in the repository |

The developer role is root on the NAS: its cron commands run as root, so does its editor's terminal and, while the CI
module is on, the workflows of the repositories it can push to. Give it only to people you would give the NAS's
admin password. For someone who only commits to repositories, the member role is enough.

Under **Your account** each person reads their own role and what it allows, with a link to this section. A page above
someone's role says which role it needs, and what only an admin changes says so where it is shown. Network, Git and CI
show their settings to everyone else as values, with "Not set" where one is empty.

Workflows run as root on the NAS, so bvsd keeps Actions off in every Git repository that a member or a viewer owns or
can write to: giving such an account write access to a repository turns that repository's Actions off within a
minute, and the Git page lists those repositories for admins. To use Actions there, first remove that write access in
Forgejo, or move a repository such an account owns to a developer's account.

### Creating and changing accounts

An admin manages accounts on the Users page.

- A new account starts with the viewer role. Developer and admin are chosen on purpose: the developer role is root on
  the NAS (above).
- A new account starts with no cron jobs, even when an account of the same name had some.
- A name has 2 to 32 characters: a lower-case letter first, then lower-case letters, digits, `.`, `_` or `-`. Two
  of `.`, `_`, `-` in a row, or one at the end, are refused. So are the names Forgejo reserves (`admin`, `api`,
  `user` and others), `bucketvs` (the package's system user), `cli` and `system` (bvsd uses both itself).
- "Must choose their own password at the first sign-in" makes the account set its own password before it can do
  anything else.
- Disabling an account ends its sessions at once and stops its API tokens. Its data and its
  [cron jobs](#scheduled-jobs) stay, but the jobs do not run while it is disabled, and a run of one that is going is
  stopped.
- Giving an account the member or viewer role does the same to its cron jobs: they stay, do not run while its role is
  below developer, and a run that is going is stopped.
- A viewer never pushes to Git, also where it was given write access in Forgejo. An account that should push gets the
  member role.
- Deleting an account removes it with its sessions and API tokens. Its home folder moves to
  `trash/home/<name>-<time>` in the data folder, where the "Empty the trash" job removes it after 7 days, and its
  `tmp/tmux-<name>` folder is removed. Its cron jobs are deleted with their runs and logs.
- The last enabled admin cannot be demoted, disabled or deleted.
- "Set a new password" ends the account's sessions and revokes its API tokens.

### Passwords and sessions

- A password has at least 10 characters and at most 72 bytes.
- Changing your own password (under Your account) ends your other sessions and revokes your API tokens. The
  session you changed it in stays signed in.
- A session ends 12 hours after its last request, and 7 days after sign-in in any case. An account keeps at most 32
  sessions; signing in once more ends the one used least recently.
- The session cookie is `HttpOnly` and `SameSite=Lax`. It is marked `Secure` when the browser reached the NAS over
  HTTPS, directly or through a [trusted proxy](#trusted-proxies).

### Wrong passwords

- Five failed sign-ins within 15 minutes from one address lock that address for 15 minutes, even with the right
  password. An IPv6 client counts as its /64 network.
- Five failures for one account from one address lock that account from that address for 15 minutes.
- Failures for one account from many addresses slow down its next sign-ins: 125 ms after one failure, doubling up
  to 2 seconds. From the local network they never lock it.
- From the internet (any address outside the local network), 20 failed sign-ins for one account since its last
  successful one lock it there, whichever addresses they came from, and the right password is then refused there
  too. The account still signs in from the local network. An admin unlocks it on the Users page (**Unlock**) or with
  `bvs user unlock <name>`, or gives it a new password. The number is set on the Users page under **Sign-ins from
  the internet**; 0 turns this lock off.
- The counts are kept in memory, so a restart of bvsd clears them; a lock stays until it is ended.

Behind a proxy, the address counted is the one the proxy reports, but only when the proxy is a
[trusted proxy](#trusted-proxies). Otherwise every client counts as the proxy's own address, and five wrong
passwords from anyone lock everyone out for 15 minutes.

### Two-factor authentication

Each account turns it on for itself under Your account: scan the QR code with an authenticator app (or type the
key), then enter the 6-digit code the app shows. After that, sign-in asks for a code after the password.

- Codes follow RFC 6238: 6 digits, a new code every 30 seconds. A code 30 seconds early or late is accepted. The
  same code is never accepted twice.
- If a fresh code fails, the phone's clock is probably off; set it to automatic time.
- Turning it off needs the account's password.
- For a lost phone, an admin turns it off on the Users page. The password alone then signs in.

### API tokens

Scripts and CI use API tokens instead of a password. Under Your account, "New token" makes one:

- It looks like `bvs_` followed by 40 hexadecimal digits, and it is shown once. bvsd keeps only its hash.
- A script sends it as `Authorization: Bearer bvs_...`.
- Its scopes are `read`, `write` and `admin`, which stand for the roles viewer, developer and admin. A token acts
  with its highest scope, never above its owner's current role, so demoting an account also demotes its tokens. No
  scope stands for member: a member makes `read` tokens.
- A new password revokes all of the owner's tokens. A disabled owner's tokens stop working.
- A token cannot change passwords or two-factor authentication, and cannot make or delete tokens: those need a signed-in
  session. A leaked token therefore cannot lock its owner out.

### When an admin is locked out

The NAS admin can reset any account from an SSH session on the NAS. The `bvs` command talks to bvsd over a socket
that only root can open (`/usr/bin/bucketvs/var/run/bvsd-cli.sock`, mode 600):

```bash
/usr/bin/bucketvs/bin/bvs user list
/usr/bin/bucketvs/bin/bvs user password administrator
/usr/bin/bucketvs/bin/bvs user totp-reset administrator
/usr/bin/bucketvs/bin/bvs user unlock administrator
```

- `bvs user password <name>` sets a new random password of 20 characters and prints it. The account must choose
  its own at the next sign-in. Its sessions end and its API tokens are revoked. On a new installation with no
  account yet, `bvs user password administrator` creates the account.
- `bvs user totp-reset <name>` turns two-factor authentication off for the account.
- `bvs user unlock <name>` ends the account's lock after too many wrong passwords from the internet.

Every request on that socket acts as the built-in admin `cli` and is written to `log/audit.log` in the data folder.
bvsd must be running for these commands.

## The data folder

The data folder is the folder that holds everything the package keeps. Its path is the one line in
`/usr/bin/bucketvs/etc/data-root`. The Storage page shows it, with the volume it is on and the free space there.

bvsd starts only on a data folder that is a folder on a real volume. The package's start checks this first: a missing
folder, or one on a tmpfs or ramfs, gives a new data folder on the install volume, as [Where the data folder
starts](#where-the-data-folder-starts) describes. A volume that is offline leaves its mount point behind as an empty
folder in the NAS's RAM, and settings written there would be gone after the next reboot. bvsd itself refuses such a
folder only when it is started by hand, and the reason then goes to the QTS System Logs.

### What bvsd gives every process

QTS keeps `/tmp` in a small RAM disk and `/root` on a RAM disk that a reboot empties. So every process bvsd starts
(module programs, cron jobs, installs) gets its temporary files and its home folder in the data folder:

| Variable | Value |
|---|---|
| `HOME` | `<data folder>/home/<user>` |
| `TMPDIR` | `<data folder>/tmp` |
| `XDG_CACHE_HOME`, `XDG_CONFIG_HOME` | `.cache` and `.config` in that home |
| `npm_config_cache` | `.npm` in that home |
| `PATH` | `~/.local/bin`, `~/.npm-global/bin`, then the package's `bin`, `sbin` and `node/bin`, then the system's folders |
| `SSL_CERT_FILE`, `GIT_SSL_CAINFO`, `CURL_CA_BUNDLE` | the package's CA bundle, `/usr/bin/bucketvs/etc/ssl/cacert.pem` |
| `TZ` | the NAS's time zone, as set in QTS |
| `SHELL`, `EDITOR` | the package's `bash` and `nano` |
| `LANG`, `LESSCHARSET` | `C.UTF-8` and `utf-8`, so file names with non-English letters show as they are |

### Moving the data folder

The Storage page moves the whole data folder to another folder under `/share`. Pick a shared folder or a volume;
the data goes into a new folder named `BucketVS` inside it. To use a folder itself, open an empty folder named
`BucketVS`. Shared folders are created in QTS (Control Panel, Shared Folders); inside a shared folder or a
volume, the page can create a new folder. The picker opens where the data is now and says so; from there, open the
shared folder or volume it should move to.

The page checks the target before the move starts and lists every reason it refuses one:

- the path must be under `/share`, and not `/share` itself;
- no `:`, `&`, `|` or control character in any part of the path, nor in the path a link leads to;
- not the current data folder, not a folder inside it, not a folder that holds it;
- a new folder, or an empty one;
- a filesystem that keeps Unix owners and permissions (FAT, exFAT and NTFS are refused: Forgejo and SVN need
  owners and modes);
- a volume mounted with `exec`, `dev` and `suid`, which build environments need;
- writable, with free space for the data plus 10 %.

```mermaid
flowchart TD
    check["Check the target again<br/>free space: the data + 10 %"] --> stop["Stop the modules<br/>the panel stays on"]
    stop --> copy["Copy with<br/>rsync -aHAX --numeric-ids"]
    copy --> verify["Verify the copy<br/>files, bytes, owners, modes, times,<br/>sha256 of up to 1000 files"]
    verify --> switch["Switch the data folder<br/>in /usr/bin/bucketvs/etc/data-root"]
    switch --> start["bvsd starts again on the new root<br/>and starts the modules"]
    start --> old["The old copy stays, marked MOVED-TO,<br/>until you remove it"]
    copy -->|fails| failed["The old root stays in use,<br/>the modules start again"]
    verify -->|a difference| failed
```

While the move runs:

- Git, CI, SVN and Bucket, and the Editor do not answer. The panel stays on and shows the progress.
- No new cron run starts. A run that was already going continues.
- Switching a module on or off is refused until the move ends.

After the switch bvsd stops and starts itself again on the new root, so the panel is gone for a few seconds.

When the move fails, the old data folder stays in use and the modules start again. The job's error says what went
wrong and what became of the copy:

- A full volume, or a cancel, removes what the move created.
- Any other failure (the copy, the check, writing to disk) keeps the copy, marked with a file named
  `INCOMPLETE`, and lists it on the Storage page under "Leftover copies of failed moves".
- A stop of the package during the move keeps the old folder as the data folder. At the next start bvsd reports the
  unfinished move in the System Logs and lists the copy.

### The old copy and leftover copies

After a successful move the old data folder stays where it was, with a file named `MOVED-TO` in it. Nothing uses
it. "Remove the old copy" on the Storage page deletes it, and "Remove this copy" deletes a leftover copy of a
failed move. Both removals stay on one filesystem and never follow a link. A folder that is a mount point or a
shared folder's own folder is emptied and kept.

## Network

The Network page holds the ports, the plain HTTP mode, the module hostnames, the trusted proxies, the public URL
and the certificate. Saving restarts what the change needs and lists it.

### Ports

| Setting | Default | What it is |
|---|---|---|
| HTTPS | 9951 | the panel and the modules over TLS |
| HTTP | 9950 | the panel and the modules over plain HTTP |
| Git over SSH | 9952 | Git over SSH |
| SSH port in clone URLs | 0 | the port shown in SSH clone addresses; 0 means the SSH port above. Set it when a router forwards another port to the NAS. |
| SSH host in clone URLs | empty | the name or address shown in SSH clone addresses. Empty means the Git hostname, else the public URL's host, else the NAS's LAN address; for a name published through the Cloudflare Tunnel, the LAN address and the Git over SSH port. Set it to a DNS-only name or to the router's address when the router forwards the SSH port. |

Ports are 1024 to 65535 and differ from each other. The module ports on 127.0.0.1 (9960 and 9961) are refused, and
so is the port httpd or the CI runner's server uses when another program took 9960 or 9961 (the message names it).
When saving leaves nginx on its old ports because it could not use a new one, a notice says why. nginx also listens
on IPv6 when the NAS has it.

### Plain HTTP

- "Serve" (the default): the panel and the modules answer on plain HTTP too. Keep this when a proxy on the LAN
  talks plain HTTP to port 9950.
- "Redirect to HTTPS": a request on plain HTTP gets a redirect (307) to the HTTPS port. A request from a trusted
  proxy that says its client used HTTPS is served on plain HTTP, so such a proxy does not loop.

The sign-in page warns when it came over plain HTTP: the password then crosses the network readable.

### The certificate

At first start bvsd makes a self-signed certificate (ECDSA P-256) for:

- the NAS's host name, the module hostnames and `localhost`;
- the NAS's LAN addresses and `127.0.0.1`.

It is valid for 825 days, the most Apple devices accept, even for a certificate trusted by hand. bvsd checks it
at start and once a day, and makes a new one when 30 days or less are left, when the set of names changes, or when
its key does not match. A new one sends a [notice](#notices). The Network page shows its names, its validity and
its SHA-256 fingerprint: compare the fingerprint before a device trusts it.

"Add your own certificate" takes a certificate chain and its private key in PEM form:

- the server certificate first, then the intermediate certificates;
- a key without a passphrase (nginx starts unattended);
- the key must belong to the first certificate, each certificate must be signed by the next, none may have
  expired, and the first one must name its hosts as subject alternative names.

The pair is stored in `etc/tls/` in the data folder with mode 600. When an uploaded certificate expires or stops
passing these checks, nginx serves the self-signed one instead and a notice says why. An uploaded certificate
with 30 days or less left sends a notice too. "New self-signed certificate" makes a new self-signed pair and
switches back to it.

Clients that come through a proxy see the proxy's certificate, not this one.

**Download the certificate** on the Network page saves the certificate the panel serves, without its key, as
`<name>.crt`. Add it to a device's trusted certificates so that the device's browsers trust the panel; the editor's
extension views need that ([Extensions](#extensions)). A new self-signed certificate has to be trusted again.

### Trusted proxies

A list of IP addresses or CIDR ranges, such as `192.168.1.5`, the address of the proxy in front of the NAS. Only for
a request whose TCP connection comes from one of them does nginx take:

- the client's address from `X-Forwarded-For`;
- the scheme the client used from `X-Forwarded-Proto` (only `http` or `https` count).

A request from any other address counts as coming straight from its client, and nginx replaces its
`X-Forwarded-*` headers with its own values. A range makes every host in it trusted: any of them can then set the
client address and the scheme, so list the proxy's own address. A range of every address (`0.0.0.0/0`, `::/0`) is
refused.

### Public URL

The address alone, such as `https://git.example.com`, without a path. The module sections say which links start
with it.

## Behind a reverse proxy (Plesk)

The usual setup puts TLS on a proxy with a public certificate and lets it talk plain HTTP to port 9950 on the
LAN. The examples below use `bvs.example.com` for the public name, `192.168.1.20` for the NAS and `192.168.1.5`
for the Plesk server; put your own in their place.

```mermaid
flowchart LR
    client["Browser, git, WebDAV client"] -->|"HTTPS 443"| plesk["Plesk nginx<br/>bvs.example.com<br/>public certificate"]
    plesk -->|"HTTP 9950<br/>Host, X-Forwarded-For,<br/>X-Forwarded-Proto: https,<br/>Upgrade for WebSocket"| nas["nginx on the NAS<br/>192.168.1.20"]
    nas --> panel["/ panel, /api/ bvsd"]
    nas --> mods["/git/, /svn/, /bucket/, /ide/"]
    sshc["git over SSH"] -->|"TCP 9952, not through the proxy"| nas
```

### On the NAS

1. On the Network page, add the Plesk server's address (as the NAS sees it) to Trusted proxies.
2. Plain HTTP can stay on "Serve". "Redirect to HTTPS" works too once the Plesk server is a trusted proxy.

### In Plesk

Put these directives into the domain's Apache and nginx settings, in the field "Additional nginx directives":

```nginx
location ~ ^/ {
    proxy_pass http://192.168.1.20:9950;
    proxy_http_version 1.1;

    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # WebSocket: the editor and its terminal
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $http_connection;

    # large pushes and uploads: no size limit, streamed, an hour of silence allowed
    client_max_body_size 0;
    proxy_request_buffering off;
    proxy_buffering off;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
}
```

`location ~ ^/` is a regular expression location. nginx prefers a matching regular expression to a plain
`location /` (Plesk writes one of its own), so this block serves every path of the site, and `proxy_pass` without
a path passes the request's path unchanged. On the same settings page, turn off "Serve static files directly by
nginx": with it on, Plesk answers requests for `.js`, `.css` and image files from the domain's own folder, and
the panel does not load.

In the domain's Hosting Settings, turn on "Permanent SEO-safe 301 redirect from HTTP to HTTPS". The additional
nginx directives also apply to the domain's plain HTTP site on port 80; without the redirect, the site answers
there too, and passwords cross the internet unencrypted.

Why each line matters:

| Directive | What breaks without it |
|---|---|
| `Host $http_host` | nginx on the NAS picks the server block by `Host`, so a module hostname never matches. bvsd compares the page's `Origin` with `Host`, port included, so every change in the panel fails with `403 csrf`; `$host` would drop a public port other than 443 and fail the same way. |
| `X-Forwarded-For` | every client counts as the Plesk server: one person's five wrong passwords lock everyone out for 15 minutes |
| `X-Forwarded-Proto $scheme` | the session cookie is not marked `Secure`, and in "Redirect to HTTPS" mode every request is sent to port 9951 of the public name |
| `Upgrade`, `Connection` | the editor's page opens but the editor and its terminal never connect |
| `client_max_body_size 0`, `proxy_request_buffering off`, the timeouts | the proxy's own body limit and timeouts apply to large pushes and uploads |

`X-Forwarded-For` and `X-Forwarded-Proto` count only when the Plesk server is in Trusted proxies. When it is not,
nginx on the NAS ignores them.

### Git over SSH

An HTTP proxy does not carry SSH. Git over SSH goes to port 9952 on the NAS directly, or through a TCP port
forward on the router. "SSH port in clone URLs" on the Network page sets the port the clone addresses show, and
"SSH host in clone URLs" the name or address, when the public name reaches only the proxy.

### Checking the setup

- The panel opens at `https://bvs.example.com/`, and signing in and saving a setting work.
- The browser's developer tools show the session cookie `bvs_session` with the `Secure` flag.
- The Logs page shows `nginx-access`: the first field of each line is the client's own address, not the Plesk
  server's.

## Hostnames for the modules

A module can have its own hostname (Network page, Hostnames). nginx then serves that module alone at the root of
the name. The name must point at the NAS, or at the proxy in front of it. On a proxy, each hostname needs its own
site, with the same directives as above.

| Module | At the root of its hostname | Also there |
|---|---|---|
| Git | Forgejo | `/git/` on every other name answers 302 to the Git hostname, with the same path, query and port |
| Editor | code-server | the panel's sign-in page and `/api/`, so a user can sign in on that name |
| SVN and Bucket | httpd | repositories stay at `/svn/<name>` and buckets at `/bucket/<name>` |

A module that is off answers `503` with a sentence saying so, on its paths and on its hostname.

### One sign-in for every hostname

The session cookie belongs to the name it was set on, unless the setting `network.cookie_domain` names a parent
domain of every module hostname. With `example.com` set, a sign-in on `bvs.example.com` also reaches
`git.example.com` and `ide.example.com`. bvsd refuses a public suffix (`com`, `com.pl`, `co.uk`,
`myqnapcloud.com`), a domain that is not a parent of every module hostname, and any domain while no module
hostname is set. Every site under that domain receives the cookie, so set the narrowest common parent.

The Network page does not show this setting. The NAS admin sets it over the CLI socket with the package's `curl`:

```bash
/usr/bin/bucketvs/bin/curl --unix-socket /usr/bin/bucketvs/var/run/bvsd-cli.sock \
  -X PUT -H 'Content-Type: application/json' \
  -d '{"network":{"cookie_domain":"example.com"}}' \
  http://bvsd/api/v1/settings
```

An empty value (`""`) keeps the cookie on each name again.

## Publishing on the internet (Cloudflare Tunnel)

The **Publish** page (Administration, admins only) puts the package on the internet under names of a domain whose
DNS is at Cloudflare. `cloudflared`, which the package ships, dials out from the NAS to Cloudflare: no router port is
opened, and Cloudflare holds the certificates for the names. Every request passes the package's nginx: the panel and
its API need a sign-in, and only what you made public (public repositories, their web view, the package
repository) opens without an account.

There are two types of name, one of each at most:

| Type | Example | What it serves |
|---|---|---|
| Main hostname | `code.example.com` | the panel and its API, Forgejo at `/git/`, the web view at `/svn-web/` and `/bucket-web/`, the package repository at `/repo/` (read only) and the guide. `/svn/` and `/bucket/` answer 404 there, and so does `/ide/` until you publish the editor (below) |
| Packages hostname | `qpkg.example.com` | GET and HEAD of `/repo/` only, for App Center on other NAS units; everything else answers 404 |

### What the token needs

Make an API token at dash.cloudflare.com (My Profile, API Tokens) with these permissions, for the zone of the names:

- Account, Cloudflare Tunnel, Edit
- Zone, DNS, Edit
- Zone, Cache Rules, Edit
- Zone, Cache Purge, Purge

The package keeps the token on the NAS in a file only root reads, and never shows it again.

If the token has Client IP Address Filtering, put both of the NAS's public addresses in it, IPv4 and IPv6: the NAS calls Cloudflare over IPv6 when it has an IPv6 address. Otherwise the Publish page refuses with `token_ip_filter` and names the address Cloudflare saw.

### Publishing

Type the names and the token, then **Publish**. The package first checks everything at Cloudflare and changes
nothing there when a check fails; the message says what to fix:

- a permission the token lacks, by its name;
- a name in a zone the token does not see;
- a DNS record the name already has (its type and where it points): delete it at Cloudflare, or pick another name;
- a main hostname while **Public browsing** (Git page) is off, since its visitors have no account;
- names in zones of two Cloudflare accounts.

Then it makes the tunnel (named `bucketvs-<NAS name>` unless you give a name; a tunnel of that name kept by
Cloudflare is taken over), points each name at it with a proxied CNAME record, adds the cache rules and starts
cloudflared. **Publish again** with the same values changes nothing. Set the public URL (Network) to the main hostname's
`https://` address, so Forgejo's links and HTTPS clone addresses name it; the page reminds you while they differ.

Git over SSH does not pass the tunnel: Cloudflare carries only HTTP to a published name. The SSH clone address is
therefore the NAS's LAN address and its Git over SSH port (9952), in the panel and in Forgejo's own clone box, and
the Git page and the Overview say "SSH clone works on the local network; from the internet clone over HTTPS." For SSH
from the internet, forward the SSH port on the router and set **SSH host in clone URLs** (Network) to the router's
public address or to a DNS-only name that points at it, with **SSH port in clone URLs** when the router uses another
port.

The main hostname answers `/ide/` with 404 until an admin turns on **Publish the editor on the main hostname** on the Publish page. The editor's
terminal is a root shell on the NAS, and whoever signs in there as a developer or admin from the internet gets it,
so turn on two-factor authentication for those accounts first. With the switch on, the editor passes the same sign-in as on the
LAN, its port proxy (`/ide/proxy/<port>/`) stays off there, and Cloudflare never keeps its pages in the cache. While
it is on, the Overview shows **Editor on the internet**.

While the CI module is on, a workflow pushed to any repository of this Forgejo runs as root on the NAS through
the NAS's own runner, whether it was pushed from the internet or from the LAN: the same power as the editor's terminal.
The Publish page says so under **What the internet reaches**. Give push access only to people you would give the
NAS's admin password.

### The cache

With **Cache presets** on, Cloudflare keeps packages (`*.qpkg`) for a year, icons for a day, `repo.xml` for 5 minutes
and the panel's files for a year; it never keeps the API, Forgejo's pages, the sign-in pages or the web view. A
publish into the package repository clears `repo.xml` (and a changed icon or a deleted version's files) from
Cloudflare's cache right away. The package adds its own rules at the end of the zone's cache rules and never changes
the others.

### Sign-in from the internet

The first password can never be set through a published name: the first-run setup takes only a local client.
Wrong passwords count per internet client, as on the LAN. A password sent with git to Forgejo counts too; a Forgejo
access token works as always. Cloudflare takes requests up to 100 MB, so a larger push or upload goes over the LAN
address.

### Stopping and removing

**Stop** ends cloudflared; the names then show Cloudflare's own error page, and nothing at Cloudflare changes.
**Remove from Cloudflare** deletes the DNS records, the cache rules and the tunnel the package made (a tunnel it
took over stays) and forgets the token. While names are published, the Overview shows an **Internet** pill with the
tunnel's state.

## Modules

The Overview shows each module with its processes, memory and CPU, and switches it on and off (admin only). A module
that is off says what it gives; an admin turns it on with its button (**Turn on Git**, **Turn on Editor** and so on) on
its tile, on its own page or, for the Editor, in the Editor tab.

- Switching a module on runs a job: the module prepares itself, the switch is saved, and its processes start.
  The job waits up to 30 seconds for them.
- CI needs Git: Git cannot be switched off while CI is on.
  While Git is off, CI's button reads **Turn on Git and CI** and turns Git on first.
- A module that is off runs no process and uses no memory.
- Core is always on.

| State | Meaning |
|---|---|
| running | every process of the module runs |
| starting | a process started less than a second ago |
| degraded | a process waits to be restarted, the module's files could not be written, or its health check fails |
| restarting in a loop | a process exited 5 times within 10 minutes; bvsd starts it again every 15 minutes, and one notice went out |
| stopped | the module's processes are not running (a data move stopped them, or they failed to start) |
| off | the module is off |

A process that exits is started again after 1 second, then 2, 4 and so on up to 60 seconds. After 10 minutes of
running the wait starts again at 1 second. A process that is restarting in a loop starts again at once when its
module is switched off and on.

## Git

The Git module runs Forgejo, a web interface for git repositories with issues, pull requests, releases, wikis, LFS,
mirrors and webhooks. An admin turns it on with **Turn on Git** on the Overview or the Git page. The job prepares
Forgejo's folders and database in `<data>/forgejo`, adds every panel account to Forgejo, and starts it. Forgejo then
opens at `/git/` on the panel's address, or at `/` of the Git hostname when one is set (see [Hostnames for the
modules](#hostnames-for-the-modules)).

**New repository** on the Git page, and on the Overview's Git tile while there is none yet, opens Forgejo's form for a
repository.
The panel offers it, and **Migrate or mirror a repository**, from the developer role up: Forgejo lets a viewer or a
member create no repository.

### Repositories and accounts

- **One sign-in.** Forgejo has no sign-in of its own: `/git/user/login` goes to the panel's sign-in, with the
  authenticator code when the account has one, and back to Forgejo after it.
- **Accounts come from the panel.** Every panel account has a Forgejo account of the same name and the same
  password. The `admin` role is a Forgejo site admin. A disabled account, or one that must change its password
  first, cannot sign in to Forgejo either. A change in the panel reaches Forgejo within seconds. The Users and Git pages show
  each account's state in Forgejo, and **Sync every account now** on the Git page runs the sync again.
- **Deleting an account** in the panel retires its Forgejo account: it can no longer sign in, and its tokens and
  SSH keys are removed, but its repositories stay. The name cannot be given to a new panel account until an admin
  renames or deletes the old Forgejo account in Forgejo's site administration (**User accounts**).
- **An account made in Forgejo itself** is never changed by the panel. A panel account of the same name shows the
  reason and waits until one of them is renamed.
- **Two-factor authentication and git.** Forgejo takes no password for an account with two-factor authentication on. git over HTTP
  or HTTPS then needs a Forgejo access token (Forgejo's **Settings**, **Applications**) in place of the password.
  SSH keys work as before.
- **Repositories** are created, imported and mirrored in Forgejo's own pages. Imports from addresses on the LAN are
  allowed by default (Settings: `git.allow_local_networks`), because a NAS usually mirrors from its own network.

### Clone over HTTPS and SSH

On the Git page, **Clone a repository** writes both commands for an owner and a repository:

| Way | Address | Sign-in |
|---|---|---|
| HTTPS (or HTTP) | `https://<NAS address>:9951/git/<owner>/<repo>.git` | the panel's username and password, or an access token |
| SSH | `ssh://git@<NAS address>:9952/<owner>/<repo>.git` | an SSH key added in Forgejo's **Settings**, **SSH / GPG Keys** |

- With a public URL (behind a proxy, [Public URL](#public-url)) the HTTPS address uses it; with a Git hostname the
  path is `/<owner>/<repo>.git` on that name.
- The SSH port is `network.ssh_port` (9952), or `network.public_ssh_port` when a router forwards another port to
  it. The proxy does not carry SSH: forward the port on the router (see [Git over SSH](#git-over-ssh)).
- The SSH host is `network.public_ssh_host` when set, else the Git hostname or the public URL's host. A name
  published through the Cloudflare Tunnel carries no SSH, so for it the address is the NAS's LAN address on the Git
  over SSH port ([Publishing on the internet](#publishing-on-the-internet-cloudflare-tunnel)).
- Large pushes are not cut: nginx sets no body size limit on `/git/`.

### Showing repositories to people without an account

**Public browsing** on the Git page lets anyone who reaches Forgejo read its public repositories without an account.
In Forgejo each repository is public or private (its **Settings**, **Visibility**). With the switch on, a visitor
without an account can:

- open a public repository, its folders and files, and download a file as it is;
- read the history, open a commit with its diff (also as a `.diff` or `.patch` file) and compare two commits;
- see its releases, download an archive, and clone it over HTTP (`git clone https://<NAS address>:9951/git/<owner>/<repo>.git`);
- find public repositories under **Explore** and on their owner's profile.

A private repository stays hidden: its address answers "not found". While the switch is on, Forgejo also lists no
accounts (Explore's **Users** tab and the user search are off, for signed-in users too), so a visitor cannot collect
the account names. Pushing, issues, pull requests and settings still need an account. With the switch off, every
page asks the visitor to sign in first.

To share a repository, make it public in Forgejo, turn **Public browsing** on and send the address. On a Git
hostname the address is `https://<Git hostname>/<owner>/<repo>`.

### New project

**New project** is a page that takes a project from nothing to a built, published first version, step by step. It
opens in its own tab from the command palette (Ctrl+Shift+P, **New project**), the top of the Editor's side bar, the
Git side bar's header, the Git page and the Projects page; an example under **Editor projects** on the Examples page has **New project from
it**. It needs the developer role, the Git and Editor modules, and your account in Forgejo (the Git page shows its sync
state). While Git or the Editor is off, the page says so above the steps, and an admin turns them on there with one
button.

1. **Template.** The templates are grouped: Programs (C, C++ with CMake, Go, Rust, Node, Python, Java), Packages for QTS (a
   QPKG package, an App Builder app, a package from source), Kernel (a kernel module, the ch341 USB-serial driver),
   Websites (a static website: pages in `src/`, a Makefile that builds `public/`, a link check as `make test`) and
   Examples. Each card says what a build makes and marks what this NAS still lacks (QDK, a builder environment, a
   module that is off), with a link to the page that adds it.
2. **Name and visibility.** The name names the repository and the folder `projects/<name>`; the page checks it as
   you type. Then the owner (your account, or an organization where Forgejo lets you create repositories), private
   or public, and a description if you want one. For an App Builder app, the preset it starts from and the target.
3. **Service**, for a QPKG package and a package from source: how the package starts and stops, built from blocks.
   Choose a preset (**Command-line tool** links the program into `/usr/bin`; **Background service** adds an account, a
   data folder on the volume, the program started as that account, a wait for its port and a stop with a timeout;
   **Web app** adds App Center's Open button; **Empty** has only the verbs), then change its blocks. Each block's card
   shows the lines it writes into the script while you type, and a refused value is marked under its field. Start
   never switches App Center's Enable on, and stop never touches it. The Project view changes the blocks later.
4. **Build.** The environment the project builds in, kept for the project (the Project view changes it later), or none,
   and bvsd picks one at each build. The step first offers the one bvsd picks itself, where `bvs qbuild` and CI build:
   the Ubuntu 24.04 builder, else the Debian 12 builder, else another Debian or Ubuntu environment. When the template's
   Makefile gives each app its environment on an ENV line, the step says so and asks for none, and the project keeps
   none. For a package from source the step also says that the starter recipe's source address and sha256 are
   placeholders: the first Build fails until the recipe names your program's real source and that file's sha256. The
   page shows the workflow CI runs at each push. **Build right after creating** is on unless you turn it off; for a
   package from source it starts off, and you can turn it on. With no Debian or Ubuntu environment for the NAS, the
   step says so: an admin creates the builder there with **Create this environment**, and then the step offers it;
   otherwise turn off **Build right after creating** and build the project later.
5. **Publish**, for a QPKG package, an App Builder app or a website: **Test deploy on this NAS** (admin, signed packages
   only; for an app, **Activate on this NAS**), **Publish to this NAS's App Center repository** and **Publish to my web
   server** (admin) with one of the [web targets](#your-own-web-server-web-targets); **Add a target** saves a new one
   there. Each is optional. The publish choice also becomes the project's switch **Publish to the App Center
   repository** (Project view, Settings): while it is on, the project's CI publishes there at each run.
6. **Review** lists every answer and the steps that will run. **Create project** sends them.
7. **Progress** follows each step's job: the repository and its first commit, the first build, then the test deploy,
   the App Center publish and the web publish, one after the other. A row shows its state and the end of its log; a
   click shows its log. A failed step says why and offers **Retry**, and the steps after it wait. At the end,
   **Open the project**, **Open in editor** and **Open in Forgejo**. Reloading the page or coming back to its tab
   reads the jobs again and sends no step twice. Leaving the page stops the chain; the Project view has the same
   steps.

When bvsd refuses the project (a name taken by a folder, an app or a repository, an owner you cannot create in), the
page goes back to the step that holds that answer, shows the reason under its field and keeps every other answer. A
module that is off (an admin turns it on from the page), no Forgejo account yet and no Debian or Ubuntu environment
for an App Builder app are explained on Review.

What the first job does:

- It makes the repository as you, private unless you chose public.
- It makes the folder `projects/<name>` of the data folder with the first commit: every file of the template and
  the workflow `.forgejo/workflows/build.yml`, on `main`. The commit is yours (your name and address are set as
  the author in the folder), and the template's scripts are executable in it. It pushes that commit to the
  repository as you. `origin` is the repository's HTTPS address, so your next push goes there. The commit also
  holds the [skill for AI assistants](#coding-with-an-ai-assistant), `.claude/skills/qnap-dev/` and `AGENTS.md`.
  The folder is in the editor's shared folder: every developer's editor opens `projects/`, whether the repository is
  private or public.
- It keeps the environment you chose for the project, unless the template's Makefile gives each app its environment. For an
  App Builder app, it makes the app of the project's name with the preset's recipes and ties it to the project, so the
  app can also build recipes you add to the project's `recipes/` folder. It also keeps your publish choice as the
  project's switch Publish to the App Center repository, before the workflow's first run reads it.

The workflow runs on the NAS's runner at every push once the CI module is on, in the environment the workflow names. The
QPKG workflow publishes the package into the [package repository](#package-repository), which takes code-signed packages
only. If the first commit or its push fails after the repository was made, the job removes the folder, deletes the
repository again and says why. While the project's switch **Publish to the App Center repository** is off, `bvs repo
publish` in the workflow prints a notice instead of publishing, and the run goes on.

### The Project view

Every folder of `projects/` in the data folder has a page of its own: a project the wizard made, a copy of an example from Editor projects,
or a repository you cloned there. It opens from the **Projects** branch of the Editor's side bar, from the **Projects**
page, from **Open the project** at the end of New project, and from the command palette (**Open the project …**, or
Ctrl+P and its name). It needs the developer role. A project's menu on the Projects page has Build, Rebuild, Publish and
Publish to the web targets too. The same menu turns CI publishing to the App Center on or off.

The page reads the folder and changes nothing in it on its own. It works out the project's type from its files: an
App Builder app of the folder's name, a `qpkg.cfg` at the top (a QPKG package), a `Makefile` at the top, or none of
these, when the project's CI workflow builds it. When an App Builder app has the folder's name, a project with a
`qpkg.cfg` at the top, or with a Makefile and a `qpkg.cfg` in `qpkg/`, keeps its own build, and Test deploy and Publish
still take its QPKG.

The strip at the top holds the steps. A step that cannot run keeps its button, disabled, and says why beside it and
under the strip. A step the project cannot take at all leaves the strip, and one sentence under it says what the project
makes and where the build's result goes: the project's folder for a program, the Kernels page for a kernel module, each
app's page for a Makefile that only builds App Builder apps.

- **Build** runs as a job in the bottom panel. A QPKG package builds with QDK's `qbuild`, a Makefile project with
  `make`, both in the project's builder environment. The log names every `.qpkg` the build wrote to `build/` and whether
  it carries a code signature. An App Builder app builds as on its own page. When a Makefile's default goal runs only
  `./bvsapp.sh create` and `./bvsapp.sh build` lines, Build does the same through the App Builder: it makes each app a
  `create` line names when that app is missing, then builds the apps one after the other. A `create` line is
  `./bvsapp.sh create NAME RECIPES-FILE [ENV] [LINK] [TARGET] [OTHERS]`. Such a Makefile names its default goal and the
  goal's prerequisites in `.PHONY` and holds only plain assignments, each variable set once, and plain rules; one with
  anything else (a condition, an include) builds with make. TARGET is `native`, `aarch64` or `armv7`, so one Build can
  make an app for this NAS and one for ARM 64-bit. OTHERS says what a static app does with recipes that cannot link
  static (`refuse` or `bundle`). An app made this way also takes the project's own recipes from `recipes/`, unless the
  folder's name is not a project name (a name with a space, for example). When an app of that name is already there with
  another target, or with another project while the folder has recipes of its own, Build stays off and says how to fix
  it. Before the first build the step names the apps it builds. A Makefile project with its package in `qpkg/` makes the
  package file with `make qpkg` in the editor's terminal; until it is there, Test deploy, Publish and Publish to web say
  "Needs a package file".
- **Rebuild** builds from scratch: `make clean`, then `make`; for a QPKG package the old `.qpkg` files of `build/` go
  first. A Makefile without a `clean` target cannot rebuild, and the page says so. An App Builder app has no Rebuild,
  and neither has a Makefile that only makes and builds App Builder apps.
- **Export as QPKG** (admin) is there for an App Builder app, as on the app's page. Publish then waits for an export, as
  on the app's page, and **Open the app** in the header opens that page.
- **Test deploy** is for admins. It installs the newest `.qpkg` on this NAS with `qpkg_cli`, the way App Center
  installs a file, and the job waits until App Center confirms it. The package stays installed until you remove it in
  App Center. A package without a code signature is refused: a test deploy installs only signed packages and never
  skips App Center's signature check (in App Center a NAS's owner may allow unsigned apps; this page does not). Sign
  it with a certificate as files (case 1 in [Code signing a QPKG](#code-signing-a-qpkg)). The package of Bucket & Code
  Server IDE itself is refused, because installing it would replace the running package. For an App Builder app, Test
  deploy activates its build on this NAS.
- **Publish** puts the newest `.qpkg` into the [package repository](#package-repository), with the repository's rules:
  an admin's publish becomes the listed version, a developer's waits for an admin, and a package without a code
  signature is refused unless the repository takes unsigned packages. A line under it says whether the project's CI
  publishes to the App Center repository too; this button publishes either way.
- **Publish to web** (admin) copies the newest `.qpkg`, or the website folder `public/` (else `dist/`), to a [web
  target](#your-own-web-server-web-targets) and reads it back over the target's public address. For a website the wizard
  made, the step waits for the first build, which writes `public/`.

A Makefile project picks a target beside **Build** (make's default, its first target, unless you choose another); a
CMake project builds, or builds and runs its tests. A kernel module project builds in the kernel workshop's workspace of
the running kernel, and that build is an admin's. A running step has **Cancel**, and the job usually ends at once. **Run
the workflow** runs the project's CI workflow by hand when the workflow has a `workflow_dispatch` trigger; otherwise the
button is off and the page says why. Every template's workflow has that trigger. **Installed on this NAS** lists the
packages the project's test deploys installed, with **Remove from this NAS** for an admin. **Delete the project**, in
the menu beside the header's buttons or in the project's menu on the Projects page, is for an admin or for the developer
whose Forgejo account holds the repository. You type the project's name first; the folder goes, and the Forgejo
repository, an App Builder app of the name and installed packages stay. The page then links the repository.

**Settings** holds what bvsd keeps for the project outside its folder. For a Makefile, CMake or QPKG project it shows
the environment Build and Rebuild run in and changes it. For a Makefile that only makes and builds App Builder apps, it
lists each app with the environment its line in the Makefile names. A project that makes a package, or whose workflow
runs `bvs repo publish`, has the switch **Publish to the App Center repository** there: while it is on, `bvs repo
publish` in the project's CI adds its packages to the repository; while it is off, the CI run skips that step with a
notice. A project with no choice kept (an example's copy, a clone) has it on. **Git** shows the branch, how far it
is ahead of or behind its remote as the last fetch left it (the page never fetches), the last commit and the changed
paths. **Files** lists one folder at a time; pick a file to read it, and a `.qpkg` downloads. Editing stays in the
editor, and the `.git` folder is left out.

**Recipes**, for a project with its own `recipes/` folder, lists the recipes its apps take: each with its version, the
library version it replaces when it has the name of a library recipe, and what the recipe check found, file and line,
under it. A recipe bvsd leaves out has a row with the reason; past three left out the same way, one row counts the
rest. What the check found about the `recipes` folder as a whole has a row of its own. A click opens the recipe's
folder in the editor; the menu also copies the command that runs the same check in a terminal of the NAS, `bvs recipe
lint` with the project's folder. The card also shows when the project's `recipes` is a link, a file, a folder bvsd
cannot read or a folder holding only links, and says why no recipe of it is read.

**Service and package** (a QPKG package, or a Makefile project with its package in `qpkg/`, such as a package from
source) holds the package's start/stop script as blocks, the fields of its `qpkg.cfg` (**Package**) and the files
written from them (**Script**). **Save** shows each file's diff, then commits it to the project's repository; if the
project changed meanwhile, plan again. **Edit by hand** turns the blocks off, so you edit the script in the editor;
**Attach the blocks again** shows what writing it again would replace. **Update the library** brings this Bucket & Code Server IDE's
`bvs-qpkg.sh` into the package, with its diff. A project made before the blocks offers **Use blocks for this script**.

**Versions** lists what the project can be switched to, and **Switch to this version** does it:

| Version | What switching does |
|---|---|
| A `.qpkg` in `build/`, or an App Builder export | a test deploy of that file (admin) |
| The App Builder's build | activates it (admin); the App Builder keeps one build |
| A version in the package repository | makes it the listed version (admin) |
| A git tag | checks the tag out in the folder |

A click picks a version; its menu also tests, publishes, publishes to a web target or downloads a `.qpkg`. A tag is
checked out as a detached HEAD, and **Back to main** in the Git card returns to the branch. The switch is refused while
the folder has changes that are not committed; the message lists them, and you commit or stash them in the editor
first. Tick two versions and **Compare the two** lists the files one has and the other lacks, and the files that
changed. Two tags, or two packages, are compared by content; a package against a commit only by size. On the
**Packages** page an admin also installs a repository version on this NAS (**Install on this NAS**), with the test
deploy's checks.

### Your own web server (web targets)

A web target is where an admin publishes a package or a website to their own web server: a folder of this NAS (QTS's
Web share, which QTS's Web Server serves, or another shared folder) or an rclone remote (SFTP, FTP, WebDAV, S3, set up
in the backups' `rclone.conf`), with the public address that serves it. They are on the **Publish** page, under **Web
targets**: **Add a target**, **Change**, **Remove** and **Test**, which copies a small file there, fetches it from the
public address and deletes it again.

A folder target is a folder of its own inside a share (`/share/Web/mysite`, never a whole share or a volume), outside
the data folder, because a website publish removes from the target what the site no longer holds. The address starts
with `http://` or `https://` and has no user or password.

Each publish reads the file back over the public address: a package's `.qpkg` with its `.sha256`, compared by
sha256, or a website's `index.html`, compared byte for byte. When the address does not serve what was copied, the job
fails with `web_check:` and both values.

## CI runners

The CI module runs Forgejo Actions: a workflow file in a repository's `.forgejo/workflows/` runs on a runner when
something is pushed. It needs the Git module. An admin turns it on under **Modules**, **CI**, **Enable**. The job
registers the NAS's own runner with Forgejo, restarts Forgejo with Actions on, and starts the runner; it ends when
the runner has reached Forgejo.

### The runner on the NAS

- **Its labels** are `bucketvs`, `qnap-x86_64` (or `qnap-aarch64`) and `self-hosted`, plus the extra labels an
  admin sets on the CI page. A job whose `runs-on` names one of them runs here, for example `runs-on: bucketvs`.
- **It runs every job on the NAS itself, as admin**, not in a container. A job has the package's tools (git, curl,
  make and the rest), Node for JavaScript actions such as `actions/checkout`, and `bvs env run <env> -- <command>`
  for a build inside an environment. Whoever can push a workflow into a repository can therefore run commands as
  admin on the NAS. A member or a viewer cannot create, fork or import repositories, or create organizations, in
  Forgejo; developers and admins can. A viewer never pushes. While CI is on, a member's push into a repository whose
  Actions are on is refused with the reason, and so is a push with a deploy key. A member's push, or one with a
  deploy key, into a repository that an account below developer owns or administers is refused too, also while CI is
  off: such an account could turn Actions on itself. bvsd turns Actions off, within a minute, in every repository a
  member or a viewer owns or can write to, and before the runner starts when CI is turned on.
- **What is left** is a developer's or an admin's own act: turning Actions on, or dispatching, in a repository a
  member wrote to earlier, or giving such an account admin access to a repository it wrote to earlier. A developer
  who merges or builds a member's commits runs them as admin too.
- **Jobs at once** is `ci.capacity` (1 by default). A job that no online runner can take waits in Forgejo.
- **Repositories made before CI was on** have Forgejo's Actions switched off, so a pushed workflow starts nothing:
  turn it on in the repository's **Settings**, **Units** (Actions). Repositories made while CI is on have it.
- **Actions** such as `actions/checkout@v4` are downloaded from `https://data.forgejo.org`, so the NAS needs internet
  access for workflows that use them. The runner has no cache: `actions/cache` finds nothing and the job goes on.
- **Artifacts** (`actions/upload-artifact`) are uploaded to Forgejo's own address (the one on the Git page), so the
  NAS must reach that address, and trust its certificate when it is HTTPS.
- **The runner's state** is in `<data>/runner`; its log is **runner** under **Logs**. Turning the module off stops the
  runner and switches Actions off in Forgejo.

### External runners

A build machine elsewhere (a server, a VM) can take jobs from this Forgejo too. On the CI page, under **Register an
external runner**, an admin gives it a name and gets the commands for Linux x86_64 and aarch64. They download
forgejo-runner 13.2.0 from code.forgejo.org, check its sha256, keep the runner's token in `runner-token` (readable by
its owner only) and start the runner with the label `linux-amd64` (or `linux-arm64`). The runner shows in the list at
once, offline until the commands run. The token is shown only then. Such a runner runs its jobs on its own machine,
as the user who started it, and reaches Forgejo at its public address.

## SVN and Bucket

The SVN and Bucket module serves Subversion repositories at `/svn/<name>` and buckets at `/bucket/<name>`. A bucket
is a network drive (WebDAV) in which every saved file becomes a revision, so any earlier version of a file can be
read or put back. An admin turns the module on under **Modules**; the **SVN & Bucket** page then creates, lists
and deletes both types.

- **Accounts** are the panel's. Permissions are set per repository: read and write, read, or none, for each account, plus
  a default for every account not listed. Admins always read and write; members and developers write where they have
  read and write; viewers read at most. A new repository gives nobody but the admins any access until an admin sets
  it (the people icon on its row, **Permissions for `<name>`**).
  Below admin, the page shows each person their own access to each repository and bucket.
- **Wrong passwords** count as at the panel's sign-in: 5 wrong passwords within 15 minutes lock that account from
  that address, and the address itself, for 15 minutes.
- **Two-factor authentication.** An account with two-factor authentication does not use its password for SVN or WebDAV. It makes one
  **app password** on its account page (**App password for WebDAV and SVN**): 25 random characters, shown once. A
  new one replaces the old; a password change or turning two-factor authentication off removes it.
- **Deleting** a repository moves it to `<data>/trash`, where the `trash` job removes it after 7 days.

### SVN repositories

```bash
svn checkout "https://<NAS address>:9951/svn/<name>"
svn commit -m "…"
```

The page shows the checkout command with the repository's address. Behind a proxy the address follows the public
URL, and with an SVN hostname it is `/svn/<name>` on that name. The **History** of a repository lists its
revisions; a file can be read at any revision, compared with another, and put back as a new revision (**Restore
this version**, for accounts with read and write).

### Buckets

- **Saving is committing.** A client that saves a file makes a revision with the account's name and the time; the
  revision is marked as made through WebDAV. Many clients write an empty file first and the data after it, so one
  save can make two revisions.
- **macOS files.** A bucket created with **Refuse macOS's ._\* and .DS_Store files** (the default) does not store
  the `._*` and `.DS_Store` files Finder writes next to what it copies. The client gets no error, but the file never
  becomes a revision. Deleting such a file is allowed.
- **Going back.** The bucket's history shows each file's versions; **Restore this version** saves the old content
  as a new revision, so nothing is lost.

### WebDAV clients and their limits

| Client | How to connect | Notes |
|---|---|---|
| macOS Finder | **Go**, **Connect to Server**, `https://<NAS address>:9951/bucket/<name>` | Finder locks a file while it writes it; the bucket answers WebDAV locks |
| rclone | a `webdav` remote with `vendor = other` and the bucket's address | suits scripted copies |
| davfs2 (Linux) | `mount -t davfs https://<NAS address>:9951/bucket/<name> /mnt/<dir>` | the panel's username and the password or app password |
| Cyberduck, WinSCP | a WebDAV (HTTPS) connection to port 9951, path `/bucket/<name>` | |
| curl | `curl -T file -u <user> https://<NAS address>:9951/bucket/<name>/file` | one upload, one revision (or two, see above) |
| Windows Explorer | **Map network drive** | Windows' WebClient sends a password only over HTTPS with a certificate Windows trusts: put the bucket behind a proxy with a real certificate ([Behind a reverse proxy](#behind-a-reverse-proxy-plesk)); the self-signed certificate is not enough |

- Large files are not cut: nginx sets no body size limit on `/bucket/` and passes the upload through as it arrives.
- A file that the bucket refuses to store (a macOS file above) is answered with `201` by the WebDAV server all the
  same, so a client shows no error; the file is not in the bucket.

### Showing a repository or bucket to people without an account

An admin can make an SVN repository or a bucket public with the switch in its row on the **SVN & Bucket** page
(**On the web**). Anyone who has the address can then read it in a browser, without an account:

| Address | What it shows |
|---|---|
| `https://<NAS address>:9951/svn-web/<name>/` | an SVN repository |
| `https://<NAS address>:9951/bucket-web/<name>/` | a bucket |

The row's **View on the web** link opens that page. It lists the files and folders at the newest revision or an
older one, shows a text file with line numbers (up to 1 MiB) or a picture, and offers every file as a download. The
**History** lists the revisions with their authors, messages and changed files. A revision's page shows the diff of
every file it changed (up to 20 files of at most 1 MiB; the others are links); any file can be compared between two
revisions.

- **Read only.** The pages cannot change anything. WebDAV clients and `svn` still need an account, also for a public
  repository.
- **Only what is public.** A repository that is not public, and a name that does not exist, both answer "There is
  no public repository at this address", so the pages do not tell which names exist.
- **Limits.** One address may open 60 pages at once and 2 a second after that, and download 2 files at a time;
  beyond that the page asks to try again in a moment. A diff is cut at 2 MiB.
- **Search engines** are asked not to index the pages. The pages are in English. They have the panel's look, dark or light
  as the visitor's browser is set.
- Behind a proxy or with an SVN hostname, the address follows the repository's own (the public URL, or the SVN
  hostname with the same paths).

## Editor and AI tools

The Editor module runs VS Code in the browser (code-server) for each developer and admin account. An admin turns it on
under **Modules**. Each account gets its own instance, started when the account opens the editor and stopped after a
time without traffic.

**The editor's terminal is a root shell on the NAS.** On QTS the package's processes run as admin (uid 0), except
Forgejo and httpd. Give the developer role only to people you would give the NAS's admin password.

### Opening the editor

- **Open the editor** on the Editor settings page (or `/ide/` on the panel's address, or the Editor hostname) opens the workbench in a
  new tab. The first request starts the instance; the panel's sign-in comes first when there is no session.
- Through the published main hostname, while the editor is not published there, the Editor settings page, the Editor tab, the
  Overview and **Open a shell** say that the editor opens only from the local network and link to its LAN address
  (`https://<NAS address>:<HTTPS port>/ide/`); for an admin they also link to the switch on the Publish page.
- The editor opens `<data>/projects`, where the examples under Editor projects put their projects. The account's settings,
  extensions and terminal history are kept in `<data>/ide/<account>` and its home folder in `<data>/home/<account>`.
- The terminal has the package's tools first on `PATH`: git, svn, node, npm, rsync, curl, bash and the rest.
  `bvs env shell <name>` enters a Toolchains environment and `eval "$(bvs use <id>)"` sets up a toolchain or runtime.
  **Open a shell** on the Toolchains page opens the editor with a terminal already inside that environment.
- The **Editor** activity, and the command palette, open the editor with a shell in an environment or with an app's
  folder. The editor shows inside the panel's Editor tab; on a phone it opens in its own browser window. Only pages of
  the panel's own address may show the editor inside them: the package tells browsers so on every answer of the editor, so
  another site cannot put the editor's root terminal under a page of its own.
- **Idle time** (default 30 minutes) stops an instance without browser traffic, but never while a program started
  from its terminals, an AI agent that an extension started (Claude Code for VS Code's `claude`, Codex, Gemini CLI) or
  something left running from them still runs. 0 turns the idle stop off. An account can stop its own instance on
  the Editor settings page; an admin can stop any.
- **Memory and processes** on the Editor settings page shows what an instance runs and how much memory it takes against the
  NAS's, marks what keeps it running, ends one process with **End**, and says why the last stop came and what it
  ended. An AI agent waiting at its prompt keeps the instance running too: end it there when it is no longer needed.
- VS Code's **Ports** view reaches a development server on the NAS at `/ide/proxy/<port>/`; the package's own
  loopback ports are refused there.
- **New recipe**, on the menu of a project's `recipes` folder, or of a recipe's folder in it, in the **Explorer**, asks
  for a name and writes a new recipe there: `recipe.json` and `build.sh` from the package's starter recipe. In the
  command palette it is **Bucket: New recipe**, which works while a file of a project with a `recipes` folder is open.
  Each time you save one of a recipe's files, the editor checks the project's recipes with `bvs recipe lint` and lists the
  findings in the **Problems** view. It warns about the starter recipe's sha256 until you put in your source's real one.

The editor answers only its own pages. A WebSocket, a form or a script's request that another page sends with the
account's session, from another site or from another port of the NAS, gets 403 and starts no instance, so another
page cannot reach a development server behind the **Ports** view either.

### Extensions

- Install extensions in the editor's Extensions view (Ctrl+Shift+X). They come from Open VSX: the browser searches
  open-vsx.org and shows each extension's page and icon from openvsx.eclipsecontent.org, and the NAS downloads the
  extension itself from open-vsx.org. The browser and the NAS both need to reach the internet for that.
- The first extension of each publisher asks "Do you trust the publisher …?". **Trust Publisher & Install** installs
  it; until then the extension shows "Installing".
- Each account keeps its extensions in `<data>/ide/<account>/extensions`, and a package update keeps them.
- The package brings a set of extensions and formatters, which the editor installs for each account at its first start,
  and again after a package update that brings a new set. With it the editor highlights and formats (**Format Document**
  in the command palette) Markdown, shell scripts, C, C++, Rust, Go, Pascal, Python, Perl, PHP, JSON, YAML, TOML,
  nginx configuration, CSS, HTML, JavaScript and TypeScript. ShellCheck marks problems in shell scripts, systemd unit
  files get highlighting and completion, a project's `.editorconfig` sets the indentation, colour codes such as
  `#ff8800` or `rgb(0, 128, 0)` show in their colour in every file, and **Hex Editor: Open Active File in Hex Editor**
  in the command palette shows a file's bytes.
- Rust is formatted once the Rust runtime is installed on the Toolchains page and the account's editor has started again;
  until then Format Document in a Rust file reports that the request failed.
- On an ARM NAS (the aarch64 package) Pascal, Perl and PHP are highlighted but not formatted: the package has no build
  of Free Pascal, Perl or Mago for it.
- An extension of the set that the account removes in the Extensions view stays removed, also after a package update,
  and one it updates keeps the newer version. A default formatter chosen for a language (**Format Document With…**,
  then **Configure Default Formatter…**) stays the account's choice.
- Every `.conf` file opens as nginx configuration. The language name in the status bar switches a file to another
  language.
- Ruff, which formats Python, says once that no Python environment extension is installed. It formats without one.
- In the editor's terminal, `code-server --install-extension <publisher.name>` installs an extension from Open VSX too.
- Some extensions draw their own views, such as the panel of Claude Code for VS Code, the hex editor and the Markdown
  preview. The browser shows these views only when it trusts the panel's certificate. With the self-signed certificate
  accepted only through the browser's warning page they stay empty, and the Editor tab and the Editor settings page say so. Any of
  these fixes it:
  - trust the NAS's certificate on the device: **Download the certificate** on the Network page or in that notice,
    then add it to the system's or the browser's trusted certificates ([The certificate](#the-certificate));
  - an admin adds a certificate for the NAS's name on the Network page;
  - open the editor through the published main hostname, once an admin publishes it there.
- Claude Code for VS Code (publisher Anthropic) installs from the Extensions view, and its panel asks how to sign
  in to Anthropic. **Claude Code: Open in Terminal** in the command palette runs Claude Code in a terminal, which
  works with any certificate. VS Code's own chat features start off (`chat.disableAIFeatures`); that setting does
  not affect this extension.

### Claude Code, Gemini CLI and Codex CLI

Under **AI command-line tools** on the Editor settings page each account installs, updates and removes these tools for itself:

| Tool | Installed with | Command |
|---|---|---|
| Claude Code | Anthropic's installer (`claude.ai/install.sh`), npm if that fails | `claude` |
| Gemini CLI | `npm install -g @google/gemini-cli` | `gemini` |
| Codex CLI | `npm install -g @openai/codex` | `codex` |

- They go into the account's home folder (`~/.local/bin`, `~/.npm-global/bin`), which is on the terminal's `PATH`;
  another account does not see them.
- Each install is a job with its log on the Jobs page. Signing in to each tool (an account or an API key) happens in
  the tool itself, in the editor's terminal.

### Coding with an AI assistant

Every project that **New project** makes carries a skill for AI assistants: `.claude/skills/qnap-dev/` and an
`AGENTS.md` that points to it. The skill, in English, holds guidelines for everything about a QNAP NAS: what it is at
the shell (a RAM-disk root, BusyBox, no systemd, no package manager, `getcfg` and `setcfg`), storage and paths, how
to build, sign and publish with this package (`bvs env run`, `bvs ldd`, `bvs qbuild`, the App Builder, the kernel
workshop, the package repository), ports, backups, and what it must not suggest, such as `apt` on the NAS or
allowing unsigned apps in App Center. It says plainly that signing with a hardware token is not supported and not
tested.

- Start the assistant in the project's folder (`cd <data>/projects/<name>`), not in `projects/`: each tool looks
  in the folder it starts in.
- Claude Code loads the skill from `.claude/skills/` when a task needs it. Codex CLI reads `AGENTS.md`. Gemini CLI
  reads `AGENTS.md` once the project's `.gemini/settings.json` holds
  `{"context": {"fileName": ["AGENTS.md", "GEMINI.md"]}}`.
- The package keeps the current skill in `/usr/bin/bucketvs/share/skills/qnap-dev`. A project made before the
  package had it, or a project copied from the Examples gallery, gets it by copying that folder into the project's
  `.claude/skills/`.
- The skill cites QNAP Club Poland's knowledge base (forum.qnap.net.pl) for what the NAS has and lacks.
- The package's build also writes the skill as `qnap-skill-<version>.zip`, the same files, for use outside the editor:
  unzip it into a project's `.claude/skills/` or into `~/.claude/skills/`. The skill's text is licensed CC BY 4.0 (its
  `LICENSE`): share and adapt it with the credit line "QNAP guidelines skill by QNAP Club Poland / qnaputilities.com, CC
  BY 4.0".

## Toolchains and environments

The Toolchains page downloads what is needed to build software on the NAS:

| Type | What it is | Examples |
|---|---|---|
| Root filesystem | a Linux system to unpack into an environment | Debian 12 and 13, Ubuntu 22.04 and 24.04, Alpine 3.22 |
| Cross toolchain | a compiler that builds for another processor | Bootlin toolchains for aarch64, x86-64 and armv7, with glibc or musl |
| Runtime | a language or build tool | Python, Go, Node.js, Zig, Rust, CMake, Ninja, Java (JDK) |
| QNAP | QNAP's own tools | QDK 2.5.0, for [`bvs qbuild`](#building-a-qpkg-with-bvs-qbuild) and Export as QPKG |
| Kernel source | a Linux kernel tree for the [kernel workshop](#kernel-workshop) | Linux 6.6.129 from kernel.org; QNAP's GPL kernel release of QTS 5.2.3 |

Entries for another processor are hidden; "Show entries for other processors" lists them, but they cannot be
installed. The Bootlin toolchains run on x86_64, so an ARM NAS has no cross toolchain.

### The catalog

The package carries a built-in catalog. The signed remote catalog at
`https://pub.qnaputilities.com/bucketvs/catalog.json` can add entries without a new package. bvsd uses it only
when its ed25519 signature matches the key built into the package and it is not older than the catalog in use.
"Refresh the catalog" fetches it at once, and the built-in cron job "Toolchain catalog refresh" fetches it every
day at 04:15.

### Installing and removing

Installing is an admin's job, shown with its progress on the Jobs page:

1. The archive is downloaded into `toolchains/cache/` in the data folder. An interrupted download resumes when the
   install is started again.
2. Its sha256 is checked against the catalog. A mismatch deletes the file.
3. A cross toolchain or a runtime is unpacked into `toolchains/<kind>/<id>` and its archive deleted. A root
   filesystem or a kernel source stays as its archive; each environment or kernel workspace unpacks its own copy.
   QNAP publishes its GPL kernel release in two parts: both are downloaded, each checked by its own sha256, and
   joined.

A full volume fails the install with "no space left on" and the mount point of the volume.

"Remove" deletes an installed entry. Environments made from a root filesystem are copies and stay.

### Using a toolchain or runtime

`bvs use <id>` prints the `export` lines (such as `PATH`) that put an installed cross toolchain or runtime to
use in a shell:

```bash
eval "$(bvs use bootlin-aarch64-glibc-stable)"
```

The Toolchains page shows the same lines under "bvs use" for each installed entry.

### Environments

An environment is a Linux system unpacked from a root filesystem into `envs/<name>` in the data folder. It is
changeable: `apt install build-essential` works inside a Debian or Ubuntu one, and what you install stays there.

- "New environment" (admin) downloads the root filesystem first when it is not installed, then unpacks it.
- While there is no environment, the list offers the Ubuntu 24.04 builder, the one the app examples build in, with one
  button (admin): its job shows in the list, and the environment appears when the job is done. Nothing is created by
  itself.
- A name has 1 to 64 characters: lower-case letters, digits, `.`, `_` or `-`, starting with a letter or a digit.
- A preset from the Examples gallery also installs packages with apt-get before the environment appears: the
  Ubuntu 24.04 builder (the default builder) and the Debian 12 builder, both full, with the App Builder's build
  tools and patchelf; Clang/LLVM (LLVM 18), Fortran (gfortran 13), Kernel builder, .NET 8 SDK and Kotlin (Kotlin
  1.3.31), all on Ubuntu 24.04. Every preset's glibc is at most 2.39, so what it builds runs on every NAS the
  package runs on ([Which NAS models an app runs on](#which-nas-models-an-app-runs-on)). When
  apt-get fails, no environment is left behind. From a shell: `bvs env create <name> --from <rootfs-id>
  --package <name> --package <name>`.
- "Clone" makes a full copy under a new name, for example to try a change without touching the original.
- "Export" packs the environment into `toolchains/exports/<name>-<time>.tar.zst`.
- "Delete" removes it. bvsd refuses while a shell or a process runs inside it. Files in `projects/` are not in
  the environment and stay.

Inside an environment:

- `projects/` of the data folder is visible as `/projects`, and a shell starts there;
- `HOME` is `/root` of the environment, and `LANG` is `C.UTF-8`;
- `/proc`, a read-only `/sys` and a small `/dev` of its own are mounted, and the NAS's `resolv.conf` and `hosts`
  are copied in;
- the mounts belong to the shell's own mount namespace and vanish when it ends, so nothing stays mounted on the
  NAS.

The same from a shell (the examples call `bvs` by name; while the package runs, a cron job and an SSH session on the
NAS can do the same):

```bash
bvs env list
bvs env create deb12 --from debian-12-x86_64
bvs env shell deb12
bvs env run deb12 -- make -C /projects/hello
```

`bvs env shell` and `bvs env run` start in the folder you run them from. A folder under the data folder's
`projects/` is the same folder under `/projects` inside the environment. Any other folder (a CI job's checkout, a
folder in a share) is mounted at `/work` and the command starts there. From `/` they start in `/projects`.

### Building a QPKG with bvs qbuild

`bvs qbuild` builds a QPKG from the folder you run it in (a folder with `qpkg.cfg`, as the QPKG template of New
project makes). It runs QNAP's `qbuild` inside a Debian or Ubuntu environment. qbuild calls system programs by
their full paths, and in the environment those paths lead to the environment's programs instead of the NAS's own.
The package lands in `build/` of the folder.

```bash
bvs tc install qdk-2.5.0-x86_64      # once; qdk-2.5.0-aarch64 on an ARM NAS
cd /share/CACHEDEV1_DATA/BucketVS/projects/MyTool
bvs qbuild                           # --env <name> picks the environment
```

It uses the environment `ubuntu2404-builder` (the example "Ubuntu 24.04 builder"), else `debian12-builder`, else
another Debian or Ubuntu one, and needs `rsync` in it. When something is missing, it stops and names the command that
adds it, for example `bvs env run ubuntu2404-builder -- apt-get install -y rsync`. Arguments after `bvs qbuild` go to
qbuild. Before qbuild runs, `bvs qbuild` says which [code-signing case](#code-signing-a-qpkg) applies to the folder.

An environment has its own files and tools. Everything in it runs as root, sees every process on the NAS and uses
the NAS's network.

### Code signing a QPKG

App Center checks a package's code signature when it installs it. Signing needs a code-signing certificate that
QNAP issued to you. `bvs qbuild` names one of three cases on its first line, for example `bvs: code signing, case 1
(a certificate as files): …`:

| Case | The folder holds | What happens |
|---|---|---|
| 1, a certificate as files | `certificate` and `private_key`, optionally `ca_certs` (the chain) | qbuild signs the package with `openssl`; the environment needs `openssl` and `bsdextrautils` |
| 2, a certificate on a hardware token | `certificate_hsm` | not supported and not tested: `bvs qbuild` stops before anything is built |
| 3, no certificate | none of those files | the package is built unsigned |

- **Case 1.** Put the three files next to `qpkg.cfg`. When the environment lacks a program, `bvs qbuild` names the
  command, for example `bvs env run ubuntu2404-builder -- apt-get install -y openssl bsdextrautils`. It refuses a
  folder with only one of `certificate` and `private_key` (qbuild would warn and build an unsigned package) and one
  where either file is empty. Keep the files out of git: the QPKG template's `.gitignore` lists them. In CI, keep
  them as the repository's secrets and write them into the folder for the build only.
- **Case 2.** qbuild would read the key's place on the token from variables that do not reach it inside the
  environment, and the package has not been tested with a token. Build without `certificate_hsm` and sign the
  package on the machine that holds the token with QDK's own tools; the QNAP skill for AI assistants
  (`references/signing.md`) describes QDK's route, which this package has never run.
- **Case 3.** On a QTS 6.1 unit, `qpkg_cli -m` of an unsigned package installed nothing, while `sh <file>.qpkg` in
  a shared folder installed and started it. The [package repository](#package-repository) takes it only while an
  admin allows unsigned packages, and App Center on another NAS installs it only if that NAS's owner
  allows apps without a valid digital signature, which this package never recommends. **Export as QPKG** always
  makes this case.
- `qbuild --sign` adds a GnuPG signature, which is not the code signature App Center and the repository read.

## App Builder

The App Builder compiles chosen programs into one folder, the app's prefix (by default
`/usr/app/my_app_prefix`), inside an environment.

The **Examples** tab of **Apps** holds ready apps, environments, CI workflows, cron jobs and Editor projects.
**View the code** on a card shows what the example makes before you use it, for any role: an Editor project's files,
the workflow file, the cron job's command, or for an app its recipes in build order with each recipe's `build.sh`
and `recipe.json`. The viewer is read only and shows text files; an environment has no file, so its card has no
such button. Using an example needs the developer role, and an environment the admin role: a developer's card of
an environment has no **Use this example**.

### Recipes

Each program is built from a recipe. A recipe names its source with a sha256, the recipes it depends on, the build
tools the environment needs, and a check that proves the result works (such as `bin/bash --version`). There are
recipes for libraries (zlib, openssl, ncurses, readline and others), shells (bash, zsh, busybox), tools
(coreutils, curl, git, jq, rsync, tmux, htop, nano and others), servers (nginx, redis, openssh, dropbear) and a
language (lua).

- A recipe is built only after the recipes it depends on.
- A rebuild reuses a recipe's result while its inputs are unchanged (the source, the recipe, its dependencies,
  the environment's compiler and C library, the prefix, and whether it links dynamic or static) and every file it
  installed is still in the prefix.
- The link mode decides how programs find their libraries. **dynamic** gives every program a RUNPATH of
  `<prefix>/lib`. **relocatable** writes it relative to the program's own folder (`$ORIGIN/../lib`), so the tree
  finds its libraries in any folder; paths compiled into programs, such as the terminfo folder, still name the
  prefix. **bundle** is relocatable and also copies into `lib` the libraries the programs load from the build
  environment (such as libstdc++), so the app needs only the NAS's C library. **static** links the recipes that
  support it statically. Switching among dynamic, relocatable and bundle rebuilds nothing.
- A static app refuses recipes that cannot link static (openssh, dropbear and tmux), naming each with the reason,
  unless the app is set to build them as bundle. The New app page and **Change** offer all four modes and, for a
  static app, that choice.
- A static program that looks up users, groups or host names reads them only from the NAS's own `/etc/passwd`,
  `/etc/group`, `/etc/hosts` and DNS, whatever `/etc/nsswitch.conf` names, so accounts from a domain or LDAP are
  unknown to it. Build such programs dynamic when those accounts matter. The panel warns when a static app
  includes one.
- After the last recipe, every recipe's check runs again over the finished prefix.
- Config files under the prefix's `etc/` survive a rebuild: an edited file stays, and the new default goes next
  to it as `<file>.bvs-new`.

### Recipe updates

New and updated recipes also come as a signed catalog from pub.qnaputilities.com, so they reach the NAS before the
next release of Bucket & Code Server IDE. bvsd checks for a newer catalog once a day. It uses one only when its signature matches the
key built into bvsd, every file matches the catalog's list, and its recipes are newer than the ones that came with
the package; a catalog that fails a check is never used. While a catalog is in use, New app marks the recipes it
brings **new in the catalog** or **updated by the catalog**.

The **Recipe library** card on the Apps page shows which library builds use (**Package library** with the package's
version, or **Catalog** with the catalog's version) and the last check with its result. An admin has two buttons
there; the second one appears while a catalog is in use:

- **Check for recipe updates** checks now instead of waiting for the daily check.
- **Use the package's library** goes back to the recipes that came with Bucket & Code Server IDE. The catalog stays set aside until
  a newer one arrives, and builds then use the newer one.

After a switch, a rebuild reuses every recipe whose files did not change. Before each build bvsd checks the
catalog's files again. If one changed on the NAS, the build stops before anything runs, and builds use the previous
catalog or the package's library.

### Building an app

**Apps**, **New app** (developer role) asks for a name, the prefix (`/usr/app/<name>` unless you choose another),
the environment the app builds in (a Debian or Ubuntu environment; the Examples gallery makes the "Ubuntu 24.04
builder", which the page chooses first), the target (this NAS's architecture, or aarch64 or armv7 as a cross build),
which NAS models it runs on, the link mode and the recipes. The
recipes they depend on are added in build order.

An app can also take the recipes of one project that has its own `recipes/` folder. Choose the project in **Recipes
from a project** in the **Packages** step. Its recipes are listed first, under the project's name and marked
**project**, and the library's follow. A project recipe with the name of a library recipe replaces that recipe in this
app and shows the library version it replaces. A recipe the project leaves out is named with the reason. A recipe you
chose that the project's list lacks is named under the select and left out of the app; choosing a list that has it
brings it back. The app's page names the project, and a developer opens its Project view from that name. A viewer
sees the recipes of the app's last build in the order its log names them, each with its state and log; before the
first build there is no order to show.

The link mode is a card each, with a picture of where the programs find their libraries, what the mode costs in
size, in updates and in where the app runs, and which recipes take it. Before you choose recipes the cards count the
whole recipe library; when you come back to the first step they count the recipes of your build. For a static app
you choose what happens to recipes that cannot link static: build them as bundle, or leave them out.

- **Build** (or **Rebuild**) runs a job: it downloads each source and checks its sha256, installs the build tools
  the recipes need in the environment, builds in a copy of the app's previous build, and checks every program.
- A build that fails, is cancelled or is interrupted leaves the app as it was: activation, download and export keep
  using the newest build that passed.
- One build of an app runs at a time, and a build waits while programs of the active app run (stop them first).
- The app's page shows the build order, each recipe's state and log, the files under the prefix, and the archive of
  the newest successful build (**Download**, a `.tar.zst`).
- **Change** on the app's page sets another environment, which NAS models it runs on, the link mode, the project it
  takes recipes from, or the recipes; the next build applies it.
- **Dependencies**, a tab of the app's page, shows what every ELF file of the newest build needs and where the NAS's
  loader finds it, read without running anything. A missing library, a version the library does not define, or a
  RUNPATH that points outside the app is listed as a finding with its fix. A developer runs a fix with its button:
  **Add … and build**, **Switch to … and build**, **Build again**, **Patch the RUNPATH**, **Set the interpreter**,
  **Choose another environment** or **Build for this NAS only**. A fix with no button (installing a package into an environment, building for
  another machine) is a sentence to act on. Below the findings, every file and what it needs is drawn as a graph
  (programs, the app's libraries, the NAS's libraries and the missing ones in columns; a dashed line is a version
  the library lacks, a dotted line a library found nowhere) or as a tree; choose a file for its facts. On the NAS,
  `bvs deps <app>` prints the same.
- **ELF inspector** (Toolchains, developer role) checks one file: from an app's build, from the projects folder or
  from an environment, and for an admin any file on the NAS, as it would run on this NAS or inside an environment.
  `bvs ldd <file>` does the same on the NAS (`--env <name>` for an environment).
- **Every build** runs the same check over its result before it replaces the app, and fails on a problem that stops
  a program on this NAS, or on a program that needs a newer glibc than the app is for, with the problem and its fix
  in the job's message.
- **Patch tool** (the Dependencies tab) sets the RUNPATH (the app's `lib`, the `$ORIGIN` form, or a folder of the
  app) or the program interpreter of chosen files of the newest build, with the environment's patchelf, on a copy
  that replaces the build only when it succeeds. **Preview** shows each file before and after and which libraries
  the new RUNPATH does not find in the app; **Apply the patch** runs it. The next build writes the RUNPATHs in the
  app's link mode again.

A Qt build runs Qt's own tools (moc, rcc, and Linguist's lrelease for translations). In a cross build the Qt recipes
(`qtbase`, `qttools`) therefore first build those tools for this NAS, into `host/` under the prefix, and then Qt for
the target; the recipes built on Qt run the tools from there. Nothing on the target runs `host/`. It is part of the
app's build, so the archive and **Export as QPKG** carry it too.

### Your own recipes

When the recipe library has no recipe for a program, a project can carry its own. The recipes go into the project's
`recipes/` folder, in the library's format, and only the apps that name the project build them. Making a package
this way takes five steps:

1. **Make the project.** In [New project](#new-project), the template **Package from source** makes a folder with a
   starter recipe named after the project, a Makefile for two apps (one for this NAS's processor, one for aarch64, a
   cross build on an x86_64 NAS) and the package's start and stop script from the **Service** step.
2. **Write the recipe.** **New recipe** in the editor adds another recipe from the same starter
   ([Opening the editor](#opening-the-editor)). Put the program's real source address and its sha256 into `recipe.json`
   (`curl -fsSL <address> | sha256sum` in the editor's terminal prints the sum), and its build steps into `build.sh`. A
   recipe folder holds `recipe.json`, `build.sh` and the vendor file `recipe.json` names, and nothing else, so a patch
   goes into `build.sh` as a here-document.
3. **Check it.** Every save of a recipe's file checks the project's recipes and lists the findings in the
   **Problems** view; `make lint` in the terminal and the **Recipes** card of the [Project view](#the-project-view)
   give the same findings. The check reads `recipe.json` and parses `build.sh` with `bash -n`. It never runs the
   recipe.
4. **Build.** **Build** in the Project view makes the two apps when they are missing and builds them one after the
   other. A build reads the project's recipes once, when it starts, so a file saved later waits for the next build. A
   project recipe with the name of a library recipe replaces that recipe in this project's apps only, and the build's
   log says so.
5. **Pack and install.** `make qpkg` in the editor's terminal packs the newest successful build of each app into the
   package with QDK from Toolchains, and leaves out an app without one. With the code-signing files in `qpkg/` beside
   `qpkg.cfg`, or in a folder named by `QPKG_CERT_DIR`, the package is signed ([Code signing a
   QPKG](#code-signing-a-qpkg)), and **Test deploy** in the Project view (admin) installs it on this NAS.

A few rules apply to project recipes:

- A recipe the project leaves out (an extra file in its folder, a link, a file over its size limit) is named with
  the reason on the Recipes card and by the check, and a build that needs it is refused.
- When the project's folder is renamed or deleted, its apps do not build: the build answers "No project named …"
  until **Change** on the app's page sets the project's new name or **None: the library only**.
- Project recipes run as root in the builder environment, as the library's do. Any developer can write and build
  them, as any developer can run a project's Makefile through Build. A viewer sees the names of an app's recipes in
  its build log, never the recipe files.
- A project recipe stays in its project. It joins the recipe library, and the signed catalog every NAS takes, only
  through a reviewed change to Bucket & Code Server IDE itself.
- A recipe whose build needs npm, Go or Cargo packages lists them in a vendor file. `npm-vendor.py`, `go-vendor.py` and
  `cargo-vendor.py` in `/usr/bin/bucketvs/share/bucketvs/tools/` write that list; they need a Python runtime from
  [Toolchains](#toolchains-and-environments).

### Which NAS models an app runs on

A program built against glibc starts only on a NAS whose glibc is at least as new as the newest version the program
asks for. The package counts on glibc 2.39 or newer on every NAS it runs on (QTS 6.0, QuTS hero 6.0 and newer), its
glibc floor. 2.39 is the glibc of Ubuntu 24.04 and of QuTS hero 6.0.1, read from that firmware's own
`/lib/libc.so.6`; QNAP publishes no glibc version, and its release notes list no QTS 6.0. The test units on QTS 6.1.0
and QuTS hero 6.1.0 run 2.43.

**Runs on** in New app and in **Change** chooses one of two:

- **Every NAS the package runs on** (the default): no program may need a glibc above 2.39. Build such an app in an
  environment whose glibc is 2.39 or older, such as the Ubuntu 24.04 builder (2.39) or the Debian 12 builder
  (2.36). This is the choice for **Export as QPKG** and the package repository.
- **This NAS only**: programs may need up to this NAS's own glibc, for software that needs a newer one. Such an app
  is built for this NAS's architecture and is never exported as a QPKG.

A cross build (aarch64 or armv7) is always for every NAS. New app warns when the chosen environment's glibc is
above what the app allows. After each build, bvsd reads the glibc version every program and library asks for, which
works for a cross build too. A file above the limit fails the build, and the fixes are named: another environment
(**Choose another environment**), the static link mode, or **Build for this NAS only**. The app's page says which glibc its newest
successful build needs, and the **Dependencies** tab lists each program above the limit.

On the NAS, `bvs tree-deps <prefix> --tree <folder> --glibc-max 2.39` runs the same check over any app tree.

### Activating an app on the NAS

**Activate** (admin) makes the prefix a link to the app's newest successful build, so its programs run on the NAS
from `/usr/app/<name>/bin`. Activation never replaces anything it did not make: a folder, a file or another link at
the prefix is refused with the reason. The QTS system disk is rebuilt at each boot, so the package makes the links
again at every start. **Deactivate** (admin) removes the link; deleting the app deactivates it first and moves it
to the trash for 7 days. A developer can delete an app only while it is not active. An app built for another architecture cannot be activated here; download its archive for a NAS of
that kind.

### Exporting an app

**Export as QPKG** (admin) packs the newest successful build as a QPKG of its own, which installs the app on another
NAS without this package. It refuses an app built for this NAS only, and it stops before packing when a program of the
build needs a glibc above 2.39 (see [Which NAS models an app runs on](#which-nas-models-an-app-runs-on)). It needs
QDK: install "QNAP QDK 2.5.0" on the Toolchains page, QNAP tab. Until QDK is installed the button is not shown and the
app's page says where to get it. The export runs qbuild inside the app's environment, the same way as [`bvs
qbuild`](#building-a-qpkg-with-bvs-qbuild). Every build installs `rsync` there; for an app last built with an older
version of the package, the refusal names the command that adds it. The exported package links its prefix at start and
removes the link at stop. It is not code-signed: its job log names [case 3](#code-signing-a-qpkg), and so does a line
under the button. In an SSH session, `/etc/init.d/<name>.sh disable` and `enable` set App Center's switch for the
exported package and leave the prefix link as it is. At every start the exported package also adds a line to
`/etc/profile` and `/root/.profile` that puts the app's `bin` and `sbin` on the `PATH` of an SSH login, after the NAS's
own commands, for admin and every other account that may log in; a removal takes the line out again.

## Package repository

The package keeps an App Center repository: QPKG packages published into it can be installed on other NAS units from
App Center, like any repository's apps.

- **The address.** `bvs repo list` prints it (and `GET /api/v1/repo`): `<base>/repo/repo.xml`, where `<base>` is
  the published packages hostname (else the published main hostname) while the package is published on the internet
  ([Publishing on the internet](#publishing-on-the-internet-cloudflare-tunnel)), else the public URL when one is set, else `http://<NAS address>:9950` (`https://…:9951` when plain HTTP only
  redirects). On the other NAS: App Center, **Settings**, **App Repository**, **Add**, and that address.
- **Publishing.** `bvs repo publish <file.qpkg>` (on the NAS, as root) or `POST /api/v1/repo/packages` with the file
  as the body (developer role). The architecture comes from the QDK file name `<name>_<version>_<arch>.qpkg`
  (`x86_64`, `arm_64`, `arm-x41`), or `--arch`. The name, version, description and author come from the package
  itself; `PUT /api/v1/repo/packages/<name>/meta` and `…/icon` (a PNG, admin) change what App Center shows. A package
  cannot be named `default`: that is the name of the repository's own icon.
- **Signed packages only.** A package without a code signature is refused, because App Center installs such
  packages only on a NAS set to allow them. The repository takes a package as signed when its code-signing block is
  a PKCS#7 signature with a certificate and a signer. It does not check that signature or its certificates against
  QNAP's; App Center does that when it installs the package. An admin can turn on `repo.accept_unsigned` in the
  settings for a network where that is known; those packages are marked in their description. Packages the App
  Builder exports are not signed, so they need this setting.
- **Versions.** Every version stays downloadable, but App Center sees one version per package, the one an admin
  lists. An admin's first version of a package is listed at once; a later one when the admin moves it
  (`POST /api/v1/repo/packages/<name>/latest`, or `bvs repo publish --latest`). A developer's version, a new
  package's first one included, is listed nowhere until an admin makes a version the latest: every NAS that added
  the repository installs the listed version as root. Deleting a version (admin) removes all its architectures;
  when it was the listed one, no version is listed until an admin picks another.
- **Uploads.** Two uploads are copied at a time; the repository refuses one that would leave less than 256 MiB
  free on the data volume. The repository is not part of a backup: after a restore, publish the packages again.
- **In the panel.** The **Packages** page shows the address to copy, the packages with their versions and whether
  they are signed; a developer uploads a QPKG there, and an admin lists a version as the latest and deletes a version.
  Its **Code signing** card shows the [three cases](#code-signing-a-qpkg), how many listed versions carry no
  signature, and the admin's switch for unsigned packages, which asks before it turns on. An app's page shows its pipeline (build, export as QPKG, publish) with a
  **Publish** button.

## Kernel workshop

The kernel workshop builds kernel modules for the kernel the NAS runs, checks each one against that kernel, and
loads it. QNAP has not published the source of its current kernels (its GPL releases end at QTS 5.2.3), so the
workshop reads what the running kernel shows about itself and configures a kernel tree to match it.

### What you need

- The Kernel builder environment from the Examples gallery (Ubuntu 24.04 with GCC 13 and the kernel build
  tools). A module must be built with the same GCC major version as the kernel; the workshop compares them.
- The admin role, and root in a terminal: the `bvs kernel` commands use bvsd's CLI socket or load modules.

### The profile

`bvs kernel profile` shows what the running kernel is: its release (`6.6.129-qnap`), the compiler QNAP built it
with, the vermagic of QNAP's own modules, the size of `struct module` in them and where init and exit sit, whether
modules must be signed, and the settings a kernel tree gets to match it.

### The workspace

`bvs kernel prepare` makes the workspace of the running kernel in `kernels/<release>` of the data folder:

1. It takes the kernel source the catalog has for this version (`bvs kernel sources`): QNAP's own GPL tree when
   QNAP published one, else the tree from kernel.org, which lacks QNAP's own changes. `--source` picks another.
2. It configures the tree: with QNAP's configuration for this model when the tree is QNAP's, else with the
   settings of the profile. `--option NAME=m` adds a setting, for example a driver.
3. It prepares the tree for module builds (`--full` builds the whole kernel instead, which takes much longer and
   lets the check see the kernel's exported symbols).
4. It checks that the tree builds for exactly the running kernel's release.

It runs as a job on the Jobs page. A failed prepare keeps the tree of the previous one.

### Building a module

- Your own module is a folder in `projects/` with a `Kbuild` file. `bvs kernel make` in that folder (or
  `bvs kernel make --project <folder>`) builds it. The example "Kernel module" is such a project.
- A driver of the kernel that QNAP does not ship is built from its directory of the kernel tree:
  `bvs kernel make --dir drivers/usb/serial`. The example "USB serial driver ch341" builds the ch341 driver this way; its setting comes first
  with `bvs kernel prepare --option USB_SERIAL=m --option USB_SERIAL_CH341=m`.

The modules a build makes are kept in `kernels/<release>/modules/`. A module QNAP ships for this kernel is not
kept: QNAP's own copy stays the one that loads. `bvs kernel make --build-only` keeps nothing there: the module stays
where it was built. The CI workflows of New project's kernel templates build this way.

### The check and loading

`bvs kernel check <module>` compares a module with the running kernel: the vermagic, the size of `struct module`
and where init and exit sit in it (against QNAP's own modules), whether a signature is required, the modules it
depends on, and every symbol it needs. Each finding says what does not match and what to do. Exit status 3 means
the module must not be loaded.

`bvs kernel load <module>` runs the check again and loads the module, the modules it depends on first (QNAP's own
from the NAS, or yours). A module with an error in the check is never loaded. `bvs kernel unload <module>` removes
it; a module in use stays loaded. Every load and unload is written to `log/kernel-modules.log`.

QNAP's kernel does not list its data symbols, so the check cannot look up symbols such as `jiffies`; it names them
as warnings. If one of them is missing, the kernel refuses the module when it loads.

### Modules in an app

An app can name modules of the workspace. Activating the app loads them before its link is made, and every start of
the package loads them again once the panel is up; the Kernels page shows a load that is still going. Deactivating
it unloads them first; a module in use keeps the app active.

- If the NAS restarted while modules were loading, the next start loads none and shows a notice: load them by hand
  or activate the app again.
- If the NAS stopped without the package's ordered stop (a crash, a power cut or a hard reset) in a boot in which
  the package loaded modules, the next start does not load those modules, since one of them may have caused the
  crash. The notice and the Kernels page name them. Loading one by hand, or activating its app again, lets the start
  load it again. A normal shutdown or restart of the NAS stops the package in order and changes nothing.
- A module whose start never ends (a driver waiting for its hardware) leaves the panel working. A notice names it
  after a minute; other loads and unloads wait for it, and a restart of the NAS ends it.

After a QTS update to another kernel, the modules must be built again in the new kernel's workspace. The same
holds after a restore onto another data folder: `kernels/` is not part of a backup.

### In the panel

The **Kernels** page shows the running kernel's profile, the kernel sources of the catalog, the workspace and its
modules. An admin prepares the workspace there (a source, the environment, extra settings, the whole kernel or
not) and builds modules from a project folder or a directory of the tree; both run as jobs. **Check** (developer)
shows each finding with its fix; **Load** and **Unload** (admin) do what `bvs kernel load` and `unload` do, and a
refused load shows the check that stopped it. A `.ko` file anywhere under the data folder can be checked by its
path.

## Cron jobs and operations

The Jobs page has two lists: scheduled jobs (cron) and operations (the long jobs bvsd runs, such as installs and
moves).

### Scheduled jobs

A developer creates and changes their own jobs; an admin changes all of them. A job has:

- Schedule: five cron fields (`0 3 * * *`), six with seconds first, or a descriptor (`@daily`,
  `@hourly`, `@every 6h`). A `CRON_TZ=<zone>` prefix picks another time zone; without it the job runs in the
  NAS's time zone, shown at the top of the page.
- Command: run as root by the package's `bash -lc`. Several lines run as one script, and its output becomes
  the run's log.
- Where it runs: on the NAS, or inside an environment (with the environment's `bash`, or `/bin/sh` where it
  has none).
- Working folder: relative to `projects/` in the data folder, or a full path. Inside an environment only
  `projects/` is visible, as `/projects`.
- Timeout: in seconds (0: none). A run still going after it is stopped and marked timed out.
- While a run is going: the next one is skipped, or it waits (at most one waits).
- On failure: a failed or timed-out run sends a [notice](#notices).

Each job keeps its last 100 runs with the end of their output (4 KB). The full log of a run is kept 30 days.

A job runs as root on its owner's behalf, with the owner's home folder as `HOME`, so it runs only while its owner is
an enabled developer or admin. While the owner is disabled or has the member or viewer role, each of the job's times
is recorded as a skipped run that says why, such as "its owner ola is disabled", and **Run now** answers with the
same reason. No notice is sent. A run that is going when the owner is disabled or given a role below developer is
stopped. The job runs again once the account is enabled, or a developer again. Deleting an account deletes its jobs.
Built-in jobs run whatever happens to an account.

When a command ends, processes it started in the background keep running. On a timeout, a delete of the job or a
stop of the package, the run's whole process group gets SIGTERM, then SIGKILL after 5 seconds.

On the nights the clocks change, the jobs follow Debian cron's rule. A job with `*` at the start of its minute or
hour field (also `*/N`), `@hourly` or `@every` follows elapsed time: in autumn it runs in both passes of the
repeated hour, and in spring the skipped hour does not come. Any other job runs each of its times once: a time in
the repeated hour runs once, and a time in the skipped hour runs right after the change.

Every signed-in account, a `viewer` too, reads the jobs' commands and their runs' output. Keep passwords and tokens
out of a command: have it read them from a file under `<data>/etc` that only root reads (`chmod 600`).

### Built-in jobs

| Job | When | What it does |
|---|---|---|
| Maintenance | 03:30 daily | removes what nothing changed for 10 days from `tmp/` in the data folder; rotates `log/audit.log` over 10 MB |
| Empty the trash | 03:45 daily | removes what was moved to `trash/` in the data folder (such as a deleted account's home folder) more than 7 days ago |
| Toolchain catalog refresh | 04:15 daily | fetches the signed toolchain catalog |
| Rotate nginx's logs | every hour, at minute 7 | renames `nginx-access.log` or `nginx-error.log` to `.1` once it is over 20 MB (five old files kept) and tells nginx to reopen its logs |

An admin can change a built-in job's schedule, timeout, policy and notice, and switch it off. Its name and command
stay.

### Operations

Installs, environment jobs, data moves and switching a module on run as operations. The list shows the last 50,
newest first, with their progress and log. Operations and their logs are kept 30 days. An operation that was running
when the package stopped shows as `interrupted`. A developer cancels their own operations; an admin cancels any.

## Backups

The **Backups** page (admin) sets what is backed up, when, where the copies go and how many are kept. **Back up now**
runs a backup at any time. Scheduled backups are off on a new installation; once **Scheduled backups** is on, a backup
runs daily by default (the built-in cron job `backup`). It keeps the newest run of each of the last 7 days, 4 weeks
and 6 months, and always the newest good run. One backup, verify, restore or delete runs at a time.

Two runs on one day count as one day: when the newer finishes, the earlier run of that day is removed. A **Back up
now** in the afternoon therefore replaces that morning's scheduled run; copy a run elsewhere first when that
morning's state matters.

### What a backup holds

A run is a folder `<data>/backups/<id>/` with a `manifest.json` that lists every file with its sha256:

| Part | What |
|---|---|
| settings | `etc/`: settings, accounts, TOTP secrets, the TLS key, the backup keys |
| Git | Forgejo's dump: repositories, wikis, LFS objects, attachments and its database |
| SVN, buckets | a copy of every repository, made incrementally: a revision already copied is not copied again |
| VS Code settings | each account's VS Code settings, key bindings, snippets and the list of its extensions (not the extensions' files: the Extensions view installs them again) |
| environments | the Toolchains environments (off by default: they can be made again) |
| projects | every project folder with its Git history and the files not committed yet, without `node_modules` |
| HOME folders | each account's HOME folder, without caches and the AI tools' own installs (their Install button puts them back) |
| App Builder | each app's settings and the history of its builds; a build makes the app's files again |
| package repository | the published packages, their icons and the index App Center reads |

The last four are in every backup, whatever is ticked on the page. A folder that has not changed since the previous
backup is linked to that backup's copy and takes no space.

The run becomes visible only when every part is written and checked; a failure or a full volume removes the
unfinished folder and leaves the earlier runs as they were. **Verify** reads a run back completely: every archive's
sha256, `svnadmin verify` of every revision, `git fsck` of every repository and an integrity check of Forgejo's
database.

### Targets

Every run is copied to each target on the page:

- **Local folder** on this NAS: the run as it is, with unchanged files linked to the previous copy, so each copy
  costs only what changed. Not encrypted. On the same disks as the data, it guards against mistakes, not against a
  failed volume.
- **rsync** (`user@host:path`) over SSH. **Sign in with the package's backup key**: the page shows the public key to
  add to the other machine's `authorized_keys`.
- **rclone** (`remote:path`, a remote from `<data>/etc/rclone.conf`): cloud storage and anything rclone reaches.

A copy that leaves the NAS (rsync, rclone) must be encrypted: turn on **Encrypt copies that leave the NAS** and
**Create the key**. The copies are encrypted with age to its public half; **the secret half is shown once** and bvsd
does not keep it. Without it no NAS, this one included, can restore an encrypted copy: store it away from this NAS.
A new key encrypts the runs from then on; older copies need the old secret key.

A target that fails is named in the run's result; the run itself stays good.

`rclone.conf` holds the remotes' credentials. Before a copy or a **Test** uses it, bvsd makes it (and the backup key)
readable by root alone (mode 600) when it is not, and the job's log says so.

### Restoring

**Restore** on a run puts back either everything, one repository (an SVN repository, a bucket or a Git
repository) or one project folder:

- Nothing is replaced before it is checked: every archive against its sha256, every SVN copy with `svnadmin verify`.
  A copy that fails stops the restore before anything live changes.
- The live data that is replaced goes to `<data>/trash` for 7 days, so a restore can itself be undone.
- **One repository** or **one project folder** is put back while the modules keep running. A project folder you
  deleted by mistake comes back with its history and its files not committed yet.
- **Everything** stops the modules, swaps the data, and restarts bvsd on the restored settings: every session ends,
  and the accounts and passwords are the ones in the backup. A project or HOME folder made after the backup stays.

## Notices

bvsd writes every notice to the QTS System Logs. It can also send them to chat and mail services through
shoutrrr URLs (Telegram, Discord, Slack, e-mail, a generic webhook and others).

| Notice | When | Switch for the URLs |
|---|---|---|
| job failure | a cron run of a job with a notice on failure failed or timed out; an operation failed | `notify.on_job_failure` |
| backup | the result of every backup, verify and restore, the good ones too; a scheduled backup that could not start | `notify.on_backup` |
| restart loop | a process started restarting in a loop | `notify.on_crashloop` |
| certificate | an uploaded certificate has 30 days or less; nginx fell back to the self-signed certificate; a new self-signed certificate was made | `notify.on_cert_expiry` |
| system | settings refused at start, an unfinished data move found, leftover processes ended, the configuration could not be written | always sent |

- The switches decide only what goes to the URLs. The System Logs get every notice.
- The same notice goes out at most once in 10 minutes. The next one after that says how many were held back.
- Each URL gets its own `shoutrrr send` with a 10-second limit. A failure is logged with the URL's scheme only,
  because the rest of a URL holds its token.

The URLs and the switches are under **Notifications** on the System page; the switches are on by default. An admin
edits them there. Other accounts see the switches and only the scheme of each URL, because the rest of a URL holds
its token. The NAS admin can also set them over the CLI socket:

```bash
/usr/bin/bucketvs/bin/curl --unix-socket /usr/bin/bucketvs/var/run/bvsd-cli.sock \
  -X PUT -H 'Content-Type: application/json' \
  -d '{"notify":{"shoutrrr_urls":["telegram://<token>@telegram?chats=<chat id>"]}}' \
  http://bvsd/api/v1/settings
```

## Logs and the support bundle

Each log is `log/<name>.log` in the data folder. The logs of bvsd and of every process it supervises are rotated at
20 MB, and five old files are kept. `nginx-access.log` and `nginx-error.log` have the same limits, checked at every
start of bvsd and every hour by the "Rotate nginx's logs" job, so one of them can pass 20 MB by up to an hour's
traffic. The "Maintenance" job rotates `audit.log` once it is over 10 MB and keeps three old files. The Logs page
(admin) shows the last lines of any log, filters them and follows new lines as they come:

| Log | What writes it |
|---|---|
| `bvsd` | bvsd itself: starts, stops, applied settings, problems |
| `nginx` | nginx's own output |
| `nginx-access`, `nginx-error` | every request, and nginx's errors |
| `audit` | every request on the CLI socket |
| one per module process | each module's program |

The System page shows the version, the platform, the time zone and every process bvsd supervises.
"Download the support bundle" (admin) makes one archive for a bug report:

- the settings with the SMTP password and each shoutrrr URL masked;
- the rendered `nginx.conf`, the certificate's details without its key, the volumes;
- the state of every module, process and recent operation;
- the last 2000 lines of each log, with every query string in them replaced by `?(query removed)`.

It never holds passwords, sessions, tokens or keys.

## Updates

bvsd reads `https://pub.qnaputilities.com/bucketvs/latest.json` at start and every 24 hours. When it names a newer
version, the System page names it, with the link the feed gives. bvsd only reports the new version: it never
downloads or installs a package. A failed check changes nothing in the panel.

### Installing a new version

Install the new package over the old one, from App Center or with **Install Manually**. The installation stops
bvsd, replaces the program files and starts it again; the data folder, the settings, the accounts and every repository
stay, and so do the panel's sessions. The editor instances stop with bvsd, so the editors' open terminals end; an instance starts
again when its account opens the editor.

**From a version that used ports 6050, 6051 and 6052.** Earlier builds listened on 6050 (HTTP), 6051 (HTTPS) and
6052 (Git over SSH). The first start of the new version moves each of these to 9950, 9951 and 9952 and says so in a
notice; a port you chose yourself stays. Change what names the old ports: bookmarks, a Plesk proxy to 6050, and git
remotes such as `ssh://git@<NAS>:6052/…` (`git remote set-url`).

### Removing, reinstalling and a broken package

Your work stays when the package is removed, installed again or broken. Everything you make lives in the data
folder: accounts and settings, Git and SVN repositories, buckets, project folders, HOME folders, VS Code settings and
extensions, App Builder apps, environments, the package repository and the backups. The install folder
(`<volume>/.qpkg/bucketvs`) holds only the programs and one file that names the data folder.

- **Remove, then install again.** Removing the package deletes the install folder and keeps the data folder and
  `/etc/config/bucketvs-data-root`, the file that names it. The next installation reads that file and starts on
  the same data folder: the same accounts and passwords, the same projects and repositories, signed-in sessions
  included.
- **Install over the installed package.** The same version or a newer one replaces the program files; the data
  folder is not touched.
- **A broken package.** When the install folder is damaged (a program missing, the folder deleted by hand), install
  the package again from App Center or with **Install Manually**. The installation finds the data folder the same
  way. A bvsd that still runs from deleted files is stopped at the start, and the installed one starts.
- **What deletes the data.** Only deleting the data folder. To start empty on purpose, remove the package, delete
  the data folder, then install: the first start creates a new data folder, as [Where the data folder
  starts](#where-the-data-folder-starts) describes.

A failing disk is a different matter: the data folder is on the NAS's own disks. Keep backups on another machine
([Backups](#backups)); every backup holds your project folders, HOME folders, App Builder settings and the
package repository.

## The bvs command

`bvs` is the package's command line, in `/usr/bin/bucketvs/bin`. It is on the `PATH` of every process bvsd
starts, cron jobs included. While the package runs, `/usr/bin/bvs` links to it, so an SSH session on the NAS calls it
by name too; stopping the package takes the link away.

Every command that talks to bvsd over its CLI socket needs root, because the socket is root's only: `user`, `jobs`,
and, while bvsd runs, `tc install`, `tc remove` and `env create`, `clone`, `export` and `delete`. `env shell` and
`env run` and `qbuild` need root too: they start `bvs-chroot`. As another user, a command that needs the socket stops with
"bvsd's CLI socket … is open to root only: run bvs as root".

```text
bvs env list
bvs env create <name> --from <rootfs-id> [--package <name>]...
bvs env shell <name>
bvs env run <name> -- <command> [<arg>...]
bvs env clone <name> <new-name>
bvs env export <name>
bvs env delete <name>
bvs tc list
bvs tc install <id>
bvs tc remove <id>
bvs use <id>
bvs jobs
bvs user list
bvs user password <name>
bvs user totp-reset <name>
bvs guide [--path] [en|pl|de|fr|it|nl|sv|ja|zh-TW]
bvs qbuild [--env <name>] [<qbuild argument>...]
bvs kernel profile [--json]
bvs kernel sources
bvs kernel prepare [--source <id>] [--env <name>] [--option NAME=VALUE]... [--full]
bvs kernel make [--project <folder> | --dir <tree directory>] [--build-only] [<target>...]
bvs kernel check <file.ko | module>
bvs kernel load <module>
bvs kernel unload <module>
```

- While bvsd runs, installs and environment jobs go to bvsd over its CLI socket, so the panel shows them, and
  `bvs` follows them until they end. Without bvsd they run in the `bvs` process itself; `--local` before the
  command does that on purpose.
- `bvs jobs` and the `user` commands need bvsd.
- `bvs guide` opens this guide with the package's `less` on a terminal, and prints the path of its file
  otherwise or with `--path`. Without a language it follows the first of `LC_ALL`, `LC_MESSAGES` and `LANG` that is
  set: `de_DE.UTF-8` opens the German guide, `zh_TW` and `zh_HK` the Traditional Chinese one, and a language the
  guide is not written in (`C`, `zh_CN`) the English one.

## Files and folders

| Path | What |
|---|---|
| `/usr/bin/bucketvs` | the package's programs (a link to its install folder) |
| `/usr/bin/bucketvs/etc/data-root` | the path of the data folder |
| `/usr/bin/bucketvs/var/run/bvsd.sock` | the panel's API socket (nginx uses it) |
| `/usr/bin/bucketvs/var/run/bvsd-cli.sock` | the socket of the `bvs` command, root only |
| `<data folder>/etc/` | settings, accounts, sessions, tokens, the certificates, the rendered `nginx.conf` (secrets with mode 600) |
| `<data folder>/log/` | the logs |
| `<data folder>/tmp/` | temporary files of every process |
| `<data folder>/home/<user>/` | the home folder of each account |
| `<data folder>/jobs/` | operations, cron jobs and their runs, with logs |
| `<data folder>/toolchains/` | the download cache, installed toolchains and runtimes, exports |
| `<data folder>/envs/` | environments |
| `<data folder>/projects/` | workspaces, visible as `/projects` inside environments |
| `<data folder>/trash/` | deleted items, kept 7 days |

The modules add folders of their own; their sections name them.