# LVM Pool Repair: a repair script for the metadata of a QNAP thin pool

One menu-driven shell script repairs the metadata of a QNAP thin pool on QTS, over SSH.

The script takes a repair through five steps: diagnosis, preparing the pool, backup, repair, and bringing the pool and the volumes up.

In the menu, Enter runs the step the script recommends next. After a dropped SSH session or a restart of the NAS, you continue the same repair session.

Version 1.8 · on the NAS, over SSH, as admin · QTS with an LVM thin pool · freeware. https://www.qnaputilities.com/pool-repair.html

- **5**: steps in the menu, in the order of a repair
- **3**: types of metadata damage, each with its own repair path
- **YES**: the word you type before a repair overwrites the pool's metadata

## The five steps

Before the menu the script shows a red WARNING and checks that the NAS has the 79 commands it uses. In a new session it then checks the RAID arrays and their disks, read-only. The menu lists the steps in the order of a repair and shows the state of each one. Enter runs the recommended step. The automatic run does all remaining steps in a row and answers each question that has a safe default with that default. It still asks you about busy volumes, about the data on the test mount and for the typed `YES`.

1. **Status and diagnosis.** The script checks the RAID and the hardware. With the pool offline it runs `thin_check` on the pool's metadata. On a pool that is still active it offers a preliminary read-only check. (`read-only`)
2. **Prepare.** The script unmounts the pool's volumes and deactivates the volume group. In the gentle mode it unmounts only what belongs to this pool, and services stop only if something stays busy. (`pool offline`)
3. **Backup.** The script saves the LVM configuration and writes an image of the metadata to the work directory. It accepts the image only after it has verified it against the device. (`verified image`)
4. **Repair the metadata.** The script offers the repair path that fits the diagnosis. Before the repair overwrites the original metadata, you type `YES`. (`typed YES`)
5. **Bring the pool and the volumes up.** The script activates the pool. QTS mounts the volumes, and the script checks each one against the list it saved before. It offers to start the services it stopped. When every volume is back, it shows LOOKS GOOD, writes a report and packs the logs. (`report and logs`)

## What the diagnosis finds

Step 1 shows the result of `thin_check` in one of the script's own phrases. Step 4 then offers the repair path for that result.

- **space map counts only**: The only errors are in the counts of the space map. The mapping trees are intact. Quick path: the script builds a repaired copy on a reserved volume and checks it. After you type `YES`, it writes the copy to the pool.
- **damaged superblock**: The primary superblock is damaged, and the script finds valid backup copies of it in the metadata. The script reads the metadata through the newest valid copy. After you type `YES`, it restores the superblock from that copy and runs the diagnosis again.
- **mapping tree errors**: Parts of the tree that maps the blocks of a volume are damaged. The repair drops these ranges, and their blocks read as empty afterwards. Full path: the pool runs on the repaired copy for a read-only test mount. You look at the data and confirm it. After you type `YES`, the script writes the copy to the pool.
- **data NOT OK**: During the test mount you answer that the data is not right. The script goes back to the original LVM configuration, and the original metadata stays unchanged. Send the case to QNAP support with the logs package from menu 7 and the `tp_collect` package from the Tools menu.

The script treats a result it does not recognise like mapping damage. For space map damage it also offers two paths that write directly to the original metadata: legacy (`thin_dump` and `thin_restore`) and rebuild (QNAP's `thin_rebuild`). Both need the verified image and a typed `YES`. On a pool without a reserved volume for the copy, legacy is the only path for mapping damage, and you type `LEGACY` to choose it.

## Recordings of the script

Each recording is a real SSH session with version 1.7 of the script on a QNAP NAS, in a terminal of 80 by 25 characters. Version 1.8 adds one section to these screens: the read-only check of the RAID and the disks, before the menu and at the start of step 1. We damaged the pool's metadata on purpose, in known places, and kept the original blocks. The damage was written after step 2, while the pool was offline. The recordings leave out that pause: after step 2 the script goes on with the diagnosis in the same session. Long waits are shortened in the recordings. Their real length depends on the pool's size and its snapshots.

- **Automatic run**: The recording starts with the download commands, from `cd /mnt/HDA_ROOT` on. Step 2 takes the pool offline, and while the menu waits, one entry of the metadata space map is set wrong. Then `a` starts the automatic run: diagnosis, metadata image, repair, and the start of the pool, the volumes and the apps. The script stops once, for the typed `YES` before it writes to the pool.
- **Space map**: One entry of the metadata space map is set wrong. The script stops the services, saves a verified image of the metadata, builds a repaired copy and checks it. After a typed `YES` it writes the copy to the pool. Step 5 brings the pool, the volumes and the apps up.
- **Superblock**: The primary superblock is damaged, and its backup copies are valid. The script reads the metadata through the newest copy. After a typed `YES` it restores the superblock from that copy, checks the metadata and brings the pool up.
- **Mapping damage**: Two leaves of the mapping tree of a test volume are damaged. The pool runs on the repaired copy for a read-only test mount, and the files are read there. After the data is confirmed and `YES` is typed, the script writes the copy to the pool and checks the filesystem before QTS mounts the volume. At the end a checksum test of the 600 test files: 534 intact, 66 changed.
- **Restart during the repair**: The same damage. The NAS restarts during the read-only test mount. The pool stays offline after the restart, because the script set the skip-activation flag before the test. After a new SSH login the script is started again. It finds the unfinished session and finishes the repair.

## Safety rules

- A repair path writes to the original metadata only after step 3 has a verified image of it. A raw image must have the size of the device and the same `thin_check` result. The script compares a compressed image with the device byte by byte.
- Before a repair overwrites the original metadata, you type `YES`. Any other answer cancels the write.
- Before a write the script looks in the kernel log for memory error reports and for I/O errors of the disks under the pool. If it finds one, it asks whether to continue, and the default answer is no.
- The script does not start a repair on an inactive RAID. On a RAID that is degraded, rebuilding, resyncing, reshaping or under a check, it asks first, and the default answer is no.
- After a repair of mapping damage, step 5 offers a read-only filesystem check (`e2fsck -fn`) of the affected volumes before it mounts them. If the check finds errors, you choose the repair: `e2fsck -fp`, `e2fsck -fy` after you type `FSCK`, or Check File System in QTS later.
- The work directory with the logs and the metadata image is outside the pool: on an external disk or on a volume of another pool. The script does not accept a folder on the pool under repair.
- The script runs inside `screen`, so a dropped SSH session does not stop it. Run the script again to return to the session.
- The session state is on the system partition of the NAS (md9), outside the pool. After a restart the script finds the unfinished session and asks whether to continue it.
- During the test mount LVM points the pool at the repaired copy. For that time the volume group has the skip-activation flag, so a restart of the NAS leaves the pool offline.
- Two cases are exempt from these rules. If the metadata is consistent and only the `needs_check` flag is set, the script clears the flag without an image, without the hardware and RAID checks and without a typed `YES`. The emergency option `--allow-no-backup` lifts the image rule only and is described under the options.
- If a volume does not come back, QTS can rebuild its storage configuration with `init_lvm.sh`. The script runs it only after you type `REBUILD`, saves the files first and lists every setting the rebuild changed.

## Limits

- The repair cannot bring back data in damaged mapping ranges. In the mapping recording two leaves of the mapping tree of a test volume with 600 files of 4 MiB were damaged. After the repair 534 files matched their saved checksums. 66 files had data in the two dropped ranges and were changed.
- The script repairs the metadata of the pool only. It does not repair a disk, a RAID or a missing LVM physical volume.
- The script works on QTS with an LVM thin pool. On QuTS hero, which uses ZFS, it stops at the start.
- With a read-write or write-only SSD cache, or with data in two or more Qtier tiers, the script stops before step 2 and before the repair and points to QNAP support. With a read-only SSD cache it first asks whether the cache is removed, or disabled for this pool.
- The pool goes offline for the repair. In the gentle mode the script stops the snapshot manager if snapshots of the pool are mounted. If something keeps a volume busy, you choose how far the script goes: stop the snapshot manager, stop all QTS services, end the processes that hold the volume, or flag the volume group so that the pool stays offline after you restart the NAS. In the full mode the script stops all QTS services at once, and step 5 offers to start them again.
- The script does not check the filesystem of an encrypted volume. With an SSD cache volume on the NAS it skips the filesystem check.
- A clean hardware check does not prove that the RAM is good. MemHound is the memory test.
- The script's messages are in English. You answer each prompt with a letter, a digit or a word and Enter. Arrow keys do not work in the menu. Ctrl+C does not interrupt the script.

## Download the script

The script is shared with a PIN. Type the PIN to show the download link.


## Start the script

The script runs on the NAS itself. It needs the admin account, because QTS has no `sudo`.

1. Log in to the NAS over SSH as admin.
2. Go to the folder with the script. The folder must not be on the pool you repair.
3. Start the script with the first command below.
4. Choose a work directory outside the pool. Without one the script can diagnose, but step 3 cannot save the metadata image.

```sh
# start the script and its menu
sh qnap-pool-repair.sh
# return to the session after a dropped connection
screen -r qnap-pool-repair
```

Storage repair can be irreversible. Before the script writes anything, make sure your important data has a backup and the diagnosis fits your NAS.

### Options

- `--auto`: Starts the automatic run after the script has detected the pool: steps 1 to 5 in a row. The automatic run needs a work directory.
- `--debug`: Shows the call trace and each command in a frame. Writes the full shell trace to `trace.log`, which needs a work directory.
- `--no-screen`: Runs the script without a `screen` session.
- `--selftest`: Checks the commands the script uses, read-only, and exits.
- `--raid-check`: Checks the RAID arrays, their members and the QNAP signatures of the disks, read-only, and exits. It needs no pool.
- `--allow-no-backup`: For an emergency only. You can then confirm the quick or the full path without a metadata image: you type `NO-BACKUP` and then the pool's name as the script shows it, for example `vg1/tp1`. The automatic run does not use this option.
- `--help`: Prints the options.
- `--version`: Prints the version.

## Related pages

- [QNAP storage pool or volume not active: what to check first](https://www.qnaputilities.com/qnap-pool-not-active.html)
- [StorageDiag: diagnostics that run on the NAS](https://www.qnaputilities.com/storagediag.html)
- [DriveTime: a disk surface test for a QNAP NAS](https://www.qnaputilities.com/drivetime.html)
- [MemHound: a RAM test for a running NAS](https://www.qnaputilities.com/memhound.html)
- [How-to guides](https://www.qnaputilities.com/howto.html)
