LVM Pool Repair: a repair script for the metadata of a QNAP thin pool
One menu-driven shell script repairs the metadata of a QNAP thin pool on QTS, over SSH.
The script takes a repair through five steps: diagnosis, preparing the pool, backup, repair, and bringing the pool and the volumes up.
In the menu, Enter runs the step the script recommends next. After a dropped SSH session or a restart of the NAS, you continue the same repair session.
- 5steps in the menu, in the order of a repair
- 3types of metadata damage, each with its own repair path
- YESthe word you type before a repair overwrites the pool's metadata
Recordings of the script
Each recording is a real SSH session with version 1.7 of the script on a QNAP NAS, in a terminal of 80 by 25 characters. Version 1.8 adds one section to these screens: the read-only check of the RAID and the disks, before the menu and at the start of step 1. We damaged the pool's metadata on purpose, in known places, and kept the original blocks. The damage was written after step 2, while the pool was offline. The recordings leave out that pause: after step 2 the script goes on with the diagnosis in the same session. Long waits are shortened in the recordings. Their real length depends on the pool's size and its snapshots.
░░░ ░░░ ░░░ ░░░ ░░░ ░░░ ░░░ ░░░ ░░░ ░░░ ░░░ ░░░ ░░░ ░░░ ░░░ ░░░ ░░░ ▀▀▀▀▀ ▀▀▀▀▀▀▀ ▀▀▀▀▀▀▀ ▀▀▀ ▀▀▀ ▀▀▀▀▀▀▀ ▀▀▀▀▀▀ ▀▀▀▀▀▀▀ ▀▀▀▀▀▀▀ ▀▀▀▀▀▀ Mounted: /share/CACHEDEV2_DATA /share/CACHEDEV1_DATA /mnt/pool1 (and 12 mounts of apps and containers on these volumes) [auto] Data reachable - finish the repair session? -> y [ OK ] Report: /share/external/DEV3302_1/pool-recovery-NAS-DEMO-vg1-tp1-20261121-010858/report.txt [ OK ] Log package: /share/external/DEV3302_1/pool-recovery-NAS-DEMO-vg1-tp1-20261121-010858/logs-20261121-012354.tgz Look through it before sending: it holds the host name, the device layout, directory listings of the test mount and, in debug mode, the shell trace. The metadata image is not in it. [ OK ] Session finished. Keep the copy of the original metadata: /share/external/DEV3302_1/pool-recovery-NAS-DEMO-vg1-tp1-20261121-010858/tmeta.raw Pool vg1/tp1 is back online with all its volumes mounted. Let's rock. [ OK ] Nothing left to do. Press Enter to return to the menu.
cd /mnt/HDA_ROOT on. Step 2 takes the pool offline, and while the menu waits, one entry of the metadata space map is set wrong. Then a starts the automatic run: diagnosis, metadata image, repair, and the start of the pool, the volumes and the apps. The script stops once, for the typed YES before it writes to the pool.One entry of the metadata space map is set wrong. The script stops the services, saves a verified image of the metadata, builds a repaired copy and checks it. After a typed YES it writes the copy to the pool. Step 5 brings the pool, the volumes and the apps up.The primary superblock is damaged, and its backup copies are valid. The script reads the metadata through the newest copy. After a typed YES it restores the superblock from that copy, checks the metadata and brings the pool up.Two leaves of the mapping tree of a test volume are damaged. The pool runs on the repaired copy for a read-only test mount, and the files are read there. After the data is confirmed and YES is typed, the script writes the copy to the pool and checks the filesystem before QTS mounts the volume. At the end a checksum test of the 600 test files: 534 intact, 66 changed.The same damage. The NAS restarts during the read-only test mount. The pool stays offline after the restart, because the script set the skip-activation flag before the test. After a new SSH login the script is started again. It finds the unfinished session and finishes the repair.Download the script
The script is shared with a PIN. Type the PIN to show the download link.
cd /
sudo -i
# quit console if displayed
cd /mnt/HDA_ROOT
wget --no-check-certificate https://pub.qnaputilities.com/pool-repair/qnap-pool-repair-1.8.sh
chmod +x qnap-pool-repair-1.8.sh
sh qnap-pool-repair-1.8.shStart the script
The script runs on the NAS itself. It needs the admin account, because QTS has no sudo.
- Log in to the NAS over SSH as admin.
- Go to the folder with the script. The folder must not be on the pool you repair.
- Start the script with the first command below.
- Choose a work directory outside the pool. Without one the script can diagnose, but step 3 cannot save the metadata image.
# start the script and its menu sh qnap-pool-repair.sh # return to the session after a dropped connection screen -r qnap-pool-repair
Storage repair can be irreversible. Before the script writes anything, make sure your important data has a backup and the diagnosis fits your NAS.
Options
--auto- Starts the automatic run after the script has detected the pool: steps 1 to 5 in a row. The automatic run needs a work directory.
--debug- Shows the call trace and each command in a frame. Writes the full shell trace to
trace.log, which needs a work directory. --no-screen- Runs the script without a
screensession. --selftest- Checks the commands the script uses, read-only, and exits.
--raid-check- Checks the RAID arrays, their members and the QNAP signatures of the disks, read-only, and exits. It needs no pool.
--allow-no-backup- For an emergency only. You can then confirm the quick or the full path without a metadata image: you type
NO-BACKUPand then the pool's name as the script shows it, for examplevg1/tp1. The automatic run does not use this option. --help- Prints the options.
--version- Prints the version.
The five steps
Before the menu the script shows a red WARNING and checks that the NAS has the 79 commands it uses. In a new session it then checks the RAID arrays and their disks, read-only. The menu lists the steps in the order of a repair and shows the state of each one. Enter runs the recommended step. The automatic run does all remaining steps in a row and answers each question that has a safe default with that default. It still asks you about busy volumes, about the data on the test mount and for the typed YES.
Status and diagnosis
The script checks the RAID and the hardware. With the pool offline it runs
read-onlythin_checkon the pool's metadata. On a pool that is still active it offers a preliminary read-only check.Prepare
The script unmounts the pool's volumes and deactivates the volume group. In the gentle mode it unmounts only what belongs to this pool, and services stop only if something stays busy.
pool offlineBackup
The script saves the LVM configuration and writes an image of the metadata to the work directory. It accepts the image only after it has verified it against the device.
verified imageRepair the metadata
The script offers the repair path that fits the diagnosis. Before the repair overwrites the original metadata, you type
typed YESYES.Bring the pool and the volumes up
The script activates the pool. QTS mounts the volumes, and the script checks each one against the list it saved before. It offers to start the services it stopped. When every volume is back, it shows LOOKS GOOD, writes a report and packs the logs.
report and logs
What the diagnosis finds
Step 1 shows the result of thin_check in one of the script's own phrases. Step 4 then offers the repair path for that result.
- space map counts only
The only errors are in the counts of the space map. The mapping trees are intact.
Quick path: the script builds a repaired copy on a reserved volume and checks it. After you type
YES, it writes the copy to the pool. - damaged superblock
The primary superblock is damaged, and the script finds valid backup copies of it in the metadata.
The script reads the metadata through the newest valid copy. After you type
YES, it restores the superblock from that copy and runs the diagnosis again. - mapping tree errors
Parts of the tree that maps the blocks of a volume are damaged. The repair drops these ranges, and their blocks read as empty afterwards.
Full path: the pool runs on the repaired copy for a read-only test mount. You look at the data and confirm it. After you type
YES, the script writes the copy to the pool. - data NOT OK
During the test mount you answer that the data is not right. The script goes back to the original LVM configuration, and the original metadata stays unchanged.
Send the case to QNAP support with the logs package from menu 7 and the
tp_collectpackage from the Tools menu.
The script treats a result it does not recognise like mapping damage. For space map damage it also offers two paths that write directly to the original metadata: legacy (thin_dump and thin_restore) and rebuild (QNAP's thin_rebuild). Both need the verified image and a typed YES. On a pool without a reserved volume for the copy, legacy is the only path for mapping damage, and you type LEGACY to choose it.
Safety rules
- A repair path writes to the original metadata only after step 3 has a verified image of it. A raw image must have the size of the device and the same
thin_checkresult. The script compares a compressed image with the device byte by byte. - Before a repair overwrites the original metadata, you type
YES. Any other answer cancels the write. - Before a write the script looks in the kernel log for memory error reports and for I/O errors of the disks under the pool. If it finds one, it asks whether to continue, and the default answer is no.
- The script does not start a repair on an inactive RAID. On a RAID that is degraded, rebuilding, resyncing, reshaping or under a check, it asks first, and the default answer is no.
- After a repair of mapping damage, step 5 offers a read-only filesystem check (
e2fsck -fn) of the affected volumes before it mounts them. If the check finds errors, you choose the repair:e2fsck -fp,e2fsck -fyafter you typeFSCK, or Check File System in QTS later. - The work directory with the logs and the metadata image is outside the pool: on an external disk or on a volume of another pool. The script does not accept a folder on the pool under repair.
- The script runs inside
screen, so a dropped SSH session does not stop it. Run the script again to return to the session. - The session state is on the system partition of the NAS (md9), outside the pool. After a restart the script finds the unfinished session and asks whether to continue it.
- During the test mount LVM points the pool at the repaired copy. For that time the volume group has the skip-activation flag, so a restart of the NAS leaves the pool offline.
- Two cases are exempt from these rules. If the metadata is consistent and only the
needs_checkflag is set, the script clears the flag without an image, without the hardware and RAID checks and without a typedYES. The emergency option--allow-no-backuplifts the image rule only and is described under the options. - If a volume does not come back, QTS can rebuild its storage configuration with
init_lvm.sh. The script runs it only after you typeREBUILD, saves the files first and lists every setting the rebuild changed.
Limits
- The repair cannot bring back data in damaged mapping ranges. In the mapping recording two leaves of the mapping tree of a test volume with 600 files of 4 MiB were damaged. After the repair 534 files matched their saved checksums. 66 files had data in the two dropped ranges and were changed.
- The script repairs the metadata of the pool only. It does not repair a disk, a RAID or a missing LVM physical volume.
- The script works on QTS with an LVM thin pool. On QuTS hero, which uses ZFS, it stops at the start.
- With a read-write or write-only SSD cache, or with data in two or more Qtier tiers, the script stops before step 2 and before the repair and points to QNAP support. With a read-only SSD cache it first asks whether the cache is removed, or disabled for this pool.
- The pool goes offline for the repair. In the gentle mode the script stops the snapshot manager if snapshots of the pool are mounted. If something keeps a volume busy, you choose how far the script goes: stop the snapshot manager, stop all QTS services, end the processes that hold the volume, or flag the volume group so that the pool stays offline after you restart the NAS. In the full mode the script stops all QTS services at once, and step 5 offers to start them again.
- The script does not check the filesystem of an encrypted volume. With an SSD cache volume on the NAS it skips the filesystem check.
- A clean hardware check does not prove that the RAM is good. MemHound is the memory test.
- The script's messages are in English. You answer each prompt with a letter, a digit or a word and Enter. Arrow keys do not work in the menu. Ctrl+C does not interrupt the script.