QNAP storage pool or volume not active: what to check first
Read-only checks over SSH show which layer stopped the pool: the disks and the RAID, LVM, the thin pool's metadata, or the volume's filesystem.
On a QNAP NAS with QTS, storage is built in layers. The disks form a RAID array, LVM puts a thin pool on the array, and the volumes live in the pool. A pool or a volume stops when one of these layers fails.
Check the layers from the disks up, in the order below. Each step gives the commands, what their output means and which layer it points to. None of the commands changes the RAID, LVM or the pool.
Before you change anything
- Change nothing until you know which layer failed. Do not pull or swap disks, and do not start a repair in QTS or over SSH before you have read the checks below.
- Storage repair can be irreversible. Before anything writes to the pool, make sure your important data has a backup.
- Keep logs and copies off the pool you check: on an external disk or on a volume of another pool.
- No check and no repair can promise your data back. A repair of damaged mapping ranges cannot restore the data that was in them.
- This page is for QTS with an LVM thin pool. QuTS hero uses ZFS, and these checks and LVM Pool Repair do not apply to it.
Four layers, from the disks up
A layer works only when the layer under it works. So the checks start at the disks. LVM Pool Repair follows the same rule: it repairs storage from the bottom up and stops when a lower layer is at fault.
Disks and RAID
QTS joins partitions of the disks into md RAID arrays. A data pool sits on such an array, for example
cat /proc/mdstatmd1.LVM
LVM uses each array, or a DRBD device on it, as a physical volume (PV). The PVs of a pool form a volume group (VG), for example
pvs · vgsvg1.Thin pool metadata
The thin pool, for example
lvs -atp1, keeps its metadata on a device of its own,tp1_tmeta. The metadata maps every block of every volume in the pool.Volume and filesystem
Each volume is a thin volume in the pool with an ext4 filesystem. QTS mounts it, and the shared folders live on it.
dmesg
Check in this order
Log in to the NAS over SSH as admin. Run the commands of a step and read the output. Go to the next step only when the layer looks right.
Read the kernel log
The kernel log shows disk errors and the messages of the pool's device-mapper devices.
dmesg | grep -iE 'device-mapper|thin|I/O error' | tail -n 12
I/O errors of the disks under the pool point to a disk. Test the disk before you look at the layers above it.
Check the RAID
/proc/mdstatlists each md array with its state and its members.md_checkerlists the data arrays with the state of each disk.cat /proc/mdstat md_checker
An array marked
inactivemeans a RAID problem. In[4/3]one of 4 members is missing, and the array runs degraded.recovery,resync,reshapeorcheckmeans that the array is busy.Check LVM
pvslists the physical volumes,vgsthe volume groups, andlvs -athe pool and its volumes.pvs vgs lvs -a
In
pvs, anmin the third place of Attr (a-m) means that LVM has marked the physical volume as missing. A PV shown asunknown devicedoes not exist on the system. Iflvs -ashows no thin pool at all, the fault is below the metadata, in the RAID or in LVM.Read the pool's row in lvs -a
The pool's Attr starts with
t. Its ninth character is the health that LVM reports:M- the metadata is read-only
F- the pool has failed
D- the pool is out of data space
p- partial: a physical volume of the pool is missing
A volume's Attr starts with
V. Anain its fifth place means active.If LVM or the kernel log printed one of these messages, the message names the layer:
Refusing activation of partial LV- a physical volume is missing: the LVM layer
transaction_id- LVM's record of the pool does not match the pool's metadata: the LVM configuration
Check of thin pool … failed- the pool's metadata
too small for target- the RAID array is smaller than the volumes on it: the RAID layer
Diagnose the metadata with the pool offline
Only
thin_checkon the offline pool gives the diagnosis of the metadata. Taking the pool offline is a change, so it is a step of LVM Pool Repair: step 2 takes the pool offline, and step 1 runs the check. On a pool that is still active, step 1 offers a preliminary read-only check.$ pdata_tools_8192 thin_check /dev/mapper/vg1-tp1_tmeta examining superblock examining superblock backups 99 valid superblock backups, last backup_id=98 blocknr=8388509 examining devices tree examining mapping tree thin device 9 is missing mappings [1470, 1725] bad checksum in btree node (block 21996) thin device 9 is missing mappings [3285, 3540] bad checksum in btree node (block 22007) thin device 9 has unexpected number of mapped blocks expected 5730 != actual 5220 examining clone tree examining tiering tree examining tiering space maps -> exit code 1 (in: do_diag > thin_check_run)
Watch real repairs → Check the filesystem of a volume that does not mount
If the pool and its volumes are active in
lvs -a, but a volume still does not mount, look at its filesystem.dmesg | grep -iE 'EXT4-fs (error|warning)'
Errors of
EXT4-fspoint to the filesystem of a volume. Check File System in QTS checks and repairs it. Start it only after you have read the checks above.
What a finding means
Each card names what you see, the layer it points to and what LVM Pool Repair 1.8 does in that case.
Disks and RAID
- inactive
An md array under the pool is inactive. This is a RAID problem.
The script does not start a repair. It says to repair the RAID first.
- [4/3] [UUU_]
The array runs, but a member is missing, or a rebuild, resync, reshape or check is running.
The script asks before it writes to the pool, and the default answer is no. Let a running rebuild finish first.
- I/O error
The kernel log reports I/O errors of disks under the pool, or memory errors.
The script asks before it writes, and the default answer is no. Test the disks with DriveTime and the RAM with MemHound first.
LVM
- a-m
LVM has marked a physical volume of the pool as missing, or shows it as
unknown device. Activation stops withRefusing activation of partial LV.The script stops: this is the LVM layer, not the pool metadata. For an
unknown deviceit points back to the RAID layer. - No thin pool found
lvs -ashows no thin pool. The fault is below the pool metadata.The script stops and names the read-only checks of the RAID and LVM:
cat /proc/mdstat,md_checker,mdadm -E /dev/sdX3,pvs,vgs,lvs -a.
Thin pool metadata
- SSD cache
The pool has a read-write or write-only SSD cache, or Qtier keeps data in two or more tiers.
The script stops before it changes anything and points to QNAP support. With a read-only SSD cache it first asks whether the cache is removed, or disabled for this pool.
- thin_check
LVM reports
Check of thin pool … failed, orthin_checkfinds errors in the pool's metadata.This is the case LVM Pool Repair is made for. The next section names the three types of damage.
- metadata consistent
thin_checkfinds no errors in the metadata.If only the
needs_checkflag is set, the script clears it: with that flag the pool starts with read-only metadata. Otherwise the fault is above the pool, in the filesystem, or below it, in the RAID or LVM.
When the thin pool's metadata is the cause
LVM Pool Repair is a shell script with a menu. Over SSH it repairs the metadata of a QNAP thin pool on QTS. Its diagnosis sorts the damage into three types, each with its own repair path.
- space map counts only
The only errors are in the counts of the space map, and the mapping trees are intact. The script builds a repaired copy, checks it and writes it to the pool after a typed
YES. - damaged superblock
The primary superblock is damaged, and valid backup copies of it exist. After a typed
YESthe script restores the superblock from the newest valid copy. - mapping tree errors
Parts of the tree that maps the blocks of a volume are damaged. The pool runs on a repaired copy for a read-only test mount, and you check your data before the script writes the copy. The blocks of the dropped ranges read as empty afterwards.