Troubleshooting NAS performance: read and write speeds, step by step

Slow reads and writes on a NAS have many possible causes: memory that corrupts data or hangs the unit, a failing disk, a bad cable or backplane, the protocol, a fragmented or nearly full volume, or the network. This guide checks them in that order, from the hardware up, with StorageDiag Workbench & Utilities.

The Workbench has two desks. Utilities works on the NAS it is installed on: DriveTime Analyzer, NetScope, MemHound and the live logs. Workbench reads a diagnostic bundle: the kernel log and Dr. Trace over the logs of that bundle. You switch between them in the Start menu (the strip that reads Workbench | Utilities); the page reloads.

1. MemoryMemHound, before anything else
2. Logs & SMARTlink errors, disk health, HCL
3. SMB or iSCSIone protocol or both
4. Disk scan & mapDriveTime: speed, layout, fullness
5. NetworkNetScope and iperf3

01Phase 1: the memory

Rule out unstable hardware first. A faulty memory module can corrupt the data it carries, cause drops in speed that nothing else explains, or hang the whole unit. Test the RAM as the first article describes: How to properly test RAM on QNAP servers using MemHound. MemHound's Benchmark and Quality also measures the memory bandwidth against the JEDEC figure of the installed modules (section 04 of that article). What is particular about memory in a NAS (mixing modules, dual channel, ECC, and why OOM errors can come while memory looks free) is section 05 of that article: RAM in a NAS.

02Phase 2, step 1: the kernel log, SMART and the HCL

The kernel log and Dr. Trace

1. On the NAS itself, switch to Utilities and open Kernel Log · LIVE or Dr. Trace · LIVE (Start menu, Live logs). For the verdict and the log analysis, the live Dr. Trace window says what to do: Run Audit, then switch to the Workbench desk. With logs a customer sent you, upload their diagnostic ZIP on the Workbench desk (Upload Diagnostic ZIP) and open Kernel Log (dmesg) and Dr. Trace (Diagnosis).
2. In the kernel log, switch the search to regex (abc / .*) and look for the lines of a failing link: hard resetting link|10B8B|SError|giving up|link is slow|exception Emask.
3. Read what the Workbench marks on them. It flags libata link resets (hard resetting link, link is slow), a port the kernel gave up on (reset failed, giving up), SATA PHY and signal errors (10B8B, Dispar, PHYRdyChg), NCQ switched off after a timeout, and a SATA link running at a lowered speed.
4. In Dr. Trace, open an incident. It gathers the repeats of one fault, a drive that disconnects and comes back included, into one incident, and splits the likely cause between the drive, the cable, the backplane and an expander.

SMART, the HCL and the drive class

1. On the NAS itself, open DriveTime Analyzer (Utilities). The Health tab keeps an hourly SMART/FARM history of every drive and turns it into critical, warning and info verdicts; the fleet list on the left marks each drive with a dot.
2. Pick a drive and click View SMART. The SMART window lists every attribute (current, worst, threshold, raw value) with a verdict for the whole disk: OK, WARNING or CRITICAL.
3. Read the two chips on the drive's card. The HCL chip compares the drive with QNAP's compatibility list for this NAS model: HCL ✓ means listed, HCL ✗ not listed for this model, HCL — not on the list for any NAS. The class chip says Desktop, NAS, Enterprise, Surveillance, Archive or Blacklist; a desktop drive is marked Consumer drive — not recommended for 24/7 NAS use.

The class matters for speed as well as for life. Consumer SSDs often write fast only until their cache fills, then slow down sharply under a sustained load. On a NAS, steady reads and writes count more than a burst.

03Phase 2, step 2: SMB or iSCSI

With the hardware ruled out, find out whether the slowdown belongs to one protocol. Does it happen only over SMB, only over iSCSI, or over both? If a client can only use iSCSI, that is no reason to skip the check: test from another client over SMB too, to narrow the search.

1. Only iSCSI, with a Windows Server client: turn off ODX (Offloaded Data Transfer) on that Windows host with the commands below, restart it and test again. It is the quickest check there is, and it needs no analysis. The Workbench gives the same advice when a bundle shows iSCSI target command aborts.
2. Both protocols: the cause is below the protocols. Go on to the disks.

# PowerShell as administrator. 0 = ODX on (the default), 1 = ODX off
Get-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem -Name FilterSupportedFeaturesMode

# Turn ODX off, then restart Windows
Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem -Name FilterSupportedFeaturesMode -Value 1

# Turn it back on
Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem -Name FilterSupportedFeaturesMode -Value 0

04Phase 2, step 3: the Disk Health Scan in DriveTime

Before an outside benchmark such as CrystalDiskMark, measure the disks from the NAS itself. DriveTime's Disk Health Scan loads the disks behind a volume and scores each one against its peers.

1. Open DriveTime Analyzer, the Speed tab, and pick the volume in the title bar.
2. Choose the test. The volume tests (sequential and random reads and writes, the 50/50 mix, or FULL REPORT with all four phases and SMART) write their test file only inside a temporary folder on that volume and delete it afterwards. Read-only disk scan reads the raw disks, each alone and then all together, and writes nothing.
3. Click ▶ Analyze. For every disk the scan reports latency (median, p95, max), utilisation, IOPS and queue depth, and marks it OK, WATCH or SUSPECT.
4. Click Save report (.md). The report goes to the Public share as report-ioutils-storagediag-utilities-<date>-<time>.md, with a -trace.csv beside it; Disk reports in the Start menu lists them.

Reading await

A high await on one disk does not by itself mean that disk is failing. Await is the time a request spends queued and served. A disk that carries more of the array's I/O than its peers waits longer because it works more, so the scan judges such a disk by its service time and says so. In the report below, the healthy disk beside the failing one got exactly that note in the random-read phase: carries 62% of the array's I/O (1.6× peer IOPS) — its await and queue are load, not a fault. The reverse happens too: a disk with an ordinary await but a transfer rate well below its peers' can be the one with a quiet mechanical fault. That is why the read-only scan reads every disk on its own as well as all together.

The report and its verdict

The report opens with a verdict a person can read at once: OK when no disk stands out, or WARNING / CRITICAL naming the weak link and the phase where it did worst. Then come the per-disk tables, the SMART evidence from before and after the run, the thresholds used and a per-second trace. Storage speed moves with the load on the NAS, so run the scan more than once and check that an anomaly repeats; the trace in every report lets you compare two runs line by line.

05Phase 2, step 4: fragmentation, fullness and what a disk can do

On hard disks the figures from a data sheet are rarely reached in daily use.

1. Fragmentation. DriveTime's Disk Map tab shows how the files of a volume or share are laid out and how broken up the free space under them is. Pick the volume in the title bar and click Scan: a Sample reads for the time limit you set, Full reads the whole tree. The Fragmented files score reads 0–30 not a fragmentation problem, 31–55 somewhat fragmented, 56 and above heavily fragmented. The scan only reads; it never moves a block.
2. Fullness. Keep a volume under 80% full, live data and snapshots together. The Capacity window warns when a volume passes 80%. On QuTS hero the Disk Map marks 80% and 90% on the pool bar: above about 80% ZFS works harder to place each write, and writes slow down for every share on the pool.
3. No defragmenting. On a volume with snapshots, every block a defragmenter moves costs the storage pool its space again until the snapshots that hold the old block expire, so the snapshots grow by what was moved. The time and the wear on the disks are not worth it either. The Workbench has no defragment button for that reason, and keeping the volume under 80% is the remedy.
4. What a disk can do. The fastest current hard disks are rated at about 280 MB/s, on the outer tracks. On the inner tracks, or on a fragmented volume, a third of that is common, and with reads and writes mixed, 100 to 120 MB/s per disk is a realistic day-to-day figure. The speed of a volume depends on the RAID level, how full and fragmented the disks are, and the head movements that costs.

Disk Map shows the layout only. To see whether the disks keep up, watch the Speed tab while the workload runs, and check each drive's history on the Health tab.

06Phase 3: the network

When the disks prove healthy and fast enough, the bottleneck may be the network. NetScope measures the link with iperf3.

1. Open NetScope (Utilities), the Bandwidth tab, and click Server. Choose the network interface. The iperf3 listener starts by itself (port 5201) and stops when you close the window.
2. On a second NAS running the Workbench & Utilities, open NetScope, Bandwidth, Client, pick this NAS (or type its IP under Manual) and press Start Test. From a Windows PC, use the NetScope client for Windows: NetScope-latest.exe, linked from NetScope's title bar.
3. If the test shows too little bandwidth, stop looking at the NAS storage and look at the network: the switches, the cables, the network cards and the routing.

AElementary knowledge: disks in a NAS

Not every disk belongs in a NAS. Its class decides whether it works with a RAID array or breaks it, and its SMART attributes tell how much longer it will last.

In a RAID array attributes 5, 197 and 198 are not only one disk's risk: a rebuild raises the load, a second weak disk can fail, and then the rebuild fails and the whole pool is lost. React early. On Seagate disks attributes 1, 7 and 195 show raw values in the millions, which is normal; look at VALUE and WORST against THRESH, not at RAW.

Rule of thumb. If any write waits in a buffer before it reaches the disks (write-back, SLOG, a disk cache without power-loss protection, PLP), a UPS is critical. If the cache only serves reads, a UPS is recommended. Keep a write cache on a protected type (RAID 1, 5, 6 or 10), never on Single, JBOD or RAID 0.

Memory (dual channel, ECC, mixing modules, symptoms and OOM errors) has its own section in the RAM guide: RAM in a NAS.