How to properly test RAM on QNAP servers using MemHound
Testing RAM on a live operating system (userspace) is an engineering challenge. The operating system, virtualization and caching services (ZFS ARC above all) keep reserving address space and block access to it.
MemHound, the Memory Doctor window of StorageDiag Workbench, works up an escalation ladder: Quick → Standard → Deep → Hammer → Boot. It locks all the free RAM it may take, less a 256 MB margin, writes test patterns into it and reads them back, and it holds every block until the pass ends, so the kernel has to hand over every free page. Below is how to check memory stability for your configuration and file system.
- Quickfast sweep, the NAS keeps serving
- Standardmoving inversions, the NAS keeps serving
- Deepstops services, frees the ARC
- Hammerrow-hammer on every core
- Bootthe pendrive, outside the NAS system
01Daily diagnostics on QTS (ext4)
With a traditional file system (ext4), QTS does not reserve large amounts of RAM for itself, so a reliable test from the running system is easier. Memory faults often show up in user space, which makes regular quick tests effective.
- Turn off the non-critical network services (Samba, NFS, AFP).
- Stop the resource-heavy apps in App Center (HDMI desktop stations, containers, virtual machines) and leave only StorageDiag running. Keep a single browser tab on the NAS.
- With the caches freed and the services stopped, most of the physical RAM is free for the test. The Quick card
is tagged
DAILYwhen at least 90% of the RAM is free. - Run the Quick test. It is fast because it runs two patterns after an address check and skips the per-line cache flush that Standard uses. Set 1 to 3 passes (the Passes field takes 1 to 16, and starts at 1). If the memory it reaches has a hardware fault, the test finds it almost at once.
02Diagnostics on QuTS hero (ZFS)
The ZFS file system uses the ARC (Adaptive Replacement Cache), which can take most of the available RAM. In this case a plain Quick test covers only a fraction of the memory.
- Select the Deep test instead of Quick.
- Once you confirm (Stop services & test), MemHound stops the NAS services and the App Center apps (the web panel stays up), flushes the ZFS ARC and drops the kernel caches. That frees most of the physical memory for the test: on a 128 GB TS-1288X, available memory went from 96.6 GB to 116.9 GB.
- Workflow: a Quick pass over the freed RAM runs first. If it finds errors, the procedure ends early and saves time. If it finds none, the tool goes on to the Standard pass.
- Downtime: the services stay down while they are being stopped and while both passes run, and a Standard pass over a lot of RAM can take hours. They come back on their own when the test ends, when you cancel it, if the test process dies, and at the latest two hours after the start. If stopping the services alone takes more than 30 minutes, the test is abandoned and everything is restored.
03Absolute testing (Rescue mode / offline)
When the kernel log (klog) points to unstable memory (spontaneous segfaults across several programs, call traces, machine checks or EDAC errors) but the tests from the running system (Quick, Deep) end with PASS, move to an isolated environment. It bypasses the NAS operating system completely.
Option A: a bootable USB pendrive (recommended)
The pendrive was built so that the whole test runs by itself, including on units you cannot connect a monitor and keyboard to. Nothing needs a key press. The boot entry makes the pendrive the default for the next restart, the test starts by itself, and it reports its progress on the NAS's own address, which the MemHound window reads for you. After a PASS the NAS restarts into QTS by itself and the window shows the verdict; after a memory fault the test powers the NAS off (see below). A monitor and keyboard are needed only when the firmware cannot take the boot entry and the pendrive has to be started from the unit's boot menu.
- In the MemHound window, click Create bootable USB in the title bar.
- Plug any USB pendrive larger than 192 MB into a USB port of the QNAP. Everything on it is erased, and its current formatting does not matter.
- Click Update the image when the wizard offers a newer build (Download the image the first time), then follow the prompts: Continue, type the device name, Erase and write, Add the boot entry.
- Restart the server from the prepared diagnostic pendrive (Restart now and test). When the NAS is back, the window shows the verdict.
The pendrive boots x86 units only: an ARM unit cannot boot the generic image.
When the boot-time test does not go as planned
- The NAS does not start the test from the pendrive. Add the boot entry puts one file,
grub/custom.cfg, on the NAS's own boot partition. At power-on the NAS's boot loader (GRUB) looks for a volume labelledMEMHOUNDand starts it when it finds one. GRUB sees a USB pendrive only through the firmware's own USB support, so the NAS boots QTS as usual when the firmware does not present the pendrive at power-on (some cheap or poor-quality pendrives do not), when USB booting is switched off in the firmware setup, or for a reason the firmware does not show. Try another pendrive, a USB port on the back, or start the pendrive from the unit's boot menu (F11, or the key this model uses, at power-on). The wizard says This firmware cannot be given a boot entry when the boot loader does not read the file at all, and The boot entry here is an old one when an earlier version wrote it (Replace the boot entry). A pendrive whose test has already run is not started again: write it anew with Create bootable USB to test once more. - The NAS switched itself off and did not come back. That is what the test does when it finds a
memory fault: it powers the NAS off on purpose, so that nothing writes to the disks with memory in that state,
and it does not restart it. A test that passes restarts the NAS into QTS instead. A NAS that switched itself
off during the boot-time test and stayed off therefore points to a memory fault. The details are on the
pendrive (FAT32, label
MEMHOUND): read/memhound/report-<time>.txtandmap-<time>.jsonon any computer, or plug the pendrive into a healthy NAS running StorageDiag Workbench and open MemHound, which reads it by itself.
Option B: Rescue mode (without configured disks) over SSH
If you prefer working remotely and know the basics of the command line, this mode gives a full test without a USB pendrive.
- Log in to the QNAP server over SSH.
- In the text menu (QTS Console Management), choose to restart the server in Rescue mode (without configured disks).
- After the restart, Qfinder finds the server as “uninitialized”. That is expected.
- Enable SSH from Qfinder. Log in as
admin(on QTS 5.2 and later the default password is the cloud password; on older versions it is the full MAC address in upper case without colons). - Download and unpack MemHound with the first block below, then start the tests in text mode with the second.
Do not follow the initialization or reset screen in the browser: use Qfinder only. Rescue mode needs a DHCP network with Internet access.
# Download and unpack MemHound
arch=$(uname -m)
[ "$arch" = armv7l ] && arch=armhf
wget https://www.qnaputilities.com/pub/memhound/bin/memhound.tar.gz
tar zxvf memhound.tar.gz
chmod +x memhound-*
# Quick test with 2 passes (recommended starting point) ./memhound-$arch --quick --passes 2 --ui # Standard test (after a Quick PASS with End reason : completed) ./memhound-$arch --standard --ui
Quick checks the addresses first, then writes a checkerboard and a random moving inversion. Standard repeats
those two patterns and adds eleven more, all in one run, and makes one pass unless you add
--passes. When the text UI closes, MemHound prints a summary in English. Go on to Standard when it
says Verdict : PASS and End reason : completed: a run cut short by memory pressure
still says PASS, with End reason : aborted: …. Section 04 explains the other verdicts.
04How to read the results
MemHound decides by fixed rules, and the rules depend on the test. Quick, Standard, Deep and Hammer hunt for faults in the memory they reach. Benchmark and Quality, further down, qualifies a whole memory kit.
Quick, Standard, Deep and Hammer
- Hard and transient faults. One bad read does not condemn a module. MemHound counts it as transient:
the map marks it Suspect and the text summary says
PASS_WITH_TRANSIENTS. A fault becomes hard when a later check fails again at the same address of MemHound's own memory (the virtual address; the physical one is printed beside it asp=), after two hits by default (--hard-threshold 2). Any hard fault makes the verdictFAIL. - What the Workbench window says. Its status line counts both kinds, for example FAIL — 0 hard fault(s), 1 transient. With no hard fault, repeat the test offline (the pendrive or Rescue mode) before you blame a module: the in-app guide treats a transient-only result as a reason for an offline retest, never for an RMA. The text summary adds that such results may point to power, heat or marginal timing.
- ECC counters and the kernel log. The ECC counters (EDAC) and the kernel-log events are shown
beside the result, and the summary adds
<== HARDWARE ECC ERRORS DURING TESTwhen a counter rose. They do not change the verdict, so check them yourself. - How the run ended. The summary ends with
End reason:completed,stopped by useroraborted: …. A run that did not end withcompleteddid not test everything it set out to, whatever its verdict line says.
Good to know: Benchmark and Quality
The Benchmark and Quality button in MemHound's title bar measures sustained memory bandwidth against the JEDEC figure of the installed modules, checks latency jitter and fails on any bit error. Its verdict is PASS (or PASS-UNQUALIFIED), MARGINAL, FAIL or INCONCLUSIVE, and its rules are stricter than the fault hunt's.
A click on the button starts the run at once, with no prompt. The first run cannot be interrupted, and by default its result is published on the public RAM board.
- Errors. One bit error, one uncorrected ECC error (UE) or one critical kernel event logged during the run makes the verdict FAIL. Critical events are a machine check or “Hardware Error”, an uncorrected memory error, a memory failure, data corruption or a kernel panic. A corrected ECC error (CE) caps the verdict at MARGINAL: the module works, but the kit cannot be certified. Segfaults, call traces, Oops and OOM kills are shown but do not change the verdict.
- A matched kit. The test qualifies one matched kit. It fails the gate when the modules differ (by part number, or by size, speed, type and ECC when DMI does not give the part number), when there are 3 modules or an odd number above 4, or when channels A and B hold different capacities. The run goes on for information, but the verdict is FAIL, and the result is neither saved nor shared. When DMI reports nothing, the gate is skipped.
- A busy NAS. When bandwidth falls short (FAIL or MARGINAL) or latency jitter fails while the NAS is busy (a 1-minute load above 0.6 per CPU), short of memory (under 40% free at the start) or throttled, the verdict is INCONCLUSIVE: the system's state is to blame, not the modules. A run stopped by you or by the memory guard is INCONCLUSIVE too. Bit errors, UE, critical kernel events, pattern errors, bit fade and the kit gate are never softened this way.
05RAM in a NAS, good practice and hardware quirks
Before blaming the disks or the network for slow transfers, it helps to know what is particular about memory in a NAS. NAS memory is not gaming PC memory. A NAS runs around the clock under a steady load: system buffers, the ZFS cache (ARC), virtual machines and containers. What it needs from memory is stability and compatibility, not record clock speeds. It also tolerates errors less than a desktop PC does: long uptime and a constant load bring out rare errors that frequent restarts would hide on a PC.
“It works” is not “supported”
- Beyond the specification. A NAS will often boot with more memory than the processor maker's official limit, for example 32 GB in a system whose stated maximum is 16 GB.
- Stable at first sight. Such a configuration usually passes the power-on self-test, reports its full capacity and runs well while idle. The errors appear under a long, heavy load.
- The maker's support. When something fails, QNAP support may decline to help until the fault is confirmed on the original, supported modules. Keep the original modules in case you need to open a support ticket.
Mixing modules: what to avoid, and why
Mixed modules make incompatibility and instability much more likely.
- Different makers or speeds. The modules carry different SPD profiles and sub-timings, and the memory controller has to train all of them to what they have in common, usually the slowest shared JEDEC profile.
- A third module. Adding one module to a pair breaks the dual-channel symmetry: part of the memory runs in the slower single-channel mode, and the extra module adds electrical load on the bus. Three modules can give less bandwidth and more latency than two matched ones. MemHound's Benchmark and Quality refuses to qualify three modules, or any odd number above four, for that reason.
- ECC with non-ECC. Never mix error-correcting (ECC) modules with ordinary (non-ECC) ones.
Dual channel and the role of ECC
- Dual channel. A matched pair of identical modules (the same capacity, speed, timings and rank organisation) is the stable choice and runs both channels together, a 128-bit path. In real workloads the gain is typically 1.2× to 1.5×. It counts in memory-heavy work such as video transcoding, where the integrated GPU shares the memory bandwidth with the processor. For plain file serving the gain is small: the network and the disks set the limit there.
- ECC and ZFS. ECC memory corrects single-bit errors as they happen and detects double-bit ones. That matters most on QuTS hero: ZFS verifies the checksums of what it reads from the disks, but it trusts what is in memory. A bit that flips in RAM before its checksum is computed is written to disk with a valid checksum, as if it were correct, and stays there.
- What ECC needs. ECC depends on the processor and the board. On consumer platforms without it, such as many Intel Celeron and Pentium processors, no setting turns it on.
Symptoms of bad memory
Before blaming the disks for poor performance, watch how the NAS behaves. Bad or incompatible memory shows as:
- no start at all, or restarts over and over, right after the memory was changed;
- less memory detected than is installed;
- random freezes or spontaneous restarts;
- on platforms with ECC, uncorrected errors (machine checks, EDAC UE) in the kernel log, which are evidence of a hardware fault. A single corrected error is not (the Workbench's own guide to machine checks: Don't panic at a single corrected event), but corrected errors that keep coming from one module point to it.
OOM errors
The kernel's OOM killer ends processes when a request for memory cannot be met, to keep the system itself running. Most often that means too little memory for what runs, or memory overloaded, not a faulty module. It can happen while the Resource Monitor still shows plenty of free memory, because free is not the same as usable for that request:
- the request may need one contiguous block, and free memory scattered in small pieces cannot serve it;
- a container or an app may have reached its own memory limit, whatever the NAS as a whole has free;
- memory held by the ZFS ARC, huge pages or locked pages is not handed over at once;
- and on faulty hardware, the kernel takes pages out of use after memory errors (
Memory failurein the kernel log), and a module the system does not detect in full leaves it less to work with.
So watch the available memory rather than the free memory, and test the memory anyway. Every MemHound test checks the addresses first and reports an address that is stuck before any pattern runs, and the Workbench puts a warning on its Memory icon when a bundle shows OOM kills or a mixed memory kit.
| Platform | ECC | Modules | Example QNAP models |
|---|---|---|---|
| Intel Celeron, Pentium (Jasper Lake, Gemini Lake) | no | SODIMM DDR4 | TS-464, TS-264, TS-664, TS-x53 series |
| AMD Ryzen Embedded V1500B, V1000 | yes | SODIMM DDR4 | TS-873A, TS-473A, TS-1273AU |
| Intel Xeon E, W | yes | UDIMM DDR4 | TS-883XU, TS-983XU, TVS-h1688X |
| Intel Atom C3000, C5000 | yes | SODIMM, UDIMM DDR4 | entry-enterprise rackmounts |
| Intel Xeon D | yes | RDIMM DDR4 | ES2486dc |
| AMD EPYC | yes | RDIMM DDR4 | TS-h1290FX |
On a platform without ECC (a TS-464, for example) no setting turns ECC on. UDIMMs are unbuffered; RDIMMs and LRDIMMs are registered and take load off the controller, so a channel takes more modules of large capacity.
Example: the TS-464, “it works” is not “supported”
The TS-464 has an Intel Celeron N5095 or N5105 and two SODIMM DDR4 slots, without ECC. The official maximum is 16 GB (2 × 8 GB), the processor's own limit. In practice 2 × 16 GB (32 GB) boots and shows its full capacity, but it is outside Intel's and QNAP's specification. Such memory can pass the power-on test and run well while idle, and fail only under a long, heavy load, so test a new configuration before you trust it.
| What you mix | What happens | Result | Advice |
|---|---|---|---|
| Different makers, different kits | different SPD profiles, sub-timings, rank organisation and DRAM chips; kits are validated only as a set | a fall back to the base JEDEC profile, possible instability under load | one kit from one box |
| Different speeds | the controller trains all memory to the lowest shared profile | all memory slows to the slowest module (3200 + 2400 → 2400) | the same speed, latency and voltage |
| Different capacities (16 + 8 GB) | usually boots with full capacity, but full dual channel is not certain | uncertain bandwidth; some models with paired slots will not boot | a pair of the same capacity |
| Three modules instead of two | break the dual-channel symmetry and load one channel | part of the memory runs single-channel; on AMD a lower guaranteed clock | an even, matched number of modules |
| ECC with non-ECC | a different word format | the system turns ECC off or does not boot | one type only |
Can three modules be slower than two? Yes, on AMD above all. A Ryzen memory controller lowers its guaranteed clock with the number of modules and ranks per channel: DDR4 3200 with one module per channel, 2933 with two single-rank ones, 2666 with two dual-rank ones. More modules give more capacity and less bandwidth; when a task runs short of memory, more capacity still beats running out and swapping.
| App | Sensitive to | What to watch |
|---|---|---|
| Virtualization Station | capacity and bandwidth | the host keeps about 1.5–2 GB for itself; short of memory, the host swaps and every VM slows at once |
| Container Station | capacity | heavy stacks (Nextcloud, Immich, databases) need about 16 GB |
| QuTS hero (ZFS, ARC) | capacity | at least 8 GB, 16 GB for deduplication, 32 GB and more recommended; high use by the ARC is normal, not a fault |
| QuMagie, Qsirch | capacity | indexing can take a lot of memory |
| Transcoding, iGPU | bandwidth | dual channel helps the graphics; Quick Sync transcoding itself cares little about the number of channels |
Good practice in short
- A matched pair of identical modules: the same capacity, speed, timings and rank organisation. That gives full dual channel and stability.
- Keep the original modules. Before a support ticket, go back to the factory modules and check whether the problem stays.
- Test after an upgrade, above all beyond the official maximum: Quick first, then Deep or the pendrive.
- Check the detected capacity in the Resource Monitor: less than installed means an incompatible or badly seated module.
- An even number of modules of the same capacity. Some models will not boot with a mismatched pair.
- More channels are not always faster. For file serving the network and the disks set the limit; integrated graphics, VMs and ZFS feel dual channel most.