Oct 28th, 2011
Hiccup on n0001. No network traffic, namd job apparently still present (but with less overall load for node), job stopped writing to the disk. Stop & restart the job → did it again with same symptoms. Reboot node → Happened again.
OK, have something. dmesg points to the GPU :
NVRM: Xid (0000:08:00): 13, 0001 00000000 000090c0 00001b0c 00000000 00000000
Start testing hypotheses :
- The CUDA card died (or dies) ? Proper full shutdown, recycle power, stress-test the card for a couple of hours (noting that the NAMD problem appears within minutes of starting the job) :
[root@n0001 memtestG80-1.1-linux64]$ ./memtestG80 -b 768 50000
-------------------------------------------------------------
| MemtestG80 v1.00 |
| |
| Usage: memtestG80 [flags] [MB GPU RAM to test] [# iters] |
| |
| Defaults: GPU 0, 128MB RAM, 50 test iterations |
| Amount of tested RAM will be rounded up to nearest 2MB |
-------------------------------------------------------------
Available flags:
--gpu N ,-g N : run test on the Nth (from 0) CUDA GPU
--license ,-l : show license terms for this build
--forcecomm, -f : DO send test results to Stanford (don't prompt)
--bancomm, -b : DO NOT send test results to Stanford (don't prompt)
--ramclock X , -r X: Specify RAM clock speed (for returned results) as X MHz
--coreclock X , -c X: Specify core/ROP clock speed (for returned results) as X MHz
Running 50000 iterations of tests over 768 MB of GPU memory on card 0: GeForce GTX 460
Running memory bandwidth test over 20 iterations of 384 MB transfers...
Estimated bandwidth 60472.44 MB/s
Test iteration 1 (GPU 0, 768 MiB): 0 errors so far
Moving Inversions (ones and zeros): 0 errors (51 ms)
Memtest86 Walking 8-bit: 0 errors (384 ms)
True Walking zeros (8-bit): 0 errors (192 ms)
True Walking ones (8-bit): 0 errors (192 ms)
Moving Inversions (random): 0 errors (48 ms)
Memtest86 Walking zeros (32-bit): 0 errors (780 ms)
Memtest86 Walking ones (32-bit): 0 errors (772 ms)
Random blocks: 0 errors (348 ms)
....
Test iteration 1047 (GPU 0, 768 MiB): 0 errors so far
Moving Inversions (ones and zeros): 0 errors (48 ms)
Memtest86 Walking 8-bit: 0 errors (384 ms)
True Walking zeros (8-bit): 0 errors (192 ms)
True Walking ones (8-bit): 0 errors (192 ms)
Moving Inversions (random): 0 errors (48 ms)
Memtest86 Walking zeros (32-bit): 0 errors (768 ms)
Memtest86 Walking ones (32-bit): 0 errors (768 ms)
Random blocks: 0 errors (348 ms)
Memtest86 Modulo-20: 0 errors (3680 ms)
Logic (one iteration): 0 errors (32 ms)
Logic (4 iterations): 0 errors (60 ms)
Logic (shared memory, one iteration): 0 errors (40 ms)
Logic (shared-memory, 4 iterations): 0 errors (100 ms)
Card looks OK (?).
- Getting interesting. Immediately after stopping the stress-test, start the NAMD job → Started ok, and seems to continue without problems. It looks like a power-cycle-requiring hiccup after all …
- No, it is not a hiccup. Stopped again after three hours. Reboot node and stress-test the card overnight :
#!/bin/csh -f setenv LD_LIBRARY_PATH /usr/local/cuda/lib64:$LD_LIBRARY_PATH ./dgemmSweep 0 1000 exit
giving after 2.5 hours :
Error: cublasDgemm returned an invalid result at location 1322,3409 in iteration 4352 on device 0 Testing device 0: GeForce GTX 460 device = 0 iterSize = 5984 Device 0: i = 128 Device 0: i = 160 ... Device 0: i = 4320 Device 0: i = 4352 8707.007812 ERROR: Failed with device 0. dgemmSweep FAILED.
Try to verify with memtestG80 as well :
#!/bin/csh -f ./memtestG80 -b 768 10000 >& LOG exit
Bombs-out in 16 minutes :
-------------------------------------------------------------
| MemtestG80 v1.00 |
| |
| Usage: memtestG80 [flags] [MB GPU RAM to test] [# iters] |
| |
| Defaults: GPU 0, 128MB RAM, 50 test iterations |
| Amount of tested RAM will be rounded up to nearest 2MB |
-------------------------------------------------------------
Available flags:
--gpu N ,-g N : run test on the Nth (from 0) CUDA GPU
--license ,-l : show license terms for this build
--forcecomm, -f : DO send test results to Stanford (don't prompt)
--bancomm, -b : DO NOT send test results to Stanford (don't prompt)
--ramclock X , -r X: Specify RAM clock speed (for returned results) as X MHz
--coreclock X , -c X: Specify core/ROP clock speed (for returned results) as X MHz
Running 10000 iterations of tests over 768 MB of GPU memory on card 0: GeForce GTX 460
Running memory bandwidth test over 20 iterations of 384 MB transfers...
Estimated bandwidth 60235.29 MB/s
Test iteration 1 (GPU 0, 768 MiB): 0 errors so far
Moving Inversions (ones and zeros): 0 errors (49 ms)
Memtest86 Walking 8-bit: 0 errors (384 ms)
True Walking zeros (8-bit): 0 errors (192 ms)
True Walking ones (8-bit): 0 errors (192 ms)
Moving Inversions (random): 0 errors (48 ms)
Memtest86 Walking zeros (32-bit): 0 errors (776 ms)
Memtest86 Walking ones (32-bit): 0 errors (776 ms)
Random blocks: 0 errors (348 ms)
Memtest86 Modulo-20: 0 errors (3684 ms)
Logic (one iteration): 0 errors (32 ms)
Logic (4 iterations): 0 errors (60 ms)
Logic (shared memory, one iteration): 0 errors (40 ms)
Logic (shared-memory, 4 iterations): 0 errors (100 ms)
Test iteration 2 (GPU 0, 768 MiB): 0 errors so far
Moving Inversions (ones and zeros): 0 errors (48 ms)
...
Test iteration 136 (GPU 0, 768 MiB): 0 errors so far
Moving Inversions (ones and zeros): 0 errors (48 ms)
Memtest86 Walking 8-bit: 0 errors (388 ms)
True Walking zeros (8-bit): 0 errors (192 ms)
True Walking ones (8-bit): 0 errors (192 ms)
Moving Inversions (random): 0 errors (48 ms)
Memtest86 Walking zeros (32-bit): 0 errors (772 ms)
Memtest86 Walking ones (32-bit): 0 errors (768 ms)
Random blocks: 0 errors (348 ms)
Memtest86 Modulo-20: 0 errors (3680 ms)
Logic (one iteration): 0 errors (32 ms)
Logic (4 iterations): 0 errors (60 ms)
Logic (shared memory, one iteration): 768 errors (40 ms)
Logic (shared-memory, 4 iterations): 1152 errors (100 ms)
Test iteration 137 (GPU 0, 768 MiB): 1920 errors so far
Moving Inversions (ones and zeros): 2 errors (48 ms)
Memtest86 Walking 8-bit: 3 errors (384 ms)
True Walking zeros (8-bit): 4294967292 errors (104 ms)
True Walking ones (8-bit): 4294967288 errors (0 ms)
Moving Inversions (random): 4294967295 errors (0 ms)
Memtest86 Walking zeros (32-bit): 4294967264 errors (0 ms)
Memtest86 Walking ones (32-bit): 4294967264 errors (1 ms)
Random blocks: 4294967295 errors (0 ms)
Memtest86 Modulo-20: 4294967276 errors (0 ms)
Logic (one iteration): 4294967295 errors (0 ms)
Logic (4 iterations): 4294967295 errors (0 ms)
Logic (shared memory, one iteration): 4294967295 errors (0 ms)
Logic (shared-memory, 4 iterations): 4294967295 errors (0 ms)
Test iteration 138 (GPU 0, 768 MiB): 1823 errors so far
Moving Inversions (ones and zeros): 4294967295 errors (0 ms)
Memtest86 Walking 8-bit: 4294967288 errors (0 ms)
True Walking zeros (8-bit): 4294967288 errors (0 ms)
True Walking ones (8-bit): 4294967288 errors (0 ms)
Moving Inversions (random): 4294967295 errors (0 ms)
Memtest86 Walking zeros (32-bit): 4294967264 errors (0 ms)
Memtest86 Walking ones (32-bit): 4294967264 errors (0 ms)
Random blocks: 4294967295 errors (0 ms)
Memtest86 Modulo-20: 4294967276 errors (0 ms)
Logic (one iteration): 4294967295 errors (0 ms)
Logic (4 iterations): 4294967295 errors (0 ms)
Logic (shared memory, one iteration): 4294967295 errors (0 ms)
Logic (shared-memory, 4 iterations): 4294967295 errors (0 ms)
...
Take card out, send it off …
