A cyclic test is a type of process in which an object of interest is subjected to repeated testing, instead of a single test. Essentially, cyclic testing involves testing and retesting, to check product performance at various intervals. This provides a more accurate idea of how the product will perform in the real world, as most products are used for an extended period of time. A cyclic test can demonstrate not only the capacity of a product when new, but also the performance over time.
In Linux, rt-test (real-time test) is used. It's a tool that provides a mechanism to measure the latency of the processor N number of times. It creates an M thread that checks and rechecks how much time (in microseconds) the processor takes to respond during a period of time.
First install the rt-tests source using GIT:
-
cd /usr/src/
-
git clone git://git.kernel.org/pub/scm/linux/kernel/git/clrkwllms/rt-tests.git
-
cd rt-tests
-
make
You may get an error about missing numa.h, which is required for the cyclictest. You may see something like this:
rt-tests # make
cc -D VERSION_STRING=0.84 -c src/cyclictest/cyclictest.c -D_GNU_SOURCE -Wall -Wno-nonnull -Isrc/include -O2 -DNUMA
In file included from src/cyclictest/cyclictest.c:37:0:
src/cyclictest/rt_numa.h:23:18: fatal error: numa.h: No such file or directory
compilation terminated.
make: *** [cyclictest.o] Error 1
To resolve this, do the following:
-
Install the libnuma-dev package (on Debian-based systems)
-
Install the numactl-devel package (on RHEL/CentOS systems)
In order to perform the test, you will need to have two SSH sessions or console sessions open. You can use one and using screen application you can split the screen in two regions. In the first one, you will run the cyclictest, and in the second you will run hdparm, which will introduce stress to the system and cause latency spikes.
-
In the first SSH or console session, run: chrt -f 99 ./cyclictest -t1 -p 80 -i 10000 -n -l 100000
-
In the second SSH or console session, run: hdparm -t <HDD Dev>
-
Note: This step should be run at least 2 or 3 times on an otherwise inactive system
-
Observe the output of cyclictest in the first SSH/console session when hdparm is executed (futher information provided below)
-
If the latency is within the acceptable range, there is nothing more to do here
-
If the latency is higher than the acceptable amount, then steps will need to be taken to attempt to reduce the amount of load on the system (halting extraneous processes, disabling unneccessary devices, updating the BIOS, removing unused devices from the PCI bus, etc.)
Information about the output of cyclictest and hdparm
When running cyclictest:
-
The AVG Represents the average latency is being measured on the system.
-
The MAX Represents the maximum latency detected on the system.
(Note: The vaules showed in cyclictest are represented in microseconds (1/1000 ms))
Asterisk is a real-time application, therefore when we run this test, we need to base our conclusion on the worse case scenario. We want to make sure that even with the highest latency reported, Asterisk/DAHDI will run just fine.
Max Latency Range (in microseconds)
-
~0 to 3000: Excellent Lower Latency than the default latency of our Drivers
-
3000 to 8000: Good Low Latency, Require to change the default latency of our Drivers
-
8000 to 15000: Fair Mid Latency, Require to change the default latency of our Driver
-
15000 to 25000: Problematic High Latency, Need to determine why the system has that latency
-
25000 to ∞: Bad Our Drivers won't be able to work
When running hdparm:
Running hdparm with the "-t" flag will perform timings of device reads for benchmark and comparison purposes. For meaningful results, this operation should be repeated 2-3 times on an otherwise inactive system (no other active processes) with at least a couple of megabytes of free memory. This displays the speed of reading through the buffer cache to the disk without any prior caching of data. The measurement is an indication of how fast the drive can sustain sequential data reads under Linux, without any filesystem overhead. (To ensure accurate measurements, the buffer cache is flushed during the processing of -t using the BLKFLSBUF ioctl.)
Transfer Speeds (MB/Sec)
-
~0 to 20: Bad Regardless latency, Recommend to the customer that he need to change his configuration
-
21 to 50: Acceptable Good for SOHO system, recommend to the customer that he dont want to do recordings
-
50 to ∞: Excellent This device should no cause latency issues
(Note: For transfer speeds below 5 Mb/Sec recommend to the customer that he need change the data transfer mode in his BIOS, it's impossible that a fairly new hardware could has this transfer speed.)
