Terabyte Linux Raid Server

Here is a system I set up for Syrrx. This has a capacity of 480GB using software Raid5 and the Reiser file system. It also has dual 22" monitors configured with Xinerama. The kernel is 2.4.18 with the ide patch. Total cost was about $5300.

CalPC 8+2 tower case
Tyan Tiger MPX
(2) Athlon MP 1900 Box + coolers
(2) PC2100 512MB Reg. ECC CL2
PowMax 500W AT+12 power supply
Triple 3.5" drive adapter
(2) Promise Ultra133 controllers
(4) Maxtor D540X (160GB) disks + cables
60GB 7200 RPM IDE root drive
Matrox G550 32MB
(2) Viewsonic P220F 22" 0.25dp flat

This system is at Scripps and is a file server. The kernel is linux 2.4.16 with the ide.2.4.16.12102001 patch. The terabyte array was built using Reiserfs and software Raid5.

CalPC 8+2 tower case
Tyan Thunder K7
(2) Athlon MP 1600 Box + coolers
(2) 512MB PC2100 CL2 memory
460W power supply
Triple 3.5" drive adapter
(2) Promise ATA/133 controllers
(8) Maxtor D540X (160GB) disks + cables
60GB 7200 RPM IDE root drive
Geforce2 MX 200 32MB
Viewsonic E55 monitor

Here are some results from bonnie:

              -------Sequential Output-------- ---Sequential Input--   --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char-  --Block---  --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  K/sec %CPU /sec  %CPU
Raid5     256 16707 98.0 57772 73.2 24978 32.1 17127 96.8  91138 46.2 301.6  2.8
Raid5    2000 15892 94.1 55295 73.1 24636 33.0 16256 94.2  83222 60.6 328.6  1.9
Raid0     256 18489 99.7 89231 65.7 42727 28.7 17324 97.2 115269 91.9 313.2  1.2

The above benchmarks were run after booting the system with 'mem=64M'. The total cost for the system was about $5000.

Here is a scsi system I built in 2001 for ADSC. It lost a disk (11/02) but didn't lose any data. The 5 year warranty on scsi disks is nice, although it does take a couple weeks to get a replacement. It has 17 data drives (raid5) and 1 hot spare.

Supermicro 370DE6 motherboard
(2) PIII 1Ghz processors
(2) 256 MB memory
(2) 9-bay hot swap enclosures
(2) adaptec 39160 scsi controllers (4 channels total)
(18) 73G ultra160 disks (ST173404LCV)

Total cost (May, 2001) was about $23K. Here are some bonnie results. The system was booted with 'mem=64M' for the benchmarks.

8 disks

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  K/sec %CPU /sec  %CPU
Raid0     256 13218 99.7 90025 99.9 64460 59.3 12577 97.9 176889 85.7 409.4  4.9

18 disks

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  K/sec %CPU /sec  %CPU
Raid5      256 11879 97.8 51639 98.7 19286 38.6 12040 96.4 119272 68.2 467.0 8.8
Raid0      256 13192 99.6 92973 99.7 94040 95.4 12625 99.0 178365 89.1 545.1 7.6

Another system configured at ADSC. This was built by Western Scientific and uses dual 3ware 7850's. Each 3Ware controller has a raid5 array with 5 drives + 1 hot spare. The two controllers form one large (1.9T) raid0 array. The file system is XFS.

Supermicro X5DPE-G2 motherboard
(2) Intel 3Ghz Xeon processors
(2) 1024 MB memory
(1) 16-bay hot swap enclosure
(2) 3Ware 7850 controllers
(12) Maxtor 250G drives

Total cost (April, 2003) was about $10K. Here are some bonnie results.

8 disks

Version  1.03      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
xfs              8G 27390  99 90170  22 65019  26 29211  99 295759  55   355   1