Water-cooled Systems

doneske

Well-Known Member
USA team member
I'm looking for any information that anyone who has experience with water-cooled systems would like to share. Specifically, how reliable are those cooling systems from a leak stand point and do they have a lot of required maintenance. I have always tended to stay away from those types of systems for fear of constant headaches from leaks or constantly having to adjust, clean, and "maintain" the cooling system. I want to acquire a couple of Threadripper 2990WX systems but if I get them pre-built they tend to come with water cooling. I could build one myself and just build it using good quality air-cooling but I haven't convinced myself I want to build another system. If the water cooling systems aren't a big headache, I may just get a couple of those. Hence, my question.

Thanks in advance....
 

Nick Name

Administrator
USA team member
I've had a loop running in #1 for around 2 years or so, and in #2 for several months. Here are the problems I've encountered. Most were avoidable.
  1. Bricked a GPU due to incorrectly installing the block, I forgot the thermal pads on back of the GPU and I think it shorted out. Luckily the manufacturer replaced it under warranty.
  2. Had a near catastrophic radiator failure in #1, it sprung a bad leak and I was alerted by hearing the pump thrashing because all the water ran out. The lesson is, don't use aluminum radiators from China, in fact just avoid Chinesium parts in general. :D And avoid mixing metals in general, even 100% antifreeze won't prevent corrosion.
  3. Had a fitting that wasn't quite tight enough, either I didn't get it tight enough or it loosened up when I buttoned everything up. I thought I might have lost a 1080 Ti. Luckily it wasn't serious at all.
  4. Had a 90 degree rotary fitting that was bad when I got it, luckily it failed in a spot that wasn't critical and again I lucked out.
You can read my post about #2 in Dry Dock about the problem with the loop in that one, again the main issue was a Chinesium radiator. I thought it would be ok there because the card is an Asus Poseidon, but it wasn't. In retrospect it was pretty stupid to try and cheap out on radiators to save a few cents when I could have lost much more in ruined GPUs etc. You should be fine if you stick to quality components. Presumably, if you buy it already built the manufacturer is going to use components from one supplier, and that reduces the likelihood of having the mixed metal problem. You'll also avoid the 1D10T errors that I had. :D

Maintenance just consists of draining the loop periodically, running some clean distilled water through it (or an acid solution if you see corrosion) and refilling. The recommended interval I've seen is six months, had I followed that I would the corrosion sooner and avoided some aggravation. JayzTwoCents has a video up where he didn't do anything with the loop for two years and it was fine. If you feel like you'll probably be lazy or you just hate tinkering in general I'd recommend going with an AIO option, especially since you don't really do any GPU crunching. Those are maintenance free and pretty reliable from what I've seen, you'll just need to make sure you get one large enough to handle the load. An AIO will also help you avoid the other major problem with water cooling, which is if you need to replace or work on anything you basically have to disassemble the whole computer. I have a hard drive right now that either has a loose connection to the motherboard or is failing, but because I'd have to tear into the loop to fix it, I've been putting off fixing it for months. :p

The main reason I tried water cooling is because I was tired of the whine of my GPU fans. Get a couple of those going around 80-85% and you'll see what I mean. :eek: For me the benefits definitely outweigh the cost even after the problems I had.

That's a long way of saying that as long as you use quality components and take your time if you build it yourself, you shouldn't have any problems. I'd be more concerned about the 2990WX performance for the projects you run. From what I've seen, unless you're rendering or running large multi-threaded workloads the performance isn't that good because of the memory design. Do you expect it to scale well for you?
 

doneske

Well-Known Member
USA team member
Thanks Nick Name for the information. I appreciate the insight.

The system I was looking at had 1 block and a closed loop so I assume that is the AIO configuration you were referencing. Just 2 hoses from the block to the radiator. I have 14 server systems running right now so I have noise no matter what. They are all in a single 10ft x 15ft room (used to be a hobby room but is now the 'data center'). I had two 20 amp circuits run (1 to each rack) and left the original 15 amp for monitors, lights, etc. Running 11 amps 24x7 on a 15 amp circuit discolored the wall plate so I decided I had better fix that. I can get by with air cooling because I had a mini-split AC system installed in that room so I can set that room to whatever temperature I want separate from the rest of the house.

Yeah, I saw some of the comments surrounding the 2990WX memory design (something like 4 execution units but only two memory controllers) but I have also seen some benchmarks run versus some of the latest INTEL chips and the 2990WX is still a beast even with "hobbled" memory access. The thing I'm looking for is to reduce my electrical load. I saw a benchmark somewhere where a 2990WX running a full (64 thread) BOINC load only used 383 watts at the wall measured with a Kill-O-Watt. No GPU and they were using a SSD. 383 watts is about 3.3 amps which is what my older 32 thread systems are pulling. I figured I could shutdown almost 15-17 amps worth of systems and replace them with 6.6 amps for a net reduction of ~10 amps and have more threads than what I had before. From what I have seen, the 2990WX produces between 80,000 and 100,000 BOINC points per day on WCG projects (don't know if that holds on other projects). I might not have to go into "summer mode" anymore where I shutdown about 64 threads of processing as it would reduce the heat load in the room. Do I think it will scale linearly up through 64 threads? No, but I would still produce more than what I'm producing now at a considerable reduction in electricity costs. Most of the WCG projects have fairly small memory footprints (30 to 60 MB per WU). Projects like Rosetta and MIP at WCG would probably take a hit with the chip's memory design. Most of the CPUs I'm running now are 2009 vintage so even a "hobbled" 2018 chip will far out perform my current systems and probably use less energy in the process.

Once again, thanks for the insight. It was helpful.
 

Nick Name

Administrator
USA team member
...

The system I was looking at had 1 block and a closed loop so I assume that is the AIO configuration you were referencing. Just 2 hoses from the block to the radiator. ...
Correct. Those are much simpler than the open loop I thought you were talking about. There's no maintenance with those, at least in most cases, you just run it until the pump fails. :D You'll want a minimum radiator size of 240 (aka 2 120mm fans side by side), and I'd get a 360 if you can.

That's good to hear about the 2990WX performance. I've been following some discussion on the SETI boards about a guy who said after 30 jobs performance really started to decline. I don't know what the memory requirements are there, probably more than the 30-60 MB you mentioned. New systems are exciting, keep us posted! :D
 

doneske

Well-Known Member
USA team member
After further review, as they say in the NFL, I think I may just wait on the Ryzen 2 processors. Supposedly, Lisa Su of AMD, is going to announce the new processor architecture at CES the week of Jan 8, 2019.
 

Nick Name

Administrator
USA team member
I'm more or less waiting for the same. They've done a pretty good job delivering on the CPU side so far, the next gen looks really promising.
 
Top