UPS

Our UPS is an APC SmartUPS SUA3000RMXLI (or SUA3000RMXLI3U, depending on where you look). This is the bit where I'd describe what it does but it's a UPS, you know what that is. It has a power rating of 3000 VA (but only 2700 W real power), is 3U tall (a rarity amongst current units) and with the load at time of writing of around 45% should have an autonomy (i.e. how long it can run on batteries for) of about 15 mins.

Monitoring/NUT

It can be tempting to think of the job of a UPS as being "it keeps things on when the power goes off". Of course, it is true that a UPS will allow bridging a brief power outage as though more-or-less nothing happened, its most important job is to allow time for its attached devices to shut down cleanly before the power to them does fail so that data isn't lost.

FilmSoc currently uses NUT (Network UPS Tools) to do this (note that this isn't APC's own solution, Powerchute, because it's terrible and also integrates poorly with appliances like pfSense and TrueNAS. You could use apcupsd if you wanted with probably equal success to NUT) to do this. It's worth reading the documentation, but roughly NUT uses a client-server architecture. The server application at time of writing runs on Filmbank (this is convenient, but also there is logic here that in case of network failure, the server for which we have the most data loss concerns is our NAS) and the client runs on every bare-metal server. NUT is available in the Debian repos, is built in to the web GUI (as "UPS") of pfSense and TrueNAS and has many Windows clients available (we use WinNUT).

The server monitors the status of the UPS and the clients read this data; when the battery is critically low, the NUT server will instruct the UPS to power itself off until grid power resumes after some delay time and then all NUT clients will shut themselves down. When grid power returns, the UPS will turn itself back on and after a delay (to make sure the power is stable) will re-power its outlets. The physical servers that the UPS powers should be set in their BIOS to power on with AC power so that they all turn back on when the power returns.

There are two users set up on NUT. A default one that can just see the status of the UPS (which is what the clients use day-to-day) and an admin user that can issue instantaneous commands (e.g. 'immediately power off'). The passwords for both are available in plain text from the Filmbank GUI.

Batteries

NUT will also warn you when you need to change the batteries (the UPS has an alarm which will also beep loudly to inform you of this fact), based on the brief self-tests that the UPS performs of its own accord. NB. this involves switching to battery power briefly, so you will get a NUT email warning you the UPS is on batter power and then one a few minutes later saying the grid power is back – it's not technically incorrect but it's not necessarily obvious what's happening.

Industry standard says to expect 3–5 years from a set of UPS batteries and APC UPSes are infamous for having a high float-charge voltage and grilling their batteries; however, ours were new in 2009, replaced in 2016 by Joel Speed and next replaced (most recent replacement) in 2023 by Adam Skrzymowski so we seem to be averaging 7 years. It's worth noting that this high float voltage in notorious for causing dead batteries to swell and rendering them impossible to get out of the UPS so please do not delay replacement.

Rule 1 of UPS battery replacement is not to buy batteries from your UPS manufacturer. None of them make their own batteries, they simply re-package (possibly putting them into a cartridge) and charging an astronomical markup. As of 2023, a whole RBC105 cartridge from APC cost around £500 inc. VAT, with just the batteries (identical model to the ones in a new cartidge) costing around £150 from Connectec. You of course don't get a new cartridge if you just buy the batteries, so you will have to take the lid off the old one and swap out the batteries which is not particularly taxing.

The original batteries used are 8x CSB HR 1234W F2 (the F2 in this case isn't that important because I don't think CSB make this battery with any other size of terminals) AGM sealed lead-acid batteries and are to date the only batteries that have been used as replacements – probably worth noting the very good life they've been providing. If in doubt, buy 8 of this exact model or a kit from somebody like MDS Battery or Advanced Power Care who show you upfront that their kit contains these batteries (this can be cheaper than buying individual batteries but make sure you're not getting CSB's GP series of batteries instead).

If you want to go off the beaten track because you have opinions about lead acid batteries, make sure you are buying:

• high-rate batteries, not something like CSB's GP series or the Yuasa NP series. UPS usage draws a lot of current all at once and conventional UPS batteries Do Not Like This.
• batteries with a 9 Ah capacity, not the 7/7.2 Ah ones you can buy in the same form factor.
• batteries with F2 push-fit terminals, not the smaller F1 terminals (make sure you scour the datasheet!)
• ideally from a properly reputable manufacturer like CSB, Yuasa, Enersys, Power Sonic, Panasonic, maybe FIAMM (NOT Duracell!!! or cheapo no-name suppliers like AJC).

(or again just consider buying the CSB batteries above)

After you've replaced the batteries, you'll want to quickly run a self-test (hold the test button for 2 seconds) so that the UPS realises that the batteries are now good and that you've not just put the old dead ones back in. Then you should be good!

Also probably worth saying, by the time the batteries next need replacing the UPS will probably be older than the then-freshers so consider replacing it.

Miscellany

Both the USB and RS232 cables are APC proprietary; for the USB cable that's obvious because the UPS end is a modular jack but the RS232 port looks standard. If you plug a standard serial cable in it will hard power off the load on the UPS which you obviously don't want. There are loads of examples on Reddit of people learning this the hard way and it's even earnt its place in APC's FAQs. Don't make this mistake!