APC SMT1500RM2U UPS:es No Longer Able to Connect to Web Interface

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
35,409
Hey,

Has anyone ever seen this problem before?

I have two of these SMT1500RM2U UPS:es in my rack I bought used on eBay about 2 years ago. They initially came without network cards, so I ordered a pair of AP9630 "UPS Network Management Card 2" and installed them in the UPS:es. When I set them up I configured them via the web interface, and I still check in on them every once and a while using the web interface.

Everything has been working well lately, so It has probably been a few months since the last time I tried logging in.

In the past it worked just fine to access them in my web browser via https://<ip address>, but this time it doesn't. Both of them refuse to connect. I have tried with both Firefox and Chrome to no avail. Firefox gives me "a secure connection failed" error:

1669686967519.png


Chrome gives me something similar:

1669687035973.png


I tried Edge and Internet explorer in a Windows 10 VM as well, and both give similar errors.

I can ping both of them just fine, and both respond via USB (one via APCUPSd in Linux and one via NUT on FreeBSD) but the web interfaces on both of them seem to have dropped dead.

Notably I also can't seem to raise them via SSH (though I can't remember if I ever could) Apparently none of the ciphers are compatible between the units and my Linux machines, giving me the following error message:

Code:
Unable to negotiate with 10.0.1.5 port 22: no matching cipher found. Their offer: aes256-cbc,3des-cbc



It seems like a little much of a coincidence that both of them have gone down at the same time.

Could this be a TLS version deprecation issue? Or maybe some sort of crazy APC licensing scheme that I never caught on to that expired?

I'd appreciate any suggestions regarding what I can do to get back into the web interface on these things!
 
Last edited:

Eulogy

2[H]4U
Joined
Nov 9, 2005
Messages
2,900
Well, the `-cbc` ciphers are utter trash and easily exploited, so, it's good that your unix install is rejecting them.
You can side-step that at least by adding something like this to you client ssh config:
Code:
Host *
  Ciphers +aes256-cbc

I'd also guess that you could try just http:// instead of https:// and possibly gain access?
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
35,409
Well, the `-cbc` ciphers are utter trash and easily exploited, so, it's good that your unix install is rejecting them.
You can side-step that at least by adding something like this to you client ssh config:
Code:
Host *
  Ciphers +aes256-cbc

I'd also guess that you could try just http:// instead of https:// and possibly gain access?

So, APC units disable http mode by default, from what I understand.

I can try to temporarily add the CBC cipher, log in via SSH and see if I can enable http mode that way, and then use the web interface to check for updates that might solve this.

Thank you for the suggestion!
 

Eulogy

2[H]4U
Joined
Nov 9, 2005
Messages
2,900
If you get a "not matching host key" error, you can skirt that by doing
Code:
ssh -oHostKeyAlgorithms=+ssh-rsa user@ip
.

Guessing these are pretty old systems though. Have you tried just reseating the NMCs? Those cbc ciphers were deprecated at least a decade ago and blacklisted in OpenSSH in 2014 iirc.
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
35,409
Eulogy

Thank you for the suggestions.

I edited my /etc/ssh and added two entries for the APC hosts, in which I enabled aes256-cbc just for those hosts as you suggested.

Then I was able to ssh in and enable the http mode using the command "web -h enable".

After a reboot http is now enabled, and I can use the web interface without TLS.

It looks like I am on 6.8 on these units, and that thre was one more release before they went EOL, so I am going to try updating them tot he latest and see if that allows me to log in using TLS again.

Appreciate all the help!
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
35,409
Hmm.

Looks like it supports TLS 1.2 as is, but that is apparently not enough.

1669784897785.png


Did all the browsers recently (in the last 6 months to a year or so) remove support for TLS 1.2?

It's only one revision behind. I didn't realize it was already going away...

Googling it, I can't find any information about TLS 1.2 going away....
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
35,409
Hmm.

Looks like it supports TLS 1.2 as is, but that is apparently not enough.

View attachment 530678

Did all the browsers recently (in the last 6 months to a year or so) remove support for TLS 1.2?

It's only one revision behind. I didn't realize it was already going away...

Googling it, I can't find any information about TLS 1.2 going away....

I poked around a little and with some help I found this known problem which seems relevant:

Looks like I need to upgrade the network management cards firmware, and then remove the old cert for it to work again.

There is a firmware update section in the web interface, but it is not for the network Management cards firmware, only for the UPS itself.

I found these instructions:

  • Go to APC, a flagship brand of Schneider Electric and search for your network management card model number.
  • Select the “UPS Network Management Card 2” link.
  • Scroll down to the product documentation tabs.
  • Click the Software and Firmware tab.
  • Find the firmware file. This one is “Network Management Card v6.5.0 Firmware for Symmetra 3-Phase with AP9630/AP9631/AP9635”
  • Verify that you have located firmware appropriate for the host UPS family.
  • Click the “Download” Link.
  • Then Select the “Download Now” button.
  • The file will download to your default download directory.
  • Click the file and extract the firmware package.
  • Open the extracted firmware package.
  • Locate and click the Firmware Upgrade Utility application.
  • Enter the IP address
  • Enter your Username and Password. The default values are both lowercase “apc”.
  • Click the Ping button to assure you have connectivity with the card to be upgraded.
  • Click “Upgrade Now” button.
  • If successful, the utility will report “Device(s) Successfully Updated”.
  • You can now log in to your UPS Network Management Card 2.
They are older than the current firmware, but the process is probably the same.

Finding the firmware the way they suggested in the download section on their website was bloody impossible, but I was able to use google with a site:apc.com filter to find what I think is the latest.

"7.0.8 for Smart-UPS and Single Phase Symmetra" (link for posterity if it helps anyone else)

It looks like the download includes an executable that will send the firmware to the UPS via the network, but it is an .exe file, sigh...

So now I have to find a windows machine... Why does everything always have to assume Windows is being used?
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
35,409
Hmm.

I tried to repeat everything on the second UPS, which is an identical configuration. It too gave me the identical cipher error message as a few posts up.

The workaround (adding the cipher in the SSH config file) works for the first one, but not for the second one. It gives me a "handshake failed" error, which is weird.

If it worked on the first, I would have expected it to work on the second...

Any suggestions here?

I commented out my edits to /etc/ssh/ssh_config and instead tried connecting using the -c option.

"ssh -c aes256-cbc root@<first_ip>" works just fine.

"ssh -c aes256-cbc root@<second_ip>" fails with a "Handshake Failed" error.

Then I figured I'd be clever and try 3des-cbc. Again, worked on the first ip, but failed with a handshake error on the second...

Starting to think something else is wrong on the second card.

Maybe if I flash the firmware first, it will work.

I'm going to try that, but not tonight. I'm getting tired. I have a policy of never flashing firmware while tired. It never ends well :p
 
Last edited:

Eulogy

2[H]4U
Joined
Nov 9, 2005
Messages
2,900
Hmm.

I tried to repeat everything on the second UPS, which is an identical configuration. It too gave me the identical cipher error message as a few posts up.

The workaround (adding the cipher in the SSH config file) works for the first one, but not for the second one. It gives me a "handshake failed" error, which is weird.

If it worked on the first, I would have expected it to work on the second...

Any suggestions here?

I commented out my edits to /etc/ssh/ssh_config and instead tried connecting using the -c option.

"ssh -c aes256-cbc root@<first_ip>" works just fine.

"ssh -c aes256-cbc root@<second_ip>" fails with a "Handshake Failed" error.

Then I figured I'd be clever and try 3des-cbc. Again, worked on the first ip, but failed with a handshake error on the second...

Starting to think something else is wrong on the second card.

Maybe if I flash the firmware first, it will work.

I'm going to try that, but not tonight. I'm getting tired. I have a policy of never flashing firmware while tired. It never ends well :p
Hah, that's a good policy!
Can you elaborate on the Handshake Failed error? That could be a few things and it usually has more details (or you may need to add a -v). I think in most cases that's a credential type issue.
Could also just try reseating the NMC. I'd leave it out for at least 30 seconds... been a while since I've worked on that generation of them, but something in my brain is saying they take a bit to fully drain power.
 

Teenyman45

2[H]4U
Joined
Nov 29, 2010
Messages
3,050
Have you tried taking UPS "1" wholly offline and then trying any of the workarounds with UPS "2" first? Sometimes weird conflicts can emerge with identical settings (one reason I try to avoid networking anything as it's like dark magic to me). An example would be a mrantz stereo receiver's local network connection interfering with a Samsung upright copier/scanner connection on the network. If UPS "2" fails in isolation, that would point more towards there being something faulty "2"'s network card.

Separately, looking at what CDW charges for that APC unit makes me glad that, years ago, I went with Cyberpower for a 1500w inverter and seemingly bigger 6v batteries at half the price. Moreover, unless you absolutely need to fit the UPS on the rack, you may want to consider using a LiFePO4 based solar "generator" when your APC units get nearer to end of life because some "generators" are coming out with built-in UPS transfer switches almost as fast as those built into a Cyberpower or APC unit, but with a 1000-2000W inverter and 1-2KW of battery power rather than having less than 150W. Presuming I can figure out how to get my batteries to run silent rather than periodically spinning up the inverter fans while idling, I will be switching out my two main UPS units for those.
 

deaedius

[H]ard|Gawd
Joined
Jun 18, 2014
Messages
1,130
Honestly I would run a wireshark packet capture and screenshot the TLS handshake seen. It is going to be TLS Fatal exception due to not proper cyphers or TLS setting of some sort.
 

GiGaBiTe

2[H]4U
Joined
Apr 26, 2013
Messages
2,400
I have this problem at work all the time, but it's old ass routers that they won't replace, rather than UPSes.

The way that I've found to get around the problem is running a Windows 7 VM and using IE11, an older version of Firefox and Pale Moon. All three are required depending on the device and what firmware version they're running.
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
35,409
I got distracted from fixing this and got back to it today.

Flashing the firmware on the first unit worked like a charm. Installing the firmware update solved all issues. The unit responds and allows me to log on via the web interface again.

The second unit, not so much.

The remote firmware update software complains that the device is either corrupt or incompatible (its the exact same model as the first one)

I tried what Teenyman45 suggested, taking the other unit offline first, but that didn't make any difference, unfortunately.

There is a reset button on the back of the network management card, I could try, but my best guess is that it just resets the network configuration to its default. Since I can ping the thing, that is not my current issue.

I was thinking maybe I should try to reboot the affected UPS. I don't think that would do anything too harmful, as most of the important stuff in the rack has redundant PSU's, with one cored plugged into each UPS. A secondary switch may go offline, but that's not the end of the world.

It struck me - however - that these things usually don't go fully offline. There is a power button on the front, but I forget if just pushing it is sufficient, or if I need to do something like unplug the unit from the wall, AND pull the battery cable disconnect or something like that in order to make sure it gets a complete cold boot.

If that doesn't work, I may just have to replace the network management card. APC doesn't seem to offer any other way to update the firmware than through the remote utility, and if it can't work because things are corrupted, then the NMC may be a goner. Luckily now that they are old, they are only ~$25 buy it now on eBay

Appreciate any thoughts!
 
Top