Crypto Miners Paint GDDR Memory Chips to Hide Wear and Tear

lRaphl

Weaksauce
Joined
Sep 3, 2003
Messages
83
As these reports note, miners are removing the stock cooling systems from GPUs to install a third-party solution or recently tried to resolder failed GPU dies back in place and paint the yellowish GDDR memory chips. According to the testing done by Iskandar Souza, you can see below the difference between a worn-out yellowish GDDR chip and its painted deception standing next to one another.

https://www.techpowerup.com/304045/crypto-miners-paint-gddr-memory-chips-to-hide-wear-and-tear

Not only did they screwed us once, they try to do it twice!
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
36,070
I'm used to saying that no appreciable wear should result from mining. The GPU manufacturers validate the cooling solutions, voltages and clocks such that you can essentially run anything on them 24/7 without seeing appreciable wear, except maybe to the fans, which eventually can and will wear out like any device with moving parts.

These are solid state devices after all.

If they are removing stock coolers and not installing proper replacements - however- then anything is on the table.
 

Lakados

Supreme [H]ardness
Joined
Feb 3, 2014
Messages
7,306
I'm used to saying that no appreciable wear should result from mining. The GPU manufacturers validate the cooling solutions, voltages and clocks such that you can essentially run anything on them 24/7 without seeing appreciable wear, except maybe to the fans, which eventually can and will wear out like any device with moving parts.

These are solid state devices after all.

If they are removing stock coolers and not installing proper replacements - however- then anything is on the table.
Many miners also install customized bios files and run the memory out of spec then just put the stock bios back in when they resell which can also lead to damage with the stock cooler.
 

Darunion

Supreme [H]ardness
Joined
Oct 6, 2010
Messages
4,633
The issue also comes from handling. ESD zaps, voltage spikes, thermal cycling. These things cause wear on the semiconductors. So while the process of mining itself may not be a path to failure, the amount of events that can happen unrelated to that, likely will.

Most of us would never trust a stranger to even touch our pc, why would we trust they took care of their hardware before selling it?
 

jardows

2[H]4U
Joined
Jun 10, 2015
Messages
2,209
The issue also comes from handling. ESD zaps, voltage spikes, thermal cycling. These things cause wear on the semiconductors. So while the process of mining itself may not be a path to failure, the amount of events that can happen unrelated to that, likely will.

Most of us would never trust a stranger to even touch our pc, why would we trust they took care of their hardware before selling it?
So the moral of this is to never buy used equipment?
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
36,070
So the moral of this is to never buy used equipment?

Well, at least be careful who you buy your used equipment from, and what used equipment you are buying.

Something cheap and throwaway like a basic $59 GPU just for basic video output? Why not?

Server pulls of enterprise hardware from reputed well reviewed recyclers? Sure

Expensive consumer consumer hardware/GPU's? I wouldn't unless I either knew the person well enough to know that they do things right or otherwise had a very good reason to trust them.

I don't trust this category of stuff from some rando online. There are a select few people on these forums I'd buy from in the FSFT thread, but it is far from everyone.
 

Sycraft

Supreme [H]ardness
Joined
Nov 9, 2006
Messages
5,305
I'm used to saying that no appreciable wear should result from mining. The GPU manufacturers validate the cooling solutions, voltages and clocks such that you can essentially run anything on them 24/7 without seeing appreciable wear, except maybe to the fans, which eventually can and will wear out like any device with moving parts.
These days... I'm not so sure. Companies seem to be pushing things harder and harder to get better performance, and they've determined that consumers don't keep things a long time or use them heavily, so they don't need to worry if it lasts. Plenty of things don't seem to be built with 24/7/365 operation in mind.

I don't know with consumer GPUs, but it wouldn't surprise me if they were not designed to handle full load all the time and if they are subject to that it can cause premature failures.
 

jardows

2[H]4U
Joined
Jun 10, 2015
Messages
2,209
Welp, there goes the [H] For Sale/Trade threads! :D

FWIW, I just bought a couple of RX 5700XT's off ebay, most likely mining cards (but sold through computer recyclers with thousands of feedback, I'm not trusting all these with random names and only 40-50 feedback), so I'll be sure to check them out for signs of repainting when they come in. Of course, I know what I'm getting into when buying these, so my expectations are managed.
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
36,070
Welp, there goes the [H] For Sale/Trade threads! :D

FWIW, I just bought a couple of RX 5700XT's off ebay, most likely mining cards (but sold through computer recyclers with thousands of feedback, I'm not trusting all these with random names and only 40-50 feedback), so I'll be sure to check them out for signs of repainting when they come in. Of course, I know what I'm getting into when buying these, so my expectations are managed.


You just have to have the right expectations.

There are some long time H:ers I definitely trust and would buy things from if I needed them and the price is right.

I bought a used GTX460 on eBay back in 2013 for like $50 to drop in a backup/workbench machine just to get basic video output. It arrived in a plastic bag without any packaging and no anti-static bag. Surprisingly it worked fine for the couple of years I used it.

I'll take risks on low value items for applications I don't really care that much about.

Everything else it is either new or a (very) trusted seller.
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
36,070
Pretty much was gonna say what Zarathustra[H] said. I buy some used stuff and it is all about expectations.

Yeah, I guess my take is, I'll buy consumer stuff used from random strangers if it is cheap enough that I really don't care if I get screwed.

Otherwise I don't.

That doesn't mean ALL used stuff is bad, you just have to be able to trust your sellers. Trusted H:ers are usually fine. Some enterprise hardware recyclers are as well. I'd consider buying high ticket items from them used.

High ticket consumer items items from random strangers is usually just a bad idea.
 
Joined
Apr 11, 2017
Messages
951
These are solid state devices after all.
Highest causes of wear/aging for PC parts:
-Thermal Cycling/Thermal Expansion (e.g. Nvidia solder issue debacle with 7000 series gtx laptop GPUs)
-Excessive heat/voltage
-Mechanical wear (Fans, Heads, etc on HDDs)

There are a lot more wear factors than traditional "mechanical wear"...
 
Joined
Apr 11, 2017
Messages
951
I'd honestly take a card that was run 24/7 rather than a gamer who plays everyday and shuts down their computer at night/daily.
Usually miner cards are given some great cooling/airflow.
The only thing I would be worried about would be the custom BIOS running out of spec memory timings when I get it, which should be fixable by flashing the stock BIOS on it.

If only there was something like SMART data but for GPUs that could log power on count, hours, memory error rate, MAX recorded temp, etc, and not be eraseable.
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
36,070
Highest causes of wear/aging for PC parts:
-Thermal Cycling/Thermal Expansion (e.g. Nvidia solder issue debacle with 7000 series gtx laptop GPUs)
-Excessive heat/voltage
-Mechanical wear (Fans, Heads, etc on HDDs)

There are a lot more wear factors than traditional "mechanical wear"...

Agreed.

Thermal expansion contraction is a problem if you use the wrong solder like Nvidia did on some 7000-9000 GPU's. My statement above assumes otherwise non-defective GPU's. That era of Nvidia GPU's were defective, and it is a goddamn crime they didn't have to pay up and make consumers and AIB's whole over it.

Excessive heat is not as much of a problem these days as GPU's throttle themselves. They also limit how much voltage you can dump into them. This - of course - assumes that they are not modded to override these limitations. If they are, all bets are off.
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
36,070
BhUt ItS pErFeCtLY fInE tO bUy MiNiNg CaRdZ

I wouldn't buy them, but not for the reasons many espouse.

100% load, running hot 24/7 is not bad for a GPU and does not wear it out...

IF the card is not defective already, and has not had its firmware or hardware modded for more power, or something like that.

As long as it hasn't been messed with, mining is actually LESS harmful to a GPU than gaming is. Gaming results in way more on/off cycles, and thermal cycling results in expansion/contraction which can over time result in cracking of solder joints, etc. Modern cards are tested for this, and should survive a gaming lifetime, but you never know for sure. Nvidia screwed the pooch with their 7000-9000 series GPU's, which died in droves, especially laptop models which tended to run hotter.

Constant heat - on the other hand - is fairly innocuous, especially since modern GPU's (well, actually any GPU that isn't ancient) throttle themselves to avoid damaging heat.

The reason I wouldn't buy GPU's used for mining is twofold:

1.) I don't want to do anyhting at all to either benefit or encourage the fuckers.

2.) There is no guarantee they ahvent otherwise messed with them. (overvolted beyond what is allowed stock, cracked chips when when installing alternative coolers, left memory chips to bake without proper cooling etc. etc.

But, if someone buys a GPU, and either leaves it in factory settings, or overclocks it as far as it will go using standard tools that don't require modding, and then proceeds to mine on it 24/7, when they go to sell it it will likely be fine. Maybe even in better shape than a gaming card. Fan bearings may be worn, but fans are replaceable.
 

staknhalo

Supreme [H]ardness
Joined
Jun 11, 2007
Messages
4,240
End of the day:

Used + pushed to the max 24/7 = no bueno

Doesn't matter how long someone wants to make a post or whatever, the answer is still 'no bueno'

Why? Refer back to the title of this very thread
 

DanNeely

Supreme [H]ardness
Joined
Aug 26, 2005
Messages
4,374
These days... I'm not so sure. Companies seem to be pushing things harder and harder to get better performance, and they've determined that consumers don't keep things a long time or use them heavily, so they don't need to worry if it lasts. Plenty of things don't seem to be built with 24/7/365 operation in mind.

I don't know with consumer GPUs, but it wouldn't surprise me if they were not designed to handle full load all the time and if they are subject to that it can cause premature failures.

I've been running distributed computing 24/7 on GPUs (and CPUs) since the GTX 260. For most of this time I had 3 systems running and passed old cards down from one to the next, before putting them away as emergency spares for a few more years. The only failures I've had were two old cards the newer one 7 generations behind the current.

GTX 260 - retired no problems
GTX 260 - also retired no problems
ATI 5850 - had a fan die under warranty (was sent a replacement cooler)
GTX 560 - died outside of warranty - 4 or 5 years old I think - errored in calculations and displayed artifacts when I tried running a GPU benchmark.
GTX 770 - retired no problems.
GTX 1080 - premature retirement due to problems removing the water block (I apparently overtightened screws resulting in them stripping heads and ripping anchors out of the plastic block).
AMD 5450 - retired early (bought as compute only but was disappointed by performance and retired early), currently a spare card
GTX 1070 - (bought used after a prior crypto crash for compute) retired, currently a spare card
GTX 1070 - (bought used after a prior crypto crash for compute) retired, currently in use
RTX 3070 - currently in use (bought because I couldn't get a 3080)
RTX 3080 - currently in use (evga waiting list finally did delivery)
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
36,070
End of the day:

Used + pushed to the max 24/7 = no bueno

Doesn't matter how long someone wants to make a post or whatever, the answer is still 'no bueno'

Why? Refer back to the title of this very thread

Again, that's because they messed with them, not because the GPU had 24/7 100% load.

Any non-defective GPU on the market can easily handle 20 years of non-stop 24/7 100% hot load. Only problem you'll have is the fan bearings (or pump motor if it is an AIO model)

This is actually less bad than a gaming load, which is hot/cold/hot/cold. The more hot/cold cycles, the greater the chance of problems.

It's messing with the hardware or the firmware that causes the problems in the OP. Many miners do that too, and since you can't tell which is which, it's better to just stay away.
 
Last edited:

1337Goat

Gawd
Joined
Apr 15, 2020
Messages
672
Remember that the scientific method exists.

It is perfectly reasonable to draw your conclusion from running benchmarks on say, 10 mining-used cards, and 10 freshly bought cards.

See if they perform within 1% of each other on average.
If a used card truly does have some level of wear, it would be good to know if it's a 5% or less average, or something far more significant.

Furthermore, it might be revealed that there is absolutely no wear or tear at all, and that this entire issue is completely fabricated, perhaps to push sales of NEW NEW NEW GPUs, rather than having people buy used.

Nvidia would not profit if we bought used, right? Miners aren't necessarily the most benevolent people either, but even so, I would let science back up my claims, and I would do this by running a scientific test. However, this test is expensive and I lack the funds. So good luck, I guess.
 

staknhalo

Supreme [H]ardness
Joined
Jun 11, 2007
Messages
4,240
Now do "do your odds increase of buying a badly modified or bad card if buying a used mining card"

That's scientific too
 

Ebernanut

[H]ard|Gawd
Joined
Dec 15, 2010
Messages
1,745
Remember that the scientific method exists.

It is perfectly reasonable to draw your conclusion from running benchmarks on say, 10 mining-used cards, and 10 freshly bought cards.

See if they perform within 1% of each other on average.
If a used card truly does have some level of wear, it would be good to know if it's a 5% or less average, or something far more significant.

Furthermore, it might be revealed that there is absolutely no wear or tear at all, and that this entire issue is completely fabricated, perhaps to push sales of NEW NEW NEW GPUs, rather than having people buy used.

Nvidia would not profit if we bought used, right? Miners aren't necessarily the most benevolent people either, but even so, I would let science back up my claims, and I would do this by running a scientific test. However, this test is expensive and I lack the funds. So good luck, I guess.
Cards don't normally drop in performance before they die which is what I would be concerned about. It's a small sample size but I've had two die on me over the years and they performed fine until suddenly failing. The first one many years ago started artifacting and then stopped working altogether within minutes and the second more recent one died in the middle of playing a game, neither were overclocked and temps weren't an issue but they were fairly old. The first one was likely due to memory issues which might show up as performance degradation instead of artifacting due to ecc these days but only once it was already failing. I think I heard a pop on the second card so a small but vital component likely died catastrophically.

Both of those were more of an exception than the norm though and I would be more concerned about how a used card has been treated and while I'm sure some miners did treat their cards well I'm equally sure that some larger operations were willing to push the memory hard enough to kill some cards as long as it improved the hashrate enough to more than offset the losses. I don't generally buy used components but I would be even more cautious about buying cards that might have been used for mining.
 

DukenukemX

Supreme [H]ardness
Joined
Jan 30, 2005
Messages
6,692
Anyone ask why miners are doing this? They ain't painting them to hide wear and tear, they're painting them to hide model numbers. Nobody is going to open up a graphics card to look for a barely visible yellowing on the RAM and suspect anything. If anything they're reusing known defective GDDR6 which never got a recall. I wouldn't be shocked if that Samsung chip is just a defective Micron painted over with some clever firmware editing to hide it. The miners probably knew this, and just removed the chips ahead of time and kept them laying around. Now repair shops are probably using those defective chips to repair GPUs.
https://youtube.com/clip/Ugkxv3mucyL3lJVu_UIYTFlpI3ji4R_NujeT
 

Slade

2[H]4U
Joined
Jun 9, 2004
Messages
2,927
So much weird speculation in this thread. In 25 years of gpus, I've killed more cards gaming than I did mining. I sold my cards locally and every person that has my cards actually still keep in touch with me as we connected on gaming stuff, pc building etc I even kept 2 of my mining cards, 1 an evga for nostalgia and a backup gpu for main rig, and another is in my htpc which I use to do sim racing while its winter and can't race my track z06.
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
36,070
lol fake news
So in other words crypto miners are scummy? What else is new?

Meh.

Not to defend miners, because I tend to fall into the category of people who really dislike what they are doing, but this was probably real and blown out of proportion at the same time.

There was probably an instance of someone doing this, and they reported on it. Crypto miners are not an organized group all following the same official crypto-mining procedure. They are individuals, some scummier than others. (though on average they do seem to fall into the free-wheeling "do anything for a buck even if it hurts someone" category of people.)

I guarantee there have been cases of gamers trying to hide damage to used parts when they sell them as well.
 

M76

[H]F Junkie
Joined
Jun 12, 2012
Messages
13,309
There was probably an instance of someone doing this, and they reported on it. Crypto miners are not an organized group all following the same official crypto-mining procedure. They are individuals, some scummier than others. (though on average they do seem to fall into the free-wheeling "do anything for a buck even if it hurts someone" category of people.)

I guarantee there have been cases of gamers trying to hide damage to used parts when they sell them as well.
Probably, but they are already doing something that rubs people the wrong way, so this is just adding insult to injury. So it's natural that the reaction is more severe than if any old rando did this.
It's not enough that they are wasting electricity with no regard for the wider implications, it's not enough that they cause supply line issues and screwed the GPU market completely where nVidia is still riding on rainbow clouds thinking €1000 is normal for a mid-range card. No, they even want to add one final insult by trying to resell degraded hardware as "like new".
 

GDI Lord

Limp Gawd
Joined
Jan 14, 2017
Messages
266
I've been running distributed computing 24/7 on GPUs (and CPUs) since the GTX 260.
1) BOINC FTW. Computing for Clean Water really made a difference.

2) Back to the point though, did you flash custom BIOSes on them or fiddled with them an ran them out of spec for all those years?
 

harmattan

Supreme [H]ardness
Joined
Feb 11, 2008
Messages
5,112
Highest causes of wear/aging for PC parts:
-Thermal Cycling/Thermal Expansion (e.g. Nvidia solder issue debacle with 7000 series gtx laptop GPUs)
...

[Eyelid starts twitching involuntarily]

I'll never get those hours of my life back spent with Dellienware customer "service" trying to get an RMA for that.
 

DanNeely

Supreme [H]ardness
Joined
Aug 26, 2005
Messages
4,374
1) BOINC FTW. Computing for Clean Water really made a difference.

2) Back to the point though, did you flash custom BIOSes on them or fiddled with them an ran them out of spec for all those years?
Mostly einstien @home for me. Whatever card was in my gaming machine of the time was overclocked because I never cared enough to see if there was a way to make it conditional, otherwise everything stock. but I doubt 5 years at 100% load was part of the spec; at least for the older cards.
 
Joined
Apr 11, 2017
Messages
951
[Eyelid starts twitching involuntarily]

I'll never get those hours of my life back spent with Dellienware customer "service" trying to get an RMA for that.
LOL!

I resorted to baking a 7950GTX two seperate times. Each lasting about 6 months.
 
Top