SMCI - Super Micro

[Foreign Tuffett]

Was feeling pretty smug yesterday until I realized that Bloomberg had publisher another "spy chip" article, which (again) crashed the stock.   

Nibbled a little more @ $12.15 yesterday, only to sell that nibble today @ $12.85. Still hold the bulk of my position.

As hilfronter83 pointed out, several hardware security professionals on Twitter are quite skeptical about many of the substantive claims made in the Bloomberg articles. I don't have the technical expertise to have a much of an opinion, but do think it's odd that no picture of a spy chip has surfaced.

Why would it be odd?  How does a picture do much?  It's a small thing that looks like a capacitor.  My guess is if someone posted a picture people would still say "but it just looks like a chip, we don't know it's real."

I don't know if this exact story is real, but the NSA has chips and spies with this sort of hardware.  Technically the functionality exists, that's not a stretch.

I remember back in 2000/2001 on the underground of the internet there was a lot of rumbling about the NSA being able to tap lines and record all traffic.  It was a rumor, there were some data points, but it was all rumor level.  Very similar to this.  Then suddenly Snowden released his cache and validated those claims.  More than ten years later those rumors were proved to be very accurate.

This story has a lot of similarities.  It's plausible, we know the US does it, so why wouldn't China, and there are weird rumors.  Apple started to build their own servers in 2015 to avoid hardware security issues.  Then Apple cut off SuperMicro in 2017.  I mean it could all be coincidence, or it could be accurate.

As to this having an impact? I'd suggest reading some of the SysAdmin reddits or forums.  There is skepticism on all levels.  I've seen a lot of "to be safe we're just going to go with someone else" type posts.  I don't know if that trickles down into future earnings or what.

I love net-nets and I love depressed stocks.  But you don't buy as the knife is falling.  You wait until they've had three or four bad quarters, they've cleaned things up, and the market still doesn't care.  That's when you buy.  If I miss out, then oh well.  But buying here seems really risky verses waiting for things to settle some.

I'm more of a "buy when there's blood in the streets" investor. I don't pay attention to technical factors "falling knife, etc) whatsoever. Instead, I'm looking to get the best entry price possible.

Some of my best investments have been situations where investors are indiscriminately selling because of massive uncertainty. Why? Probably because value is most likely to be found in situations where investors are selling first and asking questions later.

[Liberty]

The last part of this podcast (the last 20 mins? Not sure, just estimating, I didn't look at the timer) talk about the supposed chinese hack at supermicro:

https://daringfireball.net/thetalkshow/2018/10/16/ep-231

Their theory is the one that sounds most plausible to me. It's a bogus story, probably created by government people as a psyops operation to tilt public opinion in the trade war with China.

Otherwise, a lot of the technical things in the story are fishy, and the fact that with thousands of boards out there no security expert has yet come out with proof seems very suspicious.

Of course it doesn't mean that there isn't all kinds of spying going on, but this specific story seems bogus.

But it's just a theory, since it's hard to prove a negative...

[Spekulatius]

Impossible to prove a negative of course, but if these motherboards with Trojan horse hardware existed in any quantity, they would have shown up already, now that people look more closely. Plus there are reports that state that the Super Micro boards are so crappy, that they could easily be hacked with laced software and there isn’t even a need for a hardware Trojan horse.

[given2invest]

Impossible to prove a negative of course, but if these motherboards with Trojan horse hardware existed in any quantity, they would have shown up already, now that people look more closely. Plus there are reports that state that the Super Micro boards are so crappy, that they could easily be hacked with laced software and there isn’t even a need for a hardware Trojan horse.

lol is that a bull case?  their products are shit?   haha

[Liberty]

Impossible to prove a negative of course, but if these motherboards with Trojan horse hardware existed in any quantity, they would have shown up already, now that people look more closely. Plus there are reports that state that the Super Micro boards are so crappy, that they could easily be hacked with laced software and there isn’t even a need for a hardware Trojan horse.

lol is that a bull case?  their products are shit?   haha

It's only funny because it's sad: Everybody's products are shit. How many bug patches and security vulnerabilities are all of the big software and hardware vendors constantly patching with OS/app/firmware updates?

I don't know if Supermicro's worse than anyone else, I haven't followed them except for the alleged hack story, but I know that security is extremely hard, especially if it's not designed from the ground up and part of the culture at the manufacturer (which is why Apple does pretty well -- they've made it a priority for a long time).

[oddballstocks]

lol is that a bull case?  their products are shit?   haha

It's only funny because it's sad: Everybody's products are shit. How many bug patches and security vulnerabilities are all of the big software and hardware vendors constantly patching with OS/app/firmware updates?

I don't know if Supermicro's worse than anyone else, I haven't followed them except for the alleged hack story, but I know that security is extremely hard, especially if it's not designed from the ground up and part of the culture at the manufacturer (which is why Apple does pretty well -- they've made it a priority for a long time).

Their products are worse.  When you buy enterprise hardware you expect it to last a LONG time with intensive use.  For example, a Samsung SSD can handle about 400TBW, terrabytes written.  Once you write 400TB the NAND wears out and you need a new drive.  For most users this amount of data is inconceivable.

Take an Intel SSD, the average ones can handle 1 drive write per day, so 1TBW per day, or 2PB (petabytes) written over five years, that's 5x longer.  That's the low end, their write intensive units can handle 10 drive writes per day, so 10TBW a day or 18PBW in five years, 45x longer.

Intel guarantees their flash, it's warrantied, and if you don't hit these metrics you can RMA the devices. That is an example of enterprise reliability and engineering. Gear like Cisco, HPE, Dell (some), Juniper, Arista are in this category. The stuff just runs, it runs forever and keeps running.  The manufacturers stand by their products and have insane warranties.  I have some four hour response warranties on some Cisco gear.  This means that a Cisco engineer will be at the server in person replacing parts (at no charge to me) within four hours no matter what time or day of the week.

When people mention Super Micro it's always coupled with "it's cheap" and it's so cheap the recommendation is to buy two of everything.  This is so when the first one fails you have a ready replacement.  In most cases two servers are cheaper than some of the name brand servers above.  The catch is your people are the ones out of bed on Christmas Eve swapping servers.

Is there a place for this? Sure. Just like people get away with using Samsung SSD's in enterprise environments.  If your workload isn't as intense and you have extras on hand then buying cheap makes sense.

There are a LOT of small businesses out there where IT is a cost center starving for funds. These guys will buy Super Micro and deal with the quality because that's all they can get. These are the same companies that are on Windows XP or Server 2008.

They're the ultimate white box supplier.

Search reddit/r/sysadmin for Super Micro.  First two hits:
https://www.reddit.com/r/sysadmin/comments/85jwmc/supermicro_vs_dell_servers_debate/
https://www.reddit.com/r/sysadmin/comments/9hyy9i/why_supermicro_server_is_so_cheap/

People are recommending that Super Micro is good if when a server has an issue you can just throw it away and replace it with another.  One guy claims "for a lab environment they're better than nothing." 

There are thread after thread like this.  People pleading with others to buy anything but SM unless they can afford rooms full and don't care about reliability, or they don't have a budget.

I'm not trying to slam them. I've been looking at this from the investment angle too.

There's a second trend that I've noticed that I never appreciated.  Super Micro really benefitted from the inexpensive array of cheap compute notes theory of data centers.  This is what Google pioneered, a ton of cheap commodity machines you throw away as they die.  But with virtualization we've moved to extremely reliable overbuilt hosts for VM's.  AWS is selling virtualization on big machines, a lot of them. The theory is now you have a number of VM's spread across a few nodes orchestrated by Kubernetes.

[Spekulatius]

Impossible to prove a negative of course, but if these motherboards with Trojan horse hardware existed in any quantity, they would have shown up already, now that people look more closely. Plus there are reports that state that the Super Micro boards are so crappy, that they could easily be hacked with laced software and there isn’t even a need for a hardware Trojan horse.

lol is that a bull case?  their products are shit?   haha

In a roundabout way, the answer is yes. 🤪

[writser]

There's a second trend that I've noticed that I never appreciated.  Super Micro really benefitted from the inexpensive array of cheap compute notes theory of data centers.  This is what Google pioneered, a ton of cheap commodity machines you throw away as they die. But with virtualization we've moved to extremely reliable overbuilt hosts for VM's.

I highly doubt that. What's your source for this? Software eats the world. Eventually everything will run on commoditized hardware because at large scales it will always be cheaper to implement redundancy in software than in hardware. Google / Amazon / Apple / Microsoft aren't going to deploy millions of (expensive) extremely reliable overbuilt hosts to minimize the chance that one of their computers will crash at some point. And small fish aren't going to deploy these servers because it's cheaper to rent capacity from one of the bigger players.

[oddballstocks]

There's a second trend that I've noticed that I never appreciated.  Super Micro really benefitted from the inexpensive array of cheap compute notes theory of data centers.  This is what Google pioneered, a ton of cheap commodity machines you throw away as they die. But with virtualization we've moved to extremely reliable overbuilt hosts for VM's.

I highly doubt that. What's your source for this? Software eats the world. Eventually everything will run on commoditized hardware because at large scales it will always be cheaper to implement redundancy in software than in hardware. Google / Amazon / Apple / Microsoft aren't going to deploy millions of (expensive) extremely reliable overbuilt hosts to minimize the chance that one of their computers will crash at some point. And small fish aren't going to deploy these servers because it's cheaper to rent capacity from one of the bigger players.

Call any manufacturer and ask for a quote on a virtualization box. Plus hanging around the industry.

But just think this through. Virtualization sells threads/cores, ram and disk space.  It is more cost efficient to take a server from 10 cores to 20 cores and double the sticks of ram.  In a data center you pay for every amp of electricity you use. The marginal cost of more cpu cores and ram is very small compared to an entirely new machine.

A few things changed that made this shift possible.  Hyperconverged ethernet adapters. A host can have a 10/25/40Gbe ethernet adapter that's divided up into hundreds of small adapters for each host. Second is core density. In 2010 Xeon's had four or six cores.  You can buy a Xeon Platinum with 28 cores now, and in a dual NUMA machine that 56 cores or 112 threads for hosts.  Xeon's can support 1.5TB of ram, all in a single pizza box 1U.

A cloud provider then uses something like OpenStack and spins up hundreds of VM's across a small handful of machines. 

Effectively what happened is machines got quicker faster than applications could keep up.

In terms of Amazon/Azure being cheaper, that's completely false. It's 100% cheaper to buy and run your own servers all of the time.  I pay $500/mo for my half rack, the effective cost would be $20k/mo on AWS.  I have a friend who runs IT at a large company, they're moving to AWS.  He said "We know it's a lot more expensive, but it's easier than hiring, we can't hire good IT people. And the cost is opex vs capex."

I know another large company that's moving to the cloud, cost is irrelevant, it's because "IT is too slow, and we want to move quicker."  That's a theme.  My business partner has worked with a handful of Fortune 500 companies that he helped move to the cloud, the reason was the same.  Never cost, it's cheaper to buy and support your own, but it's easier to get around IT.  Execs love the idea of a recurring cost monthly vs a big budget spend one year, and then three years of nothing before IT's asking for an upgrade.

If you aren't convinced I'd say work out the math.  I have tossed around the idea of cloud hosting, the numbers are insanely lucrative.  The only way to do it is density, thread/core/disk.  Disk density is easy, you can buy a few Nimble's and plug everything into them. 

The way you thread this all together is with OpenStack and software defined networking.  You create pools of VM's that can migrate between two or three hosts.  This gives you the reliability.  The whole concept is built with automation in mind.  You can change routes, vlans, anything in the switch with a script.  These scripts can spin up and move VM's.  It's all automated and hands off.

There are some awesome YouTube videos out there from Facebook and AWS explaining how they build out data centers.  This is exactly it, extremely dense, and automated.  To get the density they're using quality enterprise gear, not cheap off the shelf stuff.  It's commodity in the sense they've moved from customized and expensive blades to 1U, but it isn't the off the shelf ATX motherboards that Google does that you're probably thinking of.

The lr;dr; of this is consider: 10 machines virtualized might move to a single hypervisor. The entire point of virtualization is less machines needed because VM's can share resources.  In the past what was 10 cheap servers are now on a single not as cheap dense server.  Look at some of the stuff where there are two or four nodes in a single server now.  It's all about density, not rooms of cheap single purpose pizza boxes.

[writser]

Thanks for the reply. So it is your take that SMCI motherboards are too unreliable to be used in such setups? That's what I am doubting and that narrative also doesn't seem to match with SMCI revenue over the past few years. But I could very well be mistaken.