Startup has grown way faster than we anticipated need 2 clusters (DL360?) 2 sans (Nimble?) Vmware

H

holden j caufield

Gawd
Joined
Mar 27, 2012
Messages
639
Besides my regular job, on the side I'm part of startup that has grown fast and we need new equipment. What are your opinions on the DL360 as servers going to need 6 (2x 3 host clusters and 2x Nimble? sans connected 10Gb switches) going to be placed geographically across different states with some type of DR (never used veeam DR or HA but we like their backup so we might go with their DR HA solutions). I don't care what we use really, I'm actually pretty happy with IBM X series (just don't know what they have now), nor is performance a major concern for our in house application servers. I'm looking for reliability and 0 down time if possible.


I don't really keep up with server specs, cpu usage isn't high across all the vm, I generally like low latency stuff and lots of ram for the hosts.
 
Isn't this what the cloud is for? High availability in geographically different regions with little downtime? Not having to manage infrastructure if you aren't ready to get down and dirty with all of the components?

Just throwing that out there because outside of having large amounts of ram, you can spin up a workload in say Amazon or Google fairly quickly.
 
What are your

a) requirements (compute and storage)
b) budget
c) timeline
d) skillset
 
Eickst said:
What are your

a) requirements (compute and storage)
b) budget
c) timeline
d) skillset
Click to expand...
This right here. You can't recommend hardware without knowing what your needs are. This is why you work with solution architects at resellers so they can help you identify what you need. If you are open about what your likes and dislikes are the resellers can account for that. If you have no idea who to work with then PM me and I can help you out.

Also, a few comments.
  • I'm weary on Nimble since they got gobbled up by HPE. I've also seen Pure Storage really starting to take over Nimble's spot in the market. Pure also has a great scale out storage option as well
  • Do you really need a SAN? Going VSAN or S2D could work just as effectively for much less money than a full blown SAN.
  • Also, in this context of DR, we really need to understand requirements again. This could be a simple backup to Azure straight from the VM or a full blown replication of block data on your SAN to a DR colo.
 
For a small startup I would give cloud offerings a serious look. You can spin up and down easily, costs are what you use with no big upfront capital costs (big for startups).

I'd be weary on Nimble also, HP bought them for their software and not their storage tech (they have 3par). Cheapest pure array is going to run you quite a bit and then annual support adds on to that. VMware licenses and support aren't cheap if you want all the bells and whistles.
 
Eickst said:
What are your

a) requirements (compute and storage)
b) budget
c) timeline
d) skillset
Click to expand...

a. lots of satellite offices to the 2 main ones, remote programs, scanning, not a fan of 0365, or vdi
b. xxx,xxx
c. looking to get it done or ordered equipment by end of fiscal year
d. mostly vmware, MS, concern is a good SAN, deltas on backups are getting large enough that the remove snapshot can take an extended amount of time. Looking at cpu that is not our limiting factor.
 
If you're set on having physical stuff, and need DR, I don't think Nutanix or Simplivity would be bad options. They have replication designed in to the platform, easy to manage, scale well, however they aren't cheap. Positives are it's all hyperconverged so no other servers or san to worry about. To be honest once you price in all your servers, san, support etc they aren't that much more expensive than typical compute and storage segregation especially once you consider ease of operations.

If you want to manage it all yourself, pretty much any servers on the vmware hcl would work. If you want Veeam then I would stick to a SAN that it has integration with, it makes backup stun on the vmware snapshots non-existent as they just create the vmware snapshot, create a storage snapshot, close the vmware snapshot and do the backup from the storage snapshot. In most cases for us the vmware snapshot is open less than a minute.

I'm not a huge fan of the replication in Veeam, I think it's a backup product that added some replication functionality and it isn't best in class at it for sure. There are other more graceful products out there. It works in a pinch is what I've found, for most businesses RPO under an hour isn't a real requirement even though they think it is.

*If you are looking to buy before end of year, I'd leverage the vendors against each other when working through a Simplivity or Nutanix quote. I'd get minimum 3 years of support up front, buying it after the fact really really really really really hurts your checkbook.

I'd still give cloud a look, at least compare pricing over the term that you will keep your hardware. I think you'll find it's pretty cost competitive and scales instantly when needed. I'm not talking VDI or O365, just talking about your datacenter load.
 
The vagueness here is incredibly hard to scope correctly as we have no idea what you have in your datacenter and what your performance needs are. So because of this, I'm going to recommend based on the average use case, where you don't have anything really pushing things one way or another.

Going Nutanix would be likely the best route here for an all in one package. Simplivity isn't as wide arcing as Nutanix, and HPE also just bought up Simplivity for the same reason they bought Nimble. I woudl recommend you use Nutanix software on your own hardware, and VMWare as well. Your very first step should be to reach out to Nutanix to talk to their engineers to scope. Still, some general guidelines to follow:
  • Recommend using ESXi 6.0 over 6.5, 6.5 is only on U1, and I've personally dealt with some pretty annoying bugs. Also, your skillset is likely not prepared to work entirely in the web interface since the thick client is gone in 6.5
  • Would definitely recommend two clusters so you have each office failing over to the other. At minimum 3 hosts, but 4 would be better. Since you are doing VSAN you can do some fun sizing here.
  • Nutanix for a virtual SAN will give you a ton of storage and good performance for cheaper than a SAN. Nutanix Acropolis, Enterprise Storage and Virtualization are what you want to talk about.
  • Avoid Nutanix AHV. It's their own home baked Hypervisor, and while it can work fine, it's going to pigeonhole you on support as well as migration paths down the line.
  • You are doing your network a disservice if you use a Cisco Catalyst switch on your servers. Buffer sizes are small and will see high buffer drops on your storage network
  • You should go 10gig networking for your servers. Cisco Nexus 92160YC is pretty solid for a small datacenter, but I would not rely on it as a core switch. It is meant to be datacenter leaf
  • Arista Networks has some very solid switches as well that are very competitive against Cisco on price. It is worth looking at their line of switches
  • If you have dark fiber between your main offices, then you should capitalize it as best you can and use at minimum 10gig optics on it.
  • Leverage licensing caveats the best you can (ie. VMWare still does per socket licensing, so a single socket can be better on costs)

Here's an example list of gear to investigate:

Per site
  • 3x HP DL380 G9
    • 1x Intel Xeon E5-2680v4 (14C @ 2.4GHz. Single socket will save cost if you go to a full VMWare license model, otherwise going 2x E5-2630v3 may be cheaper)
    • 8x 16GB DDR4 2400MHz single-rank RAM
    • 5x 1.2TB 10K RPM SAS HDD
    • 1x 600GB High Endurance SSD
    • 2x Dual 10gig SFP+ NICs
    • 1x Windows Server 2016 Datacenter 16 Core license (for licensing your Windows VMs, minimum 16 core license per physical server. Includes downgrade rights to prior versions)
  • 1x VMWare vCenter Essentials Plus Kit (it's version 6.5 with downgrade rights to 6.0)
  • Nutanix Acrpolis Software for Storage and DR
  • 2x Cisco Nexus 92160YC or 2x Arista 7280SR2-48YC6
The specs above will give you a virtual SAN with total usable storage of 9TB with a FTT of 1 and has a 10% flash to disk cache. This will give you some pretty good performance and will easily handle usual office file storage and AD stuff. If you have any large scale databases or any data warehouse, you need to talk about scoping a SAN.

That list is a total crapshoot as again we have zero scope. You need to talk to an architect to scope properly (AND THEY ARE FREE). This will at least get you down a road.
 
Last edited:
The big cloud players love startups. Just put the big three against one another for the best deal and upgrade to hardware you own when you’re making bank. If you go bust who cares you were getting a steal on cloud hosting.
 
I keep forgetting to talk about cloud. Cloud is a strong proponent here too, but requires some deep scoping to identify TCO
 
DermicSavage said:
I keep forgetting to talk about cloud. Cloud is a strong proponent here too, but requires some deep scoping to identify TCO
Click to expand...

Cloud is a strong proponent due to the stupidity of the new 2016 core licensing from MS. I literally just quoted a virtual environment for a client, 40 Cores / 8 VM's, cost was over $20,000 just for the Server standard 2016 licensing.
 
k1pp3r said:
Cloud is a strong proponent due to the stupidity of the new 2016 core licensing from MS. I literally just quoted a virtual environment for a client, 40 Cores / 8 VM's, cost was over $20,000 just for the Server standard 2016 licensing.
Click to expand...
Makes open source even more compelling -- seriously if Windows and SQL were free and they just sold support adoption would skyrocket.
 
gigatexal said:
Makes open source even more compelling -- seriously if Windows and SQL were free and they just sold support adoption would skyrocket.
Click to expand...

Oh SQL licensing got even more stupid with the last version. Puts the price out of range for many SMB's.

Hell I have a few Small Ent customers that barfed at the price for SQL server.
 
I work for Pure and they're worth taking a look at, too. All flash, zero decision making required (no RAID levels, all features always turned on - dedupe, compression, encryption), and a super compelling business model. Maintenance never goes up, lifetime warranty (not until product is EOL but true lifetime) on the media, every 3 year maintenance renewal includes a non-disruptive swap of your controllers to the newest available, and all software features they have now and in the future are included.

The long term total cost of ownership is pretty cool when you consider if you're happy with Pure you never have to replace the SAN again. Future controller swaps even include new tech like NVMe and you can then get a 25% discount to trade in your old media for denser SSDs or NVMe, all swapped out non-disruptively.

HCI is a good option for remote offices. Just be wary of how it's sized and what level of data protection you want. You'll want to work with a partner to determine a lot of settings -- FTT/replication factor, cluster resilience, Erasure coding, dedupe, compression, encryption -- and what effect they have on your usable capacity, availability, and performance.

Cloud is also a logical place to start for new companies but make sure you understand the cost to get your data OUT of the cloud and stay on top of your bills. They often turn into long, complicated messes that cost more than you ever anticipated very quickly. The more you open the faucet, the harder it is to turn it off (or even remember who or why it was turned up in the first place).

I won't lie, if your data doesn't dedupe or compress well then Pure is going to be a very expensive option. If you're mostly databases, some VMware, and especially if you do a lot of Devops then we'll be able to squeeze a lot of data in a small footprint.
 
Last edited:
Child of Wonder said:
I work for Pure and they're worth taking a look at, too. All flash, zero decision making required (no RAID levels, all features always turned on - dedupe, compression, encryption), and a super compelling business model. Maintenance never goes up, lifetime warranty (not until product is EOL but true lifetime) on the media, every 3 year maintenance renewal includes a non-disruptive swap of your controllers to the newest available, and all software features they have now and in the future are included.

The long term total cost of ownership is pretty cool when you consider if you're happy with Pure you never have to replace the SAN again. Future controller swaps even include new tech like NVMe and you can then get a 25% discount to trade in your old media for denser SSDs or NVMe, all swapped out non-disruptively.

HCI is a good option for remote offices. Just be wary of how it's sized and what level of data protection you want. You'll want to work with a partner to determine a lot of settings -- FTT/replication factor, cluster resilience, Erasure coding, dedupe, compression, encryption -- and what effect they have on your usable capacity, availability, and performance.

Cloud is also a logical place to start for new companies but make sure you understand the cost to get your data OUT of the cloud and stay on top of your bills. They often turn into long, complicated messes that cost more than you ever anticipated very quickly. The more you open the faucet, the harder it is to turn it off (or even remember who or why it was turned up in the first place).

I won't lie, if your data doesn't dedupe or compress well then Pure is going to be a very expensive option. If you're mostly databases, some VMware, and especially if you do a lot of Devops then we'll be able to squeeze a lot of data in a small footprint.
Click to expand...
I want to like Pure. You guys just tried to sell us (and for good reason — VMware sucks) too hard. Many different approaches, many different avenues, from linked in to follow up emails to all sorts of stuff. But that’s just sales. Sales guys don’t like to lose. I wish we had adopted Pure tho
 
We like pure. We went with pure.

2x M20's, each half populated and we're sitting around 3.5:1 compression/dedupe ratio. And it's FAST. Went from having some jobs which took about an hour (VNX 5200) to ~15m.

I doubt we'll cash in our Evergreen when we purchase our next 3 year block of support - maybe we'll wait until our 2nd purchase. I would need a real compelling reason because it's plenty fast enough as is. Now maybe if the storage bricks come down in price per TB due to more dense and less expensive flash, we might expand then if space is getting tight.

This was part of a whole forklift.

2x Cisco USC Blade Chassis
9x Cisco UCS Blades - with dual 18 core Xeons and 512GB RAM each
2x Cisco Nexus
2x Cisco Fabric Interconnects
2x Pure M20
2x Rubrik (I forget the exact model, 48TB RAW)

All for less than 700k.


AND because 1x Pure and 1x Rubrik went to live in our DR site, we still have room in the rack for another whole UCS Chassis, Pure shelf, and Rubrik Shelf. It's a very dense setup.
 
gigatexal said:
I want to like Pure. You guys just tried to sell us (and for good reason — VMware sucks) too hard. Many different approaches, many different avenues, from linked in to follow up emails to all sorts of stuff. But that’s just sales. Sales guys don’t like to lose. I wish we had adopted Pure tho
Click to expand...
That sucks. It's a shame most resellers try to push something rather than work with the customer. I'm an architect for one of the largest resellers in the US, and collaborating with the customer on a solution has always been the most effective approach. Granted I'm a fan of pure, but every product has their place. Still, I like to explore Pure first for high performance SAN and scale out storage before moving on to others.
 
gigatexal said:
I want to like Pure. You guys just tried to sell us (and for good reason — VMware sucks) too hard. Many different approaches, many different avenues, from linked in to follow up emails to all sorts of stuff. But that’s just sales. Sales guys don’t like to lose. I wish we had adopted Pure tho
Click to expand...

I'll be the first to admit our mix of digital marketing, inside sales, and outside sales ends up being a mess to customers. Each has their own purpose but when all 3 are simultaneously hounding one customer, it's too much. Sorry you had that experience.
 
CombatChrisNC said:
We like pure. We went with pure.

2x M20's, each half populated and we're sitting around 3.5:1 compression/dedupe ratio. And it's FAST. Went from having some jobs which took about an hour (VNX 5200) to ~15m.

I doubt we'll cash in our Evergreen when we purchase our next 3 year block of support - maybe we'll wait until our 2nd purchase. I would need a real compelling reason because it's plenty fast enough as is. Now maybe if the storage bricks come down in price per TB due to more dense and less expensive flash, we might expand then if space is getting tight.

This was part of a whole forklift.

2x Cisco USC Blade Chassis
9x Cisco UCS Blades - with dual 18 core Xeons and 512GB RAM each
2x Cisco Nexus
2x Cisco Fabric Interconnects
2x Pure M20
2x Rubrik (I forget the exact model, 48TB RAW)

All for less than 700k.


AND because 1x Pure and 1x Rubrik went to live in our DR site, we still have room in the rack for another whole UCS Chassis, Pure shelf, and Rubrik Shelf. It's a very dense setup.
Click to expand...

If you want to renew maintenance but wait on getting the included controller upgrade just talk to your sales team. They can help.

However, if your renewal isn't until late 2018 or after, I think you'll want the new controllers.
 
Child of Wonder said:
If you want to renew maintenance but wait on getting the included controller upgrade just talk to your sales team. They can help.

However, if your renewal isn't until late 2018 or after, I think you'll want the new controllers.
Click to expand...

We'll be able to renew after purchasing our next 3 year package late 2019.

I know they've got NVMe bricks out now, which is real neat. But I was told it was more of a capacity upgrade than a speed upgrade by my sales guy. Can yuo drop any hints publicly or privately about what you mean?
 
Hey guys thanks for the input, sorry I've been vague and MIA, a little vague and I don't want a competitor to know our growth, and some specifics would give up to much info, a little slow because I've got other projects in place and they need to work out MPLS and other network stuff, I also don't keep up with the hardware stuff much as I rarely see CPU as a limiting factor. I'm actually ok with keeping out old cluster of X series servers but someone above me wants to just get the best. So I'm looking at simply

1) uptime/reliable is most important, performance for the hosts is secondary, cpu and cores are so plentiful now
2) performance, latency (from what I see it's our older SAN, was leaning on Nimble but they got bought out so I don't know how that changes things)

Thanks again for some really detailed posts and advice.
 
Last edited:
holden j caufield said:
Hey guys thanks for the input, sorry I've been vague and MIA, a little vague and I don't want a competitor to know our growth, and some specifics would give up to much info, a little slow because I've got other projects in place and they need to work out MPLS and other network stuff, I also don't keep up with the hardware stuff much as I rarely see CPU as a limiting factor. I'm actually ok with keeping out old cluster of X series servers but someone above me wants to just get the best. So I'm looking at simply

1) uptime/reliable is most important, performance for the hosts is secondary, cpu and cores are so plentiful now
2) performance, latency (from what I see it's our older SAN, was leaning on Nimble but they got bought out so I don't know how that changes things)

Thanks again for some really detailed posts and advice.
Click to expand...
I left you a PM if you want to have a more private discussion.

The VSAN route is the way to go at this point for ~50-60% of use cases. Will provide you plenty of performance and reliability, but will have its limits and caveats like all things. If you are absolutely set on having a SAN, then Pure if a very good option to chase down. You should have enough info here.

At this point, you need to engage a vendor, sign an NDA, and discuss what you want to discuss openly.
 
You must log in or register to reply here.
Back
Top