OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

G

_Gea

Supreme [H]ardness
Joined
Dec 5, 2010
Messages
4,141
How to setup a ready to run OpenSolaris derived ZFS Storage Server
+configure it + add a WEB-GUI within minutes

more: napp-it mini HowTo 1 - setup a ZFS Server


How to setup a All-In-One Server -
VMware ESXi based + virtualized ZFS-SAN-Server + virtualized network Switch + other VM's like Windows, Linux..

more: napp-it mini HowTo 2 - setup a all-in one Server


with main focus on free systems
i will keep this initial post up to date, please re-read from time to time !!

ZFS is a revolutionary file-system with nearly unlimited capacity and superior data security thanks to copy on write, checksums with error-autofix, raid z1-3 without the raid5/6 write hole problem, with a online filecheck/ refresh feature and the capability to create nearly unlimited data snapshots without delay or initial space consumption. ZFS Boot snapshots are the way to go back to former OS-states. ZFS is stable, availabe since 2005 and used in Enterprise Storage Systems from Sun/ Oracle and Nexenta.

Features like Deduplication, online Encryption (from ZFS V.31), Triple Parity Raid and Hybrid Storage with SSD Read/ Write Cache Drives are State of the Art and just a included ZFS property. Volume- Raid and Storage Management are Part of any ZFS System and used with just two commands zfs and zpool.

ZFS is now part not only in Solaris derived Systems, but also in Free-BSD. There are efforts to include it in LInux ( ZoL). Even Apple (The days before they transformed from Apple Computers to the current iPod company) had planned to integrate ZFS but cancelled it just like they have done with their Xserve systems.

But Solaris derived Systems are more. ZFS is not just a file-system-add-on. Sun, inventor of ZFS, had developed a complete Enterprise Operating System with a unique integration around ZFS with other services like a Kernel-integrated AD Windows ACL compatible and fast SMB and a fast NFS Server as a ZFS property. Comstar, the included iSCSI Framework is fast and flexible, usable for complex SAN configurations. With Crossbow, the virtual Switch Framework you could build complex virtual network switches in software, Dtrace helps to analyse the system. Service-management is done the via the unique svcadm-tool. Virtualisation can be done on application level with Zones. All of these fetaures are from Sun- now Oracle, perfectly integrated into the os with the same handling and without compatibility problems between them - the main reason why i prefer ZFS on solaris derived systems-.

oi.png

In 2010 Oracle bought SUN and closed the free OpenSolaris project:
There are now the following Operating Systems based on (the last free) OpenSolaris


1. commercial options

- Oracle Solaris 11 (available 2011)
- Oracle Solaris 11 Express (free for non-commercial and evaluation use) wth GUI and time-slider
download http://www.oracle.com/technetwork/server-storage/solaris11/downloads/index.html

the most advanced ZFS at the moment, the only one with encryption
(based on "OpenSolaris" Build 151). I support it with my free napp-it Web-Gui
download http://www.napp-it.org/downloads/index_en.html
Solaris Express 11 testinstallation http://www.napp-it.org/pop11.html


-NexentaStor Enterprise Storage-Appliance
stable, based on OpenSolaris Build 134 with a lot of backported fixes
storage-only-use with the Nexenta Web-Gui
download trial at http://www.nexenta.com

-based on NexentaStor, there is a free Nexentastor Community-Edition
(=NexentaStor, with the same GUI but without Service, without some add-ons, limited to 18 TB RAW capacity and storage-only use)
NOT allowed for production use - home use only -
download at http://www.nexentastor.org

Compare different Nexenta Versions at:
http://www.nexenta.com/corp/nexentastor-overview/nexentastor-versions


2. free options

best suited for a napp-it system:

-OmniOS a minimal server distribution mainly for NAS and SAN usage, stable release with weekly betas
free with optional commercial support, very close and up to date with Illumos kernel development
download http://omnios.omniti.com/

-OpenIndiana (the free successor of OpenSolaris), based on OpenSolaris Build 151 with GUI and time-slider
currently beta, usable for Desktop or Server use.
download http://openindiana.org/download/


other options:

-NexentaCore 3, end of live
was based on a OpenSolaris Kernel Build 134 with a lot of backportet fixes from newer builds. CLI-only (without the NexentaStor Web-Gui) but running with my free napp-it Web-Gui up to napp-it 0.8

-Illumian, base of NexentaStor 4 is available.
http://www.illumian.org
Illumian is based on Illumos (like OpenIndiana and uses the same binaries) but with Debian-like software installation and packaging

-other distributions like Bellenix, Eon or Schillix


All of the above are based originally on OpenSolaris. Further development is done either by Oracle.
It was told, that they will freely publish the results after final releases If so, we may see new Oracle-features also in the free options with a delay.

On the other side, there is the Illumos Project. It is a free community driven fork of the OpenSolaris Kernel with the Base-OS-functions and some base tools. They will be as close as possible to Oracle Solaris, use their new features, when they become available but also do their own development and offer Oracle to eventually include them. Illumos is mainly financed by Nexenta and others with a focus on free ZFS based Systems. Illumos is already base for Schillix and will become the base of next Eon, Bellenix, Nexenta* and OpenIndiana. Illumos is intended to be completely free and open source.


3. Use cases:

Although there is a desktop-option with OpenIndiana, Solaris was developed by Sun to be a Enterprise Server OS with stability and performance at first place, best for:

NAS
-Kernel based high speed Windows Cifs/SMB Fileserver (AD, ACL kompatible, Snaps via previous version )
-opt. also via Samba
-Apple File-Server with TimeMaschine and ACL-support via netatalk

SAN
-NFS and iSCSI Storage for ESXi and XEN

Webserver
-AMP Stack (Apache, mySQL, PHP)

Backup and Archiv
-Snapshots , Online File-Checks with Data Refresh , Checksum

Software based Virtualization via Zones

and of course Enterprise Databases
-one of the reason, Oracle bought Sun


Appliances
-All OpenSolaris derived systems could be used as appliances and managed remotely via SSH or Browser and Web-Gui (Nexenta/ Nexentastor or napp-it on the others).


3.1 All in One Solutions

You could run OpenSolaris based systems not only on real hardware but also virtualized, best on (also free-version) ESXi with pci-passthrough to SAS Controller and disks. With ESXi 4.x you will have quite the same performance like you would have on real hardware - if you could add enough RAM to this VM. It is possible to integrate the three usual parts of a professional solution like Virtual-Server, virtualized NAS/SAN-Server and virtualized LAN/SAN/vlan-network-switch build on ESXi features in one box.

If you need hot-snaps of your VM's
-create snaps within ESXi, then with ZFS; delete ESXi snap
In case of problems, restore ZFS snap. You could then restore hot-snap within ESXi.

If you need higher availabilty, use two all-in one-boxes and replicate your vm's from the first to the second box via smb-copy from snaps or ZFS-send

Links about all-in-one:
http://wiki.xensource.com/xenwiki/VTdHowTo
http://www.servethehome.com/configure-passthrough-vmdirectpath-vmware-esxi-raid-hba-usb-drive/
http://www.napp-it.org/napp-it/all-in-one/index_en.html


3.2 short how for my All-In-One concept:

-use a mainboard with Intel 3420 or 5520 chipset + one or two Xeons
-you need a second Disk-Controller, best are LSI 1068 or 2008 based SAS/SATA in IT-mode
(flash it to IT-mode, if you have a IR/Raid flashed one)
-Install esxi (onto a >= 32 GB boot drive, connected to sata in ahci mode)
-activate vti-d in mainboard bios
-activate pci-passthrough for this SAS controller in esxi, reboot
-use the remaining space of your boot drive as a local datastor, install a virtualized Nexenta/Solaris on this local datastore; add/ pass-through your SAS Controller to this VM
-connect all other disks to your SAS/Sata Controller. Nexenta/ Solaris will have real access to these disks due to pass-through.
-set this VM to autostart first and install vmware-tools
-create a pool, share it via NFS and CIFS for easy clone/move/save/backup/snap access from Windows or from console/midnight commander.
-import this NFS-Share as a NFS datastore within ESXi and store all other VMs like any Windows, Linux, Solaris or BSD on it
-share it also via CIFS for easy cloning/ backup of a VM's from your ZFS folder or via Windows previous version.



4. Hardware:

Opensolaris derived Systems may run on a lot of current systems. But often it’s not a trouble-free experience. But there are a few always trouble-free options, mostly based on current Intel Server Chipsets, Intel Nics and LSI 1068 (ex Intel SASUC8I) or LSI 2008 based SAS/ Sata HBA (JBOD) Controller. Avoid Hardware-Raid at all and 4k drives if possible (They will work but may have lower performance, TLER disks are not needed).


See
http://www.sun.com/bigadmin/hcl/
http://nexenta.org/projects/site/wiki/About_suggested_NAS_SAN_Hardware


Cheap and low power Intel Atom or the new HP Proliant Micro Server N36 (4 HD bays, max 8 GB ECC Ram, pcie 16x und 1x slots) with 1 GB+ RAM are ok but are running with reduced performance. But you should not use some goodies like deduplication, ZFS Z1-3 or encryption or it will become really slow. But it is suggested to add as much RAM as possible.

If possible, I would always prefer server-class hardware best are the X8 series from SuperMicro with 3420 or 5520 chipset like a micro-ATX Supermicro X8-SIL-F. It is relatively cheap with i3 CPU, fast but low power with L-Class Xeons (40W Quads), can hold up to 32 GB ECC Ram with fast Intel Nics and 3 x pci-e 8x slots and ipmi (browser based remote keyboard and console access with remote on/off/reset - even when ih power-off state). It also supports vti-d, so you may use it as a ESXi virtual server with pci-passthrough to a embedded, virtualized Nexenta or Solaris-based zfs storage server.
read about that mainboard http://www.servethehome.com/supermicro-x8silf-motherboard-v102-review-whs-v2-diy-server/

minimum is 4 GB Bootdrive + 1 GB RAM + 32 Bit CPU (Nexenta)
best is 64 Bit Dual/ QuadCore, 8 GB ECC Ram+ (all used for caching)
If you want to use deduplication, you need 2-3 GB Ram per TB Data


Usually i would say: Trouble free =

- X8 Series from SuperMicro with 3420 or 5520 Intel server-chipsets (up from 150Euro),
- X-Class or L-Class low power Xeons (Dual, better Quads)
- Onboard SATA with Intel ICH10 (ahci)
- use always Intel Nics
- LSI Controller based on 1068e (always the best) or SAS2 LSI 2008 with IT firmware

about onboard LSI1068 or LSI 2008 controller
they are per default flashed in Raid-mode,
think about reflash to IT mode (native SAS without Raid functions)
see http://www.servethehome.com/flashing-intel-sasuc8i-lsi-firmware-guide/


- Avoid expander when not needed (you need it for external enclosures or if you have more disks than you can supply with additional controller) and hardware raid at all for datapools, avoid other controller or use only if others have reported success
Use mainboards with enough 4x or 8x slots for needed NICs (1Gb now /10Gb next) and needed Storage Controller.

If you follow these simple rules, you will have fun.

I would avoid SAS2 Controller ex LSI 2008 when not needed because of the WWN problem.
(LSI 2008 will report disk related WWN instead of controller based ID)
read http://www.nexenta.org/boards/1/topics/823


4b. some general suggestions:

-keep it as simple as possible
-performance improvements below 20% are not worth to think about more than a moment
-if you have the choice, avoid parts that may have problems (ex. expander, 4k disks,hardware-raid, realtek nic etc.)
-do not buy old hardware
-do not buy very new hardware
-do not buy it too small
-do not buy to satisfy "forever needs", think only up to 2 years but think about growing data, backups or if you need virtualization

follwow the best to use suggestions whenever possible
(with ZFS there are not too many)

or if i should say it in german:
"Geiz ist geil" ist genauso blöd wie
"Mann gönnt sich ja sonst nichts - wobei ich schon weiß, wie man Mann schreibt"



5. ZFS Server-OS Installation (Nexenta.*, Solaris Express 11 or OpenIndiana with suggested easy to use Live edition or with text-installer)

NexentaCore is CLI only. With OpenIndiana or Solaris Express i would prefer the GUI-Live version because they are much more user friendly and you will get the time-slider feature
(You can select a folder and go back in time, really nice to have)

Download ISO, boot from it and install OS to a separate opt. mirrored boot-drive (use always ahci with onboard sata)
You will always use the whole boot disk, so installation is ultra easy. just answer questions about your time-zone and keyboard and preferred network settings
On Nexenta3 there is currently a bug with dhcp. After Installation with dhcp, you have enable auto-config via: svcadm enable nwam

To configure a Ready ro Run ZFS-Server with all tools and extras like Web-Gui AMP and AFP, you could use my online installer script


6. First time configuration or update your NAS with the napp-it web-gui, toolss and additional NAS/www-services

nappit.png

Install NexentaCore, OpenIndiana or Solaris Express 11, reboot and
login as root (or as user, enter su for root access from your home-directory) and enter:
wget -O - www.napp-it.org/nappit | perl

The nappit online installer will configure your os (user, permissions and smb-settings), add a web-gui + minihttpd, add/ compile tools like netcat, mc, iperf, bonnie and smartmontools


Thats all, your NAS is ready to run.
Manage it via browser http://ip:81

You could setup most things via web-gui, some like acl-settings
or snapshot access via previous version are more comfortable within Windows.


7. add-ons:

On top of a running napp-it installation you could add AMP webserver stack via:
wget -O - www.napp-it.org/amp | perl

The amp online-installer will setup Apache, mySQL and PHP


or add Apple AFP+Time Machine support via:
wget -O - www.napp-it.org/afp | perl

The afp installer will compile and configure a netatalk AFP Server
with Time-Machine support. Share settings are included in the napp-it
web-gui in menue folders.


8.Some useful Links about ZFS:

excellent summary on practical tipps about ZFS
http://www.nex7.com/readme1st

newest docs from Oracle
http://docs.oracle.com/cd/E23824_01/ Manuals from Oracle Solaris 11

http://www.c0t0d0s0.org/archives/6639-Recommended-SolarisSun-Books.html (recommended books)
http://hub.opensolaris.org/bin/view/Main/documentation (overview about docs)
http://www.c0t0d0s0.org/pages/lksfbook.html (less known features)
http://dlc.sun.com/pdf/819-5461/819-5461.pdf (ZFS manual)
http://www.solarisinternals.com/ (best source of Howtos)
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide (best practice guide)
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide (best tuning guide)
http://www.oracle.com/us/products/servers-storage/storage/storage-software/031857.htm (oracle info)
http://hub.opensolaris.org/bin/view/Project+cifs-server/docs (cifs server)
http://nexenta.org/projects/site/wiki (Nexenta wiki)
http://www.zfsbuild.com/ (howto and performance tests)
ten-ways-easily-improve-oracle-solaris-zfs-filesystem-performance (10 ways to improve performance)
favorite-oracle-solaris-performance-analysis-commands (analysis tools)
http://download.oracle.com/docs/cd/E19963-01/index.html (Oracle Solaris 11 Express Information Library )
http://constantin.glez.de/blog/2011/03/how-set-zfs-root-pool-mirror-oracle-solaris-11-express (mirrored bootdrive on Solaris/OpenIndiana)

german ZFS manual (Handbuch)
http://dlc.sun.com/pdf/820-2313/820-2313.pdf (ZFS Handbuch)

about silent data corruption (schleichende Datenfehler)
(german, translate via google if needed)

http://www.solarisday.de/downloads/storageserver0609nau.pdf
http://www.fh-wedel.de/~si/seminare/ws08/Ausarbeitung/02.zfs/funktionen.html#n17


Help forums, wiki and IRC
http://echelog.matzon.dk/logs/browse/openindiana/
http://echelog.matzon.dk/logs/browse/nexenta/
http://echelog.matzon.dk/logs/browse/illumos

https://forums.oracle.com/forums/forum.jspa?forumID=1321
http://wiki.openindiana.org/oi/OpenIndiana+Wiki+Home
https://www.illumos.org/projects/site/boards


FAQ

Q What are the three biggest advantages of ZFS?
A1 Data security due to real data checksums, no raid-write hole problem, online scrubbing against silent corruption
A2 nearly unlimited datasnaps + systemsnaps and writable clones without initial space consumption due to copy on write
A3 easyness, it just works and is stable without special raid-controller or complex setup

Q Can I expand/ shrink a pool
A You cannot shrink (who wants?)
A A pool is build from one or more vdevs/ RaidZ
A You can expand a pool by adding more RaidZ/vdevs but you cannot expand a single Raidz of a pool

Q Is ZFS faster than other systems?
A ZFS was developped mainly to satisfy needs in datacenters with higest data security in mind
A With a single disk its mostly slower than other systems with much better data security
A It scales perfectly up to hundreds of disks in Petabyte ranges. Performance becomes better with any vdev you add to a pool.

Q On which platform can I use ZFS
A Newest features are from Oracle Solaris*, but you must pay for a support contract if you need patches or use it commercially
A Best free alternative is OpenIndiana or Nexentacore (dead see -> Illumos iCore project) or commercial NexentaStor
A On non-Solaris platforms you can use FreeBSD, I would not use current implementations on Linux
A Newest or all ZFS features are only available on Solaris based systems



Please read my miniHowTo's
napp-it mini HowTo 1 - setup a ZFS Server
napp-it mini HowTo 2 - setup a all-in one Server

nappitpdf.png

Gea
 
Last edited:
Good to see you back, Gea. Hopefully no over-ambitious moderator bans you this time!

By the way, what are your thoughts on ZFS with drives that have 4KB sectors but emulate 512B sectors (and report 512B sectors to the OS)? How is the write performance in Solaris-based ZFS systems?
 
Last edited:
i do not use any of them but i know a lot of people use them because they are are cheap.

i expect them to work on any of the above systems with better support on newer ZFS releasea:
Solaris Express 11 > OpenIndiana > Nexenta 3

I expect 4k drives to be standard in the near future, so the 4k problem should be obsolete with newer os-versions. any benchmarking is only valid for a few weeks or months. so if you could buy, non 4k drives, its the best at the moment or you must be aware about some performance problems up to the next major os-updates.
 
sub.mesa has implemented a workaround for 4KB/512B emulation drives in his GUI for FreeBSD ZFS (I don't have a reference, but I'm sure sub.mesa can supply it if you need one).

I was wondering if you are aware of any temporary workarounds for 4KB drives with Solaris-based ZFS systems, while we await a more permanent solution.
 
Gave that Napp-IT GUI a shot on Solaris 11 Express in VirtualBox. Pleasantly surprised. Enabled encryption and created SMB share/user. I did find that I could take snapshots, but there was no way to rollback via the gui. I had to use cmd line, which worked of course.

Between this and sub.mesa's project, ZFS NAS opportunities are looking good.
 
I've been lurking Hard Forums for about a year now, so I thought it was time to register and share my goodies for some fun and advice.

Using Bonnie I got the following benchmarks of a 5 disk, Hitachi HDS721010CLA332 raidz1-0 with the included hardware:

Code: 
System release: SunOS openindiana 5.11 oi_148
Memory size: 3840 Megabytes
System Configuration: MSI MS-7623
BIOS Configuration: American Megatrends Inc. V11.5 08/10/2010
Processor: AMD Athlon(tm) II X2 250 Processor CPU1

------Sequential Output------ --Sequential Input-      --Random-
    --Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Size  K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
7672M  71814 94 221845 59 111435 31 54921  97 424945 51 408.8   2

And a Crystal Mark test result over SMB/CIFS from W7 running in Hyper-V over a cheap "green" gigabit switch:



I'll have to research it some more to see if this is any good before I move any of my data over, or go with another solution (FreeBSD.) ZFS is interesting though. Maybe a beefer chip and some more RAM will boost results.
 
That CDM over SMB sequential read looks horrible. I would have expected 100+ MB/s.

Did your virtualized Win7 boot from the SMB volume? Because I don't know how to set CDM to use a Samba share unless it was the boot volume. Anyway, if it was the boot disk, then maybe that is affecting the speeds?
 
Last edited:
Welcome back Gea. Napp-It on Solaris Express 11 working awesome. Vielen dank! Your GUI is actually helping me to learn command line, by removing the "barrier to entry" to set everything up initially, and then finding out which cmdline options i need to use to maintain it. If I had to set it all up from cmdline to start with I would have never bothered.

@Ruroni: re: snapshots. You do realize if you're using a Windows client to connect to an SMB share on your ZFS pool, you can right-click a folder that's been snapshotted and use the "Restore previous versions" function? Works great. Works just like Volume Shadow Copy Service on windows server.
 
Last edited:
odditory said:
Welcome back Gea. Napp-It on Solaris Express 11 working awesome. Vielen dank! Your GUI is actually helping me to learn command line, by removing the "barrier to entry" to set everything up initially, and then finding out which cmdline options i need to use to maintain it. If I had to set it all up from cmdline to start with I would have never bothered.

@Ruroni: re: snapshots. You do realize if you're using a Windows client to connect to an SMB share on your ZFS pool, you can right-click a folder that's been snapshotted and use the "Restore previous versions" function? Works great. Works just like Volume Shadow Copy Service on windows server.
Click to expand...
Actually, I tried the previous version thing on the share itself rather than a folder contained within. Didn't do anything (I snapshotted before adding files, then added a single file), but the previous versions functionality was enabled (buttons not greyed out). Now that you say this, perhaps if I had folders within that share this would've worked. I'll give it another shot. :)
 
For comparison's sake, here are my CDM results. They aren't going to be perfect, as I had other network traffic going during the benchmark, but should give you an idea, and I wanted to benchmark them before my new hardware arrives.

iscsi : 4 drive WD 7200rpm RAID10
5310457423_bfe4b06d91.jpg


smb: 5 drive Seagage 7200rpm RAIDZ1
5311046798_d8830cac91.jpg
 
Here is another test, on dedicated hardware:

testtest2.jpg


Looks no better, so my guess is the switch (DGS-2205 5-PORT 10/100/1000) being "green" is causing some performance issues. Gonna try and upgrade it this weekend and see if I can get some better results before moving everything over.
 
It could also be 1) crappy nic/ bus bottleneck, 2) SATA controller bus bottleneck.
 
adi said:
It could also be 1) crappy nic/ bus bottleneck, 2) SATA controller bus bottleneck.
Click to expand...

I went with a Intel PWLA8391GT and all drives are connected to the onboard sata, operating as "IDE", not "ACHI," so maybe i'll change that first.
 
Ruroni said:
Actually, I tried the previous version thing on the share itself rather than a folder contained within. Didn't do anything (I snapshotted before adding files, then added a single file), but the previous versions functionality was enabled (buttons not greyed out). Now that you say this, perhaps if I had folders within that share this would've worked. I'll give it another shot. :)
Click to expand...
Alright. I report success. :) Too awesome.
 
some things like snapshot access or acl settings are so comfortable within windows.
not worth to add a menue item in napp-it now.

but i know there are people without windows (ex san usage or mac only) so i will add these items some day - when the more important things are done, like menue-driven async ZFS-replication between host, i am currently working on.

gea
 
nicka said:
I went with a Intel PWLA8391GT and all drives are connected to the onboard sata, operating as "IDE", not "ACHI," so maybe i'll change that first.
Click to expand...

Changed onboard from IDE to ACHI, won't boot now.. :mad: Gonna re-install and see.
 
Last edited:
Hi, my apology in case it is present but I miss it. Suggest an easy menu-option for CIFS sharing to allow guestok=true by default, or default value during initial setup. Maybe this is helpful for home users completely new to Solaris ZFS filer so that they do not need to wonder about security mapping between Windows client and the ZFS host and can go straight to file sharing.

Cheers
 
there is a menue option for guest access in menue folder, either on menue folder - create or
on the folder overview if you share a folder via smb.

if you want to have guest access per default, you may edit that menu item.
you will only need minimal perl/php or html knowledge to adjust the html form.
napp-it web-gui is fully user editable or extendable - see also mernue extend.

-activate menue editing (menue napp-it-setup, set editing to on or always)
-in editmode=on, there is a edit-link in the right top, klick on it
-klick on the "e" right of the menue item, you want to edit/modify

if something goes wrong, there is a 3-step undo in editing or just do a update via wget..
the old menues are still available after a update (selectable via napp-it setup from folder /zfs_date)

on the other hand, Solaris-mapping is quite easy. create a local user administrator, add him to smb-group administrators.
now verify that you have a mapping winuser:administrator=unixuser:root (should already be created from napp-it)
see http://www.napp-it.org/pop_en.html

now its quite the same handling like a real wndows-box.
connect as administrator and set permssions from windows.

gea
 
Thanks so much for creating this thread, it will be very helpful.

Maybe it is just me but, this link isn't working for me?


-based on NexentaStor, there is a free Community-Edition
(=NexentaStor without Service, without some add-ons, limited to 18 TB and storage-only use )
download at http://www.nexenastor.org

I think you might have forgotten a 't'?
Should it have been http://www.nexentastor.org/ instead?
 
Hi _Gea,

Thank you for your kind pointer. The instructions are easy to understand and I created the necessary environment based on your direction.

I also re-read your 1st post several times plus the ESXi all-in-one description. For convenient sake, I did the test on existing VMware player environment and found the CDM scores very encouraging. This info is for some considering quick test but find full setup of ESXi demanding. My test VM runs OpenIndiana with single vcpu and 4GB RAM with ZFS mirror for CDM test. (VMware Tools installed)

Edit : I tested OpenSolaris briefly last time and was primarily interested in Time Slider (still inside OpenIndiana). It appears now this can be done in web-gui as well (though not 100% sure because I recall reading complex blog entries by developers about putting the TimeSlider stuff in various usage scenarios.)

Cheers
 
Last edited:
adi said:
It could also be 1) crappy nic/ bus bottleneck, 2) SATA controller bus bottleneck.
Click to expand...

Got a pair of Intel PCI-E nics, and heres the results coming from my netbook:

Code: 
-----------------------------------------------------------------------
CrystalDiskMark 3.0 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :    65.569 MB/s
          Sequential Write :   100.084 MB/s
         Random Read 512KB :    65.763 MB/s
        Random Write 512KB :    90.673 MB/s
    Random Read 4KB (QD=1) :     9.110 MB/s [  2224.1 IOPS]
   Random Write 4KB (QD=1) :     4.679 MB/s [  1142.3 IOPS]
   Random Read 4KB (QD=32) :    38.719 MB/s [  9452.8 IOPS]
  Random Write 4KB (QD=32) :     7.392 MB/s [  1804.7 IOPS]

  Test : 1000 MB [Z: 0.0% (0.0/3648.6 GB)] (x5)
  Date : 2011/01/03 23:59:35
    OS : Windows 7  [6.1 Build 7600] (x64)

Gonna put another one of the Intel's in my Hyper-V setup and see how it does.
 
Your current values seems to be nearly 30% better than your first results. Did you only switched from IDE to AHCI mode or did you also changed nics?

But its clear, intel Nics are the best and AHCI is always better than IDE-mode, not only for speed but also because AHCI is hotpluggable.

Your write speed is now at Gbit-level, your other values especially on concurrent access may be only better if you add more RAM .
 
_Gea said:
Your current values seems to be nearly 30% better than your first results. Did you only switched from IDE to AHCI mode or did you also changed nics?

But its clear, intel Nics are the best and AHCI is always better than IDE-mode, not only for speed but also because AHCI is hotpluggable.
Click to expand...

Both, a Intel EXPI9301CTBLK, and did a reinstall with AHCI active. I was thinking, my Hyper-V install runs some VMs off a Raid1, one being a WHS install with a pool of 4x Seagate Barracuda LP ST32000542AS 2TB 5900 RPM. I'm moving away from WHS to ZFS, so I want to use these drives.

Should I create a mirror of 3 drives to add to the pool? And how does that work?
 
I'm moving from Nexenta Core+ napp-it to Solaris Express 11 with napp-it when I get my hardware on Thurs. Been playing with it in vmware, no big issues so far. Just wanted to say thanks for your work.
 
if you have 4 disks for zfs-datapools
you have the following options

create a n-way-mirror example (3 or 4 way)
very good performance, extremely safe
up to any of 3 drives may fail without dataloss

create two mirrored pools, one for data, one
for backups

create a raid-z1/(z2)
slow, but max capacity, good for
backup or a mediaserver
one (two) disk may fail without dataloss

if you need performance,
i would buy a 5th disk and
create a raid 10 (stripe of two mirrors)
very good performance, safe
up to two disks may fail if they are in different mirrors
you shoukd add a 5th disk as hot spare

but be aware, your disks are good for media or backups,
they are not high speed or have a lot of iops
you may add at least a 50GB ssd as read-cache.
(hybrid storage)

gea
 
_Gea said:
if you have 4 disks for zfs-datapools
you have the following options
Click to expand...

I have the following to use:

1x SSD 32g (OS install)
4x Seagate Barracuda LP ST32000542AS 2TB 5900 RPM
5x HITACHI Deskstar 7K1000.C HDS721010CLA332 1TB 7200 RPM
 
os disk is nearly performance irrelevant

i would prefer:

use a small and cheap bootdisk
ex buy a wd 160gb, 2,5" 24/7
or use any other disk

create a 2tb high speed pool of your hitachi disks
raid 10 + hotfix + ssd as read cache drive

create a raid-z backup-pool of your seagate disks
 
_Gea said:
os disk is nearly performance irrelevant
create a 2tb high speed pool of your hitachi disks
Click to expand...

Code: 
-----------------------------------------------------------------------
CrystalDiskMark 3.0 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :    65.503 MB/s
          Sequential Write :    85.153 MB/s
         Random Read 512KB :    65.504 MB/s
        Random Write 512KB :    84.163 MB/s
    Random Read 4KB (QD=1) :     9.174 MB/s [  2239.6 IOPS]
   Random Write 4KB (QD=1) :     3.536 MB/s [   863.4 IOPS]
   Random Read 4KB (QD=32) :    37.474 MB/s [  9148.9 IOPS]
  Random Write 4KB (QD=32) :     6.190 MB/s [  1511.2 IOPS]

  Test : 1000 MB [Z: 0.0% (0.0/1827.0 GB)] (x5)
  Date : 2011/01/05 21:42:57
    OS : Windows 7  [6.1 Build 7600] (x64)

Seems the RaidZ was faster on the write side.
 
@gea: any thoughts on creating a ZFS pool spindown script and GUI options in napp-it? Looks like kfoda wrote his own for FreeBSD 8, wonder if it can be adapted for Solaris.

The ability to spin down a ZFS pool is really the last remaining obstacle keeping me from migrating some of my storage off of hardware raid cards.

Check out his script here.

http://forums.servethehome.com/showthread.php?36-FreeBSD-vault&p=163&viewfull=1#post163
 
With OpenSolaris & Co, you do not need such a script.
Power-Management is part of the OS

Basically you have to do only the following:

-set desired cpu and disk threshold
edit /etc/power.conf

-enable powermanagement service via Service-Management
svcadm enable power


If you use napp-it, you can set it (for all disks) via menu
system - power mgmt

Problems about:
You have to care, that there is no service or task, regularly accessing disks like logging or ZFS-health check services or clients, regularly checking shares.


Optional:
Simple way, but always working (instead or additional to powermanagement) with max. power saving:

For my home-server i have done it much simpler.
I know, i do not need my server past 2:00 am so i have set
a power-off job via napp-it autojob.

If you also have one of these nice serverboards with ipmi, you can power-on remotely via browser, even from power-off state or you can use a simple power-timer to switch it on. (and off past shutdown)

Some links about powermanagement:
http://www.unix.com/man-page/opensolaris/4/power.conf/
http://www.nexenta.org/boards/1/topics/105
http://constantin.glez.de/blog/2010/03/opensolaris-home-server-scripting-2-setting-power-management
 
about low power solutions:
HP Proliant Microserver N36

There is a thread in http://forums.overclockers.co.uk/showthread.php?t=18216324&page=14 about the the cheap and quiet HP Proliant Microserver N36
with a very good bonnie benchmark result with raid-1 and raid-z on Solaris Express 11.

These new HP microservers with amd's new low-power dualcore cpu's seems to be a pretty nice home or backup ZFS-solution
with 4 x 3,5" disk-bays and up to 8 Gb ECC RAM. They seems to work nicely with Solaris & Co.

Due to its 16x and 1x pci-e slots, it may also be possible to add a 4 x 2,5" enclosure in the optical 5,25" slot for 2,5" disks or ssd's.
Does anyone tried a extra SAS 1068 Controller in the pci 16s slot yet? You will need a CD/DVD drive only for OS-install.
 
@Gea, thanks for the info. I'm aware of the ability to power on and off the entire server on a timer, I'm just trying to find the closest way of replicating my current storage scheme which is an Areca raid card powering down idle arrays after 60 minutes of inactivity, aka MAID. I don't care about saving $$ on power, I care about extending the life of the drives. I have a many arrays and many drives and it is mostly archival data so due to the infrequency of access I don't want those disks spinning 24x7. Also I have VM's running on some of the servers with arrays (i.e. Win2008 R2 with Hyper-V) so powering down the whole server isn't an option.

I think a simple GUI option to power down the ZFS pool after X minutes of inactivity is something a lot of people would be interested in, so they wouldn't have to edit conf files and deal with cmdline. I realize there are complexities to that - and you can't just "power down the pool" due to other background processes being hooked into ZFS that might wake the pool up unnecessarily, so I at least wanted to start a dialog on the best way to implement.
 
set disk threshold in system powermanagement to 3600 and your disk should go down 60min after last access. But i heave heard of poblems due to ZFS-health checks on Nexenta. (Regular health check to discover disk failures prior to a failed access). I have not tried it by myself yet.

i have also a lot of servers with many disks. but i do not expect less failures due to regular power-down. i expect a higher failure rate. i expect most failures are on power up cycles.
 
Understood, I'm just putting the idea out there, that you might consider making a GUI option in the future. I'm more interested in how to accomplish it than the pros and cons of spinning down disks as it relates to lifespan - I realize there are two schools of thought on this. People like me that live in an earthquake zone are served well to keep one copy of their data spun down or on tape.

I assume that excessive disk wakeups due to zfs health checks would be a similar problem on Solaris Express 11. I might play around with FreeBSD 8 on account of the guy I linked that has his zfs pool spindown script working for this exact purpose.
 
Last edited:
Somewhat related, my dream is a multi-level, nearline approach to archival data storage, be it based on ZFS or anything else, where you've got one large pool of write-infrequently/read-infrequently data (say 16 x 2TB in raidz2), which can stay spun down or low-power-idled most of the time, and then a smaller pool (2 x 2TB in raidz, or multiple SSD's) that acts as a buffer/cache and flushes to the larger pool at a user defined interval, say every 24 hours, or every week, or month, or when its X% full. A "landing pool" basically. This process is transparent to the client, all that's seen is one pool. This functionality is obviously beyond the way L2ARC and ZIL function right now. A few commercial third parties have attempted this with ZFS, but obviously it would require getting into the source for that kind of ability. Hopefully this is something we'll see in the coming years.
 
Just saw this thread from Zarf at http://forums.overclockers.co.uk/showthread.php?t=18221307 about HP's new HP Proliant Microserver and Solaris 11

101 MB/s via SMB would be a ultra excellent result with such a low power, low cost server. Could anyone with a HP Microserver and Solaris 11 verify such high values?


Zarf wrote
"Using Solaris Express 11. I was on OpenIndiana, but switched over to the dark side to see if it made any difference (which it didn't). Now its working I cba to go back.
I sorted things out, still not exactly sure what did it but I get 101MB/s sustained reads and writes over SMB now. There's a great web configuration interface for ZFS called napp-it if you are interested. It's easily installed via perl script from the internet.

Among the things I did were upgrading to 4GB RAM, disabling sync on the smb shared zfs folder.."
 
Hmm I wonder how risky disabling sync (I imagine he means the zfs sync= property). I briefly read about it.

Honestly I average around 90+MB/sec with smb, I have 12GB of ram (I also have a couple of VirtualBox VMs running inside OS), so probably not worth it, but maybe something I could try on a Share I don't care too much about.

I find most of the time my slowdowns are my workstation and not the server. (oh and the network... man I need a beefier network...)

Not to hijack... but has anyone done the OpenSolaris to Solaris 11 Express? I've been reading the blogs, just wondering if anyone has some first hand experience and any unexpected hiccups. I would hope that the upgrade would be smooth if you're running b134... of course unless you're running PostgreSQL :) I'd probably keep my zpool/zfs levels as I'd want the option to go back to b134.

PS Keep up the good work Gea. I don't personally use Napp-it, but impressive project. I should give it a whirl some time. I'm unclear do you utilize time-slider for the snapshots or your own scripts/cron jobs?
 
Last edited:
odditory said:
Somewhat related, my dream is a multi-level, nearline approach to archival data storage, be it based on ZFS or anything else, where you've got one large pool of write-infrequently/read-infrequently data (say 16 x 2TB in raidz2), which can stay spun down or low-power-idled most of the time, and then a smaller pool (2 x 2TB in raidz, or multiple SSD's) that acts as a buffer/cache and flushes to the larger pool at a user defined interval, say every 24 hours, or every week, or month, or when its X% full. A "landing pool" basically. This process is transparent to the client, all that's seen is one pool. This functionality is obviously beyond the way L2ARC and ZIL function right now. A few commercial third parties have attempted this with ZFS, but obviously it would require getting into the source for that kind of ability. Hopefully this is something we'll see in the coming years.
Click to expand...

Hierarchial storage, huh?

It looks like OpenArchive might be able to do something like this. OpenArchive can use NFS/CIFS as a backend. It can maintain the most frequently used data on one array, then push out the less-used data to a larger archive pool.

Here's a PDF on it:

http://www.openarchive.net/sites/default/files/file/0309-OA-QG-E-350.pdf

EDIT: Looks like I was wrong, it doesn't look like you can use NFS as a storage backend. However, it will accept RAID (so hey, you can keep those Arecas!)
 
You must log in or register to reply here.
Back
Top