dd if=/dev/random of=/dev/blog

I just stumbled onto this blog entry on the implementation of data deduplication into the Sun Microsystem’s ZFS file system. It is implemented in such a nice and clean way, I am looking forward to testing it. For instance, just like any other feature of the ZFS file system, data dedup can be enabled disabled at any path from the ZFS root mount point. Examples taken from Jeff Bonwick’s blog post cited above:

It is that simple (man 1 zfs).

On 23 October, 2009 it was announced on MacOSForge that Apple had decided to discontinue any and all development on the porting of the ZFS file system. I know that I am not the only one to say this but I am not surprised. Supposedly there were legal reasons behind this action but in the end, who cares? They are the ones losing out to continue with an out dated and still limiting file system.

Now Apple has recently been hiring file system developers to develop a next generation file system to replace the traditional HFS+ but (as Robin Harris has previously stated) how long will it take before it becomes stable and accepted by the general public? Traditionally it takes 5+ years before a file system is considered somewhat stable and ready for production use. It wasn’t until recently that ZFS was starting to make its impact in the enterprise scene. Though my question is, to whom will this next generation file system cater to? I am to assume that it will be for the general end user utilizing Mac devices that “don’t require the weight of the ZFS features and functionality” ; or so it has been said regarding the topic of Apple abandoning the ZFS project. If that is the case and is the primary focus of the new file system, how will this impact their server market share? We already know that there is no such thing as a perfect file system that will perform ideally in every arena it is thrown into. Some will excel more than others and is entirely dependent on its implementation and workload.

In past posts, I have always stressed the importance of the file system and what is integrated within the file system. I routinely point out the numerous drawbacks and limitations of the NTFS driver. Sure, Microsoft compensates for the “lack of features” with applications, services and additional APIs to fill in all those gaps. A good example is VSS (shadow copy). This can impact performance as it is taking file system concepts out from kernel mode and into user land and consuming user mode resources. All these feature should and need to be incorporated into the file system driver. That way we can ensure that there is stability and consistency with all functions the file system performs. Even the general layout is not ideal for traditional computing over large storage media; as the fragmented large seeks between the MFT and the file data can put a lot of stress on the magnetic device. Going back to HFS+ and sort of on the same topic (although the concept is a bit different), the same could be said about Apple’s Time Machine and it running as an application on top of the driver.

One thing that I hold to heart when it comes to file systems is the ability and flexibility to tune it even without taking the mounted device(s) off-line. Most modern UNIX and Linux file systems offer a lot of tunable features (built into the driver!). For instance (through the ZFS character device node) I can dynamically alter file system variables (man 1 zfs). For this example I will focus on access times. Let us say I am using an SSD and decide that it would be more cell friendly and better performing to disable file access times on the root mount.

To view current settings and disable this feature you would type the following in the command-line terminal:

I just hope that Apple is prepared for the journey they are about to embark on. They obviously have file system development experience, and I have no doubts that they have the talent. Do they have the patience and time to invest?

According to the H Open Source website, Sun just recently announced the availability of Solaris 10 Update 10/9.

Read more here and here. You can download the OS release here.

Yesterday afternoon I was really bored at work and had eventually navigated to a website dedicated to Easter Eggs that could be found on an operating system, software application and more. Naturally I went to the list of operating systems and started looking up the operating systems which were accessible to me. As I read through the Linux and UNIX related ones, I had already known some but there were a few that I was interested in trying.

Seeing how I was on an OpenSolaris laptop I decided to first look through the SunOS list. Unfortunately none of them seemed to work. It would appear that they were taken out. But I did remember one from many years ago that a friend (Marian Lakov) had shown me. Originally found on an installation of RHEL, it was in the man page for the xorg.conf file.

Listed under the VIDEOADAPTER SECTION you will read the following: Nobody wants to say how this works. Maybe nobody knows…

If you know of any hidden secret(s) that is not listed on the site posted earlier, please feel free to share.

Ever since I started working with OpenSolaris (release 2008.05 to build 118: 2010.02), I have been suffering through some of the longer load times. While the distribution is maturing fairly well and quick, the boot times are just horrible. And to my understanding the culprit is ZFS. OpenSolaris utilizes ZFS as its default file system. On top of that, one thing that I still cannot understand is why GRUB defaults its timeout value to 60 seconds. 60 seconds! Why!?! Who needs this 60 seconds and/or who wants to be constantly annoyed to hit enter to the default kernel image, initiating the boot process? Either way, this can be modified. On OpenSolaris, editing the GRUB boot options is a little different from your traditional UNIX/Linux operating system. Note that this article is for Intel architectures and not SPARC.

Traditionally we find the appropriate files for editing in the /boot path, specifically in /boot/grub; and depending on your distribution the configuration file can vary (grub.conf or menu.lst). In OpenSolaris and on the ZFS file system, while the /boot/grub/ path exists, it does not contain the menu.lst file that we need. Instead, it is located in the /rpool/boot/grub/ path.

When you really start using Solaris/OpenSolaris, you may notice one thing that sticks out when compared to the GNU/Linux counterpart; and that is Solaris/OpenSolaris tends to be better polished when it comes to using the command line for editing system configuration files. For example, two separate tools exist for managing boot configuration files and the boot environment: bootadm and beadm. I know that certain Linux distributions have their own sets of tools for managing such stuff (i.e. QGRUBEditor as I have seen in Ubuntu; among others) but when I hop back onto a Solaris machine, it just seems simpler and a lot more straight forward. It is also standardized across both operating platforms as opposed to each distribution having their own. Historically I have always opened up the menu.lst or grub.conf file with vim and made my modifications right there. While this can still be done, the development team behind Solaris/OpenSolaris have decided to standardize it within the two applications.

bootadm

As mentioned earlier, bootadm is used to list and/or redefine specific values in your menu.lst file. Its usage is as follows (man bootadm):

If I wanted to list my current configuration I would type at the command line:

I can easily modify a parameter such as the timeout with the following command:

beadm

The beadm tool is used to create and enable new boot environments. What beadm can do is take a snapshot of your current environment. This routinely occurs (transparent to the user) after a system update. Usually these snapshots should be made when applications are installed/removed to even when configuration files are modified. It will then append the listing into the menu.lst file. This way, if the new image ends up bringing down the system, you can revert back to the previous image (snapshot). Such are some advantages when the ZFS file system incorporates its own native snapshot mechanism. Basic usage for this utility is extremely simple (man beadm).

To list all boot environments you would type the following on the command line:

Let us say you made some changes to a configuration file or two or maybe install some applications or enable/disable services. You may want to create a new image so that if someone was wrong with the image, you can always revert back to the previous.

A new entry is created in GRUB, although the older image is still the default.

To activate the new image and have it default in GRUB, you can invoke beadm as so:

After reboot beadm list will look like this:

After writing my article on The Linux RAM Disk for Linux+ Magazine and also after writing a very generic Linux RAM disk block device module, I decided to play around with the concept of RAM disks on OpenSolaris 2009.06. I must admit that this was actually a very great learning experience. One that I wish to share with the reader. Note that this post will be separated into two section: (2) tmpfs and (3) ramdiskadm.

TMPFS

While the tmpfs module exists across multiple operating systems, including Linux, the Solaris/OpenSolaris version does differ quite a bit its Linux counterpart. It is also not as flexible as the one found in Linux. For instance, on the Linux version you have the following supported user-defined module parameters (for more details, please reference the Documentation/filesystems/tmpfs.txt file in the Linux kernel source tree):

• size - limit of allocated bytes for the size of the volume
• mode - volume permission (once mounted)
• nr_blocks - same as in size, but in blocks
• nr_inodes - maximum number of inodes

The Solaris/OpenSolaris version only defines size and when you list the mounted device with the df command, it labels it as a swap. When it comes to volume permissions, you can always work around this (read below). In all cases, the advantages to utilizing tmpfs lie in the fact that it works off of virtual memory and can swap to disk when necessary. It runs like a normal file system, but when the power goes out or when the mounted volume is unmounted, all data contents disappear; as is the case with any RAM disk and no method of synchronization employed. By default, a normal installation of Solaris/OpenSolaris will use tmpfs for various computing needs such as mounting the /tmp directory with a tmpfs volume. This module can also be used to help ease a user’s computing experience and security, for example by optimizing Firefox and instead of having cache all data contents to the physical hard drive, create and have it write all necessary content to a tmpfs mounted device. The file system can also serve well in an area where constant database queries or other web services are being cached and routinely accessed. These are a few of many scenarios in which this can be used in.

Despite the few differences between each operating system, the tmpfs module is still easy to work with. Once a directory for the mount point is created, you can then mount the tmpfs RAM-based volume:

You can even configure the /etc/vfstab to automount the tmpfs volume at bootup:

The only major drawback is that at this point only root or someone with superuser permissions (i.e. sudo/pfexec) can work with the volume or change the mount point permissions so that other users can access it. If you desire have these permissions altered at bootup, it may be advantageous to automate it in a startup script. If it doesn’t already exist, you can create it. I am referring to the /etc/rc3.d/S99local file. Some, if not all of you may already be familiar with the concept of run levels and if you are coming from a Linux environment, you may realize that the run levels are not the same between Linux and Solaris. But, this is a topic for another day. Notice under the field of startup, how I use pfexec (similar to sudo) to change the permission of the tmpfs mount. Also note that you can skip the vfstab step shown earlier and just mount the file system within the same script file.

As can be seen, tmpfs is very easy to work with and can be applied to many things to bring forth many advantages and additional securities (a result of having the contents disappear at power down of the system). If one wanted to employ some basic method of synchronization, a script can be called during scheduled periods to perform an rsync to another mount point.

RAMDISKADM

I find this module to be an excellent tool, capable of being utilized in both production use or for teaching purposes. To utilize the ramdisk module you need to invoke the ramdiskadm command on the command line. For example, let us say I want to create a memory-based volume 100 MB in size, I would type:

A device node would be created at /dev/ramdisk/ramdisk1. You can format this node with a file system and mount it locally to even exporting it as an NFS share; or configure it as an iSCSI target.

To destroy the RAM disk, you would need to type:

If you are interested, you can even create multiple ramdisks and pool them in a ZFS volume.  An advantage to utilizing this approach comes from the checksum feature to prevent data corruption of contents stored in volatile memory and also the ability to grow the volume and add it to the RAM-based pool.

I now have a zfs pool mirroring two 100 MB ramdisks. Obviously there are no real advantages to mirroring two RAM disks but this just serves as an example. I can check the status of the rampool device with the following command:

Listing the zfs device will provide the following information (note that I cropped out the unrelated material):

To destroy the rampool I can invoke the following on the command line:

Let us create a RAIDZ volume:

zpool status gives us:

A zfs list will show:

I now want to add another device to the rampool. Notice the differences when I list the zpool status and list the zfs device.

CONCLUSION

As one can see, working with RAM disks on Solaris/OpenSolaris is fairly simple and can be configured in just a couple of steps. Who knows, you may find situations in your current environment which may benefit from the use of a RAM disk. Note - Do not forget to destroy the zpool and the RAM disks when not in use. The last thing you want to do is waste much needed memory.

Please note that this is only a personal opinion of mine as I have been observing the growth and various decline of storage concepts within the data storage industry. The views of the reader may differ from my own which is why I would invite you to please post your opinions as a comment to this post.

One of the most volatile and yet needed industries is the data storage industry. As computing technologies become more cloud centric and rely upon the web for business, productivity, education to even recreation, there is a constant push to increase capacities but even more so increase I/O throughput. As a result of recent demands, our approach with these technologies need to be re-evaluated. The primary focus of this article is on the future of data storage concepts and the limited life and functionality of RAID.

Back in 1987 when the idea of RAID was first conceived, the goal or vision was to be able to scale multiple drives into a single volume which was represented to a host as such while also offering a form of redundancy with a more sensitive magnetic platter-based disk technology. Flash forward to the present and we are still reliant upon the same technologies. Is that because RAID is so perfect or have we just grown too comfortable and are too afraid of change?

Hardware Vs. Software RAID 

There was a time when processing power was limited and it became advantageous to utilize external methods for creating and managing arrays of data storage, but as time progressed, this approach became increasingly insignificant. At least that is to say for the Small-to-Medium sized Business (SMB). For the last decade, a lot of efforts have been placed toward increasing the reliability, stability and enhanced features with the software-based RAID. This has slowly been eating away at the hardware vendors. Although it has been rarely noticeable.

These software implementations are integrated with methods of Logical Volume Management (with built in redundancy via RAID 1-6), Load Balancing/Multipathing capabilities, data encryption, along with the abilities to utilize incremental snapshot(s) over designated volumes. These software implementations include dynamic resizing, quota/permission management, enhanced copy-on-write file systems that perform very well along with routine checksums to correct noisy and silent data corruption; almost all of which can be managed while volumes are on-line. Some of these volume managers have the capability to export iSCSI & FCoE targets and can also be tuned to support FC targets.

To name a few you have ZFS (an all-in-one solution), Btrfs (still in development and under test), device-mapper / LVM2 / multipath-tools, mdadm, DRBD, etc. The list goes on. What is to stop an SMB from setting up an array of JBODS and (if more redundancy is needed) cluster a couple of Solaris / OpenSolaris or Linux servers to manage their software RAID while also exporting it via a file server or into a SAN? Note that Lustre support for ZFS is still in development. Realistically most entry-level modular external RAID solutions don’t run on the latest and greatest of hardware components (as they are intended for a limited purpose and not to provide other hosting services). You will most likely achieve much greater performance with the software approach while also utilizing a much more efficient virtual memory manager (for enhanced caching) alongside a finely tuned schedular.

On the enterprise end of computing you will find some very impressive storage solutions that are intended to take the workload of the enterprise environment. Such companies as Hitachi Data Systems (HDS) have been doing an excellent job with providing high-quality and well performing storage solutions that are also easily manageable. Other companies have resorted to being a little creative in order to gain some market share with the SMB and larger companies. Such notable companies are NetApp, Data Domain to even Cleversafe.

Earlier I found an interesting link differentiating the positives and negatives of both hardware and software RAID implementations. It should be noted that times have changed and some of the key points highlighted are no longer an issue. For instance, under the category /boot partition, this seems to no longer be an issue with at least ZFS.

Enter the SSD

In more recent years, the Flash-based Solid State Drive (SSD) has been entering into enterprise markets. This is a result from such notable providers as Sun Microsystems, etc. Currently the percentage in SSD usage in the enterprise is somewhat minimal as their is a limit in maximum capacities for the drives. This may soon change as in Q3 of 2009, PureSilicon will release their Nitro 1TB SSD drive. The throughput and performance speeds seem very optimal in arenas where greater speeds are needed, but the technology introduces additional handicaps (in the form of write operations and a limited cell life) which most environments and some manufacturers have a difficult time in accomodating to. To combat the limited cell life, vendors have implemented their own method of wear leveling, transparent to the host. With this concept, the same data cell, when accessed and written to multiple times will not get written to the exact location but instead, through an “intelligent” built in firmware the data will get written to another cell on the drive. To the operating system, it is still the same “sector” location. While there is very little latency in seeking performance (sequential and random), write operations take a huge hit, especially with smaller I/O transfer sizes, when typically the flash medium erase/rewrite a 128K page at a time.

SSD Tuning

With the recent hype of Flash-based SSDs, many vendors and UNIX/Linux distributions have been writing file systems tuned to perform extremely well on SSDs (and limit the impact of these handicaps). For example, Sun Microsystem’s ZFS (available on Solaris, OpenSolaris, MacOS X [read-only], FreeBSD and Linux [over FUSE]) had recently added tunable support for SSDs in their release versions for Solaris & OpenSolaris, while the development of Btrfs for Linux has done the same. In contrast the Microsoft developed NTFS does not offer such features or functionality. In fact the file system has remained somewhat unchanged over the course of the years and is just as inferior now as it was when it was first released as a replacement to the FAT series of file systems. I wrote an entire post explaining why the NTFS file system is not well suited for today’s methods of computing here.

In recent releases it should be noted that Microsoft’s Windows 7 has been tuned for SSDs that are to be provided on netbooks. What this means, I do not know? And by tuned, this is still unclear. You can read some of that information here. The only reason for the lack of changes in NTFS is to preserve backwards compatibility. This approach limits the ability to update a current existing server’s (if not running Windows 7) NTFS module if it needed to serve backend storage utilizing SSD media.

The Impact on RAID Technologies

As SSDs become more popular the advantages to using RAID are reduced, where the only benefits are gained from a simple stripe in a RAID 0 or mirroring to a backup array within a SAN or other form of network using RAID 01 (not to be confused with a RAID 10); just in case access to the first fails for whatever reason. This is where DRBD would come in real handy. As I briefly mentioned earlier, the whole concept of this form of redundancy was dependent upon the problematic nature of a magnetic disk device; where failures were imminent. And for those who are concerned with a method of error detection for both silent and noisy data corruptions, the majority of RAID implementations (both hardware and software) do not validate the data like the ZFS or Btrfs checksum implementation.

Changes in Protocol Layers?

With the popularity of SSD technologies growing and its costs reducing, the one drawback that is setting manufacturers and consumers back are the limitations offered by the protocols that they are working with. Today, Fibre Channel, SAS and SATA are not capable of handling full SSD speeds and serve only as a bottleneck to the technology. There have been recent attempts from vendors as Fusion-io to even PureSilicon to rely on other protocol interfaces such as PCI Express (PCI-E). Capable of handling up to 1 GB per second, it only seems natural for these vendors to move in that direction. I anticipate that shortly, others will follow. Fibre Channel and SAS may continue to serve the SAN (and with the appropriate load balancing mechanisms configured, it will perform well) but when it comes to the drive within the chassis, I expect to see more PCI Express in the near future. But who knows, with the recent drop in prices for 10Gb Ethernet or the supported high throughput offered from Infiniband, things may be moving toward another direction altogether.

In conclusion, I predict that in five years time we will start to see some huge and very interesting changes. I am looking forward to it.

I have been using Sun’s Solaris (and now OpenSolaris) for years now. In fact while some of you can probably go as far back as the SunOS days, the first Solaris operating system that I worked on was Solaris 8. The company that I was working for had some older SPARC server/client nodes. They had conformed to the mindset of, “if it isn’t broken, why fix it?”

During this time I had already grown extremely comfortable with GNU/Linux. Especially when it came to the text editor tools. I have always been a fan of vim (vi improved); but when I would hop from one platform to the other, I always found myself getting stuck with the way Solaris and now, OpenSolaris default their environment.

By stuck, I mean trying to get used to their command line interface in general. The shell used to always default to the traditional Bourne Shell (/bin/sh) and not the Bourne Again Shell (/bin/bash). While this could always be customized under the user’s profile in the /etc/passwd file, I am glad to see that at least in OpenSolaris this has been changed. Now the terminal defaults to bash. Although I do not know if this will carry over to Solaris 11.

That is why as soon as a new Solaris/OpenSolaris installation is completed, I append the following additions to my my ~/.vimrc and ~/.bashrc files:

.vimrc

.bashrc

Note that sometimes (if it doesn’t already exist) you may have to create the .vimrc file. What these settings do is enable the ruler and syntax highlighting while also adding some necessary color to your terminal environment. This will aid in differentiating between file types from normal listings of directory contents while also color coding scripting and other code-like syntax.

To add some more customization, I will even append certain key binding because for some odd reason Sun has never implemented them in their terminal. For example, the Delete, Page Up and Page Down keys (still working on figuring out Home and End; although same results can be obtained with CTRL+A and CTRL+E). The delete will perform a delete of the characters on the prompt while the page up/down keys will page through your recent history. You would append the following to the .bashrc file:

These features help me get started and as I use the system more, usually more will get added to these configuration files.

Before I continue with my entry, I just wish to note that VirtualBox 2.2.4 has been released. You can review the Changelog here.

Anyways, whenever virtualizing a non-Windows operating system which utilizes the X Windowing system over VirtualBox, it may be beneficial to have some flexibility on supported resolutions for the GUI. For example, I was using OpenSolaris 2008.11 and VirtualBox seems to create a “virtual” monitor where the operating system (specifically X) is unable to read the monitor’s EDID information to obtain supported resolution information (among other things). As a result of this, by default X assigns 800×600@60 and 640×480@60 as supported display formats. When you are working on a wider screen that supports something larger, this makes for an uncomfortable computing experience; especially with limited graphical space on the virtual client.

In my case, my laptop’s wide screen has a native resolution of WXGA (1280×800). So I had plenty of extra room to work with. WhileVirtualBox allows you to fullscreen a virtual client, I like to multitask and this would limit my multitasking. I wanted to create a display configuration that would utilize most of the 1280×800 while allowing me to manage multiple other applications/windows on my host operating system. So I figured to write those entries manually in my xorg.conf file. This is located at /etc/X11/. So I began to play around with some standard display formats.

By default, OpenSolaris had the following under the “Screen” section:

Below that I added:

I then reloaded X by rebooting the virtual client and once the operating system came back up, all those options were available. No more default 800×600 and 640×480. In the image below you will notice that the OpenSolaris Virtual Client is displaying at a 1024×600.

Added Note 3Jun09: Please refer to Comments 1-3 for information on Guest Additions.

Today Sun Microsystem’s has announced the release of the latest version of OpenSolaris. It is OpenSolaris 2009.6 which can be downloaded here. You can find a nice overview (with images) here.