Storage Refresher: Latest in Fibre Channel

Just a quick refresher around some storage concepts that one can keep forgetting as time goes.

Every switch has a ID called domain ID and one switch in the fabric will be a principal switch and others will be subordinate switches. Switches distributes domain id and fabric information to peer switches in its fabric.

Disks

Near-line (NL) disks: high-capacity, slower spinning disks (typically 7.2k RPM). Cheaper and goood for capacity, not for performance.
SCSI: the original parallel interface and, more importantly, the command set that SAS, Fibre Channel and iSCSI still use today.
SAS (Serial Attached SCSI): enterprise disks, typically 10k or 15k RPM. It is a separate interface from SATA, although SAS controllers can also run SATA drives.
NVMe: flash drives that connect over PCIe rather than through a SAS or SATA controller.

Storage Class Memory

RAM in our systems are volatile, which means if the host crashes the data won’t be in the ram when the host returns but these SCM sticks are NVRAM and will survive a host crash

Levels of Protection

RAID 0 – this means striping.. Data is stped into multiple disks so it means its good write performance and read too but if we lose a disk we lose the whole data
RAID 1 – means mirror. Evey data coming in is mirrored so it offers 2.X protection but 2X writ penalty as data needs to copy twice. It’s good for reliability
RAID 5 – Strpping with distributed parity adn this can offer 1 disk failure
RAID 6 – Stripping wth distributed parity and can offer 2 disk failures
Erasure Coding – Data split into data fragments plus parity fragments, spread across disks, nodes or sites

Types of Storage

SAN – Block Storage
NAS – File Storage. A NAS array is usually fixed to a Datacenter or single premises. NAS systems have usually a hard limit on how much they can grow called inode limit and can affect performance as it grows.
Object Storage – used for unstructured data e.g.: S3 accessed over http or API. This type of storage can span across multiple datacenters, and the advantage of object storage is that it allows metadata info, tags to be published. eg: In a NAS system we can store unstructured data too but searching via tags, metadata etc for patient data, weather data etc via tags is bit better in object storage… This is infinitely scale out and offers versioning etc in each file e.g.: S3.

NAS Protocols

CIFS or SMB – for windows clients
NFS – for Linux clients

FC Protocols

FC – Fibre Channel
ISCSI – TCP/IP over Ethernet
FCOE – Lossless Ethernet
NVMe-OF – Does only NVMe.

NVMe over Fabrics (NVMe-oF) carries NVMe commands across a network fabric, so hosts get near-local NVMe latency from shared storage.

Fibre Channel fundamentals

Initiator is the host and target is storage.

WWPN (World Wide Port Name) is a 16 hex digits id that is assigned to ports in fabric eg: hba port on server. Think of it like a MAC address. Zoning and LUN masking almost always use WWPNs.

WWNN (World Wide Node Name): identifies the node, such as the HBA or server as a whole.

FLogi PLogi Prli

How a server gets from power-on to seeing its LUNs?

When any fibre channel device is attached to a fabric it will issue a FLOGI (Fabric Logi) . The switch responds back by assigning it a 24-bit address known as FCID so think of this like an IP address that is required to route traffic between server to storage.

The switch maintains a table of FCID which can be viewed by show flogi database. There is a service called FCNS (fibre channel name service) that shares this flogi info with other switches.

After the switch responds the server will send a Port logi ( PLOGI) which checked with zoning information and based on that the server is allowed to connect to storage. This is the security at the switch level

Once the connection reaches the host will send a PRLI (Process login). The lun masking is taken into account based on the host’s WWPN the san will allow or deny access to the LUN based on the zoning done. This is the security at the storage subsystem level

TCP IP Offload

This is actually used to offload the TCP/IP processing from CPU to specialised ISCSI cards. This setting is controlled in the hba properties. It can give some performance boost under heavy iSCSI load.