Cluster Shared Volume Diagnostics

[Elden Christensen]

First published on MSDN on Mar 13, 2014
This is the second blog post in a series about Cluster Shared Volumes (CSV). In this post we will go over diagnostics. We assume that reader is familiar with the previous blog post that explains CSV components and different CSV IO modes http://blogs.msdn.com/b/clustering/archive/2013/12/02/10473247.aspx

Is Direct IO on this Volume Possible?

Let’s assume you have created a cluster, added a disk to Cluster Shared Storage, you see that disk is online, and path to the volume (let’s say c:\ClusterStorage\Volume1) is accessible.  The very first question you might have is if Direct IO even possible on this volume. With Windows Server 2012 R2 there is a PowerShell cmdlet that attempts to answer exactly that question:

Get-ClusterSharedVolumeState [[-Name] <StringCollection>] [-Node <StringCollection>] [-InputObject <psobject>]    [-Cluster <string>]  [<CommonParameters>]

If you run this PowerShell cmdlet providing name of the cluster Physical Disk Resource then for each cluster node it will tell you if on that node if the volume is in File System Redirected mode or Block Level Redirected mode, and will tell you the reason.

Here is how output would look like if Direct IO is possible

PS C:\Windows\system32> get-ClusterSharedVolumeState -Name "Cluster Disk 1"
Name                         : Cluster Disk 1
VolumeName                   : \\?\Volume{1c67fa80-1171-4a9e-9f41-0bb132e88ee4}\
Node                         : clus01
StateInfo                    : Direct
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirectedName                         : Cluster Disk 1
VolumeName                   : \\?\Volume{1c67fa80-1171-4a9e-9f41-0bb132e88ee4}\
Node                         : clus02
StateInfo                    : Direct
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirected

In the output above you can see that Direct IO on this volume is possible on both cluster nodes.

If we put this disk in File System Redirected mode using

PS C:\Windows\system32> Suspend-ClusterResource -Name "Cluster Disk 1" -RedirectedAccess -ForceName                                    State                                   Node
----                                    -----                                   ----
Cluster Disk 1                          Online(Redirected)                      clus01

Then output of get-ClusterSharedVolumeState will change to

PS C:\Windows\system32> get-ClusterSharedVolumeState -Name "Cluster Disk 1"

Name                         : Cluster Disk 1
VolumeName                   : \\?\Volume{1c67fa80-1171-4a9e-9f41-0bb132e88ee4}\
Node                         : clus01
StateInfo                    : FileSystemRedirected
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : UserRequest
BlockRedirectedIOReason      : NotBlockRedirectedName                         : Cluster Disk 1
VolumeName                   : \\?\Volume{1c67fa80-1171-4a9e-9f41-0bb132e88ee4}\
Node                         : clus02
StateInfo                    : FileSystemRedirected
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : UserRequest
BlockRedirectedIOReason      : NotBlockRedirected

You can turn off File System redirected mode using following cmdlet

PS C:\Windows\system32> resume-ClusterResource -Name "Cluster Disk 1"Name                                    State                                   Node
----                                    -----                                   ----
Cluster Disk 1                          Online                                  clus01

State of CSV volume does not have to be the same on all nodes. For instance if disk is not connected to all the nodes then you might see volume in Direct mode on nodes where disk is connected and in BlockRedirected mode on the nodes where it is not connected.

CSV volume might be in a Block Level Redirected mode for one of the following reasons

• NoDiskConnectivity – Disk is not visible on/connected to this node. You need to validate your SAN settings.


• StorageSpaceNotAttached – Space is not attached on this node. Many Storage Spaces on disk formats are not trivial, and cannot be accessed for read/write by multiple nodes at the same time. Cluster enforces that a Space is accessible by only one cluster node at a time. Space is detached on all other nodes and it is attached only on the node where corresponding Physical Disk Resource is online. The only type of Space that can be attached on multiple nodes and is a Simple Space, which does not have write-back cache.

When you are using a Mirrored or Parity Space then most often you will see that volume is in Direct IO mode on the coordinator node and in Block Redirected mode on all other nodes, and the reason for block redirected mode is StorageSpaceNotAttached. Please note that if a Space uses write-back cache then it always will be in Block Redirected mode even it is a Simple Space.



CSV might be in the File System Redirected mode for one of the following reasons

• UserRequest – user put volume in redirected state. This can be done using the Failover Cluster Manager snap-in or PowerShell cmdlet Suspend-ClusterResource.


• IncompatibleFileSystemFilter – An incompatible file system filter attached to the NTFS/REFS file system. Use “fltmc instances” system event log and cluster log to learn more. Usually that means you have installed a storage solution that uses a file system filter. In the previous blog post you can find samples of fltmc output. To resolve that you can either disable or uninstall the filter. The presence of a Legacy File System filter will always disable direct IO. If solution uses a File System Minifilter Driver then filters present at the following altitudes will cause CSV to stay in File System Redirected mode
  
  
  
  • 300000 – 309999 Replication
  
  
  • 280000 – 289999 Continuous Backup
  
  
  • 180000 – 189999 HSM
  
  
  • 160000 – 169999 Compression
  
  
  • 140000 – 149999 EncryptionThe reason is that some of these filters might do something that is not compatible with Direct IO or Block Level Redirected IO. For instance a replication filter might assume that it will observe all IO so it can then replicate data to the remote site. A compression or encryption filter might need to modify data before it goes to/from the disk. If we perform Direct IO or Block Redirected IO we will bypass these filters attached to NTFS and consequently might corrupt data. Our choice is to be safe by default so we put volume in File System Redirected Mode if we notice a filter at one of the above altitudes is attached to this volume. You can explicitly inform cluster that this filter is compatible with Direct IO by adding the minifilter name to the cluster common property SharedVolumeCompatibleFilters. If you have a filter that is not on one of the altitudes that are not compatible with Direct IO, but you know that it is not compatible then you can add this minifilter to the cluster property SharedVolumeIncompatibleFilters.
  
  
  
  


• IncompatibleVolumeFilter - An incompatible volume filter attached below NTFS/REFS. Use system event log and cluster log to learn more. The reasons and solution are similar to what we’ve discussed above.


• FileSystemTiering - Volume is in file system redirected mode because the volume is a Tiered Space with heatmap tracking enabled. Tiering heatmap assumes that it can see every IO. Information about IO operations is produced by REFS/NTFS. If we perform Direct IO then statistics will be incorrect and the tiering engine could make incorrect placement decisions by moving hot data to a cold tier or vice versa. You can control if per volume heatmap is enabled/disabled using
  
  fsutil.exe tiering setflags/clearflags with flag /TrNHIf you choose to disable heatmap then you can control which files should go to what tier by pinning them to a tier using PowerShell cmdlet Set-FileStorageTier, and then running Optimize-Volume with –TierOptimize. Please note that for Optimize-Volume to work on CSV volume you need to put volume in File System Redirected mode using Suspend-ClusterResource. You can learn more about Storage Spaces tiering from this blogpost http://blogs.technet.com/b/josebda/archive/2013/08/28/step-by-step-for-storage-spaces-tiering-... .


• BitLockerInitializing – Volume is in redirected state because we are waiting for BitLocker to finish initial volume encryption of this volume.

If Get-ClusterSharedVolumeState tells volume on a node is in Direct IO state does it mean that absolutely all IO will go Direct IO way? The answer is: It is not so simple.

Here is another blog post that covers Get-ClusterSharedVolumeState PowerShell cmdlet http://blogs.msdn.com/b/clustering/archive/2013/12/05/10474312.aspx .

Is Direct IO on this File Possible?

Even if CSV volume is in Direct IO or Volume Level Redirected mode to be able to do Direct IO on a file there are number of preconditions that have to be true:

• CSVFS understands on disk file format
  
  
  
  • Such as the file is not sparse, compressed, encrypted, resilient etc
  
  
  
  


• There are no File System filters that might modify file layout or expect to see all IO
  
  
  
  • File System minifilters that provide compression, encryption, replication etc
  
  
  
  


• There are no File System filters that object to Direct IO on the stream. An example would be the Windows Server Deduplication feature. When you install deduplication and enable it on a CSV volume it will NOT disable Direct IO on all files. Instead it will veto Direct IO only for the files that have been optimized by dedup.


• CsvFs was able to make sure NTFS/REFS will not change location of the file data on the volume – file is pinned. If NTFS relocates file’s block while CSVFS does Direct IO that could result in volume corruption.


• There are no applications that need to make sure IO is observed by NTFS/REFS stack. There is an FSCTL that an application can send to the file system to tell it to keep the file in File System Redirected mode for as long as this application has the file opened. File will be switched back to the redirected mode as soon as application closes the file.


• CSVFS has appropriate oplock level. Oplocks guarantee cross node cache coherency. Oplocks are documented on MSDN http://msdn.microsoft.com/en-us/library/windows/hardware/ff551011(v=vs.85).aspx
  
  
  
  • Read-Write-Handle (RWH) or Read-Write (RW) for write. If CSVFS was able to obtain this level of oplock that means this file is opened only from this node.
  
  
  • Read-Write-Handle (RWH) or Read-Handle (RH) or Read-Write (RW) or Read (R) for reads. If CSVFS was able to obtain RH or R oplock then this file is opened from multiple nodes, but all nodes perform only file read or other operations that do not modify file content.
  
  
  
  


• CSVFS was able to purge cache on NTFS/REFS. Make sure there is no stale cache on NTFS/REFS.

If any of the preconditions are not true then IO is dispatched using File System Redirected mode. If all preconditions are true then CSVFS will translate IO from file offsets to the volume offsets and will send it to the CSV Volume Manager. Keep in mind that CSV Volume Manager might send it using Direct IO to the disk when disk is connected or it might send it over SMB to the disk on the Coordinator node using Block level Redirected IO. CSV Volume Manager always prefer Direct IO, and Block Level Redirected IO is used only when disk is not connected or when disk fails IO.

Summary

To provide high availability and good performance CSVFS has several alternative ways how IO might be dispatched to the disk. This demonstrated some of the tools that can be used to analyze why CSV volume chooses one path for IO versus the other.

Thanks!
Vladimir Petter
Principal Software Development Engineer
Clustering & High-Availability
Microsoft

To learn more, here are others in the Cluster Shared Volume (CSV) blog series:

Cluster Shared Volume (CSV) Inside Out
http://blogs.msdn.com/b/clustering/archive/2013/12/02/10473247.aspx

Cluster Shared Volume Diagnostics
http://blogs.msdn.com/b/clustering/archive/2014/03/13/10507826.aspx

Cluster Shared Volume Performance Counters
http://blogs.msdn.com/b/clustering/archive/2014/06/05/10531462.aspx

Cluster Shared Volume Failure Handling
http://blogs.msdn.com/b/clustering/archive/2014/10/27/10567706.aspx