md-cluster: update the documentation

Update design documentation based on recent development.
original version comes from Neil.

Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
Signed-off-by: NeilBrown <neilb@suse.com>

Documentation/md-cluster.txt

@@ -3,7 +3,7 @@ The cluster MD is a shared-device RAID for a cluster.
 
 1. On-disk format
 
-Separate write-intent-bitmap are used for each cluster node.
+Separate write-intent-bitmaps are used for each cluster node.
 The bitmaps record all writes that may have been started on that node,
 and may not yet have finished. The on-disk layout is:
 
@@ -14,117 +14,161 @@ and may not yet have finished. The on-disk layout is:
 | bm super[2] + bits  | bm bits [2, contd]  | bm super[3] + bits  |
 | bm bits [3, contd]  |                     |                     |
 
-During "normal" functioning we assume the filesystem ensures that only one
-node writes to any given block at a time, so a write
-request will
+During "normal" functioning we assume the filesystem ensures that only
+one node writes to any given block at a time, so a write request will
+
  - set the appropriate bit (if not already set)
  - commit the write to all mirrors
  - schedule the bit to be cleared after a timeout.
 
-Reads are just handled normally.  It is up to the filesystem to
-ensure one node doesn't read from a location where another node (or the same
+Reads are just handled normally. It is up to the filesystem to ensure
+one node doesn't read from a location where another node (or the same
 node) is writing.
 
 
 2. DLM Locks for management
 
-There are two locks for managing the device:
+There are three groups of locks for managing the device:
 
 2.1 Bitmap lock resource (bm_lockres)
 
- The bm_lockres protects individual node bitmaps. They are named in the
- form bitmap001 for node 1, bitmap002 for node and so on. When a node
- joins the cluster, it acquires the lock in PW mode and it stays so
- during the lifetime the node is part of the cluster. The lock resource
- number is based on the slot number returned by the DLM subsystem. Since
- DLM starts node count from one and bitmap slots start from zero, one is
- subtracted from the DLM slot number to arrive at the bitmap slot number.
+ The bm_lockres protects individual node bitmaps. They are named in
+ the form bitmap000 for node 1, bitmap001 for node 2 and so on. When a
+ node joins the cluster, it acquires the lock in PW mode and it stays
+ so during the lifetime the node is part of the cluster. The lock
+ resource number is based on the slot number returned by the DLM
+ subsystem. Since DLM starts node count from one and bitmap slots
+ start from zero, one is subtracted from the DLM slot number to arrive
+ at the bitmap slot number.
+
+ The LVB of the bitmap lock for a particular node records the range
+ of sectors that are being re-synced by that node.  No other
+ node may write to those sectors.  This is used when a new nodes
+ joins the cluster.
+
+2.2 Message passing locks
+
+ Each node has to communicate with other nodes when starting or ending
+ resync, and for metadata superblock updates.  This communication is
+ managed through three locks: "token", "message", and "ack", together
+ with the Lock Value Block (LVB) of one of the "message" lock.
+
+2.3 new-device management
+
+ A single lock: "no-new-dev" is used to co-ordinate the addition of
+ new devices - this must be synchronized across the array.
+ Normally all nodes hold a concurrent-read lock on this device.
 
 3. Communication
 
-Each node has to communicate with other nodes when starting or ending
-resync, and metadata superblock updates.
+ Messages can be broadcast to all nodes, and the sender waits for all
+ other nodes to acknowledge the message before proceeding.  Only one
+ message can be processed at a time.
 
 3.1 Message Types
 
- There are 3 types, of messages which are passed
+ There are six types of messages which are passed:
 
- 3.1.1 METADATA_UPDATED: informs other nodes that the metadata has been
-   updated, and the node must re-read the md superblock. This is performed
-   synchronously.
+ 3.1.1 METADATA_UPDATED: informs other nodes that the metadata has
+   been updated, and the node must re-read the md superblock. This is
+   performed synchronously. It is primarily used to signal device
+   failure.
 
- 3.1.2 RESYNC: informs other nodes that a resync is initiated or ended
-   so that each node may suspend or resume the region.
+ 3.1.2 RESYNCING: informs other nodes that a resync is initiated or
+   ended so that each node may suspend or resume the region.  Each
+   RESYNCING message identifies a range of the devices that the
+   sending node is about to resync. This over-rides any pervious
+   notification from that node: only one ranged can be resynced at a
+   time per-node.
+
+ 3.1.3 NEWDISK: informs other nodes that a device is being added to
+   the array. Message contains an identifier for that device.  See
+   below for further details.
+
+ 3.1.4 REMOVE: A failed or spare device is being removed from the
+   array. The slot-number of the device is included in the message.
+
+ 3.1.5 RE_ADD: A failed device is being re-activated - the assumption
+   is that it has been determined to be working again.
+
+ 3.1.6 BITMAP_NEEDS_SYNC: if a node is stopped locally but the bitmap
+   isn't clean, then another node is informed to take the ownership of
+   resync.
 
 3.2 Communication mechanism
 
  The DLM LVB is used to communicate within nodes of the cluster. There
  are three resources used for the purpose:
 
-  3.2.1 Token: The resource which protects the entire communication
+  3.2.1 token: The resource which protects the entire communication
    system. The node having the token resource is allowed to
    communicate.
 
-  3.2.2 Message: The lock resource which carries the data to
+  3.2.2 message: The lock resource which carries the data to
    communicate.
 
-  3.2.3 Ack: The resource, acquiring which means the message has been
+  3.2.3 ack: The resource, acquiring which means the message has been
    acknowledged by all nodes in the cluster. The BAST of the resource
-   is used to inform the receive node that a node wants to communicate.
+   is used to inform the receiving node that a node wants to
+   communicate.
 
 The algorithm is:
 
- 1. receive status
+ 1. receive status - all nodes have concurrent-reader lock on "ack".
 
-   sender                         receiver                   receiver
-   ACK:CR                          ACK:CR                     ACK:CR
+   sender                         receiver                 receiver
+   "ack":CR                       "ack":CR                 "ack":CR
 
- 2. sender get EX of TOKEN
-    sender get EX of MESSAGE
+ 2. sender get EX on "token"
+    sender get EX on "message"
     sender                        receiver                 receiver
-    TOKEN:EX                       ACK:CR                   ACK:CR
-    MESSAGE:EX
-    ACK:CR
+    "token":EX                    "ack":CR                 "ack":CR
+    "message":EX
+    "ack":CR
 
-    Sender checks that it still needs to send a message. Messages received
-    or other events that happened while waiting for the TOKEN may have made
-    this message inappropriate or redundant.
+    Sender checks that it still needs to send a message. Messages
+    received or other events that happened while waiting for the
+    "token" may have made this message inappropriate or redundant.
 
- 3. sender write LVB.
-    sender down-convert MESSAGE from EX to CW
-    sender try to get EX of ACK
-    [ wait until all receiver has *processed* the MESSAGE ]
+ 3. sender writes LVB.
+    sender down-convert "message" from EX to CW
+    sender try to get EX of "ack"
+    [ wait until all receivers have *processed* the "message" ]
 
-                                     [ triggered by bast of ACK ]
-                                     receiver get CR of MESSAGE
+                                     [ triggered by bast of "ack" ]
+                                     receiver get CR on "message"
                                      receiver read LVB
                                      receiver processes the message
                                      [ wait finish ]
-                                     receiver release ACK
-
-   sender                         receiver                   receiver
-   TOKEN:EX                       MESSAGE:CR                 MESSAGE:CR
-   MESSAGE:CR
-   ACK:EX
-
- 4. triggered by grant of EX on ACK (indicating all receivers have processed
-    message)
-    sender down-convert ACK from EX to CR
-    sender release MESSAGE
-    sender release TOKEN
-                               receiver upconvert to PR of MESSAGE
-                               receiver get CR of ACK
-                               receiver release MESSAGE
+                                     receiver releases "ack"
+                                     receiver tries to get PR on "message"
+
+   sender                         receiver                  receiver
+   "token":EX                     "message":CR              "message":CR
+   "message":CW
+   "ack":EX
+
+ 4. triggered by grant of EX on "ack" (indicating all receivers
+    have processed message)
+    sender down-converts "ack" from EX to CR
+    sender releases "message"
+    sender releases "token"
+                               receiver upconvert to PR on "message"
+                               receiver get CR of "ack"
+                               receiver release "message"
 
    sender                      receiver                   receiver
-   ACK:CR                       ACK:CR                     ACK:CR
+   "ack":CR                    "ack":CR                   "ack":CR
 
 
 4. Handling Failures
 
 4.1 Node Failure
- When a node fails, the DLM informs the cluster with the slot. The node
- starts a cluster recovery thread. The cluster recovery thread:
+
+ When a node fails, the DLM informs the cluster with the slot
+ number. The node starts a cluster recovery thread. The cluster
+ recovery thread:
+
 	- acquires the bitmap<number> lock of the failed node
 	- opens the bitmap
 	- reads the bitmap of the failed node
@@ -132,45 +176,143 @@ The algorithm is:
 	- cleans the bitmap of the failed node
 	- releases bitmap<number> lock of the failed node
 	- initiates resync of the bitmap on the current node
+		md_check_recovery is invoked within recover_bitmaps,
+		then md_check_recovery -> metadata_update_start/finish,
+		it will lock the communication by lock_comm.
+		Which means when one node is resyncing it blocks all
+		other nodes from writing anywhere on the array.
 
- The resync process, is the regular md resync. However, in a clustered
+ The resync process is the regular md resync. However, in a clustered
  environment when a resync is performed, it needs to tell other nodes
  of the areas which are suspended. Before a resync starts, the node
- send out RESYNC_START with the (lo,hi) range of the area which needs
- to be suspended. Each node maintains a suspend_list, which contains
- the list  of ranges which are currently suspended. On receiving
- RESYNC_START, the node adds the range to the suspend_list. Similarly,
- when the node performing resync finishes, it send RESYNC_FINISHED
- to other nodes and other nodes remove the corresponding entry from
- the suspend_list.
+ send out RESYNCING with the (lo,hi) range of the area which needs to
+ be suspended. Each node maintains a suspend_list, which contains the
+ list of ranges which are currently suspended. On receiving RESYNCING,
+ the node adds the range to the suspend_list. Similarly, when the node
+ performing resync finishes, it sends RESYNCING with an empty range to
+ other nodes and other nodes remove the corresponding entry from the
+ suspend_list.
 
- A helper function, should_suspend() can be used to check if a particular
- I/O range should be suspended or not.
+ A helper function, ->area_resyncing() can be used to check if a
+ particular I/O range should be suspended or not.
 
 4.2 Device Failure
+
  Device failures are handled and communicated with the metadata update
- routine.
+ routine.  When a node detects a device failure it does not allow
+ any further writes to that device until the failure has been
+ acknowledged by all other nodes.
 
 5. Adding a new Device
-For adding a new device, it is necessary that all nodes "see" the new device
-to be added. For this, the following algorithm is used:
+
+ For adding a new device, it is necessary that all nodes "see" the new
+ device to be added. For this, the following algorithm is used:
 
     1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues
-       ioctl(ADD_NEW_DISC with disc.state set to MD_DISK_CLUSTER_ADD)
-    2. Node 1 sends NEWDISK with uuid and slot number
+       ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CLUSTER_ADD)
+    2. Node 1 sends a NEWDISK message with uuid and slot number
     3. Other nodes issue kobject_uevent_env with uuid and slot number
        (Steps 4,5 could be a udev rule)
     4. In userspace, the node searches for the disk, perhaps
        using blkid -t SUB_UUID=""
-    5. Other nodes issue either of the following depending on whether the disk
-       was found:
+    5. Other nodes issue either of the following depending on whether
+       the disk was found:
        ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CANDIDATE and
-                disc.number set to slot number)
+             disc.number set to slot number)
        ioctl(CLUSTERED_DISK_NACK)
-    6. Other nodes drop lock on no-new-devs (CR) if device is found
-    7. Node 1 attempts EX lock on no-new-devs
-    8. If node 1 gets the lock, it sends METADATA_UPDATED after unmarking the disk
-       as SpareLocal
-    9. If not (get no-new-dev lock), it fails the operation and sends METADATA_UPDATED
-    10. Other nodes get the information whether a disk is added or not
-	by the following METADATA_UPDATED.
+    6. Other nodes drop lock on "no-new-devs" (CR) if device is found
+    7. Node 1 attempts EX lock on "no-new-dev"
+    8. If node 1 gets the lock, it sends METADATA_UPDATED after
+       unmarking the disk as SpareLocal
+    9. If not (get "no-new-dev" lock), it fails the operation and sends
+       METADATA_UPDATED.
+   10. Other nodes get the information whether a disk is added or not
+       by the following METADATA_UPDATED.
+
+6. Module interface.
+
+ There are 17 call-backs which the md core can make to the cluster
+ module.  Understanding these can give a good overview of the whole
+ process.
+
+6.1 join(nodes) and leave()
+
+ These are called when an array is started with a clustered bitmap,
+ and when the array is stopped.  join() ensures the cluster is
+ available and initializes the various resources.
+ Only the first 'nodes' nodes in the cluster can use the array.
+
+6.2 slot_number()
+
+ Reports the slot number advised by the cluster infrastructure.
+ Range is from 0 to nodes-1.
+
+6.3 resync_info_update()
+
+ This updates the resync range that is stored in the bitmap lock.
+ The starting point is updated as the resync progresses.  The
+ end point is always the end of the array.
+ It does *not* send a RESYNCING message.
+
+6.4 resync_start(), resync_finish()
+
+ These are called when resync/recovery/reshape starts or stops.
+ They update the resyncing range in the bitmap lock and also
+ send a RESYNCING message.  resync_start reports the whole
+ array as resyncing, resync_finish reports none of it.
+
+ resync_finish() also sends a BITMAP_NEEDS_SYNC message which
+ allows some other node to take over.
+
+6.5 metadata_update_start(), metadata_update_finish(),
+    metadata_update_cancel().
+
+ metadata_update_start is used to get exclusive access to
+ the metadata.  If a change is still needed once that access is
+ gained, metadata_update_finish() will send a METADATA_UPDATE
+ message to all other nodes, otherwise metadata_update_cancel()
+ can be used to release the lock.
+
+6.6 area_resyncing()
+
+ This combines two elements of functionality.
+
+ Firstly, it will check if any node is currently resyncing
+ anything in a given range of sectors.  If any resync is found,
+ then the caller will avoid writing or read-balancing in that
+ range.
+
+ Secondly, while node recovery is happening it reports that
+ all areas are resyncing for READ requests.  This avoids races
+ between the cluster-filesystem and the cluster-RAID handling
+ a node failure.
+
+6.7 add_new_disk_start(), add_new_disk_finish(), new_disk_ack()
+
+ These are used to manage the new-disk protocol described above.
+ When a new device is added, add_new_disk_start() is called before
+ it is bound to the array and, if that succeeds, add_new_disk_finish()
+ is called the device is fully added.
+
+ When a device is added in acknowledgement to a previous
+ request, or when the device is declared "unavailable",
+ new_disk_ack() is called.
+
+6.8 remove_disk()
+
+ This is called when a spare or failed device is removed from
+ the array.  It causes a REMOVE message to be send to other nodes.
+
+6.9 gather_bitmaps()
+
+ This sends a RE_ADD message to all other nodes and then
+ gathers bitmap information from all bitmaps.  This combined
+ bitmap is then used to recovery the re-added device.
+
+6.10 lock_all_bitmaps() and unlock_all_bitmaps()
+
+ These are called when change bitmap to none. If a node plans
+ to clear the cluster raid's bitmap, it need to make sure no other
+ nodes are using the raid which is achieved by lock all bitmap
+ locks within the cluster, and also those locks are unlocked
+ accordingly.