Bright Kb

Navigation:

cmsh -c initiates the Cluster Management Shell (cmsh) in batch mode, allowing you to execute a series of commands

To determine which software image is assigned to servers or nodes, you can use the cmsh command followed by the softwareimage list command. This command provides an overview of all the software images configured in Bright. The output will display a list of software images, their paths, kernel versions, and the number of nodes associated with each image. For example:

root@lambda-bright01:~# cmsh
root@lambda-bright01:~# softwareimage list

Name (key)           Path                                  Kernel version                 Nodes
-------------------- ---------------------------------------- ---------------------------- --------
default-image        /cm/images/default-image               3.10.0-1062.12.1.el7.x86_64   3

In this example, the default-image is located at /cm/images/default-image and is associated with 3 nodes.

The chroot command can be used to install non-packaged software into a software image. For example:

cd /cm/images/default-image/usr/src
tar -xvzf /tmp/app-4.5.6.tar.gz
chroot /cm/images/default-image
cd /usr/src/app-4.5.6
./configure --prefix=/usr
make install

After installing the software, you can implement the changes on all running nodes using the updateprovisioners and imageupdate commands as well as rebooting the servers.

The updateprovisioners command utilizes the rsync utility to update the provisioners in the cluster, ensuring they have the latest software image information. In a large cluster, you can configure CPU nodes to provision a group of servers (such as one CPU node per rack and have that node do all of the provisioning for all servers in the rack).

The imageupdate command utilizes the rsync utility to transfer and sync the headnode gold image to other servers without needing to restart them.

When working under chroot, certain commands may not reflect the actual state of the software image. For instance, running uname -a under chroot returns the kernel currently running outside the chroot, not the kernel of the software image. To find the kernel version of the image, you can use:

cmsh -c "softwareimage; use default-image; get kernelversion"

With Bright, you can manage software on an individual node and then sync its state back to the headnode’s image repository. This is useful when you need to test and validate software configurations on a single node before rolling them out to multiple nodes in the cluster.

After configuring the software on the node and ensuring it operates as expected, the next step is to synchronize this node’s state back to the headnode’s software image repository. This is accomplished with the grabimage command in cmsh, or the “Synchronize image” option in Bright View.

The following steps are considered best practice for synchronizing nodes to image:

1. Clone the current image so that any changes can be backed out just by moving to the old image

[root@headnode ~]# cmsh -c "softwareimage; clone default new-image; commit -w"

2. Do a test run of the grabimage without the -w option and review the change log

The grabimage command without the -w option simply does a dry-run you can see in the provisioning logs what would be grabbed, without having the changes actually carried out. Running grabimage -w instructs CMDaemon to really write the image.

2.a. review for any unattended changes that another admin may have committed

2.b. review for any changes that were not captured. The software image has settings that ignore portions of the filesystem

3. You can create a new image during the grabimage command with the -i option flag, where -i specifies the image it will write to. If the specified image does not exist, a new image with the name will be created.

Example:

[root@headnode ~]# cmsh -c "device use node001; grabimage -w -i new-image; commit"

This cmsh command performs the following actions:

• device use node001 selects the device “node001”
• grabimage -w -i new-image captures the current state of node001 and saves it as a software image on the headnode. The -w flag tells grabimage to actually write the image. Without this flag, grabimage would only perform a dry-run. The -i flag specifies that a new image should be created, and new-image is the name of the image to be created.
• commit commits the changes made.

This process captures the node’s state, excluding network mounts and parallel filesystems like Lustre and GPFS, but includes any regular disk mounted on the node itself.

Once the software changes are integrated into the software image, you can deploy this updated image to all running nodes using the updateprovisioners and reboot commands. This command sequence first ensures that all provisioners in the cluster are updated with the latest software image information then deploys this updated image to all running nodes, ensuring they have the latest software and configurations.

[root@headnode ~]# cmsh -c "updateprovisioners"
[root@headnode ~]# cmsh -c "category; use default; set softwareimage new-image; commit -w"
[root@headnode ~]# cmsh -c "device; reboot node-001..node-255"

This cmsh command sequence performs the following actions:

• cmsh -c "updateprovisioners"

  • updateprovisioners updates the provisioners in the cluster to ensure they have the latest software image information.

• cmsh -c "category; use default; set softwareimage new-image; commit -w"

  • category switches to the category context in cmsh. Categories in Bright Cluster Manager are used to group nodes that share common configurations.
  • use default selects the category named default. All nodes associated with this category will be affected by subsequent commands.
  • set softwareimage new-image sets the software image for the selected category (default). This means that all nodes associated with this category will use the new-image software image when they are next provisioned.

• cmsh -c "device; reboot node-001..node-255"

  • device switches to the device context in cmsh, which deals with node devices in the cluster.
  • reboot node-001..node-255 reboots a range of nodes, from node-001 to node-255. When these nodes reboot, they will be provisioned with the software image assigned to their associated category (in this case, new-image from the default category).

Until the new image is saved, any modifications made to the node will be lost and will revert to the old image upon reboot.

This stage encompasses the initial steps that occur immediately after powering up a node and before the Linux kernel is loaded.

Sub-stages:

• PXE Booting: By default, nodes boot from the network when using the cluster manager. On x86 architectures it is known as PXE boot. The head node runs a tftpd server, managed by systemd, that supplies the boot loader from within the default software image.
• iPXE Booting From A Disk Drive: On disked nodes, iPXE software is placed on the drive during node installation. If PXE booting fails, the BIOS might attempt a boot from the hard drive, triggering another PXE network boot attempt.

Once the kernel is loaded, the node-installer takes over to set up the node. The steps the node-installer goes through for each node are:

1. Request Certificate: node-installer communicates with CMDaemon on the head node for a certificate. If absent, requests one. Stored in /cm/node-installer/certificates/ by MAC address.

2. Node Configuration: After establishing communication with the head node CMDaemon, the node-installer determines the node’s configuration based on detected network hardware. If issues arise, the admin is prompted to select a configuration.

3. Network Interface Setup: Using the known node configuration, the node-installer sets up IP addresses on the required provisioning interfaces.

4. Determine Install & Execution Mode: Decides synchronization type for the node’s local drive, whether to enter maintenance mode, and when to apply these settings. Also checks if full installation confirmation is needed.

5. Run Initialize Scripts: Executes custom commands before checking partitions, typically for unsupported hardware or specific RAID configurations.

6. Check Partitions & Mount: Executes the determined install-mode. Checks and, if necessary, recreates drive partitions and filesystems. Mounts all filesystems for synchronization with the software image.

7. Synchronize Local Drive: After mounting, synchronizes local filesystems with the associated software image. Skipped for certain install-mode values. Delegated to CMDaemon provisioning system.

8. Write Network Config: Sets up network configuration files for each interface, placing them on the local drive.

9. Create /etc/fstab: Generates and places a valid /etc/fstab file on the local drive based on the drive layout configuration.

10. Install GRUB Bootloader: Optionally installs a boot record on the local drive if installbootrecord property is set to on, allowing boot from the local drive.

When the node-installer finishes steps 9-10, it brings down all network devices and hands over control to the local /sbin/init process. Eventually a local init script uses the network configuration files to bring the interfaces back up.

During the boot process, several state change messages are sent to the head node CMDaemon or detected by polling from the head node CMDaemon.

States:

• UP: Indicated by an up-arrow.
• DOWN: Indicated by a blue down-arrow.
• INSTALLING: Entered as soon as the node-installer determines on which node it’s running.
• INSTALLER_CALLINGINIT: Entered when the node-installer hands over control to the local init process.

The node-installer log is used in diagnosing any issues that might arise during this process and is the primary source of information when troubleshooting node boot failures. The first place to look for hints on node boot failure is usually the node-installer log file, as it provides a comprehensive record of the node’s setup process.

The node-installer log is located at /var/log/node-installer. This location is consistent across nodes.

The node-installer log contains the following information:

• Detailed insights into the various stages the node-installer goes through, from the initial communication with the head node CMDaemon to the final stages of system setup.
• Specific issues related to disk setup, hardware problems, and PXE booting.

The log will indicate if there are any discrepancies or errors in the partition sizes, filesystems, or any other disk-related configurations. The log can also flag hardware-related problems that might affect the boot process or the overall functioning of the node.

The BOOTIF mechanism in Bright Cluster Manager simplifies the booting process by abstracting the specific NICs. By relying on PXE booting and passing the network device name to the OS, Bright ensures a consistent and efficient booting process across all nodes.

• Purpose of BOOTIF: Instead of specifying the exact NIC for every machine in Bright, a generic device named ‘BOOTIF’ is used. This allows for a more streamlined and consistent booting process across various nodes, regardless of the specific NIC they might have.
• Reliance on PXE Booting: Bright’s booting process is unique in that it completely depends on PXE booting. The system first loads an initial ramfs (without using grub) and then pivots the root to the intended OS during every boot. This approach ensures that the OS is loaded freshly during each boot, providing a consistent environment.
• Passing Network Device Name: Since Bright relies on PXE booting, it can also pass the network device name used for PXE booting over to the OS. This is achieved using the /proc/cmdline method.

root@lambda-bright02:~# cat /proc/cmdline
BOOT_IMAGE=images/gpu-image/vmlinuz initrd=images/gpu-image/initrd  \
nouveau.modeset=0 console=tty0 ip=10.141.255.253:10.141.255.254:10.141.255.254:255.255.0.0 \
BOOTIF=01-7c-c2-55-25-1d-af    <==========

Inspecting the /proc/cmdline will display the BOOTIF parameter, which indicates the network device used.

The value after BOOTIF= represents the MAC address of the NIC used for PXE booting.

This section details the process for updating the OS within a Bright image. When updating the OS within a Bright image, there are specific steps and considerations to keep in mind:

To work within any image directory, as if it is a live system, you have to bind mount the host system directories /sys, /proc , and /dev (and sometimes other directories available within the chroot jail) into that image directory.

The /proc namespace used should be different than the head node’s due to namespace issues that may affect some pre- and post-installation scripts bundled with the package.

Normally pre-configuring all this with bind mounting can be tedious, but Bright offers a utility command cm-chroot-sw-img to simply this process. This command not only sets up the necessary bind mounts but also provides additional functionalities:

root@lambda-bright01:~# cm-chroot-sw-img /cm/images/default-image/
...

root@default-image:/# pwd
/

From here you can interact with the image as if it is live:

root@default-image:/# apt update && apt upgrade -y
root@default-image:/# exit

By default, Bright uses rsync to synchronize the default image to a node each time the node goes through the PXE boot process.

To apply the updated image, you need to reboot the node:

root@lambda-compute02:~# reboot

The process outlined above should get the image to the latest updates within a release. However, if you’re looking to upgrade to a new release altogether, the do-release-upgrade -d process will fail within the chroot. While there may be workarounds, it’s recommended to perform release upgrades from a live system for better reliability.

After updated the OS within the Bright image, the next step is to deploy the updated image to the devices. This is done using the cmsh scripting tool.

cmsh -c initiates the Cluster Management Shell (cmsh) in batch mode, allowing you to execute a series of commands

cmsh -c "device; foreach -c diskboot (set installmode AUTO; set nextinstallmode AUTO; commit)"
cmsh -c "device; foreach -c diskboot (power on)"
watchstatus

This cmsh command sequence performs the following actions:

• "device; foreach -c diskboot" navigates to the device section and iterates over each device categorized as “diskboot”.
• (set installmode AUTO; set nextinstallmode AUTO;) sets the current install mode and the next install mode to “AUTO” for each “diskboot” device.
• commit commits the changes made.
• (power on) powers on each “diskboot” device.
• watchstatus monitors the status of the devices as they boot and install the updated image.

cmsh
cmsh -c "device use gpu001; get softwareimage;set softwareimage gpu;commit"
cmsh -c "device use gpu001; grabimage -w; commit"

wait 3 mins

This cmsh command sequence performs the following actions:

• "device use gpu001;" selects the device named “gpu001”.
• get softwareimage; retrieves the current software image associated with “gpu001”.
• set softwareimage gpu; sets the software image for “gpu001” to “gpu”.
• commit commits the changes made.
• "device use gpu001;" selects the device named “gpu001”.
• grabimage -w; grabs the image for “gpu001” and waits (-w) until the operation is complete.
• commit commits the changes made.

After setting the software image, wait for approximately 3 minutes to ensure that the image is properly loaded onto the device before moving on to the next step. CMSH will display console messages alerting you to when the creation is complete

cmsh -c "device; foreach -c gpu (set softwareimage gpu-image);commit"
cmsh -c "device; foreach -c gpu (set nextinstallmode full);commit"
cmsh -c "device; foreach -c gpu (reboot)"

This cmsh command sequence performs the following actions:

• "device; foreach -c gpu" navigates to the device section and iterates over each device categorized as “gpu”.
• (set softwareimage gpu-image); sets the software image for each “gpu” device to “gpu-image”.
• commit commits the changes made.
• (set nextinstallmode full); sets the next install mode for each “gpu” device to “full”.
• commit commits the changes made.
• (reboot) reboots each “gpu” device.

The node-installer log needs to be reviewed to ensure the node has been set up correctly.

a. Navigate to the directory where the node-installer log is stored and open the log.

cd /var/log/
cat node-installer

b. Ensure the initial handshake and certificate exchange took place by looking for entries indicating the node’s communication with the CMDaemon on the head node.

"Requesting node certificate from CMDaemon..."

c. Confirm the node configuration was decided or selected successfully.

"Node configuration established based on detected network hardware."

d. Check for the successful setup of IP addresses on the provisioning interfaces.

"Setting up IP addresses on provisioning interfaces..."

e. Check for entries that indicate the determination of install-mode and execution mode values.

"Determined install-mode as AUTO."

f. Confirm that any required initialization scripts ran successfully.

"Running initialize script for node's category..."

g. Check for messages that confirm the successful checking and mounting of partitions and filesystems.

"Checking partitions of the local drive... All partitions mounted successfully."

h. Validate that the local filesystems were synchronized with the software image associated with the node.

"Synchronizing local drive with software image... Synchronization successful."

i. Look for entries that indicate the successful writing of network configuration files to the local drive.

"Writing network configuration files to /etc/sysconfig/network-scripts/..."

j. Ensure that the /etc/fstab file was generated and placed on the local drive.

"Generating and placing /etc/fstab file on local drive."

k. If your setup includes installing a bootloader on the local drive, confirm its successful installation.

"Installing GRUB bootloader on local drive..."

This section covers both the process for assigning a new software image to an entire category of nodes as well as the process for safely removing a software image from Bright’s list.

Effective management of the lifecycle of software images in a cluster is crucial to ensuring optimal performance.

To change the software image for an entire category, you can use the following cmsh command:

 cmsh -c "category;use default; set softwareimage gpu02-image;commit"

This cmsh command performs the following actions:

• "category;use default;" navigates to the category section and selects the default category.
• set softwareimage gpu02-image; sets the software image for the selected category to “gpu02-image”.
• commit finalizes and commits the changes made.

To delete a software image, you can use the following sequence of commands:

root@lambda-bright01:~# cmsh -c "softwareimage; remove gpu02-image; commit"
root@lambda-bright01:~# cmsh -c "softwareimage list"
# no gpu02-image listed
# you have to manually delete the images folder contents
root@lambda-bright01:~# rm -rf /cm/images/gpu02-image

This cmsh command sequence performs the following actions:

• "softwareimage; remove gpu02-image;" navigates to the software image section and removes the “gpu02-image”.
• commit commits the changes made.
• softwareimage list lists the available software images.

If the “gpu02-image” is no longer listed, the image has been successfully removed.

While the image reference is removed from the software image list, the actual image files and folder contents still exist on the system. To completely remove the image files, you need to manually delete the image folder.

The command rm -rf /cm/images/gpu02-image will recursively (-r) and forcefully (-f) remove the directory /cm/images/gpu02-image and all its contents.

Always exercise caution when using the rm -rf command, as it will delete files and directories without any confirmation. Ensure you have backups or are certain about the path you are deleting.

This section provides details on how to configure and verify SOL (Serial-Over-LAN) settings for nodes managed by Bright. The SOL feature is crucial for remote management and diagnostics, especially when direct access to the server console is not feasible.

Only enable SOL while troubleshooting a problem.

Keeping SOL running eats up disk space, as console messages are continually captured into log files.

SOL allows for out-of-band management of servers, enabling administrators to monitor and control a server even when the operating system is not running.

With Bright, the SOL settings are managed at the image level. The following needs to be performed to enable SOL within a Bright image:

1. Access cluster management shell

• cmsh

2. Select the software image.

• softwareimage use <image> navigates to the software image section and selects the defined image.

3. Enable SOL

• softwareimage[default-image]]% set enablesol yes enables the SOL feature for the selected software image.

4. Commit changes

• commit

5. Verify SOL settings

• show | grep -i sol displays the SOL-related settings for the selected software image. The output provides details such as whether SOL is enabled, the port used for SOL, the speed, and flow control settings.

The following example assumes default-image is being used:

root@lambda-bright01:~# cmsh
[lambda-bright01]% softwareimage use default-image
[lambda-bright01->softwareimage[default-image]]% set enablesol yes
[lambda-bright01->softwareimage*[default-image*]]% commit
[lambda-bright01->softwareimage[default-image]]% show | grep -i sol
Kernel output console            tty0
Enable SOL                       yes
SOL Port                         ttyS1
SOL Speed                        115200
SOL Flow Control                 yes

Nodes in a Bright cluster do not typically use the local /boot/grub/grub.cfg for boot configurations. Instead, the PXE boot configurations are found in /tftpboot/pxelinux.cfg/category.default.

This section defines the boot label “linux” and specifies the kernel image, initial ramdisk, and kernel boot parameters. The console=ttyS1,115200n8r parameter indicates that the kernel should output to the serial console on port ttyS1 at a speed of 115200 baud.

The provided configuration snippet indicates the serial settings and the kernel boot parameters for the nodes:

SERIAL 1 115200

LABEL linux
  KERNEL images/default-image/vmlinuz
  IPAPPEND 3
  APPEND initrd=images/default-image/initrd console=ttyS1,115200n8r rd.driver.blacklist=nouveau iommu=pt console=tty0
  MENU LABEL ^AUTO       - Normal node boot
  MENU DEFAULT

Bright Software allows administrators to oversee, control, and optimize their clusters without being physically present, giving them the ability to quickly diagnose and rectify issues remotely.

The significance of remote management and connectivity cannot be overstated. The ability to manage and connect to systems remotely is a necessity for maintaining uptime, troubleshooting, or deploying new configurations.

IPMI (Intelligent Platform Management Interface) is a standardized interface that provides out-of-band management capabilities, including SOL. This section covers how to connect to IPMI SOL using both the Bright method and the standard ipmitool.

To connect to IPMI SOL via the Bright method, the following steps need to be performed:

1. Access the cluster management shell

• cmsh

2. Select the device

• device use lambda-compute01

3. Initiate remote console

• rconsole

root@lambda-bright01:~# cmsh
[lambda-bright01]% device use lambda-compute01
[lambda-bright01->device[lambda-compute01]]% rconsole

This example demonstrates initiating a remote console session to the selected device.

The escape sequence to exit a remote console session is <ENTER> "&" ".".

To connect to IMPI SOL via the ipmitool method, the following command needs to be executed:

root@lambda-bright01:~# ipmitool -I lanplus -H <hostname-ipmi> -U <user> -P "<pass>" sol activate

This command uses ipmitool to initiate an SOL session to the specified IPMI host. The -I lanplus option specifies the interface to use, while -H, -U, and -P options specify the IPMI host, user, and password respectively. The escape sequence to exit this session is <ENTER> "~" ".".

SSH Escape Sequences

For standard SSH escape sequences, the ~ is relative to the number of connection hops you are currently within. In the connection → → the escape sequence to drop back to the would be: ~~.