general protection fault, probably for non-canonical address 0xe8156fafab95fc98: 0000 [#1] SMP PTI
CPU: 0 UID: 0 PID: 1355947 Comm: lraft_0000 Kdump: loaded Tainted: G W OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary)
Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_voting+0x9/0x20 [lraft]
Code: 43 18 e9 77 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 d0 e8 83 e0 01 c3 cc cc cc cc 66 0f 1f 84 00 00 00 00
RSP: 0000:ffffd561c1ac3e78 EFLAGS: 00010202
RAX: ffff8ec684623880 RBX: 0000000000000001 RCX: 0000000000000000
RDX: ffff8ec679dd4870 RSI: 0000000000000000 RDI: e8156fafab95fc80
RBP: ffff8ec66f933e00 R08: 0000000000000040 R09: 0000000000000004
R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000001
R13: 0000116fa15c1769 R14: ffffffffc11fc050 R15: 0000000000000007
FS: 0000000000000000(0000) GS:ffff8ec6bbc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055ca48513010 CR3: 00000001480d2001 CR4: 0000000000372ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
? show_trace_log_lvl+0x1b0/0x2f0
? show_trace_log_lvl+0x1b0/0x2f0
? raft_get_num_voting_nodes+0x45/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? __pfx_lraft_timerd+0x10/0x10 [lraft]
? raft_node_is_voting+0x9/0x20 [lraft]
raft_get_num_voting_nodes+0x45/0x70 [lraft]
lraft_timerd+0x155/0x350 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xfd/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x34/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: dm_flakey lzstd(OE) llz4hc(OE) llz4(OE) obdecho(OE) ec(OE) ptlrpc_gss(OE) osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill intel_rapl_msr intel_rapl_common intel_uncore_frequency_common kvm_intel kvm mlx5_ib irqbypass rapl ib_uverbs iTCO_wdt iTCO_vendor_support macsec virtio_balloon mlx5_fwctl i2c_i801 fwctl lpc_ich i2c_smbus pcspkr joydev fuse loop vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 mlx5_core ahci libahci libata virtio_net mlxfw tls crct10dif_pclmul crc32_pclmul net_failover crc32c_intel ghash_clmulni_intel psample failover pci_hyperv_intf virtio_blk serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod ib_ipoib ib_cm ib_core nfnetlink
Unloaded tainted modules: llog_test(OE):1 lnet_selftest(OE):1 [last unloaded: dm_flakey]
Crash core archive creation started: kernel-crash.onyx-142vm9.vmcore.tar.gz
Crash core archive created: kernel-crash.onyx-142vm9.vmcore.tar.gz

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 2dbc9173-ea8d-4920-aa78-c587c121b26f.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup table /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup remove /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe -r dm-flakey
Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 20 previous similar messages
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds1_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 2dbc9173-ea8d-4920-aa78-c587c121b26f r/w with ordered data mode. Quota mode: journalled.
LustreError: MGC10.240.27.62@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect