Fortinet black logo

New Features

Implement sysrq for kernel crash 7.2.4

Copy Link
Copy Doc ID 77966226-6996-11ec-bdf2-fa163e15d75b:923888
Download PDF

Implement sysrq for kernel crash 7.2.4

In some scenarios where it is necessary to simulate a system crash, these new commands allow a super administrator to safely trigger a kernel crash using the sysrq key. A kernel crash dump is outputted to the console, and FortiOS reboots and recovers without function loss. Only FortiGate-VM supports this feature.

Following are the new commands for this feature:

Command

Description

diagnose debug kernel sysrq status

Verify if the feature is enabled or disabled.

diagnose debug kernel sysrq enable | disable

Enable or disable the feature.

diagnose debug kernel sysrq command crash

Trigger the safe kernel crash.

Scenarios where you may need to simulate a system crash without a reboot include high availability failover tests, virtual network function (VNF) healing tests, and third party VNF certification tests.

The following shows a sample kernel crash dump:

fgt01 # diagnose debug kernel sysrq status 
Magic SysRq is disable
fgt01 # 
fgt01 # diagnose debug kernel sysrq command 
crash    Perform a system crash.

fgt01 # diagnose debug kernel sysrq command crash 
Magic SysRq is disable!

fgt01 # 
fgt01 # diagnose debug kernel sysrq enable 

fgt01 # 
fgt01 # diagnose debug kernel sysrq status
Magic SysRq is enabled (val=0x8)
fgt01 # 
fgt01 # diagnose debug kernel sysrq command crash
This operation will generate a kernel crash and cause the firewall to reboot.
Do you want to continue? (y/n)y

BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
PGD 28989a067 P4D 28989a067 PUD 28989b067 PMD 0 
Oops: 0002 [#1] SMP
CPU: 2 PID: 2111 Comm: newcli Tainted: P                  4.19.13 #1
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
RIP: 0010:sysrq_handle_crash+0xd/0x16
Code: 89 f7 e8 d6 15 dd ff eb d2 45 89 ec eb f1 41 bc fb ff ff ff eb c5 e8 ca 06 c6 ff 90 90 c7 05 7e 04 19 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 c3 55 48 89 e5 bf 02 00 00 00 e8 44 89 c8
RSP: 0018:ffffc9000480fd58 EFLAGS: 00010246
RAX: ffffffff8068fe00 RBX: 0000000000000001 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000092 RDI: 0000000000000063
RBP: ffffc9000480fd80 R08: 0000000000000233 R09: 0000000000000004
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000063
R13: 0000000000000004 R14: ffffffff81673560 R15: 0000000000000000
FS:  00007f0177113fc0(0000) GS:ffff8882b7b00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000289899001 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 ? __handle_sysrq.cold+0x48/0xf7
 write_sysrq_trigger+0x26/0x35
 proc_reg_write+0x36/0x74
 __vfs_write+0x36/0x16c
 ? __se_sys_newfstat+0x89/0xa5
 ? _cond_resched+0x15/0x3c
 vfs_write+0xa0/0x198
 ksys_write+0x4f/0xad
 __x64_sys_write+0x15/0x17
 do_syscall_64+0x66/0x281
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f017a341fa3
Code: 8b 15 f1 ce 0c 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40 00 48 83 ec 28 48 89 54 24 18
RSP: 002b:00007ffe608b46d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f017a341fa3
RDX: 0000000000000001 RSI: 000000001138a000 RDI: 0000000000000007
RBP: 000000001138a000 R08: 0000000000000000 R09: 0000000000000001
R10: 00000000000001b6 R11: 0000000000000246 R12: 0000000000000001
R13: 000000001137f2a0 R14: 0000000000000001 R15: 00007f017a40b880
Modules linked in: filter4(P)
CR2: 0000000000000000
---[ end trace 1a3b9e86e8978153 ]---
RIP: 0010:sysrq_handle_crash+0xd/0x16
Code: 89 f7 e8 d6 15 dd ff eb d2 45 89 ec eb f1 41 bc fb ff ff ff eb c5 e8 ca 06 c6 ff 90 90 c7 05 7e 04 19 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 c3 55 48 89 e5 bf 02 00 00 00 e8 44 89 c8
RSP: 0018:ffffc9000480fd58 EFLAGS: 00010246
RAX: ffffffff8068fe00 RBX: 0000000000000001 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000092 RDI: 0000000000000063
RBP: ffffc9000480fd80 R08: 0000000000000233 R09: 0000000000000004
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000063
R13: 0000000000000004 R14: ffffffff81673560 R15: 0000000000000000
FS:  00007f0177113fc0(0000) GS:ffff8882b7b00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000289899001 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Kernel panic - not syncing: Fatal exception
Kernel Offset: disabled
Rebooting in 5 seconds..

System is starting...
Starting system maintenance...
Scanning /dev/sda1... (100%) 
Scanning /dev/sda2... (100%) 
Serial number is FGT123456

Implement sysrq for kernel crash 7.2.4

In some scenarios where it is necessary to simulate a system crash, these new commands allow a super administrator to safely trigger a kernel crash using the sysrq key. A kernel crash dump is outputted to the console, and FortiOS reboots and recovers without function loss. Only FortiGate-VM supports this feature.

Following are the new commands for this feature:

Command

Description

diagnose debug kernel sysrq status

Verify if the feature is enabled or disabled.

diagnose debug kernel sysrq enable | disable

Enable or disable the feature.

diagnose debug kernel sysrq command crash

Trigger the safe kernel crash.

Scenarios where you may need to simulate a system crash without a reboot include high availability failover tests, virtual network function (VNF) healing tests, and third party VNF certification tests.

The following shows a sample kernel crash dump:

fgt01 # diagnose debug kernel sysrq status 
Magic SysRq is disable
fgt01 # 
fgt01 # diagnose debug kernel sysrq command 
crash    Perform a system crash.

fgt01 # diagnose debug kernel sysrq command crash 
Magic SysRq is disable!

fgt01 # 
fgt01 # diagnose debug kernel sysrq enable 

fgt01 # 
fgt01 # diagnose debug kernel sysrq status
Magic SysRq is enabled (val=0x8)
fgt01 # 
fgt01 # diagnose debug kernel sysrq command crash
This operation will generate a kernel crash and cause the firewall to reboot.
Do you want to continue? (y/n)y

BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
PGD 28989a067 P4D 28989a067 PUD 28989b067 PMD 0 
Oops: 0002 [#1] SMP
CPU: 2 PID: 2111 Comm: newcli Tainted: P                  4.19.13 #1
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
RIP: 0010:sysrq_handle_crash+0xd/0x16
Code: 89 f7 e8 d6 15 dd ff eb d2 45 89 ec eb f1 41 bc fb ff ff ff eb c5 e8 ca 06 c6 ff 90 90 c7 05 7e 04 19 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 c3 55 48 89 e5 bf 02 00 00 00 e8 44 89 c8
RSP: 0018:ffffc9000480fd58 EFLAGS: 00010246
RAX: ffffffff8068fe00 RBX: 0000000000000001 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000092 RDI: 0000000000000063
RBP: ffffc9000480fd80 R08: 0000000000000233 R09: 0000000000000004
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000063
R13: 0000000000000004 R14: ffffffff81673560 R15: 0000000000000000
FS:  00007f0177113fc0(0000) GS:ffff8882b7b00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000289899001 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 ? __handle_sysrq.cold+0x48/0xf7
 write_sysrq_trigger+0x26/0x35
 proc_reg_write+0x36/0x74
 __vfs_write+0x36/0x16c
 ? __se_sys_newfstat+0x89/0xa5
 ? _cond_resched+0x15/0x3c
 vfs_write+0xa0/0x198
 ksys_write+0x4f/0xad
 __x64_sys_write+0x15/0x17
 do_syscall_64+0x66/0x281
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f017a341fa3
Code: 8b 15 f1 ce 0c 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40 00 48 83 ec 28 48 89 54 24 18
RSP: 002b:00007ffe608b46d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f017a341fa3
RDX: 0000000000000001 RSI: 000000001138a000 RDI: 0000000000000007
RBP: 000000001138a000 R08: 0000000000000000 R09: 0000000000000001
R10: 00000000000001b6 R11: 0000000000000246 R12: 0000000000000001
R13: 000000001137f2a0 R14: 0000000000000001 R15: 00007f017a40b880
Modules linked in: filter4(P)
CR2: 0000000000000000
---[ end trace 1a3b9e86e8978153 ]---
RIP: 0010:sysrq_handle_crash+0xd/0x16
Code: 89 f7 e8 d6 15 dd ff eb d2 45 89 ec eb f1 41 bc fb ff ff ff eb c5 e8 ca 06 c6 ff 90 90 c7 05 7e 04 19 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 c3 55 48 89 e5 bf 02 00 00 00 e8 44 89 c8
RSP: 0018:ffffc9000480fd58 EFLAGS: 00010246
RAX: ffffffff8068fe00 RBX: 0000000000000001 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000092 RDI: 0000000000000063
RBP: ffffc9000480fd80 R08: 0000000000000233 R09: 0000000000000004
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000063
R13: 0000000000000004 R14: ffffffff81673560 R15: 0000000000000000
FS:  00007f0177113fc0(0000) GS:ffff8882b7b00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000289899001 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Kernel panic - not syncing: Fatal exception
Kernel Offset: disabled
Rebooting in 5 seconds..

System is starting...
Starting system maintenance...
Scanning /dev/sda1... (100%) 
Scanning /dev/sda2... (100%) 
Serial number is FGT123456