场外解 coredump 的方法

能看一些信息,但不多

获取 GDB 工具

找 PAE

部署工具

将工具解压,上传到环境的 /tmp 目录,然后部署到 data 分区

mv /tmp/gdb_1711 /data
mv /tmp/libreadline.so.8 /data
chmod 777 /data/gdb_1711

上传 coredump 文件

coredump 文件是成对生成的,其中 core-* 对应 coredump 文件本体,core-maps-* 对应进程当时的虚拟内存地址空间布局,用于偏移量辅助计算,coredump 文件名末尾的时间对应的是 BMC 版本编译日期,coredump 文件的时间戳才是 coredump 发生的时间点;

只需上传 core-* 文件到 BMC 的 /tmp 目录下

使用 gdb 解析 coredump

确保环境是发生 coredump 当时的版本(为了确保偏移计算准确)

LD_LIBRARY_PATH=/data /data/gdb_1711 /opt/bmc/skynet/skynet /tmp/core-skynet-11-security-20260112155052

会打印如下内容:

GNU gdb (GDB) 8.2
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-euler-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/bmc/skynet/skynet...(no debugging symbols found)...done.
[New LWP 3016]
[New LWP 2671]
[New LWP 2678]
[New LWP 2674]
[New LWP 2680]
[New LWP 2675]
[New LWP 2709]
[New LWP 2710]
[New LWP 2681]
[New LWP 2729]
[New LWP 2713]
[New LWP 2851]
[New LWP 2679]
[New LWP 2677]
[New LWP 2673]
[New LWP 2672]
[New LWP 2629]
[New LWP 2646]
[New LWP 3014]
[New LWP 2730]
[New LWP 2787]
[New LWP 2715]
[New LWP 2676]
[New LWP 2728]
[New LWP 2950]

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Can't read pathname for load map: Unknown error -1.

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `bmc_core /opt/bmc/apps/hica/subsys/bmc_core/config.cfg'.
Program terminated with signal SIGABRT, Aborted.
#0  0x0000ffff88918334 in ?? () from /lib64/libc.so.6
[Current thread is 1 (LWP 3016)]

其中,0x0000ffff88918334 是崩溃的发生地址,SIGABRT 是崩溃触发的直接原因,/lib64/libc.so.6 是发生问题的 so 库
我们记录一下这个崩溃地址

解析偏移量到函数名

打开对应生成的 maps 文件 core-maps-skynet-6-bmc_core-20260122151932
找到这个崩溃地址所对应的位置

ffff88872000-ffff88873000 r--p 00002000 b3:50 17905                      /usr/lib64/libgmodule-2.0.so.0.7800.3
ffff88873000-ffff88874000 rw-p 00003000 b3:50 17905                      /usr/lib64/libgmodule-2.0.so.0.7800.3
ffff88894000-ffff88a30000 r-xp 00000000 b3:50 1038                       /lib64/libc.so.6
ffff88a30000-ffff88a3f000 ---p 0019c000 b3:50 1038                       /lib64/libc.so.6
ffff88a3f000-ffff88a43000 r--p 0019b000 b3:50 1038                       /lib64/libc.so.6
ffff88a43000-ffff88a45000 rw-p 0019f000 b3:50 1038                       /lib64/libc.so.6
ffff88a45000-ffff88a55000 rw-p 00000000 00:00 0 
ffff88aa3000-ffff88aa4000 r-xp 00000000 b3:50 1079                       /lib64/librt.so.1
ffff88aa4000-ffff88ab3000 ---p 00001000 b3:50 1079                       /lib64/librt.so.1

比如这里找到,我们的崩溃地址落在这个区间

ffff88894000-ffff88a30000 r-xp 00000000 b3:50 1038                       /lib64/libc.so.6

接下来,计算崩溃偏移相对于虚拟地址区间头部的偏移
0x0000ffff88918334 - 0xffff88894000 = 0x84334‬

接下来,把环境上的 /lib64/libc.so.6 拷贝到 /tmp 目录,下载到本地(可能需要 chmod 777 才能下载)
之后,在本地执行

addr2line -ife libc.so.6 0x84334
pthread_key_delete
??:?

这里打印出的 pthread_key_delete 就是发生崩溃的函数名了,可供进一步分析

2 个赞