问题描述
长时间 AC,概率性出现板卡已加载,但传感器未注册,概率约 5%
问题现象:AC 起来后,自动化脚本检查发现 PCIe/OCP 相关传感器未出现,消失的是这些传感器:
问题时间线:
19:55 做 AC:
2026-02-25 19:55:51 CLI,Administrator@192.168.109.48:50888,fructrl,Set FRU0 to ACCycle successfully
20:01 时卡的 sr 已加载:
2026-02-25 20:01:17.712144 hwdiscovery NOTICE: hwcomponent.lua(313): [self-discovery] name: Connector_PCIE_SLOT2_01010103, position: 0101010302, current: 1, previous: 0,uptime: 269 s
2026-02-25 20:01:17.820435 hwdiscovery NOTICE: init.lua(201): position: 0101010302, get csr data from /opt/bmc/sr/14140130_100010e2_10004010.sr, format version: 3.00, data version: 3.00
2026-02-25 20:01:17.829969 hwdiscovery NOTICE: hwcomponent.lua(209): position: 0101010302, load sr data successfully, uptime: 269 s, cost: 20ms
2026-02-25 20:01:17.837994 hwdiscovery NOTICE: hwcomponent.lua(230): position: 0101010302, start to process sr data, source: /opt/bmc/sr/14140130_100010e2_10004010.sr, format version: 3.00, data version: 3.00, uptime: 269 s
2026-02-25 20:01:17.970283 hwdiscovery NOTICE: hwcomponent.lua(313): [self-discovery] name: Connector_PCIE_SLOT3_01010103, position: 0101010303, current: 1, previous: 0,uptime: 269 s
2026-02-25 20:01:18.049693 storage NOTICE: mctp_service.lua(94): [Storage] mctp prepare finished. bmc_eid = 12 bmc_phy = 768 state = true
2026-02-25 20:01:18.093276 hwdiscovery NOTICE: sdr.lua(76): position: 0101010302, get objects, count: 39
2026-02-25 20:01:18.107033 maca NOTICE: init.lua(531): bmc.kepler.bios unlocked ForceResetLocked status
2026-02-25 20:01:18.126559 hwdiscovery NOTICE: init.lua(201): position: 0101010303, get csr data from /opt/bmc/sr/14140130_808657b0_80860002.sr, format version: 3.00, data version: 3.00
2026-02-25 20:01:18.128557 hwdiscovery NOTICE: hwcomponent.lua(209): position: 0101010303, load sr data successfully, uptime: 270 s, cost: 10ms
2026-02-25 20:01:18.133372 hwdiscovery NOTICE: hwcomponent.lua(230): position: 0101010303, start to process sr data, source: /opt/bmc/sr/14140130_808657b0_80860002.sr, format version: 3.00, data version: 3.00, uptime: 270 s
2026-02-25 20:01:18.181420 hwdiscovery NOTICE: hwcomponent.lua(313): [self-discovery] name: Connector_OCP_1_0101, position: , current: 1, previous: 0,uptime: 270 s
2026-02-25 20:01:18.219446 hwdiscovery NOTICE: parser_work.lua(76): position: 0101010302, process sr data successfully, uptime: 270 s, cost: 350ms
2026-02-25 20:01:18.308729 hwdiscovery NOTICE: init.lua(201): position: , get csr data from /opt/bmc/sr/14220247_15b3101f_1f242011.sr, format version: 3.00, data version: 3.00
2026-02-25 20:01:18.315986 hwdiscovery NOTICE: hwcomponent.lua(209): position: 010107, load sr data successfully, uptime: 270 s, cost: 20ms
2026-02-25 20:01:18.318517 hwdiscovery NOTICE: hwcomponent.lua(230): position: 010107, start to process sr data, source: /opt/bmc/sr/14220247_15b3101f_1f242011.sr, format version: 3.00, data version: 3.00, uptime: 270 s
2026-02-25 20:01:18.337640 hwdiscovery NOTICE: hwcomponent.lua(313): [self-discovery] name: Connector_OCP_2_0101, position: , current: 1, previous: 0,uptime: 270 s
2026-02-25 20:01:18.466059 hwdiscovery NOTICE: sdr.lua(76): position: 0101010303, get objects, count: 26
2026-02-25 20:01:18.480409 hwdiscovery NOTICE: init.lua(201): position: , get csr data from /opt/bmc/sr/14220247_14e416d7_1f242013.sr, format version: 3.00, data version: 3.00
2026-02-25 20:01:18.499224 hwdiscovery NOTICE: hwcomponent.lua(209): position: 010108, load sr data successfully, uptime: 270 s, cost: 30ms
2026-02-25 20:01:18.508022 hwdiscovery NOTICE: hwcomponent.lua(230): position: 010108, start to process sr data, source: /opt/bmc/sr/14220247_14e416d7_1f242013.sr, format version: 3.00, data version: 3.00, uptime: 270 s
2026-02-25 20:01:18.529628 hwdiscovery NOTICE: parser_work.lua(76): position: 0101010303, process sr data successfully, uptime: 270 s, cost: 200ms
sensor 模块此时也接收到了框架的对象分发:
2026-02-25 20:01:21.336397 sensor NOTICE: object_manage.lua(332): add objects callback, path: /bmc/kepler/ObjectGroup/0101010302, life cycle id: 1, count: 7, took 1060ms, uptime: 273 s
2026-02-25 20:01:21.344539 sensor NOTICE: object_manage.lua(342): add objects completely, path: /bmc/kepler/ObjectGroup/0101010302, life cycle id: 1, took 0ms, uptime: 273 s
2026-02-25 20:01:21.373723 sensor NOTICE: object_manage.lua(312): start to add objects, path: /bmc/kepler/ObjectGroup/0101010303, life cycle id: 1, count: 2, uptime: 273 s
2026-02-25 20:01:21.763484 sensor NOTICE: object_manage.lua(332): add objects callback, path: /bmc/kepler/ObjectGroup/0101010303, life cycle id: 1, count: 2, took 390ms, uptime: 273 s
2026-02-25 20:01:21.770718 sensor NOTICE: object_manage.lua(342): add objects completely, path: /bmc/kepler/ObjectGroup/0101010303, life cycle id: 1, took 0ms, uptime: 273 s
2026-02-25 20:01:21.841223 sensor NOTICE: object_manage.lua(312): start to add objects, path: /bmc/kepler/ObjectGroup/010107, life cycle id: 1, count: 3, uptime: 273 s
2026-02-25 20:01:22.366241 sensor NOTICE: object_manage.lua(332): add objects callback, path: /bmc/kepler/ObjectGroup/010107, life cycle id: 1, count: 3, took 530ms, uptime: 274 s
2026-02-25 20:01:22.367582 sensor NOTICE: object_manage.lua(342): add objects completely, path: /bmc/kepler/ObjectGroup/010107, life cycle id: 1, took 0ms, uptime: 274 s
2026-02-25 20:01:22.395031 sensor NOTICE: object_manage.lua(312): start to add objects, path: /bmc/kepler/ObjectGroup/010108, life cycle id: 1, count: 3, uptime: 274 s
但是,在 sensor.log 里这段时间却没有看到任何传感器注册,只能看到相关卡的 fru 注册了,fru 和卡都是在同一份 sr 上的,理论上 fru 注册了传感器肯定也得注册,但是却没有:
[2026-02-25 20:00:15] discrete sdr DiscreteSensor_PwrCapStatus_01010A of host 0 is registered.
[2026-02-25 20:00:15] discrete event [DiscreteEvent_PSRedundancy_01010A] is registered
[2026-02-25 20:00:15] discrete event [DiscreteEvent_PwrCapFailed_01010A] is registered
[2026-02-25 20:00:16] Entity [Entity_Fan_01010305] of host 0 is registered
[2026-02-25 20:00:16] sensor ThresholdSensor_FanRspeed_01010305 of host 0 is registered, number is 152.
[2026-02-25 20:00:16] threshold sdr ThresholdSensor_FanRspeed_01010305 of host 0 is registered.
[2026-02-25 20:01:23] fru sdr XC385(9) of host 0 is registered by id.
[2026-02-25 20:01:23] fru sdr XC386(7) of host 0 is registered by id.
[2026-02-25 20:09:39] fru sdr ExpBoard1(1) of host 0 is registered by obj.
[2026-02-25 20:09:41] bmc global enables[RecvMsgIntrptEnabled] = 1
[2026-02-25 20:09:41] bmc global enables[EvtMsgBufFullIntrptEnabled] = 1
[2026-02-25 20:09:41] bmc global enables[EvtMsgBufEnabled] = 1
[2026-02-25 20:09:41] bmc global enables[SELEnabled] = 1
正常场景下,是有这种打印的,异常场景下却没有:
[2026-02-25 20:12:52] Entity [Entity_RaidCard_0101010302] of host 0 is registered
[2026-02-25 20:12:52] sensor ThresholdSensor_PCIeBBUTemp_0101010302 of host 0 is registered, number is 153.
[2026-02-25 20:12:52] threshold sdr ThresholdSensor_PCIeBBUTemp_0101010302 of host 0 is registered.
[2026-02-25 20:12:55] fru sdr XC385(9) of host 0 is registered by id.
[2026-02-25 20:12:55] sensor ThresholdSensor_RaidCardTemp_0101010302 of host 0 is registered, number is 154.
[2026-02-25 20:12:55] threshold sdr ThresholdSensor_RaidCardTemp_0101010302 of host 0 is registered.
[2026-02-25 20:12:55] sensor DiscreteSensor_BBUStatus_0101010302 of host 0 is registered, number is 155.
[2026-02-25 20:12:55] discrete sdr DiscreteSensor_BBUStatus_0101010302 of host 0 is registered.
[2026-02-25 20:12:56] fru sdr XC386(7) of host 0 is registered by id.
[2026-02-25 20:12:56] discrete event [DiscreteEvent_BBUStatus_0_0101010302] is registered
[2026-02-25 20:12:56] discrete event [DiscreteEvent_BBUStatus_2_0101010302] is registered
[2026-02-25 20:12:56] discrete event [DiscreteEvent_BBUStatus_1_0101010302] is registered
[2026-02-25 20:12:57] Entity [Entity_PCIeCard_0101010303] of host 0 is registered
[2026-02-25 20:12:57] sensor ThresholdSensor_Temp_0101010303 of host 0 is registered, number is 156.
[2026-02-25 20:12:57] threshold sdr ThresholdSensor_Temp_0101010303 of host 0 is registered.
[2026-02-25 20:12:58] Entity [Entity_OCPCard_010108] of host 0 is registered
[2026-02-25 20:12:58] sensor ThresholdSensor_OCPTemp_010108 of host 0 is registered, number is 157.
[2026-02-25 20:12:58] threshold sdr ThresholdSensor_OCPTemp_010108 of host 0 is registered.
[2026-02-25 20:12:58] sensor ThresholdSensor_OpticalModuleTemp_010108 of host 0 is registered, number is 158.
[2026-02-25 20:12:58] threshold sdr ThresholdSensor_OpticalModuleTemp_010108 of host 0 is registered.
[2026-02-25 20:12:59] Entity [Entity_OCPCard_010107] of host 0 is registered
[2026-02-25 20:12:59] sensor ThresholdSensor_OCPTemp_010107 of host 0 is registered, number is 159.
[2026-02-25 20:12:59] threshold sdr ThresholdSensor_OCPTemp_010107 of host 0 is registered.
[2026-02-25 20:13:00] sensor ThresholdSensor_OpticalModuleTemp_010107 of host 0 is registered, number is 160.
[2026-02-25 20:13:00] threshold sdr ThresholdSensor_OpticalModuleTemp_010107 of host 0 is registered.
怀疑存在某种概率性时序问题,sensor 模块在这段时间也没有明显报错
需要注意的是:日志是在脚本自动恢复环境后收集的
这是产生问题的那次 AC:
2026-02-25 19:55:51 CLI,Administrator@192.168.109.48:50888,fructrl,Set FRU0 to ACCycle successfully
这是恢复的 AC,这次 AC 后传感器都是正常加载的:
2026-02-25 20:07:17 CLI,Administrator@192.168.109.48:52192,fructrl,Set FRU0 to ACCycle successfully
日志已传递
环境信息
社区 1230 基线版本

