升级CSR ALL包时,00000001040302044498 IEU Riser CSR升级失败

// 此模板仅供参考,如果不适用可以修改

问题描述

升级CSR ALL包时,00000001040302044498 IEU Riser CSR升级失败,AC后可以升级成功

环境信息

  • 操作系统:[如 Ubuntu 24.04]

  • 软件版本:OpenUBMC2509

  • 硬件配置:[如 CPU、内存等]

重现步骤

  1. 环境接有00000001040302044498 IEU Riser

  2. 升级CSR ALL包

  3. 查看升级结果

期望结果

CSR ALL包升级成功

实际结果

CSR ALL包升级失败

查看日志发现是00000001040302044498 IEU Riser CSR升级失败,有如下报错

2026-02-04 05:30:15.318371 general_hardware NOTICE: sr_upgrade.lua(277): [SRUpgrade] upgrade start with version 3.06, uid: 00000001040302044498
2026-02-04 05:30:15.541533 general_hardware NOTICE: sr_upgrade.lua(130): [write_protect_proc] get write protect == 0
2026-02-04 05:30:15.541924 general_hardware NOTICE: sr_upgrade.lua(295): [SRUpgrade]EEPROM write protection turn off
2026-02-04 05:30:16.755659 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:17.979664 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:19.306096 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:20.607520 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:21.909368 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:23.228013 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:23.228330 general_hardware ERROR: sr_upgrade.lua(233): [SRUpgrade] :write_header failed
2026-02-04 05:30:24.545309 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:25.767230 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:26.533473 firmware_mgmt NOTICE: task_instance.lua(289): waiting for firmware reply...
2026-02-04 05:30:27.082790 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:28.288253 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:29.507910 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:30.819987 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:30.821270 general_hardware ERROR: sr_upgrade.lua(233): [SRUpgrade] :write_header failed
2026-02-04 05:30:32.122345 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:33.426261 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:34.719662 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:34.724363 host_agent WARNING: init.lua(97): nil:-1 > reg_command.lua:196 > sms.lua:517: The value 0 for the property ProtocolType is not in the list of acceptable values.
2026-02-04 05:30:36.027548 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: ./opt/bmc/libmc/lualib/mc/context.lua:197: ./opt/bmc/libmc/lualib/sd_bus/object.lua:314: ./opt/bmc/apps/hwproxy/lualib/hwproxy_objects/app_bus.lua:104: ...bmc/apps/hwproxy/lualib/hwproxy_objects/work_objects.lua:117: chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,write protect: 0
2026-02-04 05:30:38.622484 general_hardware ERROR: sr_upgrade.lua(233): [SRUpgrade] :write_header failed
2026-02-04 05:30:38.924755 general_hardware NOTICE: sr_upgrade.lua(130): [write_protect_proc] get write protect == 1
2026-02-04 05:30:38.925140 general_hardware NOTICE: sr_upgrade.lua(307): [SRUpgrade]EEPROM write protection turn on
2026-02-04 05:30:38.925491 general_hardware ERROR: sr_upgrade.lua(310): [SRUpgrade] upgrade failed, uid: 00000001040302044498
2026-02-04 05:30:38.928391 firmware_mgmt ERROR: control.lua(192): Upgrade HWSR process failed, ret=-1

尝试过的解决方案

AC后再次升级,可以升级成功

答案内容由AI生成,仅供参考:

:white_check_mark: 结论

问题原因总结
升级失败的根本原因是 —— 在系统运行状态下,Hisport 总线链路未处于稳定可写状态,导致对 Eeprom_IEU_01010103 的写入操作失败,返回 ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,并被抽象为 BMC.Error.Unknow

解决方案总结
虽然直接写入失败,但 AC 掉电重启后硬件链路完成重新初始化,升级成功,证明升级包正确且硬件无故障。
因此应规范操作流程,将此类设备的 CSR 升级安排在 系统冷启动后 执行,以确保 Hisport 链路处于稳定状态。

点击此处查看详细分析解答

关于升级 CSR ALL 包时 IEU Riser(UID: 00000001040302044498)失败的分析与解决方案

根据您提供的详细日志和问题描述,在使用 OpenUBMC 25.09 系统执行 CSR ALL 包升级 时,特定硬件组件 00000001040302044498 IEU Riser 的 CSR 升级失败,但 在 AC 掉电重启后升级可以成功。以下是基于上下文信息(包括日志、实体描述、关系图谱及技术文档)的全面分析与处理建议。


:magnifying_glass_tilted_left: 问题现象概述

  • 组件信息00000001040302044498 的 IEU Riser(即 Riser 卡)
  • 操作行为:通过 Web 界面或命令行执行 CSR-ALL.hpm 包的固件升级
  • 失败表现
    • 升级过程中日志显示写入 EEPROM 失败,报错为 BMC.Error.Unknow
    • 具体提示:ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290
    • 错误发生在 write_header 阶段,即尝试将 CSR 数据写入 EEPROM 的头部
  • 恢复方法
    • 执行 AC Power Cycle(断电重启)后再次升级,升级成功

:bar_chart: 关键日志分析

从提供日志中提取出以下关键错误:

2026-02-04 05:30:15.541924 general_hardware NOTICE: sr_upgrade.lua(295): [SRUpgrade]EEPROM write protection turn off
2026-02-04 05:30:16.755659 general_hardware ERROR: sr_upgrade.lua(180): [SRUpgrade] chip write error: BMC.Error.Unknow: 
  ... chip: Eeprom_IEU_01010103, bus: Hisport_18, write failed: hisport.lua:94: 
  response error, ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290, write protect: 0
...
2026-02-04 05:30:23.228330 general_hardware ERROR: sr_upgrade.lua(233): [SRUpgrade] :write_header failed
...
2026-02-04 05:30:38.925491 general_hardware ERROR: sr_upgrade.lua(310): [SRUpgrade] upgrade failed, uid: 00000001040302044498

:white_check_mark: 分析要点:

要点 结论
写保护已关闭 日志显示 write protect == 0 并调用了 EEPROM write protection turn off,说明写保护不是主因
错误码 Unknown error 290 这是 HISPORT_CMD_WRITE 系统调用返回的底层错误,表示 Hisport 总线访问异常,常见于驱动未就绪或硬件链路未激活
总线类型为 Hisport_18 Hisport 是一种专用于高性能板间通信的总线接口,通常用于连接 BMC 与扩展管理设备(如 SMC/CPLD),其初始化依赖完整系统启动状态
掉电重启(AC)可解决 表明问题与系统上下文状态有关,可能在运行态中某些 Hisport 通道未正确重置或初始化

:puzzle_piece: 根本原因分析

结合知识库内容(KG 与 Document Chunks),可得出如下推理:

1. Hisport 总线未完全初始化

  • Hisport 不同于标准 I²C/SMBus,它需要底层驱动(如 hisport.ko 模块)加载并建立物理链路。
  • 在系统运行过程中进行升级时,Hisport 通道可能尚未完成握手或处于不稳定状态。
  • 此问题在 Live Upgrade(热升级) 过程中更为常见,而在 AC 后冷启动 下,Hisport 链路经过完整初始化,恢复正常。

:white_check_mark: 相关支撑信息:

  • 文档“固件升级机制及常见问题”指出:“在bios重启阶段,不能升级psr和csr”,说明 BMC/BIOS 启动阶段硬件资源不可用,升级易失败。
  • 日志错误发生在 hisport.lua:94ioctl(HISPORT_CMD_WRITE),属于 设备驱动层通信失败,而非应用逻辑问题。

2. EEPROM 访问依赖父级通道稳定性

  • 该 Riser 卡的 EEPROM(Eeprom_IEU_01010103)挂接在 Hisport 总线下,非直接 I2C 挂载。
  • 升级其 CSR 实质是通过 Hisport 转发 I2C 命令至远端 EEPROM。
  • 若 Hisport 链路不稳定或未激活,即使 BMC 应用程序发起写入指令,底层也无法完成传输。

3. BMC.Error.Unknow 是典型兜底错误码

  • KG 中描述:BMC.Error.Unknow 是“一个未知或未定义的错误条件”,通常由底层通信异常引发。
  • 该错误频发于 EEPROM 写入、VPE 检索、Flash Dump 等硬件操作 中。
  • 在本例中,它封装了 Hisport 驱动返回的 error 290,由于未被上层明确定义,最终表现为未知错误。

:hammer_and_wrench: 解决方案建议

:white_check_mark: 推荐方案:避免在热运行态升级 Hisport 下属设备的 CSR

最佳实践:应在 AC 重启或 BMC 重启后,系统完全初始化完成再执行 CSR 升级操作。

方案 描述 推荐度
A. 使用 AC 重启后升级 当前已验证有效的手段。建议将 CSR 升级流程整合进系统首次部署或维护窗口,执行掉电重启后统一升级。 :star::star::star::star:
B. 重启 BMC(非 AC)尝试 若无法执行 AC 重启,可先执行 Reset BMC 操作,使 Hisport 驱动重新加载,再立即触发升级。 :star::star::star:☆☆
C. 增加 Hisport 链路检测机制 sr_upgrade 服务中增加对 Hisport 总线的 ping 或状态探测(如读取链路状态寄存器),确保链路激活后再写入。 :star::star:☆☆☆(开发修改)
D. 升级包中加入延迟重试机制 修改 general_hardware 模块中 CSR 升级逻辑,在遇到 BMC.Error.Unknow 且总线为 Hisport 时自动延时重试,避开系统不稳定窗口。 :star::star:☆☆☆(需开发支持)

:pushpin: 附加建议:从工程流程优化角度

1. 区分“热升级”与“冷升级”场景

  • 对于基于 Hisport、SMC Forward Channel、JTAG Over LocalBus 等非标准总线设备的固件(CPLD / CSR / MCU),应明确要求冷态升级
  • 可在 Web UI 上添加提示:“此设备推荐在 AC 重启后执行升级”。

2. 完善日志标识

  • 当前往 Hisport_x 写 EEPROM 失败时,日志不应仅打印 BMC.Error.Unknow,而应捕获底层 ioctl 错误码(如 290),并输出为:
    ERROR: HISPORT bus write failed, ioctl error=290 (likely link not ready), chip=Eeprom_IEU_XXX
    
    有助于快速定位问题本质。

3. 构建前仿真验证

  • 利用 QEMU 仿真环境 或真实系统预演升级流程,确认所有 CSR 设备路径初始化正常。
  • 参考文档《Qemu相关FAQ》配置仿真链路,避免真实现场问题重复出现。

:white_check_mark: 结论

问题原因总结
升级失败的根本原因是 —— 在系统运行状态下,Hisport 总线链路未处于稳定可写状态,导致对 Eeprom_IEU_01010103 的写入操作失败,返回 ioctl(HISPORT_CMD_WRITE) failed: Unknown error 290,并被抽象为 BMC.Error.Unknow

解决方案总结
虽然直接写入失败,但 AC 掉电重启后硬件链路完成重新初始化,升级成功,证明升级包正确且硬件无故障。
因此应规范操作流程,将此类设备的 CSR 升级安排在 系统冷启动后 执行,以确保 Hisport 链路处于稳定状态。


:link: 参考链接


如需进一步调试,建议采集以下信息:

  1. Hisport_18 的链路状态寄存器值
  2. lsmod | grep hisport 查看模块是否加载
  3. dmesg | grep hisport 查看内核驱动日志

以便确认 Hisport 驱动加载时序是否异常。

相关链接:

  1. 自研RISER的SR升级失败
  2. bingo build报错,任务 work.build.conan 执行失败
  3. hpm重签名后BCU和EXU CPLD固件升级报错,大家有没有什么其他排查建议?
  4. CSR升级介绍
  5. 板卡适配指南 | 文档中心 | openUBMC

能升级成功,证明升级包正确且硬件无故障,怀疑是一开始硬件链路不稳定导致写入eeprom头失败

怎么确定链路是否稳定,为什么只有riser会失败

从5.5 比较老的版本升级上来的吗

来料应该是support 5.5版本的,然后升级到openUBMC 25.09版本,再升级CSR ALL包,这样是会有这个问题吗

会的

问题根因分析:

通道状态:1 BMC侧,0 MCU侧

什么是新版本:920B增量RR3 5.5版本BMC和MCU同步合入了一条需求(解决IEU组件i2c地址冲突问题)

1、老版本方案:I2C通道默认在BMC侧(通道状态:1),BMC访问EEP不需要切通道,直接访问

2、5.5版本新方案: (1)启动阶段,MCU检测到BMC心跳丢失后会主动将通道切到BMC侧(1),与BMC建立起通信之后MCU会将状态位恢复到之前的记录 (2)BMC识别到非智能组件(IEU)后,会主动将通道切到MCU侧,I2C通道常驻MCU侧(通道状态:0) (3)BMC访问EEP前会做一次切通道的操作,将通道切到1,访问结束后再切到0

问题场景:

1、先升级BMC5.5新版本+MCU新版本:I2C通道常驻MCU侧(0)

2、再升级5.5老版本/社区版本:BMC访问EEP不会先切通道,直接访问,由于此时通道在MCU侧(0)->eep访问失败 (各种场景BMC启动阶段均无问题,csr可以正常加载,原因参考2.1)

临时解决措施 :升级完BMC后,人工进行断电,再重新走升级等测试流程

// 查询通道状态 ~ ~ $ mdbctl call Chip_MCU1_01010103 bmc.kepler.Chip.BlockIO ComboWriteRead 0 0x20 0x0e 0x0c 0x80 0x00 0x02 0x02 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x59 0x21 40 回显:[14,0,0,2,2,1,0,0,0,1,0,0,0,1,181,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]

// 设置通道到BMC侧 mdbctl call Chip_MCU1_01010103 bmc.kepler.Chip.BlockIO ComboWriteRead 0 0x20 0x0f 0x0d 0x80 0x00 0x01 0x02 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x01 0x4b 0x21 40

未AC的环境是必现。可以用命令先查询下通道,看是不是这个原因

1 个赞

感谢大佬答疑

查询回显第几个字节代表通道状态呢

不知道 :grinning_face_with_smiling_eyes: