本地conan缓存占用空间过大,是否可以优化

问题背景:

使用机械盘构建hpm包很慢,将docker容器移到SSD后构建速度明显提升,但SSD容量有限。最近SSD被写满导致docker无法启动,定位发现主要是/root/.conan2缓存过大造成的

问题初步分析

[home/workspace/source]du --max-depth=1 /root/.conan2/p  -h | sort -h
……
100M    /root/.conan2/p/abseief8e89c61da2c
102M    /root/.conan2/p/vppb37842755681e
202M    /root/.conan2/p/boosta9d41685a7e55
202M    /root/.conan2/p/boostb4e4042b6f406
220M    /root/.conan2/p/generbce75dd824179
272M    /root/.conan2/p/grpc_9eb8f8ac74608
490M    /root/.conan2/p/gener0c244fecacb88
490M    /root/.conan2/p/gener12c82568dca6e
490M    /root/.conan2/p/gener7bb205eb585ce
490M    /root/.conan2/p/gener936a41e23a449
490M    /root/.conan2/p/genere584e269da843
490M    /root/.conan2/p/genere8981ed725e2d
493M    /root/.conan2/p/gener1cb6eeb14dc9d
512M    /root/.conan2/p/stora6b93f060c421f
512M    /root/.conan2/p/stora7b0348dcd6102
512M    /root/.conan2/p/storaabb335f3de754
512M    /root/.conan2/p/storacd2ded28010b1
529M    /root/.conan2/p/stora0bc545435fce0
529M    /root/.conan2/p/stora3163be9b695eb
529M    /root/.conan2/p/stora9f9590e716587
529M    /root/.conan2/p/storaa87ec8323aa95
530M    /root/.conan2/p/stora0c3a3e9eb78d8
530M    /root/.conan2/p/stora1f3e6ce57b807
530M    /root/.conan2/p/stora22d48b355b10c
530M    /root/.conan2/p/stora36945ff3aee68
530M    /root/.conan2/p/stora4267afd72d8ca
530M    /root/.conan2/p/stora475810ee09573
530M    /root/.conan2/p/stora539690620e375
530M    /root/.conan2/p/stora5eb369970b400
530M    /root/.conan2/p/stora6765c1fc14509
530M    /root/.conan2/p/stora69c79cce72bfa
530M    /root/.conan2/p/stora7959aae2f392b
530M    /root/.conan2/p/stora80330d5538287
530M    /root/.conan2/p/storaa0f4f407c375b
530M    /root/.conan2/p/storaa1adb2aaac6ea
530M    /root/.conan2/p/storac3d197fff1c5d
530M    /root/.conan2/p/storacae369f7ac049
530M    /root/.conan2/p/storacd054ca501325
530M    /root/.conan2/p/storacd178f52cb7ed
530M    /root/.conan2/p/storad50a84596aa19
530M    /root/.conan2/p/storad8a64c62990bd
530M    /root/.conan2/p/storadd84da0569636
530M    /root/.conan2/p/storafd5c32450eb7d
26G     /root/.conan2/p/b
48G     /root/.conan2/p

可以看到单个组件的conan缓存可以达到530M!!以storage组件为例接着分析

storage组件缓存内容分析

[home/workspace/source]du --max-depth=1 /root/.conan2/p/storafd5c32450eb7d/s/temp/  -h | sort -h
92K     /root/.conan2/p/storafd5c32450eb7d/s/temp/etc
828K    /root/.conan2/p/storafd5c32450eb7d/s/temp/.deploy
2.7M    /root/.conan2/p/storafd5c32450eb7d/s/temp/coverage
4.1M    /root/.conan2/p/storafd5c32450eb7d/s/temp/lua_codegen
6.6M    /root/.conan2/p/storafd5c32450eb7d/s/temp/lib
43M     /root/.conan2/p/storafd5c32450eb7d/s/temp/dtframeforlua
92M     /root/.conan2/p/storafd5c32450eb7d/s/temp/opt
164M    /root/.conan2/p/storafd5c32450eb7d/s/temp/usr
195M    /root/.conan2/p/storafd5c32450eb7d/s/temp/include
507M    /root/.conan2/p/storafd5c32450eb7d/s/temp/

可以看到主要是temp路径下的usr和include目录占用过多

usr缓存内容

[home/workspace/source]ll /root/.conan2/p/storafd5c32450eb7d/s/temp/usr/lib64/ -Srh
……
-rwxr-xr-x 1 root root 1.1M  1月  5 10:33 liblua.so*
-rw-r--r-- 1 root root 1.9M  1月  5 10:33 libxcap.so
-rw-r--r-- 1 root root 2.2M  1月  5 10:33 libssl_ossl.so
-rw-r--r-- 1 root root 2.6M  1月  5 10:33 libmctp.so
-rw-r--r-- 1 root root 4.5M  1月  5 10:33 libssl_ossl.a
-rw-r--r-- 1 root root 5.0M  1月  5 10:33 libpse.so
-rwxr-xr-x 1 root root 5.6M  1月  5 10:33 libsqlite3.so.0.8.6*
-rw-r--r-- 1 root root 6.2M  1月  5 10:33 libipsi_crypto.so
-rw-r--r-- 1 root root  12M  1月  5 10:33 libcrypto_ossl.so
-rw-r--r-- 1 root root  15M  1月  5 10:33 libsoc_adapter.so
-rw-r--r-- 1 root root  25M  1月  5 10:33 libcrypto_ossl.a
-rw-r--r-- 1 root root  43M  1月  5 10:33 libsomp.so

可以看到主要是各种动态库缓存

include缓存内容

[home/workspace/source]du --max-depth=1 /root/.conan2/p/storafd5c32450eb7d/s/temp/include/  -h | sort -h
12K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/logging_utils
12K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/soc_adapter
16K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/mip
20K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/utils
32K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/mds
40K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/sdpcac
52K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/secure
52K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/shmlock
60K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/ncsi
68K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/frudata
80K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/lsw
80K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/luawrapper
80K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/tee
84K     /root/.conan2/p/storafd5c32450eb7d/s/temp/include/lite-kmc
104K    /root/.conan2/p/storafd5c32450eb7d/s/temp/include/cac
108K    /root/.conan2/p/storafd5c32450eb7d/s/temp/include/common
108K    /root/.conan2/p/storafd5c32450eb7d/s/temp/include/dbus
180K    /root/.conan2/p/storafd5c32450eb7d/s/temp/include/drivers
204K    /root/.conan2/p/storafd5c32450eb7d/s/temp/include/sdp
216K    /root/.conan2/p/storafd5c32450eb7d/s/temp/include/kmc
1.9M    /root/.conan2/p/storafd5c32450eb7d/s/temp/include/sml
2.3M    /root/.conan2/p/storafd5c32450eb7d/s/temp/include/openssl
184M    /root/.conan2/p/storafd5c32450eb7d/s/temp/include/boost
195M    /root/.conan2/p/storafd5c32450eb7d/s/temp/include/

可以看到icnlude下面主要是boost库的头文件占用过多

/p和/p/b路径下都有组件缓存

[home/workspace/source]du --max-depth=1 /root/.conan2/p/b  -h | sort -h
……
293M    /root/.conan2/p/b/webui08feb395e756e
293M    /root/.conan2/p/b/webui141977ed4cab7
293M    /root/.conan2/p/b/webui7221ac78372ba
293M    /root/.conan2/p/b/webuif607eadfd2993
296M    /root/.conan2/p/b/webui49e5567325b12
296M    /root/.conan2/p/b/webui6f58364d6338f
296M    /root/.conan2/p/b/webuia8c7348e5dd8c
296M    /root/.conan2/p/b/webuid6340d650284f
297M    /root/.conan2/p/b/webui1c65548d50e47
297M    /root/.conan2/p/b/webui1f927b47bea10
297M    /root/.conan2/p/b/webui300fcea8030b2
297M    /root/.conan2/p/b/webui465a0eef939b3
297M    /root/.conan2/p/b/webuib4175cd928278
396M    /root/.conan2/p/b/libmc5a6bac203db92
401M    /root/.conan2/p/b/opentd8e953c4bed33
496M    /root/.conan2/p/b/gener2e9ca284b736f
496M    /root/.conan2/p/b/gener860e300cadf21
496M    /root/.conan2/p/b/generca734a65d820b
496M    /root/.conan2/p/b/generf0b8371f27671
496M    /root/.conan2/p/b/generfab6b4b1688c0
496M    /root/.conan2/p/b/generfb985058e9793
498M    /root/.conan2/p/b/genere3b9f005258f5
539M    /root/.conan2/p/b/storad657b349355a5
545M    /root/.conan2/p/b/storaf150d820e8780
545M    /root/.conan2/p/b/storaf5557aa0cad95
547M    /root/.conan2/p/b/stora0f727c62bf9c7
561M    /root/.conan2/p/b/stora65bc9ce3a3320
563M    /root/.conan2/p/b/stora16c7918e6f485
563M    /root/.conan2/p/b/stora380412145ce59
563M    /root/.conan2/p/b/stora535189a88d48c
563M    /root/.conan2/p/b/stora5ada21da2d93d
563M    /root/.conan2/p/b/stora8cadcfc6fa5a6
563M    /root/.conan2/p/b/stora94e00a7bedde8
578M    /root/.conan2/p/b/stora1941854ef30dc
578M    /root/.conan2/p/b/stora67c97150c86b7
578M    /root/.conan2/p/b/stora76530e061b7c4
578M    /root/.conan2/p/b/storab87b240939ae5
579M    /root/.conan2/p/b/stora0eb65be049a88
579M    /root/.conan2/p/b/stora2fcca6798fd38
579M    /root/.conan2/p/b/stora31634e495a553
579M    /root/.conan2/p/b/stora4c8744d2f881f
579M    /root/.conan2/p/b/stora53cf005d39de6
579M    /root/.conan2/p/b/stora6faba90b886ec
579M    /root/.conan2/p/b/stora7fc30c13b40b1
579M    /root/.conan2/p/b/stora89f65aefe32ba
579M    /root/.conan2/p/b/storaa691b04a67539
579M    /root/.conan2/p/b/storacb207584ac310
579M    /root/.conan2/p/b/storad54dc15e5d0b0
579M    /root/.conan2/p/b/storade5f2ca3e4910
579M    /root/.conan2/p/b/storae190686a3f265
579M    /root/.conan2/p/b/storae54f05c368966
579M    /root/.conan2/p/b/storaf4d8e8ce1d42d
26G     /root/.conan2/p/b

/p/b路径下的storage同样是temp路径下的动态库和头文件缓存占用了大量空间,这里就不再展开

述求

优化conan本地缓存的大小,降低编译环境的资源开销
@xuhaijun

  • 比如限制缓存的大小、包的个数或者生存时间等
  • 针对storage、general_hardware等大缓存的组件专门优化

补充

发现可以使用conan remove "*" --lru=7d -c清理指定时间前的conan缓存
其他重要的 Conan 功能 — pygallery 文档

从本地构建时会将工作区目录复制到组件的conan源码目录,而一般temp目录是临时文件存储目录,将temp一并复制导致conan源码目录偏大。

这可能是一个缺陷,bmcgo/component/template_v2/conanbase.py.mako文件复制源码时应该忽略temp目录。

1 个赞

补充一下,temp可以在/root/.conan2/p/storafd5c32450eb7d/s/temp和/root/.conan2/p/b/stora94e00a7bedde8/b/temp两个地方看到,说明source和build都需要忽略temp目录

这是conan的构建流程:源码目录的内容会复制到构建目录中,再由构建脚本打包到最终打包目录。

处理源码即可。