Support for DPKG local caching #3292

Kalimuthu-Velappan · 2019-08-06T12:01:26Z

- What I did
DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker.

The DPKG caching framework provides an infrastructure which caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package.

This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files.

- How I did it
It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache.
SONIC_DPKG_CACHE_METHOD=cache
SONIC_DPKG_CACHE_SOURCE=

    where  SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching
                            none:     no caching
                            cache:    cache from local directory
                            #url:      cache over remote HTTP server (*To be supported)
                            #registry: cache on the docker local/remote registry (*To be supported)

Dependency file tracking:
Dependency files are tracked for each target in two levels.
1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc.
2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc.

    For example: dependency files for Linux Kernel - src/sonic-linux-kernel,

            SMPATH=$(SRC_PATH)/sonic-linux-kernel/
            SM_DEP_LIST  := Makefile
            SM_DEP_LIST  += patch/*
            SM_DEP_LIST  += patch/preconfig/*
            SMDEP_LIST   := $(wildcard $(addprefix $(SMPATH),$(SM_DEP_LIST)))
            DEP_LIST     := $(SONIC_MAKEFILE_LIST) rules/linux-kernel.mk

            LINUX_HEADERS_COMMON = linux-headers-$(KVERSION_SHORT)-common_$(KERNEL_VERSION)-$(KERNEL_SUBVERSION)_all.deb
            $(LINUX_HEADERS_COMMON)_CACHE_MODE = GIT_COMMIT_SHA
            $(LINUX_HEADERS_COMMON)_DEP_SOURCE = $(DEP_LIST)
            $(LINUX_HEADERS_COMMON)_SMDEP_SOURCE = $(SMDEP_LIST)
            SONIC_MAKE_DEBS += $(LINUX_HEADERS_COMMON)

Cache file tracking:
The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files.
The cache filename is formed with the following format

    FORMAT:
            <module deb filename>.<48 byte SHA hash>.tgz
            Eg:
              linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-fd21bb776fa16f47c0b63ef593d4a32643bca228.tgz

            < 48-byte SHA value > - the SHA value is derived from either of the following.
                    GIT_COMMIT_SHA  - SHA value of the last git commit ID if it is a submodule
                    GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files.

Target Specific rules:
Caching can be enabled/disabled on a global level and also on the per-target level.

            $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
                    $$($$*_DEP_SOURCE) $$($$*_SMDEP_SOURCE)
            $(HEADER)


            # Load the target deb from DPKG cache
            $(if $(and $(filter-out none,$(SONIC_DPKG_CACHE_METHOD)),$($*_CACHE_MODE)), $(call LOAD_CACHE,$*) )


            # Skip building the target if it is already loaded from cache
            if [ -z '$($*_CACHE_LOADED)' ] ; then

            .....
            # Rules for Generating the target DEB file.
            .....


            # Save the target deb into DPKG cache
            $(if $(and $(filter-out none,$(SONIC_DPKG_CACHE_METHOD)),$($*_CACHE_MODE)), $(call SAVE_CACHE,$*,$@))
            fi


            $(FOOTER)


    The make rule-'$$($$*_DEP_SOURCE) $$($$*_SMDEP_SOURCE)' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target.

    Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents.
    The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed.
    It also updates the target-specific variable to indicate the cache is loaded or not.
    The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock.

- How to verify it

    The caching functionality is verified by enabling it in Linux kernel submodule.
    It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build.

- Description for the changelog
The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files.
If the module's dependency files are not changed, it restores the module deb files from the cache storage.

- A picture of a cute animal (not mandatory but encouraged)

msftclas · 2019-08-06T12:01:41Z

All CLA requirements met.

lguohan · 2019-08-06T14:40:46Z

can you add full description of what you have done? how you verified?

Kalimuthu-Velappan · 2019-08-09T12:19:41Z

can you add a full description of what you have done? how you verified?

Added full description for design note and verification of the patch

lguohan · 2019-08-15T08:43:52Z

I am not sure I fully understand the idea. I think we should discuss this in a community meeting. can you book a community meeting to discuss this?

lguohan · 2019-12-03T01:19:59Z

is there any update on this PR?

Kalimuthu-Velappan · 2019-12-03T14:36:04Z

Hi lguohan , We have added more enhancements to PR to cover the caching for docker images along with Debian packages and submitted the code changes for internal code review. We are working on the HLD update for this feature and will share it with you once it is ready. We have planned to run the caching framework internally for a few weeks to make sure that any missing dependency to be captured. Once everything is stable, we will update the PR for upstream. Thanks Kals

…

On Tue, Dec 3, 2019 at 6:50 AM lguohan ***@***.***> wrote: is there any update on this PR? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3292>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM2UC2RSLKTNLSJFEV665ZDQWWX4BANCNFSM4IJVS7BQ> .

-- Thanks - Kals

lguohan · 2019-12-11T00:11:22Z

retest vsimage please

lguohan · 2019-12-11T00:14:10Z

slave.mk

@@ -349,6 +454,11 @@ $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(a
 	if [ -f $($*_SRC_PATH).patch/series ]; then pushd $($*_SRC_PATH) && quilt pop -a -f; popd; fi
 	# Take built package(s)
 	mv $(addprefix $($*_SRC_PATH)/../, $* $($*_DERIVED_DEBS) $($*_EXTRA_DEBS)) $(DEBS_PATH) $(LOG)
+
+		# Save the target deb into DPKG cache
+		$(if $(and $(filter-out none,$(SONIC_DPKG_CACHE_METHOD)),$($*_CACHE_MODE)), $(call SAVE_CACHE,$*,$@))


can check the alignment here?

lguohan · 2019-12-11T00:14:24Z

slave.mk

@@ -321,6 +412,12 @@ $(addprefix $(DEBS_PATH)/, $(SONIC_MAKE_DEBS)) : $(DEBS_PATH)/% : .platform $$(a
 	DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS_GENERIC}" make DEST=$(shell pwd)/$(DEBS_PATH) -C $($*_SRC_PATH) $(shell pwd)/$(DEBS_PATH)/$* $(LOG)
 	# Clean up
 	if [ -f $($*_SRC_PATH).patch/series ]; then pushd $($*_SRC_PATH) && quilt pop -a -f; popd; fi
+
+		# Save the target deb into DPKG cache
+		$(if $(and $(filter-out none,$(SONIC_DPKG_CACHE_METHOD)),$($*_CACHE_MODE)), $(call SAVE_CACHE,$*,$@))


check you check the alignment here?

lguohan · 2019-12-11T00:15:23Z

rules/linux-kernel.mk

+SMPATH=$(SRC_PATH)/sonic-linux-kernel/
+SM_DEP_LIST  := Makefile
+SM_DEP_LIST  += patch/*
+SM_DEP_LIST  += patch/preconfig/*


why do we have only rule for linux-kernel? We should have rules for other source code?

lguohan · 2019-12-11T00:16:03Z

rules/linux-kernel.mk

@@ -7,8 +7,19 @@ KERNEL_SUBVERSION = 1+deb9u5

 export KVERSION_SHORT KVERSION KERNEL_VERSION KERNEL_SUBVERSION

+
+SMPATH=$(SRC_PATH)/sonic-linux-kernel/
+SM_DEP_LIST  := Makefile


what does this SM stand for?

SM stands for Sub Module in the sonic repo
Initially, the DPKG caching framework is supported only for the Linux submodule, later, we have enhanced the DPKG caching framework to cover all the sub modules and Debian packages in the sonic repro.
We will close this one and will raise a new pull request for the enhanced version of the DPKG caching framework.

Support for DPKG local caching

b943612

stcheng added the Enhancement ➕ label Aug 6, 2019

lguohan assigned qiluo-msft Aug 10, 2019

lguohan requested a review from marian-pritsak August 15, 2019 08:29

Kalimuthu-Velappan force-pushed the master branch from 0055bc3 to b943612 Compare September 12, 2019 11:48

Kalimuthu-Velappan added 2 commits October 4, 2019 07:38

Merge remote-tracking branch 'upstream/master'

e7e3891

Merge remote-tracking branch 'upstream/master'

40b4d47

nirenjan approved these changes Oct 17, 2019

View reviewed changes

marian-pritsak removed their request for review November 1, 2019 17:18

lguohan reviewed Dec 11, 2019

View reviewed changes

Kalimuthu-Velappan closed this Dec 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for DPKG local caching #3292

Support for DPKG local caching #3292

Kalimuthu-Velappan commented Aug 6, 2019 •

edited

msftclas commented Aug 6, 2019 •

edited

lguohan commented Aug 6, 2019

Kalimuthu-Velappan commented Aug 9, 2019

lguohan commented Aug 15, 2019

lguohan commented Dec 3, 2019

Kalimuthu-Velappan commented Dec 3, 2019 via email

lguohan commented Dec 11, 2019

lguohan Dec 11, 2019

lguohan Dec 11, 2019

lguohan Dec 11, 2019

lguohan Dec 11, 2019

Kalimuthu-Velappan Dec 11, 2019

Support for DPKG local caching #3292

Support for DPKG local caching #3292

Conversation

Kalimuthu-Velappan commented Aug 6, 2019 • edited

msftclas commented Aug 6, 2019 • edited

lguohan commented Aug 6, 2019

Kalimuthu-Velappan commented Aug 9, 2019

lguohan commented Aug 15, 2019

lguohan commented Dec 3, 2019

Kalimuthu-Velappan commented Dec 3, 2019 via email

lguohan commented Dec 11, 2019

lguohan Dec 11, 2019

Choose a reason for hiding this comment

lguohan Dec 11, 2019

Choose a reason for hiding this comment

lguohan Dec 11, 2019

Choose a reason for hiding this comment

lguohan Dec 11, 2019

Choose a reason for hiding this comment

Kalimuthu-Velappan Dec 11, 2019

Choose a reason for hiding this comment

Kalimuthu-Velappan commented Aug 6, 2019 •

edited

msftclas commented Aug 6, 2019 •

edited