Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rust: Replace keep_fingerprints logic with unconditional cargo clean #63

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

rust: Replace keep_fingerprints logic with unconditional cargo clean #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b8f3ea5
rust: Avoid `rename-output.sh` output if there is no output
ijc Aug 19, 2024
a9b9b6e
rust: Echo `cargo $args` into the logs
ijc Aug 19, 2024
d24d25c
rust: Replace `keep_fingerprints` logic with unconditional `cargo clean`
ijc Aug 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 17 additions & 46 deletions rust/Earthfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ VERSION 0.8
# INIT sets some configuration in the environment (used by following functions), and installs required dependencies.
# Arguments:
# - cache_prefix: Overrides cache prefix for cache IDS. Its value is exported to the build environment under the entry: $EARTHLY_CACHE_PREFIX. By default ${EARTHLY_TARGET_PROJECT_NO_TAG}#${OS_RELEASE}#earthly-cargo-cache
# - keep_fingerprints (false): Instructs the following +CARGO calls to not remove the Cargo fingerprints of the source packages. Use only when source packages have been COPYed with --keep-ts option.
# - sweep_days (4): +CARGO uses cargo-sweep to clean build artifacts that haven't been accessed for this number of days.
INIT:
FUNCTION
Expand All @@ -17,17 +16,14 @@ INIT:
ENV PATH="$PATH:$CARGO_HOME/bin"
END
DO +INSTALL_CARGO_SWEEP
COPY +get-jq/jq /tmp/jq

# $EARTHLY_CACHE_PREFIX
ARG EARTHLY_TARGET_PROJECT_NO_TAG #https://docs.earthly.dev/docs/earthfile/builtin-args
ARG OS_RELEASE=$(md5sum /etc/os-release | cut -d ' ' -f 1)
ARG cache_prefix="${EARTHLY_TARGET_PROJECT_NO_TAG}#${OS_RELEASE}#earthly-cargo-cache"
ENV EARTHLY_CACHE_PREFIX=$cache_prefix

# $EARTHLY_KEEP_FINGERPRINTS
ARG keep_fingerprints=false
ENV EARTHLY_KEEP_FINGERPRINTS=$keep_fingerprints

# $EARTHLY_SWEEP_DAYS
ARG sweep_days=4
ENV EARTHLY_SWEEP_DAYS=$sweep_days
Expand Down Expand Up @@ -56,16 +52,18 @@ CARGO:
ARG --required args
ARG output
DO +SET_CACHE_MOUNTS_ENV
IF [ "$EARTHLY_KEEP_FINGERPRINTS" = "false" ]
DO +REMOVE_SOURCE_FINGERPRINTS
END
RUN --mount=$EARTHLY_RUST_CARGO_HOME_CACHE --mount=$EARTHLY_RUST_TARGET_CACHE \
set -e; \
echo "+CARGO: cargo $args"; \
cargo $args; \
echo "+CARGO: sweeping target cache"; \
cargo sweep -r -t $EARTHLY_SWEEP_DAYS; \
cargo sweep -r -i; \
$EARTHLY_FUNCTIONS_HOME/copy-output.sh "$output";
RUN $EARTHLY_FUNCTIONS_HOME/rename-output.sh
echo "+CARGO: copying output"; \
$EARTHLY_FUNCTIONS_HOME/copy-output.sh "$output"; \
echo "+CARGO: removing local crates from target cache"; \
cargo metadata --format-version=1 --no-deps | /tmp/jq -r '.packages[].name' | xargs -I{} cargo clean -p {};
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The call to cargo clean needs to match the profile used by cargo $args e.g. needs to include --release or --profile=foo as needed. Or I need to figure out a way to say "all profiles".

I'll have a think.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't find a way to get a list of interesting profiles to clean. Other than "ask the user to supply it" or something ugly involving inspecting $args I can't currently see a way to make this approach work.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is further complicated by tools such as cargo llvm-cov which overrides target dir (to target/llvm-cov-target/).

RUN $EARTHLY_FUNCTIONS_HOME/rename-output.sh "$output"

# SET_CACHE_MOUNTS_ENV sets the following entries in the environment, to be used to mount the cargo caches.
# - EARTHLY_RUST_CARGO_HOME_CACHE: Code of the mount cache for the cargo home.
Expand Down Expand Up @@ -135,15 +133,6 @@ get-cross:
RUN wget -nv -O- "https://github.com/cross-rs/cross/releases/download/v${version}/cross-x86_64-unknown-linux-musl.tar.gz" | tar -xzf - -C .
SAVE ARTIFACT cross

get-tomljson:
FROM alpine:3.18.3
ARG USERARCH
ARG version=2.1.0
RUN wget -O tomljson.tar.xz https://github.com/pelletier/go-toml/releases/download/v${version}/tomljson_${version}_linux_${USERARCH}.tar.xz && \
tar -xf tomljson.tar.xz; \
chmod +x tomljson
SAVE ARTIFACT tomljson

get-jq:
FROM alpine:3.18.3
ARG USERARCH
Expand Down Expand Up @@ -177,36 +166,18 @@ INSTALL_EARTHLY_FUNCTIONS:
chmod +x $EARTHLY_FUNCTIONS_HOME/copy-output.sh; \
# rename-output.sh moves files back from $OUTPUT_TMP_FOLDER to ./target
# this function is expected to be called from a build context with ./target not belonging to a shared cache
echo "mkdir -p target;
if [ \"\$(find \"$OUTPUT_TMP_FOLDER\" -type f -printf . | wc -c)\" -eq 0 ]; then
echo \"no files found within ./target matching the provided output regexp\";
else
cp -ruT \"$OUTPUT_TMP_FOLDER\" target;
rm -rf \"$OUTPUT_TMP_FOLDER\";
fi;" > $EARTHLY_FUNCTIONS_HOME/rename-output.sh; \
echo "if [ -n \"\$1\" ]; then
mkdir -p target;
if [ \"\$(find \"$OUTPUT_TMP_FOLDER\" -type f -printf . | wc -c)\" -eq 0 ]; then
echo \"no files found within ./target matching the provided output regexp\";
else
cp -ruT \"$OUTPUT_TMP_FOLDER\" target;
rm -rf \"$OUTPUT_TMP_FOLDER\";
fi;
fi;" > $EARTHLY_FUNCTIONS_HOME/rename-output.sh; \
chmod +x $EARTHLY_FUNCTIONS_HOME/rename-output.sh; \
fi;

REMOVE_SOURCE_FINGERPRINTS:
FUNCTION
DO +CHECK_INITED
COPY +get-tomljson/tomljson /tmp/tomljson
COPY +get-jq/jq /tmp/jq
RUN if [ ! -n "$EARTHLY_RUST_TARGET_CACHE" ]; then \
echo "+SET_CACHE_MOUNTS_ENV has not been called yet in this build environment" ; \
exit 1; \
fi;
RUN --mount=$EARTHLY_RUST_TARGET_CACHE \
set -e;\
source_libs=$(find . -name Cargo.toml -exec bash -c '/tmp/tomljson {} | /tmp/jq -r .package.name; printf "\n"' \;) ; \
fingerprint_folders=$(find target -name .fingerprint) ; \
for fingerprint_folder in $fingerprint_folders; do \
cd $fingerprint_folder; \
for source_lib in $source_libs; do \
find . -maxdepth 1 -regex "\./$source_lib-[^-]+" -exec bash -c 'echo "deleting $(readlink -f {})"; rm -rf {}' \; ; \
done \
done;

CHECK_INITED:
FUNCTION
RUN if [ ! -n "$EARTHLY_CACHE_PREFIX" ]; then \
Expand Down
6 changes: 0 additions & 6 deletions rust/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,6 @@ DO rust+INIT ...
Overrides cache prefix for cache IDS. Its value is exported to the build environment under the entry: `$EARTHLY_CACHE_PREFIX`.
By default `${EARTHLY_TARGET_PROJECT_NO_TAG}#${OS_RELEASE}#earthly-cargo-cache`

#### `keep_fingerprints (false)`

By default `+CARGO` removes the [compiler fingerprints](https://doc.rust-lang.org/nightly/nightly-rustc/cargo/core/compiler/fingerprint/struct.Fingerprint.html) of those packages found in your source code (not their dependencies), to force their recompilation and work even when the Earthly `COPY` commands overwrote file mtimes (by default).

Set `keep_fingerprints=true` to keep the source packages fingerprints and avoid their recompilation, when source packages have been copied with `--keep-ts `option.

#### `sweep_days (4)`
`+CARGO` calls use cargo-sweep to clean build artifacts that haven't been accessed for this number of days.

Expand Down
Loading