Upload CI generated fuzz corpus coverage to codecov #4153

Anyitechs · 2025-10-10T12:39:12Z

Following the work (#3718 and #3925) that introduced uploading coverage from no-corpus fuzzing runs into codecov in CI. This PR focuses on uploading the CI-generated fuzz corpus coverage into codecov in CI.

Closes #3926

ldk-reviews-bot · 2025-10-10T12:39:15Z

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

codecov · 2025-10-10T13:41:43Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.28%. Comparing base (c1bca16) to head (5807852).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4153      +/-   ##
==========================================
+ Coverage   88.85%   89.28%   +0.42%     
==========================================
  Files         180      180              
  Lines      137901   137901              
  Branches   137901   137901              
==========================================
+ Hits       122537   123125     +588     
+ Misses      12552    12173     -379     
+ Partials     2812     2603     -209

Flag	Coverage Δ
fuzzing	`32.87% <ø> (+11.43%)`	⬆️
tests	`88.71% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

TheBlueMatt · 2025-10-14T13:03:38Z

contrib/generate_fuzz_coverage.sh

+        for target_dir in hfuzz_workspace/*; do
+            [ -d "$target_dir" ] || continue
+            src_name="$(basename "$target_dir")"
+            for dest in "$src_name" "${src_name%_target}"; do


I don't think you need to copy into $src_name.

TheBlueMatt · 2025-10-14T13:04:38Z

contrib/generate_fuzz_coverage.sh

+                mkdir -p "test_cases/$dest"
+                # Copy corpus files into the test_cases directory
+                find "$target_dir" -maxdepth 2 -type f \
+                \( -path "$target_dir/CORPUS/*" -o -path "$target_dir/INPUT/*" -o -path "$target_dir/NEW/*" -o -path "$target_dir/input/*" \) \


Because we're just looking in hfuzz_workspace, I believe we only need to look in input, not CORPUS, INPUT, or NEW.

TheBlueMatt · 2025-10-14T13:09:51Z

.github/workflows/build.yml

          cargo clean
      - name: Run fuzzers
        run: cd fuzz && ./ci-fuzz.sh && cd ..
+      - name: Upload honggfuzz corpus


Rather than only uploading, is there a way to make this directory persistent so that we can keep it between fuzz jobs?

I'm not sure if we really need to persist the directory here. My understanding is that the fuzz job runs on the latest code changes on every PR, so the generated corpus is tailored to the code changes on that PR. If we persist the corpus from a previous run and use that on a new run, won't that produce incorrect/misleading coverage data?

I don't think the point of the fuzz job is only to generate coverage data, but rather test the code :). Having a bit more coverage data from fuzzing than we "deserve" is okay, at least now that we split the coverage data out so that codecov shows fuzzing separately, and having persistent fuzzing corpus means our fuzzing is much more likely to catch issues.

Right, how long do you think we can have this directory persisted? The upload-artifact action have a retention-days input that can be used to persist the artifact for a while. The default is 90 days but can be adjusted (https://github.com/actions/upload-artifact?tab=readme-ov-file#retention-period).

I believe the simple "upload-artifact" task just stores data for this CI run. What I was thinking is some kind of persistent directory that's shared across jobs so that each CI fuzz task picks up the latest directory, does some fuzzing, finds new test cases, then uploads a new copy with more tests in it.

What I was thinking is some kind of persistent directory that's shared across jobs so that each CI fuzz task picks up the latest directory, does some fuzzing, finds new test cases, then uploads a new copy with more tests in it.

Makes sense. I pushed eea2e4b to handle this using Github's cache action (https://github.com/actions/cache?tab=readme-ov-file).

TheBlueMatt · 2025-10-14T13:10:16Z

contrib/generate_fuzz_coverage.sh

+                # Copy corpus files into the test_cases directory
+                find "$target_dir" -maxdepth 2 -type f \
+                \( -path "$target_dir/CORPUS/*" -o -path "$target_dir/INPUT/*" -o -path "$target_dir/NEW/*" -o -path "$target_dir/input/*" \) \
+                -print0 | xargs -0 -I{} cp -n {} "test_cases/$dest/" 2>/dev/null || true


Suggested change

-print0 | xargs -0 -I{} cp -n {} "test_cases/$dest/" 2>/dev/null || true

-print0 | xargs -0 -I{} cp -n {} "test_cases/$dest/"

Done. Thank you.

TheBlueMatt · 2025-10-14T13:10:33Z

contrib/generate_fuzz_coverage.sh

+        done
+        # Check if any files were actually imported
+        if [ -n "$(find test_cases -type f -print -quit 2>/dev/null)" ]; then
+            imported=1


Not sure its worth the extra effort just to print differently.

ldk-reviews-bot · 2025-10-14T13:10:39Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

Anyitechs · 2025-10-14T17:28:59Z

Thank you for the review.

I've addressed all feedbacks and pushed a fixup here 1e4a7c5

TheBlueMatt

Responded at #4153 (comment)

Anyitechs · 2025-10-27T16:41:12Z

I rebased on main and that pulled in a dependency update to proptest 1.9.0 which has broken the 1.75.0 MSRV check. This seems unrelated to my changes, but CI is failing because of that.

EDIT: This seems to be blocking the build (and fuzzing as well).

TheBlueMatt · 2025-10-28T01:29:40Z

Yea, sorry, CI is kinda a mess for three reasons all at once. Can you rebase on #4179? That should get at least the fuzz job running again, even if not others.

Anyitechs · 2025-10-28T09:57:16Z

Can you rebase on #4179? That should get at least the fuzz job running again, even if not others.

Yes, done! Thank you.

tnull · 2025-10-28T10:46:24Z

FWIW, remaining CI failures should be resolved shortly by #4180

TheBlueMatt · 2025-10-28T12:43:08Z

.github/workflows/build.yml

+        uses: actions/cache@v4
+        with:
+          path: fuzz/hfuzz_workspace
+          key: fuzz-corpus-${{ github.ref }}-${{ github.sha }}


Isn't this going to be per-pr? We don't want it to be per-pr we want it to be global.

Isn't this going to be per-pr? We don't want it to be per-pr we want it to be global.

Addressed this by adding a two-step logic to the workflow: a read-only per-pr step that seeds the fuzzer for a more effective run on PRs and a main branch step that does same but also writes to the global cache.

TheBlueMatt · 2025-10-29T12:39:44Z

.github/workflows/build.yml

+        uses: actions/cache@v4
+        with:
+          path: fuzz/hfuzz_workspace
+          key: fuzz-corpus-refs/heads/main-${{ github.sha }}


Where do we save to fuzz-corpus-refs/heads/main-? this includes the sha.

Where do we save to fuzz-corpus-refs/heads/main-?

No, the key isn't a save location but an identifier used to save and search for a cache.

this includes the sha.

Yes. Because we can't mutate an already existing cache, but still need to ensure new updated corpus are cached. The sha ensures the cache is unique.

No, the key isn't a save location but an identifier used to save and search for a cache.

Wait, this statement contradicted itself?

Yes. Because we can't mutate an already existing cache, but still need to ensure new updated corpus are cached. The sha ensures the cache is unique.

But how does the read end know where to look for it?

Wait, this statement contradicted itself?

Right, generally, the key is used to save and search. But the idea here is to rely on the key to save the updated corpus uniquely (since we can't mutate an already existing cache) and use restore-keys to search.

But how does the read end know where to look for it?

The restore-keys does a prefix search and restores/downloads the closest matching cache that was recently created, when there's a miss on key.

For example, when this runs the first time and the corpus gets saved as fuzz-corpus-refs/heads/main-sha123, on the second run the key becomes fuzz-corpus-refs/heads/main-sha456 and will miss, so it falls back to the restore-keys to restore the most recent cache with the prefix fuzz-corpus-refs/heads/main-. That will download fuzz-corpus-refs/heads/main-sha123, the run will use that, updates it and save it as fuzz-corpus-refs/heads/main-sha456, and the loop continues.

Ohhhhhh, okay, i wasn't clear that it does a prefix search, can you add a comment noting that? Otherwise LGTM!

Sure! I just updated the comment and pushed 5edb8cf.

Let me know when I can squash my second fixup commit into my first commit.

TheBlueMatt · 2025-10-29T12:40:13Z

.github/workflows/build.yml

+        with:
+          path: fuzz/hfuzz_workspace
+          key: fuzz-corpus-refs/heads/main-${{ github.sha }}
+          restore-keys: |


Presumably when running on main we don't need the restore-keys trick?

Presumably when running on main we don't need the restore-keys trick?

No, we still do. Because the save key includes the sha so it will always miss on restore, but the restore-keys will help restore the matching most recent cache.

Right, but then where do we save in a way that something knows where to find it?

The restore-keys does a prefix search and restores the most recently created cache with the prefix provided.

TheBlueMatt · 2025-10-31T16:08:02Z

Cool! Yea, this LGTM, we'll obv have to land it to fully test it. Feel free to squash the fixup commits down, and given the fuzz CI task appears to be passing now (???) probably worth rebasing on current git and dropping the dependency on the commits from #4179.

When you do so, please add some linebreaks to the commit message so that no line is longer than ~70 chars.

Because each CI job runs on a fresh runner and can't share data between jobs. We rely on Github Actions upload-artifact and download-artifact to share the CI generated fuzz corpus, then replay them in the `contrib/generate_fuzz_coverage.sh` script to generate the coverage report.

Implements a persistent, global fuzz corpus cache. PRs perform a "read-only" restore from the `main` cache to seed fuzzer runs. The `main` branch performs a "read-write" to save new findings and grow the corpus.

Anyitechs · 2025-10-31T17:27:36Z

Cool! Yea, this LGTM, we'll obv have to land it to fully test it. Feel free to squash the fixup commits down, and given the fuzz CI task appears to be passing now (???) probably worth rebasing on current git and dropping the dependency on the commits from #4179.

When you do so, please add some linebreaks to the commit message so that no line is longer than ~70 chars.

Done! I've rebased onto main, squashed the fixup, and formatted both commit messages to the ~70 chars limit. Thanks for all the guidance!

TheBlueMatt

Awesome! Thanks so much.

ldk-reviews-bot requested a review from tankyleo October 10, 2025 12:49

Anyitechs marked this pull request as draft October 10, 2025 18:42

Anyitechs force-pushed the upload-fuzz-coverage branch from dc493c2 to fdf6799 Compare October 13, 2025 00:25

Anyitechs marked this pull request as ready for review October 13, 2025 01:52

tankyleo requested review from TheBlueMatt and removed request for tankyleo October 13, 2025 22:16

TheBlueMatt reviewed Oct 14, 2025

View reviewed changes

Anyitechs requested a review from TheBlueMatt October 23, 2025 12:07

TheBlueMatt reviewed Oct 24, 2025

View reviewed changes

Anyitechs force-pushed the upload-fuzz-coverage branch 2 times, most recently from 19c1495 to eea2e4b Compare October 27, 2025 16:17

Anyitechs requested a review from TheBlueMatt October 27, 2025 16:42

Anyitechs force-pushed the upload-fuzz-coverage branch from eea2e4b to de9e1fd Compare October 28, 2025 08:55

TheBlueMatt reviewed Oct 28, 2025

View reviewed changes

Anyitechs force-pushed the upload-fuzz-coverage branch 2 times, most recently from fcba095 to 6cd3f8f Compare October 28, 2025 17:10

Anyitechs requested a review from TheBlueMatt October 28, 2025 19:24

TheBlueMatt reviewed Oct 29, 2025

View reviewed changes

Anyitechs force-pushed the upload-fuzz-coverage branch from 6cd3f8f to 5edb8cf Compare October 30, 2025 15:25

Anyitechs added 2 commits October 31, 2025 17:46

Persist fuzz corpus between CI runs

5807852

Implements a persistent, global fuzz corpus cache. PRs perform a "read-only" restore from the `main` cache to seed fuzzer runs. The `main` branch performs a "read-write" to save new findings and grow the corpus.

Anyitechs force-pushed the upload-fuzz-coverage branch from 5edb8cf to 5807852 Compare October 31, 2025 17:12

TheBlueMatt approved these changes Nov 3, 2025

View reviewed changes

TheBlueMatt merged commit cd25ef1 into lightningdevkit:main Nov 3, 2025
23 of 25 checks passed

	-print0 \| xargs -0 -I{} cp -n {} "test_cases/$dest/" 2>/dev/null \|\| true
	-print0 \| xargs -0 -I{} cp -n {} "test_cases/$dest/"

Upload CI generated fuzz corpus coverage to codecov #4153

Upload CI generated fuzz corpus coverage to codecov #4153

Uh oh!

Conversation

Anyitechs commented Oct 10, 2025

Uh oh!

ldk-reviews-bot commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ldk-reviews-bot commented Oct 14, 2025

Uh oh!

Anyitechs commented Oct 14, 2025

Uh oh!

TheBlueMatt left a comment

Choose a reason for hiding this comment

Uh oh!

Anyitechs commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TheBlueMatt commented Oct 28, 2025

Uh oh!

Anyitechs commented Oct 28, 2025

Uh oh!

tnull commented Oct 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TheBlueMatt commented Oct 31, 2025

Uh oh!

Anyitechs commented Oct 31, 2025

Uh oh!

TheBlueMatt left a comment

ldk-reviews-bot commented Oct 10, 2025 •

edited

Loading

codecov bot commented Oct 10, 2025 •

edited

Loading

Anyitechs commented Oct 27, 2025 •

edited

Loading