-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add apt2aptly, parallel package uploads #9
Commits on Jul 26, 2023
-
Add new tool, apt2aptly, built on new crate sync2aptly
This adds new tool, apt2aptly, to mirror an existing apt repository in aptly. To reuse obs2aptly's core logic, that was pulled out into a shared crate, sync2aptly. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Configuration menu - View commit details
-
Copy full SHA for ec29968 - Browse repository at this point
Copy the full SHA ec29968View commit details -
sync2aptly: Support uploading packages in parallel
On full runs, this can drastically reduce the runtime (which also makes testing significatly easier). Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Configuration menu - View commit details
-
Copy full SHA for 014cd95 - Browse repository at this point
Copy the full SHA 014cd95View commit details -
apt2aptly: Read the repository from the network, not a local path
This requires refactoring in sync2aptly to generalize the location of files, as well as backoff integration to retry on download / upload failure. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Configuration menu - View commit details
-
Copy full SHA for 5d4c0af - Browse repository at this point
Copy the full SHA 5d4c0afView commit details -
apt2aptly: Split one apt repo into per-component aptly repos
This is how our actual layout on OBS is, so it needs to be matched here. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Configuration menu - View commit details
-
Copy full SHA for e9b1013 - Browse repository at this point
Copy the full SHA e9b1013View commit details
Commits on Jul 27, 2023
-
Move all sync functionality exclusively into sync2aptly
obs2aptly keeps its version computations, because sync2aptly is generic over URLs and files, which makes moving it harder, and other origin sources like apt2aptly can more easily obtain the versions anyway. Unfortunately, the version can't just be pre-computed while gathering the origin content; because that would result in a rather large amount of version lookups in contexts where we currently don't need to. To work around this, the version—and the function used to compute it—are stored in a OnceCell + Box<dyn Fn> pair. *Technically* the caching isn't entirely needed, but while reworking all of this code anyway, might as well also avoid the reduntant version parsing. Additionally, analyzing arch-independent packages switches from using (source package, source version) pairs to (source package, package version) pairs. This should have the equivalent effect of deduplicating the packages for each architecture without relying on the source version, which is hard to obtain from apt2aptly. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Configuration menu - View commit details
-
Copy full SHA for e4f63bd - Browse repository at this point
Copy the full SHA e4f63bdView commit details -
apt2aptly: Process a full list of snapshots
--apt-snapshot has been replaced with --apt-snapshots, which will point to a file containing one snapshot to process per line. Already-existing snapshots will now result in skipping the entire scan process, which required some refactoring to get out a component list without also doing a full scan. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Configuration menu - View commit details
-
Copy full SHA for 87dcc60 - Browse repository at this point
Copy the full SHA 87dcc60View commit details
Commits on Jul 28, 2023
-
apt2aptly: Refactor template parsing
This cleans up the code a bit and improves the parallel structure. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Configuration menu - View commit details
-
Copy full SHA for 7f66e24 - Browse repository at this point
Copy the full SHA 7f66e24View commit details -
sync2aptly: Switch a use of reduce() to max_by_key()
It's a bit clearer as to the actual purpose of the code. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Configuration menu - View commit details
-
Copy full SHA for 93c3c35 - Browse repository at this point
Copy the full SHA 93c3c35View commit details -
apt2aptly: Add support for publishing repos & snapshots
This adds a new option, --publish-to, to publish to the given prefix, as well as additional options to control signing and re-publishing snapshots. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Configuration menu - View commit details
-
Copy full SHA for bd47c0b - Browse repository at this point
Copy the full SHA bd47c0bView commit details
Commits on Aug 1, 2023
-
apt2aptly: Add a flag to create aptly repos
This is useful because the tool already knows the repo names to use. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Configuration menu - View commit details
-
Copy full SHA for 759f121 - Browse repository at this point
Copy the full SHA 759f121View commit details -
apt2aptly: Log release file paths when syncing
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Configuration menu - View commit details
-
Copy full SHA for a2af8b2 - Browse repository at this point
Copy the full SHA a2af8b2View commit details -
apt2aptly: Fix hash conflict errors when importing .dsc files
aptly sorts the filenames before hashing them, but the .dsc files coming from reprepro often aren't sorted. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Configuration menu - View commit details
-
Copy full SHA for f2b851e - Browse repository at this point
Copy the full SHA f2b851eView commit details -
apt2aptly: Skip Contents & bz2 on publish
We already do this in the normal OBS integration for performance reasons. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Configuration menu - View commit details
-
Copy full SHA for beb442e - Browse repository at this point
Copy the full SHA beb442eView commit details