Commit Graph
100 Commits
Author SHA1 Message Date
Justin Chadwell 48d7dafbd5 git: update gitutil test utilities
- Adds a new GitServeHTTP function to start an http server to serve a
  target git repository.
- Adds a new GitDir helper method to get the path to the .git
  directory
- Updates the GitAdd method to take a variable number of files

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-06-06 17:15:19 +02:00
Justin Chadwell d03e93f6f1 test: tmpdir can be a test helper
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-06-06 17:15:19 +02:00
Justin ChadwellandGitHub 7cef021a8a Merge pull request #1804 from jedevc/fixup-solve 2023-06-06 09:12:10 +02:00
Justin Chadwell c6db4cf342 build: clarify NewResultHandle comment
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-06-06 09:08:30 +02:00
Justin ChadwellandGitHub a906149930 Merge pull request #1862 from jedevc/update-contributing-tests
docs: update CONTRIBUTING.md with specific test notes
2023-06-01 21:55:12 +01:00
Justin Chadwell af328fe413 docs: update CONTRIBUTING.md with specific test notes
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-06-01 11:34:20 +01:00
Justin Chadwell b7f0b3d763 build: clear exports for secondary solve request
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-31 15:13:56 +01:00
Justin Chadwell 5b27d5a9f6 build: cleanup res if returned in basic build
In practice, this shouldn't happen, but the check is good to include
anyways.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-31 15:13:03 +01:00
Justin Chadwell cd1648192e build: rename ResultContext to ResultHandle
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-31 09:50:46 +01:00
Justin Chadwell 8d822fb06c build: move main solve request into main gateway call
Now, we always perform the full solve request in the main gateway call.
This ensures that progress works properly, and makes the lifetime
semantics much clearer.

NewResultContext abstracts the details of a successful/failed build, to
always return a single ResultContext, even though the details of how a
gateway is created is different:
- For a failed build, we can just keep the gateway open.
- For a successful build, we immediately open another gateway and
  re-evaluate the build definition in that gateway. This should give an
  instant cache hit (since the build was just successful).

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-31 09:46:46 +01:00
Justin ChadwellandGitHub 5e2f8bd64a Merge pull request #1853 from jedevc/logrus-show-error 2023-05-30 10:29:49 +01:00
Justin Chadwell 5788ab33d2 logutil: print error from WithError if found
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-30 10:00:24 +01:00
Justin Chadwell 6c62225d1b vendor: update buildkit to master@c36941f4a10e
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-30 09:10:52 +01:00
Justin Chadwell 2dae553d18 build: update outline fallback image
Subrequests have been included in docker/dockerfile:1.5 labs, so we can
update the fallback to point to this release.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-26 10:38:38 +01:00
Justin Chadwell 91c17f25fb build: tidy up print func
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-26 10:38:38 +01:00
Justin ChadwellandGitHub 354ccc9469 Merge pull request #1844 from tonistiigi/imageid-fixes 2023-05-26 10:21:53 +01:00
Justin ChadwellandGitHub 341fb65f6f Merge pull request #1830 from jedevc/fix-missing-driver-in-build 2023-05-25 11:35:01 +01:00
Justin ChadwellandGitHub 69a9c6609a Merge pull request #1699 from jedevc/bake-attestation-override 2023-05-25 09:54:34 +01:00
Justin Chadwell c77bd8a578 build: fix missing "docker" driver name in build progress
This was missing, since the driver property can only be fully populated
after loading nodes from disk. So we add logic to load the nodes, and
check for an error, which ensures that the "docker" driver is always
correctly present in the progress description.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-23 10:37:22 +01:00
Justin ChadwellandGitHub e5f701351c Merge pull request #1829 from jedevc/vendor-buildkit-master 2023-05-23 10:23:24 +01:00
Justin Chadwell 09798cdebd vendor: update buildkit to master@212ab16a39b1
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-23 10:16:09 +01:00
Justin ChadwellandGitHub 7e39644f69 Merge pull request #1816 from bpaquet/fix_session_token 2023-05-22 09:34:07 +01:00
Justin Chadwell f558fd8b22 vendor: use docker/distribution instead of distribution/distribution
This doesn't require pulling in a new direct dependency, we already use
docker/distribution throughout imagetools and build.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-22 09:24:41 +01:00
Justin ChadwellandGitHub 432e16ef70 Merge pull request #1806 from thaJeztah/update_engine2 2023-05-22 09:19:35 +01:00
Justin ChadwellandGitHub 4429ccbcc2 Merge pull request #1817 from jedevc/docs-monitor-invoke-parts 2023-05-19 16:40:47 +01:00
Justin ChadwellandGitHub c59fc18325 Merge pull request #1770 from jedevc/integration-tests 2023-05-19 16:18:31 +01:00
Justin Chadwell 4ce80856b3 docs: add info for on-error and debug-shell values for invoke
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-19 14:51:51 +01:00
Justin Chadwell af3feec4ea docs: fixup typo in buildkit console output
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-19 14:51:30 +01:00
Justin Chadwell 90c849f5ef bake: merge attest entries
This ensures that `target.attest=["type=sbom,<value>"]` can be
appropriately merged when `--sbom=true` or `--set
target.attest=type=sbom`.

To merge, we simply naively take the last valid value.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-19 14:32:58 +01:00
Justin Chadwell 6024212ac8 ci: add integration tests
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-18 14:26:12 +01:00
Justin Chadwell 2d124e0ce9 test: add basic integration tests
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-18 14:26:12 +01:00
Justin Chadwell e61a8cf637 vendor: update buildkit to master@31c870e82a48
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-18 14:25:31 +01:00
Justin ChadwellandGitHub 167cd16acb Merge pull request #1797 from crazy-max/fix-image-create 2023-05-18 09:06:45 +01:00
Justin ChadwellandGitHub 060ac842bb Merge pull request #1794 from jedevc/fix-printer-start-race 2023-05-12 09:43:58 +01:00
Justin ChadwellandGitHub 31d1b778ff Merge pull request #1795 from thaJeztah/vendor_distribution_v2.8.2 2023-05-12 09:43:07 +01:00
Justin ChadwellandGitHub 10aea8e970 Merge pull request #1791 from jedevc/docs/bake-reference-matrix
Document new bake matrix builds
2023-05-11 15:43:49 +01:00
Justin Chadwell be6542911f progress: make printer ready only after pausing logrus
This fixes a possible race where messages printed directly after calls
to NewPrinter may appear before the printer starts. With this change, we
delay all of the logs until after.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-11 15:23:41 +01:00
Justin Chadwell 9ee19520dd controller: move path resolution into controller package
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-11 13:56:05 +01:00
Justin ChadwellandDavid Karlsson <david.karlsson@docker.com 878faae332 docs: add matrix docs for bake
Co-authored-by: David Karlsson <david.karlsson@docker.com
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-11 13:44:28 +01:00
Justin Chadwell eaf38570e7 github: don't auto-add kind/bug labels
kind labels should only be manually added by maintainers after
confirming the issue. Otherwise, it can be tricky to sort through all
the issues when filtering for legitimate bugs.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-11 13:43:25 +01:00
Justin ChadwellandGitHub e61a1da7fc Merge pull request #1748 from dvdksn/docs/bake-reference
docs: move and rewrite bake reference
2023-05-11 13:00:21 +01:00
Justin ChadwellandGitHub 2c8a9aad76 Merge pull request #1782 from jedevc/docs-monitor
docs: add initial guide about debugging monitor mode
2023-05-11 11:29:21 +01:00
Justin ChadwellandGitHub 32009a701c Merge pull request #1786 from jedevc/debug-multi-platform
debug: evaluate all refs to allow multi-platform debugging
2023-05-11 09:54:13 +01:00
Justin Chadwell 0cbc316f76 docs: add initial guide about debugging monitor mode
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-11 09:53:21 +01:00
Justin Chadwell 45fccef3f3 debug: evaluate all refs to allow multi-platform debugging
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-11 09:44:50 +01:00
Justin ChadwellandGitHub fdcb4e2fb9 Merge pull request #1784 from jedevc/support-new-compose-features
Support new compose features in bake
2023-05-11 09:44:02 +01:00
Justin ChadwellandGitHub 4a0a67d7a2 Merge pull request #1783 from jedevc/fix-bake-sub-dir-context
bake: copy remote context contents to root
2023-05-11 09:40:25 +01:00
Justin ChadwellandGitHub 2eeef180ea Merge pull request #1746 from ktock/resultcleanup
controller: avoid "context canceled" error on cleanup
2023-05-10 13:24:12 +01:00
Justin ChadwellandGitHub 1eb9ad979e Merge pull request #1785 from jedevc/default-to-local-controller
controller: default to using local controller
2023-05-10 11:18:55 +01:00
Justin Chadwell 77e0e860f8 controller: default to using local controller
--detach shouldn't be the default yet, since it still has the potential
to leak some sessions in odd edge cases, and is slightly more painful to
debug.

For now, we should set the local controller as the default, with the
idea that we can change it back in the future.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-10 09:51:58 +01:00
Justin Chadwell e228c398f4 bake: support compose additional contexts
compose-go v1.13.0 supports the new additional_contexts to allow passing
additional build context during build, so we should map this to bake's
contexts property.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-10 09:41:37 +01:00
Justin Chadwell 5d06406f26 bake: support compose inline dockerfile
compose-go v1.13.0 supports the new dockerfile_inline to allow including
a dockerfile inline in the compose definition, so we should map this to
bake's dockerfile-inline property.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-10 09:39:00 +01:00
Justin ChadwellandGitHub cb061b684c Merge pull request #1753 from nicksieger/compose-go-v1.13.4
compose go v1.13.4
2023-05-10 09:25:10 +01:00
Justin Chadwell 12b6a3ad9a bake: copy remote context contents to root
When resolving remote contexts locally in bake, then we need to ensure
that we properly unpack the contents of that context to the root
directory, instead of leaving it in the subdirectory.

Otherwise, any files will be found in the wrong location. Along with
this change, we also need a change to the dockerfile location lookup to
ensure that it is found at the root instead of in the subdirectory.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-09 17:19:26 +01:00
Justin Chadwell 22e1901581 bake: reorganize input creation together
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-09 17:15:44 +01:00
Justin ChadwellandGitHub e23c37fa96 Merge pull request #1771 from jedevc/unset-other-build-client-opt
debug: secondary client build should not reuse solve opts
2023-05-09 14:57:23 +01:00
Justin Chadwell e5a0ed1149 debug: secondary client build should not reuse solve opt ref
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-09 14:23:41 +01:00
Justin ChadwellandGitHub c9c1303e31 Merge pull request #1777 from ktock/fixerrormsg
monitor: improve error messages
2023-05-09 13:49:07 +01:00
Justin ChadwellandGitHub ae3299d9d4 Merge pull request #1628 from cpuguy83/policy_file
Support for passing through raw buildkit policies
2023-05-09 11:50:10 +01:00
Justin ChadwellandGitHub 7ad970f93a Merge pull request #1776 from tonistiigi/fix-conn-leak
docker: fix connection leak on features check
2023-05-09 09:52:18 +01:00
Justin ChadwellandGitHub c7c37c3591 Merge pull request #1760 from ktock/monitor-commands
monitor: Move commands to a separated package
2023-05-03 14:33:08 +01:00
Justin ChadwellandKohei Tokunaga a43837d26c monitor: extend monitor interface from controller
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-03 22:08:30 +09:00
Justin ChadwellandGitHub 43a07f3997 Merge pull request #1597 from robertlestak/k8s-sa
kubernetes driver: add serviceaccount opt
2023-05-02 12:56:07 +01:00
Justin ChadwellandGitHub 9d5af461b2 Merge pull request #1755 from jedevc/move-across-controller-boundary
Move metadata and subrequest output outside of controller
2023-04-27 11:15:54 +01:00
Justin Chadwell 6bec8f6e00 docs: add serviceaccount driver-opt for k8s driver
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-26 16:44:03 +01:00
Justin Chadwell ba92989a94 controller: print result outside of controller
This will allow result printing to work with the remote controller
(though this currently causes a panic, to be fixed in a follow-up).

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-26 16:34:36 +01:00
Justin Chadwell 2bf996d9ad controller: don't write metadata file in controller
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-26 16:34:36 +01:00
Justin ChadwellandGitHub 75ed3e296b Merge pull request #1737 from jedevc/improved-controller-progress
controller: refactor progress api
2023-04-26 10:17:00 +01:00
Justin ChadwellandCrazyMax 784dc2223d lint: add empty line between console and json code blocks
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-24 12:05:03 +02:00
Justin Chadwell 16d5b38f2b debug: display build warnings after each build
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-24 09:58:02 +01:00
Justin ChadwellandGitHub afcaa8df5f Merge pull request #1752 from nicksieger/compose-go-default-filenames
bake: use default filenames from compose-go
2023-04-21 17:28:32 +01:00
Justin Chadwell 2ab8749052 controller: replace logrus status messages with progress messages
logrus info messages aren't particularly in-theme with the rest of the
progress output (and are also frustratingly racy). The progress output
is a lot neater, so we refactor it into that.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-21 12:33:04 +01:00
Justin Chadwell e826141af4 controller: refactor progress api
Refactor the progress printer creation to the caller-side of the
controller api. Then, instead of passing around status channels (and
progressMode strings), we can simply pass around the higher level
interface progress.Writer.

This has a couple of benefits:
- A simplified interface to the controller
- Allows us to correctly extract warnings out of the controller, so that
  they can be displayed correctly from the client side.

Some extra work is required to make sure that we can pass a
progress.Printer into the debug monitor. If we want to keep it
persistent, then we need a way to temporarily suspend output from it,
otherwise it will continue printing as the monitor is prompting for
input from the user, and forwarding output from debug containers.

To handle this, we add two methods to the printer, `Pause` and
`Unpause`. `Pause` acts similarly to `Wait`, closing the printer, and
cleanly shutting down the display - however, the printer does not
terminate, and can later be resumed by a call to `Unpause`. This
provides a neater interface to the caller, instead of needing to
continually reconstruct printers for every single time we want to
produce progress output.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-21 11:17:43 +01:00
Justin Chadwell 0c1fd31226 build: refactor out common build command components
We had some duplicated code between the basic runBuild and
launchControllerAndRunBuild.

This patch refactors out the common logic (since it's only really like
to keep growing), and has runBuild call into either the controller or
directly start the build depending on whether BUILDX_EXPERIMENTAL is
set.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-21 11:13:58 +01:00
Justin Chadwell 0e9804901b controller: refactor status message conversion to pb package
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-21 10:59:16 +01:00
Justin Chadwell 2402607846 build: use gateway's solve context to allow cancelling getResultAt
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-21 10:52:03 +01:00
Justin ChadwellandGitHub de4cdab411 Merge pull request #1747 from jedevc/only-capture-result-for-controller
controller: only capture the result when invoked through a controller
2023-04-18 18:28:08 +01:00
Justin Chadwell a7e471b7b3 controller: only capture the result when invoked through a controller
This ensures that the code used to capture and evaluated a result is
only executed when built through the controller. Otherwise, no build
result should be recorded.

This ensures that new code added to capture and store the build result
for debugging isn't used when BUILDX_EXPERIMENTAL is not set.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-18 16:35:42 +01:00
Justin ChadwellandGitHub ba6e5cddb0 Merge pull request #1745 from jedevc/remote-unneccessary-exec-server-name
monitor: update check for exec args
2023-04-18 11:29:01 +01:00
Justin Chadwell e4ff82f864 monitor: update exec failure message when no args are passed
"exec" does not take a server name, so we can rename the error check
here.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-18 11:16:12 +01:00
Justin ChadwellandGitHub 48b733d6da Merge pull request #1740 from jedevc/resolve-paths-context-dockerfile
build: avoid resolution of dockerfile if context is remote
2023-04-18 10:54:41 +01:00
Justin ChadwellandGitHub 0b432cc5f2 Merge pull request #1640 from ktock/monitor-invoke-mode-restore
monitor: add `debug-shell` and `on-error`
2023-04-18 10:41:04 +01:00
Justin Chadwell f6cccefffc build: avoid resolution of dockerfile if context is remote
In 566f41b598, we added a check to ensure
that we avoid resolving http URLs for Dockerfile. However, we have
another circumstance we should not resolve the path in - if the context
is a remote context, the dockerfile is resolved in that context (see
build.go#LoadInputs for more information).

Therefore, we should only resolve the dockerfile to a local directory if
the context is also resolved to a local directory.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-17 11:18:49 +01:00
Justin ChadwellandGitHub b716e48926 Merge pull request #1734 from jedevc/more-ssh-context-fixes
Various context fixups
2023-04-14 09:27:36 +01:00
Justin ChadwellandGitHub 7d35a3b8d8 Merge pull request #1733 from jedevc/use-dockerui-context-detection
Use dockerui context detection
2023-04-14 09:25:07 +01:00
Justin Chadwell 566f41b598 build: avoid file resolution of dockerfile urls
Dockerfiles can be HTTP URLs as well as local paths 🤦

We just copy the same logic we use for resolving context paths, and
apply it here as well.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-12 12:11:20 +01:00
Justin Chadwell 6c0547e7e6 bake: ensure remote files behind ssh expose agent
The updateContext function may make modifications to the build inputs,
creating either an SSH URL, or an SSH llb.State. In these cases, we need
to ensure that we appropriately expose the client's default agent.

Previously, we would only expose it if the remote context was a git URL,
however, we need to also ensure that if the input was used to override
the context (in the case of ReadRemoteFiles), that we expose the agent
here as well.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-12 12:11:20 +01:00
Justin Chadwell 871f865ac8 bake: update ReadRemoteFiles to use buildkit api
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-12 11:15:25 +01:00
Justin Chadwell 62a21520ea vendor: update buildkit to master@333ee9158128
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-12 10:11:46 +01:00
Justin ChadwellandGitHub 1e149bb84f Merge pull request #1722 from jedevc/bake-matrix-check-map-type
bake: fix map type checking to detect matrix property set to list
2023-04-05 16:03:50 +01:00
Justin Chadwell 9827abbf76 bake: fix map type checking to detect matrix property set to list
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-05 13:41:03 +01:00
Justin ChadwellandGitHub a3293cdaaa Merge pull request #1720 from jedevc/bake-replace-func
Add `replace` function to bake stdlib
2023-04-05 11:59:40 +01:00
Justin ChadwellandGitHub f7d8bd2055 Merge pull request #1690 from jedevc/bake-matrix
Implement matrix for bake targets
2023-04-05 11:58:59 +01:00
Justin Chadwell 5d33a3af22 bake: add replace func to stdlib
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-05 11:48:35 +01:00
Justin Chadwell 87f900ce77 chore: sort bake stdlib map
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-05 11:48:35 +01:00
Justin ChadwellandGitHub 5c21e80a83 Merge pull request #1711 from jedevc/bake-remote-file-git
bake: forward default ssh agent for remote bake file
2023-04-04 11:50:42 +01:00
Justin ChadwellandGitHub 91fdb0423d Merge pull request #1713 from ktock/monitor-buildapi-options
controller: Extract nested CommonOptions on controller API
2023-04-03 14:48:06 +01:00
Justin ChadwellandGitHub 16e41ba297 Merge pull request #1710 from jedevc/use-buildkit-gitutil-parsegitref
Use buildkit's gitutil package to detect remote git files
2023-04-03 11:14:08 +01:00
Justin Chadwell 387ce5be7c bake: forward default ssh agent for remote bake file
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-03 10:50:43 +01:00
Justin Chadwell 87a120e8e3 bake: use build.IsRemoteURL
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-03 10:48:31 +01:00