Commit Graph
100 Commits
Author SHA1 Message Date
Jonathan A. SternbergandGitHub 8e8054f1d9 Merge pull request #3510 from tonistiigi/fix-mediatype-concurrency
imagetools: fix concurrent map write from containerd issue
2025-11-11 15:35:29 -06:00
Jonathan A. SternbergandGitHub c9b2da6ba4 Merge pull request #3513 from jsternberg/vendor-buildkit
vendor: update buildkit v0.26.0-rc2
2025-11-11 15:31:33 -06:00
Jonathan A. Sternberg 1651da02d9 vendor: update buildkit v0.26.0-rc2
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-11-11 14:14:36 -06:00
Jonathan A. Sternberg 9099a030f4 dap: attempt to find multiple digests when a failure happens
The solve error that gets returned sometimes has a platform when the LLB
returned by `ToState` doesn't. In order to ensure we find the failed
digest, we now search for both the digest of what was returned and also
clear out the platform and search for the digest without that in case
that one matches instead.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-10-29 16:08:46 -05:00
Jonathan A. Sternberg 5b8b65a59c dap: next and out now respect breakpoints
If a breakpoint occurs before the step pointed to by next or out, dap
will now stop there instead of the desired location.

This also updates the loop to always set the breakpoints rather than
only when continue is chosen.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-10-29 15:20:15 -05:00
Jonathan A. SternbergandGitHub 166ab97882 Merge pull request #3481 from rcjsuen/dap-set-breakpoints-null-fix
dap: stop sending null to conform to the specification
2025-10-24 13:43:45 -05:00
Jonathan A. Sternberg 4345a8e50b dap: refactor how steps are evaluated in the program
This change modifies how steps are evaluated in the program. Previously,
the execution relied on looking for the digest we were going to stop at
and evaluating the step right before that digest. This could result in
some gaps where it erroneously didn't execute inputs for the current
step as it skipped those digests.

Now, the evaluation reads the inputs and executed those directly rather
than relying on the evaluation of previous steps in the sequence. This
should make the evaluation of inputs more accurate and allow us to have
better breakpoints on things like the copy operation.

Due to the change, it's also easier for us to include the different
inputs in the file explorer.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-10-24 11:13:25 -05:00
Jonathan A. SternbergandGitHub abf6ab4a37 Merge pull request #3471 from jsternberg/dap-attach-standalone
dap: support run in terminal request when buildx is run in standalone mode
2025-10-17 13:07:58 -05:00
Jonathan A. Sternberg 6ac8458692 dap: support run in terminal request when buildx is run in standalone mode
The run in terminal request sent by exec would not work if `buildx` had
been invoked directly instead of through `docker`. If we don't find the
reexec environment variable, we use `os.Args[0]` to launch buildx
directly when invoking attach.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-10-15 12:03:49 -05:00
Jonathan A. SternbergandGitHub beaebcbf39 Merge pull request #3438 from thaJeztah/bump_engine
vendor: github.com/docker/docker, docker/cli v28.5.1, buildkit v0.25.1
2025-10-09 09:54:43 -05:00
Jonathan A. Sternberg ec01355b03 docs: clarify cli help for the --tag option
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-10-06 12:46:59 -05:00
Jonathan A. Sternberg b47488aea2 ci: update buildkit ref for ci
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-10-01 09:57:59 -05:00
Jonathan A. SternbergandGitHub e2534c77e3 Merge pull request #3442 from jsternberg/buildkit-vendor
vendor: update buildkit to v0.25.0
2025-09-30 14:22:56 -05:00
Jonathan A. Sternberg 3d9cc5128a vendor: update buildkit to v0.25.0
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-09-30 14:09:19 -05:00
Jonathan A. Sternberg 92ea5e64b4 dap: file explorer uses the wrong path when looking up mounts
It accidentally retrieved the relative path to the mount directory
instead of the full path which resulted in always returning the root
directory.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-09-24 09:28:38 -05:00
Jonathan A. Sternberg 39feed47ea docs: add plugin guideline requirements for dap plugins
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-09-22 12:13:15 -05:00
Jonathan A. Sternberg 17c2e1dc35 dap: collect references for inputs to ensure the file explorer works on an error
The debug adapter will now look at the error and collect the input
mounts from the exec operation and translate them to a gateway reference
to be able to access the filesystem in the file explorer.

When we are attempting to find the real operation that failed and
succeed, we now re-solve the dependencies (which should already have
been solved correctly to begin with) so that we can have the proper
reference and can access the filesystem contents.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-09-17 15:58:28 -05:00
Jonathan A. SternbergandGitHub f55ed6015b Merge pull request #3412 from crazy-max/dap-fix-data-path
dap: use slash separator when reading ref files
2025-09-08 09:42:07 -05:00
Jonathan A. Sternberg bf40c24e00 dap: improve location resolution for duplicate digests
This improves location resolution for duplicate digests by making some
assumptions about the file structure to determine a "best match" when
there are multiple possible locations.

It uses the next operation (that this digest is the input for) to
determine a location. If the location of the current operation is before
the next operation, this location is preferred. If there are multiple
locations that happen before the next operation, the one closest to the
next operation (aka later in the file) is used instead.

This resolves the most common case of multiple identical `FROM`
statements without adding more code to the frontends themselves.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-09-04 15:04:06 -05:00
Jonathan A. Sternberg 8841b2dfc8 dap: ensure test client is closed on cleanup
The dap test wasn't waiting for the client's goroutines to complete
before exiting which caused a race condition that could cause it to log
to the dead test logger. This became apparent when `--count` of greater
than one was used since it caused the test to run long enough to trigger
the behavior. It would have also triggered if we had added more tests.

Add the client close to the cleanup so it waits for the goroutine to
finish before the test exits as it was properly supposed to do.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-09-03 10:51:01 -05:00
Jonathan A. Sternberg a7c54da345 docs: update dap docs to reflect updates to the debugger
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-08-19 10:19:09 -05:00
Jonathan A. Sternberg 5c97696d64 dap: improve determination of the proper parent for certain ops
Improves the determination of the proper parent for exec and file ops.
With file ops, it will only consider inputs and ignore secondary inputs.
This prevents the following case:

```
FROM busybox AS build1
RUN echo foo > /hello

FROM scratch
COPY --from=build1 /hello .
```

Previously, `build1` would be considered the parent of the copy
instruction. Now, copy properly does not have a parent.

If there are multiple file ops and the operations disagree on the
canonical "parent", we give up on trying to find a canonical parent and
assume there is none.

For exec operations, whichever input is associated with the root mount
is considered the primary parent.

For all other operations, the first parent is considered the primary
parent if it exists.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-08-18 09:27:02 -05:00
Jonathan A. Sternberg b3c389690c dap: look for base name of dockerfile name instead of path from context
When the builder loads a dockerfile, it does it by using the base name
of the dockerfile path and only loads the innermost directory. This
means that the source name we're looking for is the base name and not
the full relative path.

Update the set breakpoints functionality so it takes this into account.
Fixes scenarios where DAP is used with a dockerfile nested in the
context.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-08-15 14:41:00 -05:00
Jonathan A. Sternberg dbda218489 dap: make exec shell persistent across the build
Invoking the shell will cause it to persist across the entire build and
to re-execute whenever the builder pauses at another location again.

This still requires using `exec` to launch the shell. Launching by frame
id is also removed since it no longer applies to this version.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-08-11 12:40:09 -05:00
Jonathan A. SternbergandGitHub 5c27294f27 Merge pull request #3327 from jsternberg/dap-fs-inspect
dap: filesystem inspection when paused on a digest
2025-08-05 11:06:16 -05:00
Jonathan A. Sternberg 8e356c3454 dap: filesystem inspection when paused on a digest
Add a file explorer to the debugger that allows exploring the filesystem
of the current container. It will show directory contents, file
contents, and symlink destinations. It will also show the file mode
associated with a file.

The file explorer defaults to marking itself as an expensive operation
so the debugger doesn't automatically retrieve the information.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-28 09:52:30 -05:00
Jonathan A. Sternberg 1e3c44709d dap: refactor how step in/step out works
Change how breakpoints and stepping works. These now work more how you
would expect another programming language to work. Breakpoints happen
before the step has been invoked rather than after which means you can
inspect the state before the command runs.

This has the advantage of being more intuitive for someone familiar with
other debuggers. The negative is that you can't run to after a certain
step as easily as you could before. Instead, you would run to that stage
and then use next to go to the step directly afterwards.

Step in and out also now have different behaviors. When a step has
multiple inputs, the inputs of non-zero index are considered like
"function calls". The most common cause of this is to use `COPY --from`
or a bind mount. Stepping into these will cause it to jump to the
beginning of the call chain for that branch. Using step out will exit
back to the location where step in was used.

This change also makes it so some steps may be invoked multiple times in
the callgraph if multiple steps depend on them. The reused steps will
still be cached, but you may end up stepping through more lines than the
previous implementation.

Stack traces now represent where these step in and step out areas
happen rather than the previous steps. This can help you know from where
a certain step is being used.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-23 17:10:40 -05:00
Jonathan A. Sternberg fea53ad1f8 dap: return error from evaluate command in repl context
In the repl context, we will now return the error instead of directly
printing it. We also suppress reporting errors from cobra. The logic
flow has also been changed to prevent returning errors from cobra unless
there was something related to the command line invocation so usage will
only be printed when a command was typed wrong and it will not show up
for every error.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-23 13:51:03 -05:00
Jonathan A. Sternberg ac9050261e docs: add docs related to dap
Adds some entry-level and developer-friendly docs for the debug adapter.
The one in `docs/dap.md` is meant for someone trying to use the debugger
while the one in `docs/reference/buildx_dap_build.md` is more focused on
documenting the command to be integrated in a debugger extension.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-15 10:58:01 -05:00
Jonathan A. Sternberg 3453f3b00a dap: support evaluate request to invoke a container
Supports using the `evaluate` request in REPL mode to start a container
with the `exec` command. Presently doesn't support any arguments.

This improves the dap server so it is capable of sending reverse
requests and receiving the response. It also adds a hidden command
`dap attach` that attaches to the socket created by `evaluate`.

This requires the client to support `runInTerminal`.

Likely needs some additional work to make sure resources are cleaned up
cleanly especially when the build is unpaused or terminated, but it
should work as a decent base.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-15 10:45:25 -05:00
Jonathan A. Sternberg e7b8de2b0c dap: correctly set the target when provided by the launch config
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-14 14:51:05 -05:00
Jonathan A. Sternberg e9d4b86161 dap: always return the error from execution if we paused on an error and resume
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-14 14:00:02 -05:00
Jonathan A. Sternberg 7925996c0c dap: do not modify variable references on variables that are zero
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-14 13:57:26 -05:00
Jonathan A. Sternberg 0a78659776 dap: alias step into to next and step out to continue
Step into and step out are required by the UI for DAP. We don't have a
way to implement these in a logical manner but they need to exist. We'll
discuss in further iterations how these might differ from next and
continue, but for now, we just need some implementation for the UI.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-14 11:00:33 -05:00
Jonathan A. Sternberg 1886e232c5 dap: implement variable references
Implement variable references to inspect the state of a stack frame.

Variable reference ids are composed of two sections. A thread mask that
is the first 8 bytes and the remainder is an increasing number that gets
reset each time a thread is resumed. This allows the adapter to know
which thread to delegate the variables request to and allows the
variable references to still remain confined to each thread. An int32 is
used for this because variable references need to be in the range of
(0, 2^32).

At the moment, only the platform variables and some of the exec
operations for an operation. These are labeled as "arguments" to the
stack frame.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-14 10:59:05 -05:00
Jonathan A. Sternberg 0dddf0a7b8 dap: implement first pass at breakpoints
Implement the first iteration of breakpoints. When a breakpoint is set,
it starts unverified. When a thread begins evaluation, it tries to see
if a breakpoint corresponds to one of the parsed instructions and will
verify it.

Breakpoints work when continue is used.

At the current moment, setting breakpoints while a thread is currently
running doesn't work. Breakpoints are rechecked each time execution is
about to restart.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-14 09:19:17 -05:00
Jonathan A. Sternberg a291698eaf dap: support stopOnEntry to configure behavior when starting the debugger
This will configure the default behavior when beginning to evaluate a
build target. When `stopOnEntry` is used, it will default to `stepNext`.
Otherwise, `stepContinue` will be used.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-07 09:39:23 -05:00
Jonathan A. SternbergandGitHub 3a3fc54e33 Merge pull request #3293 from jsternberg/dap-adapter-test-flaky
dap: increase timeout for receiving configuration done in adapter test
2025-07-03 09:47:42 -05:00
Jonathan A. Sternberg 4f2e23a9b8 dap: add stack traces with next and continue functionality
It is now possible to send next and continue as separate signals. When
executing a build, the debug adapter will divide the LLB graph into
regions. Each region corresponds to an uninterrupted chain of
instructions. It will also record which regions depend on which other
ones.

This determines the execution order and it also determines what the
stack traces look like.

When continue is used, we will attempt to evaluate the last leaf node
(the head). If we push next, we will determine which digest would be the
next one to be processed.

In the future, this will be used to also support breakpoints.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-03 09:18:55 -05:00
Jonathan A. Sternberg f03ed8cc9f dap: increase timeout for receiving configuration done in adapter test
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-07-03 09:12:34 -05:00
Jonathan A. Sternberg 42599a7d49 dap: add debug adapter implementation
Adds a simple implementation of the debug adapter that supports the very
basics of a debug adapter.

It supports the launch request, the configuration done request, the
creation of threads, stopping, resuming, and disconnecting from server.

It does not support custom breakpoints, stack traces, or variable
inspection yet. These are planned to be added in the future.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-30 10:51:20 -05:00
Jonathan A. Sternberg 3c2decea38 dockerfile: update buildkit to 0.23.0
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-17 13:29:22 -05:00
Jonathan A. Sternberg 7660acf9c7 progress: ensure bake waits for progress to finish printing on error conditions
Some minor fixes to the printer and how bake invokes it. Bake previously
had a race condition that could result in the display not updating on an
error condition, but it was much rarer because the channel communication
was much closer. The refactor added a proxy for the status channel so
there was more of an opportunity to surface the race condition.

When bake exits with an error when reading the bakefiles, it doesn't
wait for the printer to finish so it is possible for the printer to
update the display after an error is printed. This adds an extra `Wait`
in a defer to make sure the printer is finished.

`Wait` has also been fixed to allow it to be called multiple times and
have the same behavior. Previously, it only waited for the done channel
once so only the first wait would block.

The `onclose` method is now called every time the display is paused or
stopped. That was the previous behavior and it's been restored here.

The display only gets refreshed if we aren't exiting. There's no point
in initializing another display if we're about to exit.

The metric writer attached to the printer was erroneously removed. It is
now assigned properly.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-16 12:24:04 -05:00
Jonathan A. Sternberg 7f5ff6b797 commands: remove debug package in commands
The package just causes the entire flow to be more complicated as build
has to pretend it doesn't know about debug options and the debugger has
to pretend it doesn't know about the build.

This abstraction has been difficult when integrating a DAP command into
this same workflow so I don't think this abstraction has much of a
value.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-13 09:32:35 -05:00
Jonathan A. Sternberg e1adeee898 vendor: github.com/moby/buildkit v0.23.0-rc1
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-11 16:29:31 -05:00
Jonathan A. Sternberg 38cf84346c build: change build handler to evaluate instead of onresult
This changes the build handler to customize the behavior of evaluate
rather than onresult and also simplifies the `ResultHandle`. The
`ResultHandle` is now only valid within the gateway callback and can be
used to start containers from the handler.

`Evaluate` now executes inside of the gateway callback rather than
having a separate implementation that executes or re-invokes the build.
This keeps the gateway callback session open until the debugger has
returned.

The `ErrReload` for monitor has now been moved into the `build` package
and been renamed to `ErrRestart`. This is because it restarts the build
so the name makes a bit more sense. The actual use of this functionality
is still tied to the monitor reload.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-10 11:48:41 -05:00
Jonathan A. Sternberg 34e59ca1bd progress: fix progress writer pause and unpause to prevent panics
This changes the progress printer's pause and unpause implementation to
be reentrant to prevent race conditions and it also allows the status
updates to be buffered when the display is paused.

The previous implementation mixed the pause implementation with the
finish implementation and could cause a send on closed channel panic
because it could close the status channel before it had finished being
used. Now, the status channel is not closed.

When the display is enabled, the status channel will be forwarded to an
internal channel that is used to display the updates. When the display
is paused, the status channel will have the statuses buffered in memory
to be sent when the progress display is resumed.

The `Unpause` method has also been renamed to `Resume`.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-09 14:07:52 -05:00
Jonathan A. Sternberg 1d7cda1232 controller: remove remaining parts of the controller
Removes all references to the controller and moves the remaining
sections of code to other packages.

Processes has been moved to monitor where it is used and the data
structs have been removed so buildflags is used directly. The controller
build function has been moved to the commands package.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-05 11:57:03 -05:00
Jonathan A. Sternberg 8f2604b6b4 monitor: move remaining controller functionality into monitor
This creates a `Monitor` type that keeps the global state between
monitor invocations and allows the monitor to exist during the build so
it can be utilized for callbacks.

The result handler is now registered with the monitor during the build
and `Run` will use the result if it is present and the configuration
intends the monitor to be invoked with the given result.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-04 15:27:24 -05:00
Jonathan A. Sternberg 21ebf82c99 monitor: refactor how reload works
The build now happens in a loop and the monitor is run after every
build. The monitor can return `ErrReload` to signal to the main thread
that it should reload the build result.

This will be used in the future to move the monitor into a callback
rather than as a separate existence. It allows the monitor to not
control the build itself which now makes it possible to completely
remove the controller.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-04 15:06:31 -05:00
Jonathan A. Sternberg 65e46cc6af commands: simplify passing stdin to the build when the monitor is configured
The monitor needs stdin to run and isn't compatible with loading a
context or dockerfile from stdin. We already disallow this combination
and, with the removal of the remote controller, there's no way to use
stdin during the build when invoke is configured.

This just removes the extra code to allow forwarding stdin to the build
when the monitor is configured to simplify that section of code.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-03 15:41:23 -05:00
Jonathan A. Sternberg 6a0f5610e3 controller: remove the controller interface
The controller interface is removed and the local controller is used for
only the initial build, invoke, and rebuilds.

Process control has been moved to the monitor.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-03 15:41:23 -05:00
Jonathan A. Sternberg e78aa98c92 build: refactor some of the build functions into smaller utility functions
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-03 15:41:22 -05:00
Jonathan A. Sternberg 9bd1ba2f5c commands: update deprecation notice for keep-storage
The `--keep-storage` flag was changed to `--reserved-space`. Before it was
changed to that name, it was changed to `--max-storage`. This flag never
made it into a release as the name was changed before release, but the
update to the flag in buildx forgot to update the deprecation notice.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-06-02 09:35:39 -05:00
Jonathan A. Sternberg 781a3f117a hack: remove code generation related to generated files
With the removal of the protobuf for the controller, there are no longer
any generated files. Remove the makefile targets and the associated
dockerfiles and bake targets.

This wasn't being included in CI because it wasn't part of the
`validate` target.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-05-22 14:59:42 -05:00
Jonathan A. Sternberg 384f0565f5 controller: remove controller/errdefs protobuf files
Remove the protobuf files associated with controller/errdefs.

This doesn't completely remove the type as the monitor still uses it as
a signal to start the monitor.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-05-01 12:14:36 -05:00
Jonathan A. SternbergandGitHub ee77cdb175 Merge pull request #3102 from jsternberg/buildkit-rc1
vendor: github.com/moby/buildkit v0.21.0-rc1
2025-04-09 10:56:27 -05:00
Jonathan A. Sternberg 8fb1157b5f vendor: github.com/moby/buildkit v0.21.0-rc1
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-04-09 10:28:03 -05:00
Jonathan A. Sternberg 53e576b306 vendor: github.com/moby/buildkit v0.20.2
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-03-24 15:56:16 -05:00
Jonathan A. Sternberg e75ac22ba6 buildflags: skip empty cache entries when parsing
Broken in 11c84973ef. The section to skip
an empty input was accidentally removed when some code was refactored to
fix a separate issue.

This skips empty cache entries which allows disabling the `cache-from` and
`cache-to` entries from the command line overrides.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-02-24 10:09:02 -06:00
Jonathan A. Sternberg 75160643e1 ci: update buildkit to 0.20.0
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-02-19 15:21:14 -06:00
Jonathan A. SternbergandGitHub ad18ffc018 Merge pull request #3010 from jsternberg/vendor-update
vendor: github.com/moby/buildkit v0.20.0
2025-02-19 13:30:37 -06:00
Jonathan A. Sternberg 80c3832c94 vendor: github.com/moby/buildkit v0.20.0
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-02-19 13:17:40 -06:00
Jonathan A. SternbergandGitHub 7762ab2c38 Merge pull request #3008 from thaJeztah/bump_engine_28.0_rc3
vendor: github.com/docker/docker, docker/cli v28.0.0-rc.3
2025-02-19 11:59:57 -06:00
Jonathan A. Sternberg d6fdf83f45 bake: allow annotations to be set on the command line
Annotations were not merged correctly. The overrides in `ArrValue` would
be merged, but the section of code setting them from the command line
did not include `annotations` in the list of available attributes so the
command line option was completely discarded.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-02-14 11:57:30 -06:00
Jonathan A. Sternberg 3ae9970da5 buildflags: make work on go 1.22 by reverting rangefunc usage
Reverts the usage of rangefunc and attempts to keep the foundation of it
in for when we move to go 1.23. We have downstream dependencies that
aren't ready to move to go 1.23. We can likely move after go 1.24 is
released.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-02-10 11:03:46 -06:00
Jonathan A. Sternberg abc85c38f8 buildflags: handle unknown values from cty
Update the buildflags cty code to handle unknown values. When hcl
decodes a value with an invalid variable name, it appends a diagnostic
for the error and then returns an unknown value so it can continue
processing the file and finding more errors.

The iteration code has now been changed to use a rangefunc from go 1.23
and it skips empty or unknown values. Empty values are valid when they
are skipped and unknown values will have a diagnostic for itself.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-02-06 09:45:18 -06:00
Jonathan A. Sternberg 11c84973ef buildflags: fix ref only format for command line and bake
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-01-22 13:18:38 -06:00
Jonathan A. Sternberg 3aed658dc4 buildflags: marshal attestations into json with extra attributes correctly
`MarshalJSON` would not include the extra attributes because it iterated
over the target map rather than the source map.

Also fixes JSON unmarshaling for SSH and secrets. The intention was to
unmarshal into the struct, but `UnmarshalText` takes priority over the
default struct unmarshaling so it didn't work as intended.

Tests have been added for all marshaling and unmarshaling methods.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2025-01-21 15:05:23 -06:00
Jonathan A. SternbergandCrazyMax 1e992b295c bake: test empty override
Co-authored-by: CrazyMax <github@crazymax.dev>
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-12-18 11:56:19 -06:00
Jonathan A. Sternberg 4f81bcb5c8 bake: implement composable attributes for attestations
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-12-18 11:48:50 -06:00
Jonathan A. Sternberg 5dd4ae0335 bake: various fixes for composable attributes
This changes how the composable attributes are implemented and provides
various fixes to the first iteration.

Cache-from and cache-to now no longer print sensitive values that are
automatically added. These automatically added attributes are added when
the protobuf is created rather than at the time of parsing so they will
no longer be printed. If they are part of the original configuration
file, they will still be printed.

Empty strings will now be skipped. This was the original behavior and
composable attributes removed this functionality accidentally. This
functionality is now restored.

This also expands the available syntax that works with each of the
composable attributes. It is now possible to interleave the csv syntax
with the object syntax without any problems. The canonical form is still
the object syntax and variables are resolved according to that syntax.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-12-18 10:26:15 -06:00
Jonathan A. Sternberg 3ccbb88e6a bake: initial set of composable bake attributes
This allows using either the csv syntax or object syntax to specify
certain attributes.

This applies to the following fields:
- output
- cache-from
- cache-to
- secret
- ssh

There are still some remaining fields to translate. Specifically
ulimits, annotations, and attest.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-11-21 12:31:11 -06:00
Jonathan A. Sternberg a5bb117ff0 bake: improve error when using incorrect format for setting labels
Improves the error message when using an incorrect format for setting
labels. This includes the intended format directly in the error message
instead of assuming the user knows what the format is.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-11-04 14:38:23 -06:00
Jonathan A. Sternberg cf7a9aa084 pprof: take cpu and memory profiles by setting environment variables
When run in standalone mode, the environment variables
`DOCKER_BUILDX_CPU_PROFILE` and `DOCKER_BUILDX_MEM_PROFILE` will cause
profiles to be written by the CLI.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-10-24 09:56:27 -05:00
Jonathan A. Sternberg 64c5139ab6 hack: generate vtproto files for buildx
Integrates vtproto into buildx. The generated files dockerfile has been
modified to copy the buildkit equivalent file to ensure files are laid
out in the appropriate way for imports.

An import has also been included to change the grpc codec to the version
in buildkit that supports vtproto. This will allow buildx to utilize the
speed and memory improvements from that.

Also updates the gc control options for prune.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-10-08 13:35:06 -05:00
Jonathan A. Sternberg 9fc6f39d71 bake: raise maximum size limit and fix size check
Similar to https://github.com/docker/buildx/pull/2716.

Use the file size rather than the proto size, raise the allowed limit to
the same value for consistency, and improve the error message to include
the limit in human units.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-10-04 09:11:07 -05:00
Jonathan A. Sternberg d7de22c61f build: raise maximum size limit for dockerfile and fix size check
Raise the maximum size limit for the dockerfile and correct the size
check. The size check was intended to use the size attribute from the
file stat, but the original gogo version confused the `Size()`
method (which returned the size of the proto message) with the `Size`
attribute (which was named `Size_`).

During the conversion, we noticed the mistake but kept the incorrect
behavior for the sake of keeping the conversion simple.

This also raises the maximum limit because 512 kB is likely a bit too
conservative. The limit has been raised to 2 MB and the limit has been
included in the error message.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-10-03 12:12:40 -05:00
Jonathan A. Sternberg b35a0f4718 protobuf: remove gogoproto
Removes gogo/protobuf from buildx and updates to a version of
moby/buildkit where gogo is removed.

This also changes how the proto files are generated. This is because
newer versions of protobuf are more strict about name conflicts. If two
files have the same name (even if they are relative paths) and are used
in different protoc commands, they'll conflict in the registry.

Since protobuf file generation doesn't work very well with
`paths=source_relative`, this removes the `go:generate` expression and
just relies on the dockerfile to perform the generation.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-10-02 15:51:59 -05:00
Jonathan A. Sternberg 6c5279da54 metrics: add mutex to the metric writer
It was possible for multiple status messages to be written at the same
time which caused some of the metric writer code to have a race
condition.

This code should be fast enough that it doesn't interrupt the display,
but some further work might be needed here.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-08-14 10:57:04 -05:00
Jonathan A. Sternberg 58571ff6d6 metrics: add metrics for bake command
This adds metrics for the bake command using a different method of
calculating the build identifier but with the same attributes otherwise.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-08-08 10:00:11 -05:00
Jonathan A. SternbergandGitHub 16860e6dd2 Merge pull request #2640 from crazy-max/call-metadata
support metadata file with call flag for build and bake commands
2024-08-08 09:21:05 -05:00
Jonathan A. Sternberg 2810f20f3a metricwriter: compile regular expressions only on first use
Compile the regular expressions only on first use rather than implicitly
as part of the `init()` function of the package. This prevents taking a
speed hit on the initialization of the package regardless if this type
is used and moves it to the time when a regular expression is first
used.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-08-05 10:23:06 -05:00
Jonathan A. Sternberg ef4a165e48 commands: add an alias for --check to be the same as --call=check
This adds an alias for `--check` that causes it to behave the same as
`--call=check`. This is done using `BoolFunc` to call a function when
the option is seen and to set it to the correct value. This should allow
command line flags like `--check --call=targets` to work correctly (even
though they conflict) by making it so the first invocation sets the
print function to `check` and the second overwrites the first. This is
the expected behavior for these types of boolean flags.

`BoolFunc` itself is part of the standard library flags package, but
never seems to have made it into pflag possibly because it was added in
go 1.21.

https://pkg.go.dev/flag#FlagSet.BoolFunc

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-06-03 13:25:21 -05:00
Jonathan A. Sternberg b1cb658a31 vendor: update buildkit to v0.14.0-rc1
Update buildkit dependency to v0.14.0-rc1. Update the tracing
infrastructure to use the new detect API which updates how the delegated
exporter is configured.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-05-31 16:23:41 -05:00
Jonathan A. Sternberg 1a93bbd3a5 metrics: record the number of times lint rules are triggered during a build
This metric records the number of times it sees a lint warning in the
progress stream and categorizes the number of times each rule has been
triggered. This will only record whether a lint warning was triggered
and not whether the linter was even used or which rules were present.
That information isn't presently part of the stream.

With this change, we might be reaching some of the limitations that
spying on the progress stream gives us for metrics and may want to
consider another way for the build to communicate metrics back to the
client.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-05-30 15:05:00 -05:00
Jonathan A. Sternberg cf68b5b878 vendor: update buildx to latest docker/cli
This version of docker/cli has changes to remove compose-cli wrapper and
move all CLI metrics to OTEL.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
(cherry picked from commit 4fc4bc07ae)
2024-05-16 12:14:08 -05:00
Jonathan A. Sternberg 4fc4bc07ae vendor: update buildx to latest docker/cli
This version of docker/cli has changes to remove compose-cli wrapper and
move all CLI metrics to OTEL.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-05-16 12:07:13 -05:00
Jonathan A. Sternberg b4799f9d16 metricutil: switch to using the cli meter provider
The meter provider initialization that was located here has now been
moved to a common area in the docker cli. This upgrades our CLI version
and then uses this common code instead of our own version.

As a piece of additional functionality, the docker OTEL endpoint can now
be overwritten with `DOCKER_CLI_OTEL_EXPORTER_OTLP_ENDPOINT` for
testing.

This removes the OTLP exporter from the CLI that was previously locked
behind `BUILDX_EXPERIMENTAL`. I do plan for this to return, but as a
proper part of the `docker/cli` implementation rather than something
special with `buildx`.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-04-02 09:36:55 -05:00
Jonathan A. Sternberg adb9bc86e5 vendor: github.com/docker/docker and github.com/docker/cli v26.0.0
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-04-01 13:05:55 -05:00
Jonathan A. Sternberg 5c89004bb6 driver: initialize tracer delegate in driver handle instead of individual plugins
This refactors the driver handle to initialize the tracer delegate
inside of the driver handle instead of the individual plugins.

This provides more uniformity to how the tracer delegate is created by
allowing the driver handle to pass additional client options to the
drivers when they create the client. It also avoids creating the tracer
delegate client multiple times because the driver handle will only
initialize the client once. This prevents some drivers, like the remote
driver, from accidentally registering multiple clients as tracer
delegates.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-03-27 15:13:43 -05:00
Jonathan A. Sternberg 4999908fbc progress: remove the experimental label from progress metrics
Removes the experimental label from progress metrics. User-metrics
themselves are still experimental so this is still blocked behind the
experimental flag, but this will allow the docker otlp endpoint to
receive these metrics.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-03-19 08:23:32 -05:00
Jonathan A. Sternberg a4a8846e46 metrics: measure idle time during builds
This measures the amount of time spent idle during the build. This is
done by collecting the set of task times, determining which sections
contain gaps where no task is running, and aggregating that duration
into a metric.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-03-18 08:43:15 -05:00
Jonathan A. Sternberg a0cce9b31e metrics: measure export image operation
This measures the amount of time it takes to export to a specific
format.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-03-14 16:08:19 -05:00
Jonathan A. Sternberg 9016d85718 metrics: measure run operations for exec operations
This measures the duration of exec operations. It does not factor in
whether the operation was cached or not so this should include the
amount of time to determine whether an operation was cached.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-03-14 14:51:27 -05:00
Jonathan A. Sternberg 9a71895a48 metrics: measure image transfers for image source operations
This measures the transfer size and duration for image pulls along with
the time spent extracting the image contents.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-03-05 16:33:20 -06:00
Jonathan A. Sternberg 97052cf203 metrics: measure context transfers for local source operations
Measure the transfer size and duration of context transfers for various
categories of local source transfers from the progress stream that's
returned during the build.

Local source transfers are split into one of four categories:
* context
* dockerfile
* dockerignore
* namedcontext

Named contexts that are different names will be categorized under the
same metric.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-02-21 14:52:03 -06:00
Jonathan A. Sternberg fe08cf2981 metricutil: remove BUILDX_EXPERIMENTAL from internal docker reporting
The `BUILDX_EXPERIMENTAL` check is removed from the docker otel
collector. We'll send metrics to the OTLP endpoint for docker desktop if
it is present and enabled regardless of experimental status.

The user-facing `OTEL` endpoints for enabling the metric reporting for
external use is still hidden behind the experimental flag. We'll likely
remove the experimental flag for this feature for v0.14.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-02-15 14:17:58 -06:00
Jonathan A. Sternberg bda968ad5d metrics: add build command duration metric
This adds a build duration metric for the build command with attributes
related to the buildx driver, the error type (if any), and which options
were used to perform the build from a subset of the options.

This also refactors some of the utility methods used by the git tool to
determine filepaths into its own separate package so they can be reused
in another place.

Also adds a test to ensure the resource is initialized correctly and
doesn't error. The otel handler logging message is suppressed on buildx
invocations so we never see the error if there's a problem with the
schema url. It's so easy to mess up the schema url when upgrading OTEL
that we need a proper test to make sure we haven't broken the
functionality.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-02-14 15:58:52 -06:00
Jonathan A. Sternberg c65b7ed24f otel: include service instance id attribute to resource and move to metricutil package
Add the service instance id to the resource attributes to prevent
downstream OTEL processors and exporters from thinking that the CLI
invocations are a single process that keeps restarting. The unique id
can be removed through downstream aggregation to prevent cardinality
issues, but we need some way to tell OTEL that it shouldn't reset the
counters.

Move the check for the experimental flag to its own package and then use
that invocation to prevent creating exporters so metrics are disabled
completely. This makes it so we don't have to check for the experimental
flag in every place we add metrics until we decide to make metrics
stable in general.

This also moves the OTEL initialization to a `util/metricutil` package
to be more consistent with the existing util naming and to differentiate
it from the upstream `metric` name. Using both `metrics` and `metric` as
import names was confusing since `metric` was an upstream dependency and
`metrics` was a local utility. `metricutil` matches with the existing
utilities and makes clear that it isn't a spelling mistake.

The record version metric has been removed since we weren't planning on
keeping that metric anyway and most of the information is now included
in the instrumentation library name and version. That function is
included as a utility in the `otel/sdk/metric` package to retrieve the
appropriate meter from the meter provider.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-01-30 16:27:02 -06:00