alextab93/buildx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

vendor: github.com/moby/buildkit v0.23.0-rc1

Browse Source 
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
This commit is contained in:
Jonathan A. Sternberg
2025-06-11 16:29:31 -05:00
parent 1b4bd20e6f
commit e1adeee898
1145 changed files with 62545 additions and 34279 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vendor/github.com/AdaLogics/go-fuzz-headers/LICENSE
Generated Vendored
-201
View File
@@ -1,201 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
vendor/github.com/AdaLogics/go-fuzz-headers/README.md
Generated Vendored
-93
View File
@@ -1,93 +0,0 @@
# go-fuzz-headers
This repository contains various helper functions for go fuzzing. It is mostly used in combination with [go-fuzz](https://github.com/dvyukov/go-fuzz), but compatibility with fuzzing in the standard library will also be supported. Any coverage guided fuzzing engine that provides an array or slice of bytes can be used with go-fuzz-headers.
## Usage
Using go-fuzz-headers is easy. First create a new consumer with the bytes provided by the fuzzing engine:
```go
import (
fuzz "github.com/AdaLogics/go-fuzz-headers"
)
data := []byte{'R', 'a', 'n', 'd', 'o', 'm'}
f := fuzz.NewConsumer(data)
```
This creates a `Consumer` that consumes the bytes of the input as it uses them to fuzz different types.
After that, `f` can be used to easily create fuzzed instances of different types. Below are some examples:
### Structs
One of the most useful features of go-fuzz-headers is its ability to fill structs with the data provided by the fuzzing engine. This is done with a single line:
```go
type Person struct {
Name string
Age int
}
p := Person{}
// Fill p with values based on the data provided by the fuzzing engine:
err := f.GenerateStruct(&p)
```
This includes nested structs too. In this example, the fuzz Consumer will also insert values in `p.BestFriend`:
```go
type PersonI struct {
Name string
Age int
BestFriend PersonII
}
type PersonII struct {
Name string
Age int
}
p := PersonI{}
err := f.GenerateStruct(&p)
```
If the consumer should insert values for unexported fields as well as exported, this can be enabled with:
```go
f.AllowUnexportedFields()
```
...and disabled with:
```go
f.DisallowUnexportedFields()
```
### Other types:
Other useful APIs:
```go
createdString, err := f.GetString() // Gets a string
createdInt, err := f.GetInt() // Gets an integer
createdByte, err := f.GetByte() // Gets a byte
createdBytes, err := f.GetBytes() // Gets a byte slice
createdBool, err := f.GetBool() // Gets a boolean
err := f.FuzzMap(target_map) // Fills a map
createdTarBytes, err := f.TarBytes() // Gets bytes of a valid tar archive
err := f.CreateFiles(inThisDir) // Fills inThisDir with files
createdString, err := f.GetStringFrom("anyCharInThisString", ofThisLength) // Gets a string that consists of chars from "anyCharInThisString" and has the exact length "ofThisLength"
```
Most APIs are added as they are needed.
## Projects that use go-fuzz-headers
- [runC](https://github.com/opencontainers/runc)
- [Istio](https://github.com/istio/istio)
- [Vitess](https://github.com/vitessio/vitess)
- [Containerd](https://github.com/containerd/containerd)
Feel free to add your own project to the list, if you use go-fuzz-headers to fuzz it.
## Status
The project is under development and will be updated regularly.
## References
go-fuzz-headers' approach to fuzzing structs is strongly inspired by [gofuzz](https://github.com/google/gofuzz).
vendor/github.com/AdaLogics/go-fuzz-headers/consumer.go
Generated Vendored
-960
View File
@@ -1,960 +0,0 @@
// Copyright 2023 The go-fuzz-headers Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package gofuzzheaders
import (
"archive/tar"
"bytes"
"encoding/binary"
"errors"
"fmt"
"io"
"math"
"os"
"path/filepath"
"reflect"
"strconv"
"strings"
"time"
"unsafe"
)
var (
MaxTotalLen uint32 = 2000000
maxDepth = 100
)
func SetMaxTotalLen(newLen uint32) {
MaxTotalLen = newLen
}
type ConsumeFuzzer struct {
data []byte
dataTotal uint32
CommandPart []byte
RestOfArray []byte
NumberOfCalls int
position uint32
fuzzUnexportedFields bool
forceUTF8Strings bool
curDepth int
Funcs map[reflect.Type]reflect.Value
}
func IsDivisibleBy(n int, divisibleby int) bool {
return (n % divisibleby) == 0
}
func NewConsumer(fuzzData []byte) *ConsumeFuzzer {
return &ConsumeFuzzer{
data: fuzzData,
dataTotal: uint32(len(fuzzData)),
Funcs: make(map[reflect.Type]reflect.Value),
curDepth: 0,
}
}
func (f *ConsumeFuzzer) Split(minCalls, maxCalls int) error {
if f.dataTotal == 0 {
return errors.New("could not split")
}
numberOfCalls := int(f.data[0])
if numberOfCalls < minCalls || numberOfCalls > maxCalls {
return errors.New("bad number of calls")
}
if int(f.dataTotal) < numberOfCalls+numberOfCalls+1 {
return errors.New("length of data does not match required parameters")
}
// Define part 2 and 3 of the data array
commandPart := f.data[1 : numberOfCalls+1]
restOfArray := f.data[numberOfCalls+1:]
// Just a small check. It is necessary
if len(commandPart) != numberOfCalls {
return errors.New("length of commandPart does not match number of calls")
}
// Check if restOfArray is divisible by numberOfCalls
if !IsDivisibleBy(len(restOfArray), numberOfCalls) {
return errors.New("length of commandPart does not match number of calls")
}
f.CommandPart = commandPart
f.RestOfArray = restOfArray
f.NumberOfCalls = numberOfCalls
return nil
}
func (f *ConsumeFuzzer) AllowUnexportedFields() {
f.fuzzUnexportedFields = true
}
func (f *ConsumeFuzzer) DisallowUnexportedFields() {
f.fuzzUnexportedFields = false
}
func (f *ConsumeFuzzer) AllowNonUTF8Strings() {
f.forceUTF8Strings = false
}
func (f *ConsumeFuzzer) DisallowNonUTF8Strings() {
f.forceUTF8Strings = true
}
func (f *ConsumeFuzzer) GenerateStruct(targetStruct interface{}) error {
e := reflect.ValueOf(targetStruct).Elem()
return f.fuzzStruct(e, false)
}
func (f *ConsumeFuzzer) setCustom(v reflect.Value) error {
// First: see if we have a fuzz function for it.
doCustom, ok := f.Funcs[v.Type()]
if !ok {
return fmt.Errorf("could not find a custom function")
}
switch v.Kind() {
case reflect.Ptr:
if v.IsNil() {
if !v.CanSet() {
return fmt.Errorf("could not use a custom function")
}
v.Set(reflect.New(v.Type().Elem()))
}
case reflect.Map:
if v.IsNil() {
if !v.CanSet() {
return fmt.Errorf("could not use a custom function")
}
v.Set(reflect.MakeMap(v.Type()))
}
default:
return fmt.Errorf("could not use a custom function")
}
verr := doCustom.Call([]reflect.Value{v, reflect.ValueOf(Continue{
F: f,
})})
// check if we return an error
if verr[0].IsNil() {
return nil
}
return fmt.Errorf("could not use a custom function")
}
func (f *ConsumeFuzzer) fuzzStruct(e reflect.Value, customFunctions bool) error {
if f.curDepth >= maxDepth {
// return err or nil here?
return nil
}
f.curDepth++
defer func() { f.curDepth-- }()
// We check if we should check for custom functions
if customFunctions && e.IsValid() && e.CanAddr() {
err := f.setCustom(e.Addr())
if err != nil {
return err
}
}
switch e.Kind() {
case reflect.Struct:
for i := 0; i < e.NumField(); i++ {
var v reflect.Value
if !e.Field(i).CanSet() {
if f.fuzzUnexportedFields {
v = reflect.NewAt(e.Field(i).Type(), unsafe.Pointer(e.Field(i).UnsafeAddr())).Elem()
}
if err := f.fuzzStruct(v, customFunctions); err != nil {
return err
}
} else {
v = e.Field(i)
if err := f.fuzzStruct(v, customFunctions); err != nil {
return err
}
}
}
case reflect.String:
str, err := f.GetString()
if err != nil {
return err
}
if e.CanSet() {
e.SetString(str)
}
case reflect.Slice:
var maxElements uint32
// Byte slices should not be restricted
if e.Type().String() == "[]uint8" {
maxElements = 10000000
} else {
maxElements = 50
}
randQty, err := f.GetUint32()
if err != nil {
return err
}
numOfElements := randQty % maxElements
if (f.dataTotal - f.position) < numOfElements {
numOfElements = f.dataTotal - f.position
}
uu := reflect.MakeSlice(e.Type(), int(numOfElements), int(numOfElements))
for i := 0; i < int(numOfElements); i++ {
// If we have more than 10, then we can proceed with that.
if err := f.fuzzStruct(uu.Index(i), customFunctions); err != nil {
if i >= 10 {
if e.CanSet() {
e.Set(uu)
}
return nil
} else {
return err
}
}
}
if e.CanSet() {
e.Set(uu)
}
case reflect.Uint:
newInt, err := f.GetUint()
if err != nil {
return err
}
if e.CanSet() {
e.SetUint(uint64(newInt))
}
case reflect.Uint16:
newInt, err := f.GetUint16()
if err != nil {
return err
}
if e.CanSet() {
e.SetUint(uint64(newInt))
}
case reflect.Uint32:
newInt, err := f.GetUint32()
if err != nil {
return err
}
if e.CanSet() {
e.SetUint(uint64(newInt))
}
case reflect.Uint64:
newInt, err := f.GetInt()
if err != nil {
return err
}
if e.CanSet() {
e.SetUint(uint64(newInt))
}
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
newInt, err := f.GetInt()
if err != nil {
return err
}
if e.CanSet() {
e.SetInt(int64(newInt))
}
case reflect.Float32:
newFloat, err := f.GetFloat32()
if err != nil {
return err
}
if e.CanSet() {
e.SetFloat(float64(newFloat))
}
case reflect.Float64:
newFloat, err := f.GetFloat64()
if err != nil {
return err
}
if e.CanSet() {
e.SetFloat(float64(newFloat))
}
case reflect.Map:
if e.CanSet() {
e.Set(reflect.MakeMap(e.Type()))
const maxElements = 50
randQty, err := f.GetInt()
if err != nil {
return err
}
numOfElements := randQty % maxElements
for i := 0; i < numOfElements; i++ {
key := reflect.New(e.Type().Key()).Elem()
if err := f.fuzzStruct(key, customFunctions); err != nil {
return err
}
val := reflect.New(e.Type().Elem()).Elem()
if err = f.fuzzStruct(val, customFunctions); err != nil {
return err
}
e.SetMapIndex(key, val)
}
}
case reflect.Ptr:
if e.CanSet() {
e.Set(reflect.New(e.Type().Elem()))
if err := f.fuzzStruct(e.Elem(), customFunctions); err != nil {
return err
}
return nil
}
case reflect.Uint8:
b, err := f.GetByte()
if err != nil {
return err
}
if e.CanSet() {
e.SetUint(uint64(b))
}
case reflect.Bool:
b, err := f.GetBool()
if err != nil {
return err
}
if e.CanSet() {
e.SetBool(b)
}
}
return nil
}
func (f *ConsumeFuzzer) GetStringArray() (reflect.Value, error) {
// The max size of the array:
const max uint32 = 20
arraySize := f.position
if arraySize > max {
arraySize = max
}
stringArray := reflect.MakeSlice(reflect.SliceOf(reflect.TypeOf("string")), int(arraySize), int(arraySize))
if f.position+arraySize >= f.dataTotal {
return stringArray, errors.New("could not make string array")
}
for i := 0; i < int(arraySize); i++ {
stringSize := uint32(f.data[f.position])
if f.position+stringSize >= f.dataTotal {
return stringArray, nil
}
stringToAppend := string(f.data[f.position : f.position+stringSize])
strVal := reflect.ValueOf(stringToAppend)
stringArray = reflect.Append(stringArray, strVal)
f.position += stringSize
}
return stringArray, nil
}
func (f *ConsumeFuzzer) GetInt() (int, error) {
if f.position >= f.dataTotal {
return 0, errors.New("not enough bytes to create int")
}
returnInt := int(f.data[f.position])
f.position++
return returnInt, nil
}
func (f *ConsumeFuzzer) GetByte() (byte, error) {
if f.position >= f.dataTotal {
return 0x00, errors.New("not enough bytes to get byte")
}
returnByte := f.data[f.position]
f.position++
return returnByte, nil
}
func (f *ConsumeFuzzer) GetNBytes(numberOfBytes int) ([]byte, error) {
if f.position >= f.dataTotal {
return nil, errors.New("not enough bytes to get byte")
}
returnBytes := make([]byte, 0, numberOfBytes)
for i := 0; i < numberOfBytes; i++ {
newByte, err := f.GetByte()
if err != nil {
return nil, err
}
returnBytes = append(returnBytes, newByte)
}
return returnBytes, nil
}
func (f *ConsumeFuzzer) GetUint16() (uint16, error) {
u16, err := f.GetNBytes(2)
if err != nil {
return 0, err
}
littleEndian, err := f.GetBool()
if err != nil {
return 0, err
}
if littleEndian {
return binary.LittleEndian.Uint16(u16), nil
}
return binary.BigEndian.Uint16(u16), nil
}
func (f *ConsumeFuzzer) GetUint32() (uint32, error) {
u32, err := f.GetNBytes(4)
if err != nil {
return 0, err
}
return binary.BigEndian.Uint32(u32), nil
}
func (f *ConsumeFuzzer) GetUint64() (uint64, error) {
u64, err := f.GetNBytes(8)
if err != nil {
return 0, err
}
littleEndian, err := f.GetBool()
if err != nil {
return 0, err
}
if littleEndian {
return binary.LittleEndian.Uint64(u64), nil
}
return binary.BigEndian.Uint64(u64), nil
}
func (f *ConsumeFuzzer) GetUint() (uint, error) {
var zero uint
size := int(unsafe.Sizeof(zero))
if size == 8 {
u64, err := f.GetUint64()
if err != nil {
return 0, err
}
return uint(u64), nil
}
u32, err := f.GetUint32()
if err != nil {
return 0, err
}
return uint(u32), nil
}
func (f *ConsumeFuzzer) GetBytes() ([]byte, error) {
var length uint32
var err error
length, err = f.GetUint32()
if err != nil {
return nil, errors.New("not enough bytes to create byte array")
}
if length == 0 {
length = 30
}
bytesLeft := f.dataTotal - f.position
if bytesLeft <= 0 {
return nil, errors.New("not enough bytes to create byte array")
}
// If the length is the same as bytes left, we will not overflow
// the remaining bytes.
if length != bytesLeft {
length = length % bytesLeft
}
byteBegin := f.position
if byteBegin+length < byteBegin {
return nil, errors.New("numbers overflow")
}
f.position = byteBegin + length
return f.data[byteBegin:f.position], nil
}
func (f *ConsumeFuzzer) GetString() (string, error) {
if f.position >= f.dataTotal {
return "nil", errors.New("not enough bytes to create string")
}
length, err := f.GetUint32()
if err != nil {
return "nil", errors.New("not enough bytes to create string")
}
if f.position > MaxTotalLen {
return "nil", errors.New("created too large a string")
}
byteBegin := f.position
if byteBegin >= f.dataTotal {
return "nil", errors.New("not enough bytes to create string")
}
if byteBegin+length > f.dataTotal {
return "nil", errors.New("not enough bytes to create string")
}
if byteBegin > byteBegin+length {
return "nil", errors.New("numbers overflow")
}
f.position = byteBegin + length
s := string(f.data[byteBegin:f.position])
if f.forceUTF8Strings {
s = strings.ToValidUTF8(s, "")
}
return s, nil
}
func (f *ConsumeFuzzer) GetBool() (bool, error) {
if f.position >= f.dataTotal {
return false, errors.New("not enough bytes to create bool")
}
if IsDivisibleBy(int(f.data[f.position]), 2) {
f.position++
return true, nil
} else {
f.position++
return false, nil
}
}
func (f *ConsumeFuzzer) FuzzMap(m interface{}) error {
return f.GenerateStruct(m)
}
func returnTarBytes(buf []byte) ([]byte, error) {
return buf, nil
// Count files
var fileCounter int
tr := tar.NewReader(bytes.NewReader(buf))
for {
_, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
fileCounter++
}
if fileCounter >= 1 {
return buf, nil
}
return nil, fmt.Errorf("not enough files were created\n")
}
func setTarHeaderFormat(hdr *tar.Header, f *ConsumeFuzzer) error {
ind, err := f.GetInt()
if err != nil {
hdr.Format = tar.FormatGNU
//return nil
}
switch ind % 4 {
case 0:
hdr.Format = tar.FormatUnknown
case 1:
hdr.Format = tar.FormatUSTAR
case 2:
hdr.Format = tar.FormatPAX
case 3:
hdr.Format = tar.FormatGNU
}
return nil
}
func setTarHeaderTypeflag(hdr *tar.Header, f *ConsumeFuzzer) error {
ind, err := f.GetInt()
if err != nil {
return err
}
switch ind % 13 {
case 0:
hdr.Typeflag = tar.TypeReg
case 1:
hdr.Typeflag = tar.TypeLink
linkname, err := f.GetString()
if err != nil {
return err
}
hdr.Linkname = linkname
case 2:
hdr.Typeflag = tar.TypeSymlink
linkname, err := f.GetString()
if err != nil {
return err
}
hdr.Linkname = linkname
case 3:
hdr.Typeflag = tar.TypeChar
case 4:
hdr.Typeflag = tar.TypeBlock
case 5:
hdr.Typeflag = tar.TypeDir
case 6:
hdr.Typeflag = tar.TypeFifo
case 7:
hdr.Typeflag = tar.TypeCont
case 8:
hdr.Typeflag = tar.TypeXHeader
case 9:
hdr.Typeflag = tar.TypeXGlobalHeader
case 10:
hdr.Typeflag = tar.TypeGNUSparse
case 11:
hdr.Typeflag = tar.TypeGNULongName
case 12:
hdr.Typeflag = tar.TypeGNULongLink
}
return nil
}
func (f *ConsumeFuzzer) createTarFileBody() ([]byte, error) {
return f.GetBytes()
/*length, err := f.GetUint32()
if err != nil {
return nil, errors.New("not enough bytes to create byte array")
}
// A bit of optimization to attempt to create a file body
// when we don't have as many bytes left as "length"
remainingBytes := f.dataTotal - f.position
if remainingBytes <= 0 {
return nil, errors.New("created too large a string")
}
if f.position+length > MaxTotalLen {
return nil, errors.New("created too large a string")
}
byteBegin := f.position
if byteBegin >= f.dataTotal {
return nil, errors.New("not enough bytes to create byte array")
}
if length == 0 {
return nil, errors.New("zero-length is not supported")
}
if byteBegin+length >= f.dataTotal {
return nil, errors.New("not enough bytes to create byte array")
}
if byteBegin+length < byteBegin {
return nil, errors.New("numbers overflow")
}
f.position = byteBegin + length
return f.data[byteBegin:f.position], nil*/
}
// getTarFileName is similar to GetString(), but creates string based
// on the length of f.data to reduce the likelihood of overflowing
// f.data.
func (f *ConsumeFuzzer) getTarFilename() (string, error) {
return f.GetString()
/*length, err := f.GetUint32()
if err != nil {
return "nil", errors.New("not enough bytes to create string")
}
// A bit of optimization to attempt to create a file name
// when we don't have as many bytes left as "length"
remainingBytes := f.dataTotal - f.position
if remainingBytes <= 0 {
return "nil", errors.New("created too large a string")
}
if f.position > MaxTotalLen {
return "nil", errors.New("created too large a string")
}
byteBegin := f.position
if byteBegin >= f.dataTotal {
return "nil", errors.New("not enough bytes to create string")
}
if byteBegin+length > f.dataTotal {
return "nil", errors.New("not enough bytes to create string")
}
if byteBegin > byteBegin+length {
return "nil", errors.New("numbers overflow")
}
f.position = byteBegin + length
return string(f.data[byteBegin:f.position]), nil*/
}
type TarFile struct {
Hdr *tar.Header
Body []byte
}
// TarBytes returns valid bytes for a tar archive
func (f *ConsumeFuzzer) TarBytes() ([]byte, error) {
numberOfFiles, err := f.GetInt()
if err != nil {
return nil, err
}
var tarFiles []*TarFile
tarFiles = make([]*TarFile, 0)
const maxNoOfFiles = 100
for i := 0; i < numberOfFiles%maxNoOfFiles; i++ {
var filename string
var filebody []byte
var sec, nsec int
var err error
filename, err = f.getTarFilename()
if err != nil {
var sb strings.Builder
sb.WriteString("file-")
sb.WriteString(strconv.Itoa(i))
filename = sb.String()
}
filebody, err = f.createTarFileBody()
if err != nil {
var sb strings.Builder
sb.WriteString("filebody-")
sb.WriteString(strconv.Itoa(i))
filebody = []byte(sb.String())
}
sec, err = f.GetInt()
if err != nil {
sec = 1672531200 // beginning of 2023
}
nsec, err = f.GetInt()
if err != nil {
nsec = 1703980800 // end of 2023
}
hdr := &tar.Header{
Name: filename,
Size: int64(len(filebody)),
Mode: 0o600,
ModTime: time.Unix(int64(sec), int64(nsec)),
}
if err := setTarHeaderTypeflag(hdr, f); err != nil {
return []byte(""), err
}
if err := setTarHeaderFormat(hdr, f); err != nil {
return []byte(""), err
}
tf := &TarFile{
Hdr: hdr,
Body: filebody,
}
tarFiles = append(tarFiles, tf)
}
var buf bytes.Buffer
tw := tar.NewWriter(&buf)
defer tw.Close()
for _, tf := range tarFiles {
tw.WriteHeader(tf.Hdr)
tw.Write(tf.Body)
}
return buf.Bytes(), nil
}
// This is similar to TarBytes, but it returns a series of
// files instead of raw tar bytes. The advantage of this
// api is that it is cheaper in terms of cpu power to
// modify or check the files in the fuzzer with TarFiles()
// because it avoids creating a tar reader.
func (f *ConsumeFuzzer) TarFiles() ([]*TarFile, error) {
numberOfFiles, err := f.GetInt()
if err != nil {
return nil, err
}
var tarFiles []*TarFile
tarFiles = make([]*TarFile, 0)
const maxNoOfFiles = 100
for i := 0; i < numberOfFiles%maxNoOfFiles; i++ {
filename, err := f.getTarFilename()
if err != nil {
return tarFiles, err
}
filebody, err := f.createTarFileBody()
if err != nil {
return tarFiles, err
}
sec, err := f.GetInt()
if err != nil {
return tarFiles, err
}
nsec, err := f.GetInt()
if err != nil {
return tarFiles, err
}
hdr := &tar.Header{
Name: filename,
Size: int64(len(filebody)),
Mode: 0o600,
ModTime: time.Unix(int64(sec), int64(nsec)),
}
if err := setTarHeaderTypeflag(hdr, f); err != nil {
hdr.Typeflag = tar.TypeReg
}
if err := setTarHeaderFormat(hdr, f); err != nil {
return tarFiles, err // should not happend
}
tf := &TarFile{
Hdr: hdr,
Body: filebody,
}
tarFiles = append(tarFiles, tf)
}
return tarFiles, nil
}
// CreateFiles creates pseudo-random files in rootDir.
// It creates subdirs and places the files there.
// It is the callers responsibility to ensure that
// rootDir exists.
func (f *ConsumeFuzzer) CreateFiles(rootDir string) error {
numberOfFiles, err := f.GetInt()
if err != nil {
return err
}
maxNumberOfFiles := numberOfFiles % 4000 // This is completely arbitrary
if maxNumberOfFiles == 0 {
return errors.New("maxNumberOfFiles is nil")
}
var noOfCreatedFiles int
for i := 0; i < maxNumberOfFiles; i++ {
// The file to create:
fileName, err := f.GetString()
if err != nil {
if noOfCreatedFiles > 0 {
// If files have been created, we don't return an error.
break
} else {
return errors.New("could not get fileName")
}
}
if strings.Contains(fileName, "..") || (len(fileName) > 0 && fileName[0] == 47) || strings.Contains(fileName, "\\") {
continue
}
fullFilePath := filepath.Join(rootDir, fileName)
// Find the subdirectory of the file
if subDir := filepath.Dir(fileName); subDir != "" && subDir != "." {
// create the dir first; avoid going outside the root dir
if strings.Contains(subDir, "../") || (len(subDir) > 0 && subDir[0] == 47) || strings.Contains(subDir, "\\") {
continue
}
dirPath := filepath.Join(rootDir, subDir)
if _, err := os.Stat(dirPath); os.IsNotExist(err) {
err2 := os.MkdirAll(dirPath, 0o777)
if err2 != nil {
continue
}
}
fullFilePath = filepath.Join(dirPath, fileName)
} else {
// Create symlink
createSymlink, err := f.GetBool()
if err != nil {
if noOfCreatedFiles > 0 {
break
} else {
return errors.New("could not create the symlink")
}
}
if createSymlink {
symlinkTarget, err := f.GetString()
if err != nil {
return err
}
err = os.Symlink(symlinkTarget, fullFilePath)
if err != nil {
return err
}
// stop loop here, since a symlink needs no further action
noOfCreatedFiles++
continue
}
// We create a normal file
fileContents, err := f.GetBytes()
if err != nil {
if noOfCreatedFiles > 0 {
break
} else {
return errors.New("could not create the file")
}
}
err = os.WriteFile(fullFilePath, fileContents, 0o666)
if err != nil {
continue
}
noOfCreatedFiles++
}
}
return nil
}
// GetStringFrom returns a string that can only consist of characters
// included in possibleChars. It returns an error if the created string
// does not have the specified length.
func (f *ConsumeFuzzer) GetStringFrom(possibleChars string, length int) (string, error) {
if (f.dataTotal - f.position) < uint32(length) {
return "", errors.New("not enough bytes to create a string")
}
output := make([]byte, 0, length)
for i := 0; i < length; i++ {
charIndex, err := f.GetInt()
if err != nil {
return string(output), err
}
output = append(output, possibleChars[charIndex%len(possibleChars)])
}
return string(output), nil
}
func (f *ConsumeFuzzer) GetRune() ([]rune, error) {
stringToConvert, err := f.GetString()
if err != nil {
return []rune("nil"), err
}
return []rune(stringToConvert), nil
}
func (f *ConsumeFuzzer) GetFloat32() (float32, error) {
u32, err := f.GetNBytes(4)
if err != nil {
return 0, err
}
littleEndian, err := f.GetBool()
if err != nil {
return 0, err
}
if littleEndian {
u32LE := binary.LittleEndian.Uint32(u32)
return math.Float32frombits(u32LE), nil
}
u32BE := binary.BigEndian.Uint32(u32)
return math.Float32frombits(u32BE), nil
}
func (f *ConsumeFuzzer) GetFloat64() (float64, error) {
u64, err := f.GetNBytes(8)
if err != nil {
return 0, err
}
littleEndian, err := f.GetBool()
if err != nil {
return 0, err
}
if littleEndian {
u64LE := binary.LittleEndian.Uint64(u64)
return math.Float64frombits(u64LE), nil
}
u64BE := binary.BigEndian.Uint64(u64)
return math.Float64frombits(u64BE), nil
}
func (f *ConsumeFuzzer) CreateSlice(targetSlice interface{}) error {
return f.GenerateStruct(targetSlice)
}
vendor/github.com/AdaLogics/go-fuzz-headers/funcs.go
Generated Vendored
-62
View File
@@ -1,62 +0,0 @@
// Copyright 2023 The go-fuzz-headers Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package gofuzzheaders
import (
"fmt"
"reflect"
)
type Continue struct {
F *ConsumeFuzzer
}
func (f *ConsumeFuzzer) AddFuncs(fuzzFuncs []interface{}) {
for i := range fuzzFuncs {
v := reflect.ValueOf(fuzzFuncs[i])
if v.Kind() != reflect.Func {
panic("Need only funcs!")
}
t := v.Type()
if t.NumIn() != 2 || t.NumOut() != 1 {
fmt.Println(t.NumIn(), t.NumOut())
panic("Need 2 in and 1 out params. In must be the type. Out must be an error")
}
argT := t.In(0)
switch argT.Kind() {
case reflect.Ptr, reflect.Map:
default:
panic("fuzzFunc must take pointer or map type")
}
if t.In(1) != reflect.TypeOf(Continue{}) {
panic("fuzzFunc's second parameter must be type Continue")
}
f.Funcs[argT] = v
}
}
func (f *ConsumeFuzzer) GenerateWithCustom(targetStruct interface{}) error {
e := reflect.ValueOf(targetStruct).Elem()
return f.fuzzStruct(e, true)
}
func (c Continue) GenerateStruct(targetStruct interface{}) error {
return c.F.GenerateStruct(targetStruct)
}
func (c Continue) GenerateStructWithCustom(targetStruct interface{}) error {
return c.F.GenerateWithCustom(targetStruct)
}
vendor/github.com/AdaLogics/go-fuzz-headers/sql.go
Generated Vendored
-556
View File
@@ -1,556 +0,0 @@
// Copyright 2023 The go-fuzz-headers Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package gofuzzheaders
import (
"fmt"
"strings"
)
// returns a keyword by index
func getKeyword(f *ConsumeFuzzer) (string, error) {
index, err := f.GetInt()
if err != nil {
return keywords[0], err
}
for i, k := range keywords {
if i == index {
return k, nil
}
}
return keywords[0], fmt.Errorf("could not get a kw")
}
// Simple utility function to check if a string
// slice contains a string.
func containsString(s []string, e string) bool {
for _, a := range s {
if a == e {
return true
}
}
return false
}
// These keywords are used specifically for fuzzing Vitess
var keywords = []string{
"accessible", "action", "add", "after", "against", "algorithm",
"all", "alter", "always", "analyze", "and", "as", "asc", "asensitive",
"auto_increment", "avg_row_length", "before", "begin", "between",
"bigint", "binary", "_binary", "_utf8mb4", "_utf8", "_latin1", "bit",
"blob", "bool", "boolean", "both", "by", "call", "cancel", "cascade",
"cascaded", "case", "cast", "channel", "change", "char", "character",
"charset", "check", "checksum", "coalesce", "code", "collate", "collation",
"column", "columns", "comment", "committed", "commit", "compact", "complete",
"compressed", "compression", "condition", "connection", "constraint", "continue",
"convert", "copy", "cume_dist", "substr", "substring", "create", "cross",
"csv", "current_date", "current_time", "current_timestamp", "current_user",
"cursor", "data", "database", "databases", "day", "day_hour", "day_microsecond",
"day_minute", "day_second", "date", "datetime", "dec", "decimal", "declare",
"default", "definer", "delay_key_write", "delayed", "delete", "dense_rank",
"desc", "describe", "deterministic", "directory", "disable", "discard",
"disk", "distinct", "distinctrow", "div", "double", "do", "drop", "dumpfile",
"duplicate", "dynamic", "each", "else", "elseif", "empty", "enable",
"enclosed", "encryption", "end", "enforced", "engine", "engines", "enum",
"error", "escape", "escaped", "event", "exchange", "exclusive", "exists",
"exit", "explain", "expansion", "export", "extended", "extract", "false",
"fetch", "fields", "first", "first_value", "fixed", "float", "float4",
"float8", "flush", "for", "force", "foreign", "format", "from", "full",
"fulltext", "function", "general", "generated", "geometry", "geometrycollection",
"get", "global", "gtid_executed", "grant", "group", "grouping", "groups",
"group_concat", "having", "header", "high_priority", "hosts", "hour", "hour_microsecond",
"hour_minute", "hour_second", "if", "ignore", "import", "in", "index", "indexes",
"infile", "inout", "inner", "inplace", "insensitive", "insert", "insert_method",
"int", "int1", "int2", "int3", "int4", "int8", "integer", "interval",
"into", "io_after_gtids", "is", "isolation", "iterate", "invoker", "join",
"json", "json_table", "key", "keys", "keyspaces", "key_block_size", "kill", "lag",
"language", "last", "last_value", "last_insert_id", "lateral", "lead", "leading",
"leave", "left", "less", "level", "like", "limit", "linear", "lines",
"linestring", "load", "local", "localtime", "localtimestamp", "lock", "logs",
"long", "longblob", "longtext", "loop", "low_priority", "manifest",
"master_bind", "match", "max_rows", "maxvalue", "mediumblob", "mediumint",
"mediumtext", "memory", "merge", "microsecond", "middleint", "min_rows", "minute",
"minute_microsecond", "minute_second", "mod", "mode", "modify", "modifies",
"multilinestring", "multipoint", "multipolygon", "month", "name",
"names", "natural", "nchar", "next", "no", "none", "not", "no_write_to_binlog",
"nth_value", "ntile", "null", "numeric", "of", "off", "offset", "on",
"only", "open", "optimize", "optimizer_costs", "option", "optionally",
"or", "order", "out", "outer", "outfile", "over", "overwrite", "pack_keys",
"parser", "partition", "partitioning", "password", "percent_rank", "plugins",
"point", "polygon", "precision", "primary", "privileges", "processlist",
"procedure", "query", "quarter", "range", "rank", "read", "reads", "read_write",
"real", "rebuild", "recursive", "redundant", "references", "regexp", "relay",
"release", "remove", "rename", "reorganize", "repair", "repeat", "repeatable",
"replace", "require", "resignal", "restrict", "return", "retry", "revert",
"revoke", "right", "rlike", "rollback", "row", "row_format", "row_number",
"rows", "s3", "savepoint", "schema", "schemas", "second", "second_microsecond",
"security", "select", "sensitive", "separator", "sequence", "serializable",
"session", "set", "share", "shared", "show", "signal", "signed", "slow",
"smallint", "spatial", "specific", "sql", "sqlexception", "sqlstate",
"sqlwarning", "sql_big_result", "sql_cache", "sql_calc_found_rows",
"sql_no_cache", "sql_small_result", "ssl", "start", "starting",
"stats_auto_recalc", "stats_persistent", "stats_sample_pages", "status",
"storage", "stored", "straight_join", "stream", "system", "vstream",
"table", "tables", "tablespace", "temporary", "temptable", "terminated",
"text", "than", "then", "time", "timestamp", "timestampadd", "timestampdiff",
"tinyblob", "tinyint", "tinytext", "to", "trailing", "transaction", "tree",
"traditional", "trigger", "triggers", "true", "truncate", "uncommitted",
"undefined", "undo", "union", "unique", "unlock", "unsigned", "update",
"upgrade", "usage", "use", "user", "user_resources", "using", "utc_date",
"utc_time", "utc_timestamp", "validation", "values", "variables", "varbinary",
"varchar", "varcharacter", "varying", "vgtid_executed", "virtual", "vindex",
"vindexes", "view", "vitess", "vitess_keyspaces", "vitess_metadata",
"vitess_migration", "vitess_migrations", "vitess_replication_status",
"vitess_shards", "vitess_tablets", "vschema", "warnings", "when",
"where", "while", "window", "with", "without", "work", "write", "xor",
"year", "year_month", "zerofill",
}
// Keywords that could get an additional keyword
var needCustomString = []string{
"DISTINCTROW", "FROM", // Select keywords:
"GROUP BY", "HAVING", "WINDOW",
"FOR",
"ORDER BY", "LIMIT",
"INTO", "PARTITION", "AS", // Insert Keywords:
"ON DUPLICATE KEY UPDATE",
"WHERE", "LIMIT", // Delete keywords
"INFILE", "INTO TABLE", "CHARACTER SET", // Load keywords
"TERMINATED BY", "ENCLOSED BY",
"ESCAPED BY", "STARTING BY",
"TERMINATED BY", "STARTING BY",
"IGNORE",
"VALUE", "VALUES", // Replace tokens
"SET", // Update tokens
"ENGINE =", // Drop tokens
"DEFINER =", "ON SCHEDULE", "RENAME TO", // Alter tokens
"COMMENT", "DO", "INITIAL_SIZE = ", "OPTIONS",
}
var alterTableTokens = [][]string{
{"CUSTOM_FUZZ_STRING"},
{"CUSTOM_ALTTER_TABLE_OPTIONS"},
{"PARTITION_OPTIONS_FOR_ALTER_TABLE"},
}
var alterTokens = [][]string{
{
"DATABASE", "SCHEMA", "DEFINER = ", "EVENT", "FUNCTION", "INSTANCE",
"LOGFILE GROUP", "PROCEDURE", "SERVER",
},
{"CUSTOM_FUZZ_STRING"},
{
"ON SCHEDULE", "ON COMPLETION PRESERVE", "ON COMPLETION NOT PRESERVE",
"ADD UNDOFILE", "OPTIONS",
},
{"RENAME TO", "INITIAL_SIZE = "},
{"ENABLE", "DISABLE", "DISABLE ON SLAVE", "ENGINE"},
{"COMMENT"},
{"DO"},
}
var setTokens = [][]string{
{"CHARACTER SET", "CHARSET", "CUSTOM_FUZZ_STRING", "NAMES"},
{"CUSTOM_FUZZ_STRING", "DEFAULT", "="},
{"CUSTOM_FUZZ_STRING"},
}
var dropTokens = [][]string{
{"TEMPORARY", "UNDO"},
{
"DATABASE", "SCHEMA", "EVENT", "INDEX", "LOGFILE GROUP",
"PROCEDURE", "FUNCTION", "SERVER", "SPATIAL REFERENCE SYSTEM",
"TABLE", "TABLESPACE", "TRIGGER", "VIEW",
},
{"IF EXISTS"},
{"CUSTOM_FUZZ_STRING"},
{"ON", "ENGINE = ", "RESTRICT", "CASCADE"},
}
var renameTokens = [][]string{
{"TABLE"},
{"CUSTOM_FUZZ_STRING"},
{"TO"},
{"CUSTOM_FUZZ_STRING"},
}
var truncateTokens = [][]string{
{"TABLE"},
{"CUSTOM_FUZZ_STRING"},
}
var createTokens = [][]string{
{"OR REPLACE", "TEMPORARY", "UNDO"}, // For create spatial reference system
{
"UNIQUE", "FULLTEXT", "SPATIAL", "ALGORITHM = UNDEFINED", "ALGORITHM = MERGE",
"ALGORITHM = TEMPTABLE",
},
{
"DATABASE", "SCHEMA", "EVENT", "FUNCTION", "INDEX", "LOGFILE GROUP",
"PROCEDURE", "SERVER", "SPATIAL REFERENCE SYSTEM", "TABLE", "TABLESPACE",
"TRIGGER", "VIEW",
},
{"IF NOT EXISTS"},
{"CUSTOM_FUZZ_STRING"},
}
/*
// For future use.
var updateTokens = [][]string{
{"LOW_PRIORITY"},
{"IGNORE"},
{"SET"},
{"WHERE"},
{"ORDER BY"},
{"LIMIT"},
}
*/
var replaceTokens = [][]string{
{"LOW_PRIORITY", "DELAYED"},
{"INTO"},
{"PARTITION"},
{"CUSTOM_FUZZ_STRING"},
{"VALUES", "VALUE"},
}
var loadTokens = [][]string{
{"DATA"},
{"LOW_PRIORITY", "CONCURRENT", "LOCAL"},
{"INFILE"},
{"REPLACE", "IGNORE"},
{"INTO TABLE"},
{"PARTITION"},
{"CHARACTER SET"},
{"FIELDS", "COLUMNS"},
{"TERMINATED BY"},
{"OPTIONALLY"},
{"ENCLOSED BY"},
{"ESCAPED BY"},
{"LINES"},
{"STARTING BY"},
{"TERMINATED BY"},
{"IGNORE"},
{"LINES", "ROWS"},
{"CUSTOM_FUZZ_STRING"},
}
// These Are everything that comes after "INSERT"
var insertTokens = [][]string{
{"LOW_PRIORITY", "DELAYED", "HIGH_PRIORITY", "IGNORE"},
{"INTO"},
{"PARTITION"},
{"CUSTOM_FUZZ_STRING"},
{"AS"},
{"ON DUPLICATE KEY UPDATE"},
}
// These are everything that comes after "SELECT"
var selectTokens = [][]string{
{"*", "CUSTOM_FUZZ_STRING", "DISTINCTROW"},
{"HIGH_PRIORITY"},
{"STRAIGHT_JOIN"},
{"SQL_SMALL_RESULT", "SQL_BIG_RESULT", "SQL_BUFFER_RESULT"},
{"SQL_NO_CACHE", "SQL_CALC_FOUND_ROWS"},
{"CUSTOM_FUZZ_STRING"},
{"FROM"},
{"WHERE"},
{"GROUP BY"},
{"HAVING"},
{"WINDOW"},
{"ORDER BY"},
{"LIMIT"},
{"CUSTOM_FUZZ_STRING"},
{"FOR"},
}
// These are everything that comes after "DELETE"
var deleteTokens = [][]string{
{"LOW_PRIORITY", "QUICK", "IGNORE", "FROM", "AS"},
{"PARTITION"},
{"WHERE"},
{"ORDER BY"},
{"LIMIT"},
}
var alter_table_options = []string{
"ADD", "COLUMN", "FIRST", "AFTER", "INDEX", "KEY", "FULLTEXT", "SPATIAL",
"CONSTRAINT", "UNIQUE", "FOREIGN KEY", "CHECK", "ENFORCED", "DROP", "ALTER",
"NOT", "INPLACE", "COPY", "SET", "VISIBLE", "INVISIBLE", "DEFAULT", "CHANGE",
"CHARACTER SET", "COLLATE", "DISABLE", "ENABLE", "KEYS", "TABLESPACE", "LOCK",
"FORCE", "MODIFY", "SHARED", "EXCLUSIVE", "NONE", "ORDER BY", "RENAME COLUMN",
"AS", "=", "ASC", "DESC", "WITH", "WITHOUT", "VALIDATION", "ADD PARTITION",
"DROP PARTITION", "DISCARD PARTITION", "IMPORT PARTITION", "TRUNCATE PARTITION",
"COALESCE PARTITION", "REORGANIZE PARTITION", "EXCHANGE PARTITION",
"ANALYZE PARTITION", "CHECK PARTITION", "OPTIMIZE PARTITION", "REBUILD PARTITION",
"REPAIR PARTITION", "REMOVE PARTITIONING", "USING", "BTREE", "HASH", "COMMENT",
"KEY_BLOCK_SIZE", "WITH PARSER", "AUTOEXTEND_SIZE", "AUTO_INCREMENT", "AVG_ROW_LENGTH",
"CHECKSUM", "INSERT_METHOD", "ROW_FORMAT", "DYNAMIC", "FIXED", "COMPRESSED", "REDUNDANT",
"COMPACT", "SECONDARY_ENGINE_ATTRIBUTE", "STATS_AUTO_RECALC", "STATS_PERSISTENT",
"STATS_SAMPLE_PAGES", "ZLIB", "LZ4", "ENGINE_ATTRIBUTE", "KEY_BLOCK_SIZE", "MAX_ROWS",
"MIN_ROWS", "PACK_KEYS", "PASSWORD", "COMPRESSION", "CONNECTION", "DIRECTORY",
"DELAY_KEY_WRITE", "ENCRYPTION", "STORAGE", "DISK", "MEMORY", "UNION",
}
// Creates an 'alter table' statement. 'alter table' is an exception
// in that it has its own function. The majority of statements
// are created by 'createStmt()'.
func createAlterTableStmt(f *ConsumeFuzzer) (string, error) {
maxArgs, err := f.GetInt()
if err != nil {
return "", err
}
maxArgs = maxArgs % 30
if maxArgs == 0 {
return "", fmt.Errorf("could not create alter table stmt")
}
var stmt strings.Builder
stmt.WriteString("ALTER TABLE ")
for i := 0; i < maxArgs; i++ {
// Calculate if we get existing token or custom string
tokenType, err := f.GetInt()
if err != nil {
return "", err
}
if tokenType%4 == 1 {
customString, err := f.GetString()
if err != nil {
return "", err
}
stmt.WriteString(" " + customString)
} else {
tokenIndex, err := f.GetInt()
if err != nil {
return "", err
}
stmt.WriteString(" " + alter_table_options[tokenIndex%len(alter_table_options)])
}
}
return stmt.String(), nil
}
func chooseToken(tokens []string, f *ConsumeFuzzer) (string, error) {
index, err := f.GetInt()
if err != nil {
return "", err
}
var token strings.Builder
token.WriteString(tokens[index%len(tokens)])
if token.String() == "CUSTOM_FUZZ_STRING" {
customFuzzString, err := f.GetString()
if err != nil {
return "", err
}
return customFuzzString, nil
}
// Check if token requires an argument
if containsString(needCustomString, token.String()) {
customFuzzString, err := f.GetString()
if err != nil {
return "", err
}
token.WriteString(" " + customFuzzString)
}
return token.String(), nil
}
var stmtTypes = map[string][][]string{
"DELETE": deleteTokens,
"INSERT": insertTokens,
"SELECT": selectTokens,
"LOAD": loadTokens,
"REPLACE": replaceTokens,
"CREATE": createTokens,
"DROP": dropTokens,
"RENAME": renameTokens,
"TRUNCATE": truncateTokens,
"SET": setTokens,
"ALTER": alterTokens,
"ALTER TABLE": alterTableTokens, // ALTER TABLE has its own set of tokens
}
var stmtTypeEnum = map[int]string{
0: "DELETE",
1: "INSERT",
2: "SELECT",
3: "LOAD",
4: "REPLACE",
5: "CREATE",
6: "DROP",
7: "RENAME",
8: "TRUNCATE",
9: "SET",
10: "ALTER",
11: "ALTER TABLE",
}
func createStmt(f *ConsumeFuzzer) (string, error) {
stmtIndex, err := f.GetInt()
if err != nil {
return "", err
}
stmtIndex = stmtIndex % len(stmtTypes)
queryType := stmtTypeEnum[stmtIndex]
tokens := stmtTypes[queryType]
// We have custom creator for ALTER TABLE
if queryType == "ALTER TABLE" {
query, err := createAlterTableStmt(f)
if err != nil {
return "", err
}
return query, nil
}
// Here we are creating a query that is not
// an 'alter table' query. For available
// queries, see "stmtTypes"
// First specify the first query keyword:
var query strings.Builder
query.WriteString(queryType)
// Next create the args for the
queryArgs, err := createStmtArgs(tokens, f)
if err != nil {
return "", err
}
query.WriteString(" " + queryArgs)
return query.String(), nil
}
// Creates the arguments of a statements. In a select statement
// that would be everything after "select".
func createStmtArgs(tokenslice [][]string, f *ConsumeFuzzer) (string, error) {
var query, token strings.Builder
// We go through the tokens in the tokenslice,
// create the respective token and add it to
// "query"
for _, tokens := range tokenslice {
// For extra randomization, the fuzzer can
// choose to not include this token.
includeThisToken, err := f.GetBool()
if err != nil {
return "", err
}
if !includeThisToken {
continue
}
// There may be several tokens to choose from:
if len(tokens) > 1 {
chosenToken, err := chooseToken(tokens, f)
if err != nil {
return "", err
}
query.WriteString(" " + chosenToken)
} else {
token.WriteString(tokens[0])
// In case the token is "CUSTOM_FUZZ_STRING"
// we will then create a non-structured string
if token.String() == "CUSTOM_FUZZ_STRING" {
customFuzzString, err := f.GetString()
if err != nil {
return "", err
}
query.WriteString(" " + customFuzzString)
continue
}
// Check if token requires an argument.
// Tokens that take an argument can be found
// in 'needCustomString'. If so, we add a
// non-structured string to the token.
if containsString(needCustomString, token.String()) {
customFuzzString, err := f.GetString()
if err != nil {
return "", err
}
token.WriteString(fmt.Sprintf(" %s", customFuzzString))
}
query.WriteString(fmt.Sprintf(" %s", token.String()))
}
}
return query.String(), nil
}
// Creates a semi-structured query. It creates a string
// that is a combination of the keywords and random strings.
func createQuery(f *ConsumeFuzzer) (string, error) {
queryLen, err := f.GetInt()
if err != nil {
return "", err
}
maxLen := queryLen % 60
if maxLen == 0 {
return "", fmt.Errorf("could not create a query")
}
var query strings.Builder
for i := 0; i < maxLen; i++ {
// Get a new token:
useKeyword, err := f.GetBool()
if err != nil {
return "", err
}
if useKeyword {
keyword, err := getKeyword(f)
if err != nil {
return "", err
}
query.WriteString(" " + keyword)
} else {
customString, err := f.GetString()
if err != nil {
return "", err
}
query.WriteString(" " + customString)
}
}
if query.String() == "" {
return "", fmt.Errorf("could not create a query")
}
return query.String(), nil
}
// GetSQLString is the API that users interact with.
//
// Usage:
//
// f := NewConsumer(data)
// sqlString, err := f.GetSQLString()
func (f *ConsumeFuzzer) GetSQLString() (string, error) {
var query string
veryStructured, err := f.GetBool()
if err != nil {
return "", err
}
if veryStructured {
query, err = createStmt(f)
if err != nil {
return "", err
}
} else {
query, err = createQuery(f)
if err != nil {
return "", err
}
}
return query, nil
}
vendor/github.com/containerd/console/console_other.go
Generated Vendored
+2 -2
View File
@@ -1,5 +1,5 @@
//go:build !darwin && !freebsd && !linux && !netbsd && !openbsd && !solaris && !windows && !zos
// +build !darwin,!freebsd,!linux,!netbsd,!openbsd,!solaris,!windows,!zos
//go:build !darwin && !freebsd && !linux && !netbsd && !openbsd && !windows && !zos
// +build !darwin,!freebsd,!linux,!netbsd,!openbsd,!windows,!zos
/*
Copyright The containerd Authors.
vendor/github.com/containerd/console/console_unix.go
Generated Vendored
+9
View File
@@ -31,6 +31,15 @@ func NewPty() (Console, string, error) {
if err != nil {
return nil, "", err
}
return NewPtyFromFile(f)
}
// NewPtyFromFile creates a new pty pair, just like [NewPty] except that the
// provided [os.File] is used as the master rather than automatically creating
// a new master from /dev/ptmx. The ownership of [os.File] is passed to the
// returned [Console], so the caller must be careful to not call Close on the
// underlying file.
func NewPtyFromFile(f File) (Console, string, error) {
slave, err := ptsname(f)
if err != nil {
return nil, "", err
vendor/github.com/containerd/console/tc_darwin.go
Generated Vendored
+2 -3
View File
@@ -18,7 +18,6 @@ package console
import (
"fmt"
"os"
"golang.org/x/sys/unix"
)
@@ -30,12 +29,12 @@ const (
// unlockpt unlocks the slave pseudoterminal device corresponding to the master pseudoterminal referred to by f.
// unlockpt should be called before opening the slave side of a pty.
func unlockpt(f *os.File) error {
func unlockpt(f File) error {
return unix.IoctlSetPointerInt(int(f.Fd()), unix.TIOCPTYUNLK, 0)
}
// ptsname retrieves the name of the first available pts for the given master.
func ptsname(f *os.File) (string, error) {
func ptsname(f File) (string, error) {
n, err := unix.IoctlGetInt(int(f.Fd()), unix.TIOCPTYGNAME)
if err != nil {
return "", err
vendor/github.com/containerd/console/tc_freebsd_cgo.go
Generated Vendored
+2 -3
View File
@@ -21,7 +21,6 @@ package console
import (
"fmt"
"os"
"golang.org/x/sys/unix"
)
@@ -39,7 +38,7 @@ const (
// unlockpt unlocks the slave pseudoterminal device corresponding to the master pseudoterminal referred to by f.
// unlockpt should be called before opening the slave side of a pty.
func unlockpt(f *os.File) error {
func unlockpt(f File) error {
fd := C.int(f.Fd())
if _, err := C.unlockpt(fd); err != nil {
C.close(fd)
@@ -49,7 +48,7 @@ func unlockpt(f *os.File) error {
}
// ptsname retrieves the name of the first available pts for the given master.
func ptsname(f *os.File) (string, error) {
func ptsname(f File) (string, error) {
n, err := unix.IoctlGetInt(int(f.Fd()), unix.TIOCGPTN)
if err != nil {
return "", err
vendor/github.com/containerd/console/tc_freebsd_nocgo.go
Generated Vendored
+2 -3
View File
@@ -21,7 +21,6 @@ package console
import (
"fmt"
"os"
"golang.org/x/sys/unix"
)
@@ -42,12 +41,12 @@ const (
// unlockpt unlocks the slave pseudoterminal device corresponding to the master pseudoterminal referred to by f.
// unlockpt should be called before opening the slave side of a pty.
func unlockpt(f *os.File) error {
func unlockpt(f File) error {
panic("unlockpt() support requires cgo.")
}
// ptsname retrieves the name of the first available pts for the given master.
func ptsname(f *os.File) (string, error) {
func ptsname(f File) (string, error) {
n, err := unix.IoctlGetInt(int(f.Fd()), unix.TIOCGPTN)
if err != nil {
return "", err
vendor/github.com/containerd/console/tc_linux.go
Generated Vendored
+2 -3
View File
@@ -18,7 +18,6 @@ package console
import (
"fmt"
"os"
"unsafe"
"golang.org/x/sys/unix"
@@ -31,7 +30,7 @@ const (
// unlockpt unlocks the slave pseudoterminal device corresponding to the master pseudoterminal referred to by f.
// unlockpt should be called before opening the slave side of a pty.
func unlockpt(f *os.File) error {
func unlockpt(f File) error {
var u int32
// XXX do not use unix.IoctlSetPointerInt here, see commit dbd69c59b81.
if _, _, err := unix.Syscall(unix.SYS_IOCTL, f.Fd(), unix.TIOCSPTLCK, uintptr(unsafe.Pointer(&u))); err != 0 {
@@ -41,7 +40,7 @@ func unlockpt(f *os.File) error {
}
// ptsname retrieves the name of the first available pts for the given master.
func ptsname(f *os.File) (string, error) {
func ptsname(f File) (string, error) {
var u uint32
// XXX do not use unix.IoctlGetInt here, see commit dbd69c59b81.
if _, _, err := unix.Syscall(unix.SYS_IOCTL, f.Fd(), unix.TIOCGPTN, uintptr(unsafe.Pointer(&u))); err != 0 {
vendor/github.com/containerd/console/tc_netbsd.go
Generated Vendored
+2 -3
View File
@@ -18,7 +18,6 @@ package console
import (
"bytes"
"os"
"golang.org/x/sys/unix"
)
@@ -31,12 +30,12 @@ const (
// unlockpt unlocks the slave pseudoterminal device corresponding to the master pseudoterminal referred to by f.
// unlockpt should be called before opening the slave side of a pty.
// This does not exist on NetBSD, it does not allocate controlling terminals on open
func unlockpt(f *os.File) error {
func unlockpt(f File) error {
return nil
}
// ptsname retrieves the name of the first available pts for the given master.
func ptsname(f *os.File) (string, error) {
func ptsname(f File) (string, error) {
ptm, err := unix.IoctlGetPtmget(int(f.Fd()), unix.TIOCPTSNAME)
if err != nil {
return "", err
vendor/github.com/containerd/console/tc_openbsd_cgo.go
Generated Vendored
+2 -4
View File
@@ -20,8 +20,6 @@
package console
import (
"os"
"golang.org/x/sys/unix"
)
@@ -34,7 +32,7 @@ const (
)
// ptsname retrieves the name of the first available pts for the given master.
func ptsname(f *os.File) (string, error) {
func ptsname(f File) (string, error) {
ptspath, err := C.ptsname(C.int(f.Fd()))
if err != nil {
return "", err
@@ -44,7 +42,7 @@ func ptsname(f *os.File) (string, error) {
// unlockpt unlocks the slave pseudoterminal device corresponding to the master pseudoterminal referred to by f.
// unlockpt should be called before opening the slave side of a pty.
func unlockpt(f *os.File) error {
func unlockpt(f File) error {
if _, err := C.grantpt(C.int(f.Fd())); err != nil {
return err
}
vendor/github.com/containerd/console/tc_openbsd_nocgo.go
Generated Vendored
+2 -4
View File
@@ -29,8 +29,6 @@
package console
import (
"os"
"golang.org/x/sys/unix"
)
@@ -39,10 +37,10 @@ const (
cmdTcSet = unix.TIOCSETA
)
func ptsname(f *os.File) (string, error) {
func ptsname(f File) (string, error) {
panic("ptsname() support requires cgo.")
}
func unlockpt(f *os.File) error {
func unlockpt(f File) error {
panic("unlockpt() support requires cgo.")
}
vendor/github.com/containerd/console/tc_zos.go
Generated Vendored
+2 -3
View File
@@ -17,7 +17,6 @@
package console
import (
"os"
"strings"
"golang.org/x/sys/unix"
@@ -29,11 +28,11 @@ const (
)
// unlockpt is a no-op on zos.
func unlockpt(_ *os.File) error {
func unlockpt(File) error {
return nil
}
// ptsname retrieves the name of the first available pts for the given master.
func ptsname(f *os.File) (string, error) {
func ptsname(f File) (string, error) {
return "/dev/ttyp" + strings.TrimPrefix(f.Name(), "/dev/ptyp"), nil
}
vendor/github.com/containerd/containerd/v2/core/content/proxy/content_writer.go
Generated Vendored
+2 -7
View File
@@ -20,6 +20,7 @@ import (
"context"
"fmt"
"io"
"slices"
contentapi "github.com/containerd/containerd/api/services/content/v1"
"github.com/containerd/errdefs/pkg/errgrpc"
@@ -78,15 +79,9 @@ func (rw *remoteWriter) Digest() digest.Digest {
func (rw *remoteWriter) Write(p []byte) (n int, err error) {
const maxBufferSize = defaults.DefaultMaxSendMsgSize >> 1
for i := 0; i < len(p); i += maxBufferSize {
for data := range slices.Chunk(p, maxBufferSize) {
offset := rw.offset
end := i + maxBufferSize
if end > len(p) {
end = len(p)
}
data := p[i:end]
resp, err := rw.send(&contentapi.WriteContentRequest{
Action: contentapi.WriteAction_WRITE,
Offset: offset,
vendor/github.com/containerd/containerd/v2/core/images/handlers.go
Generated Vendored
-2
View File
@@ -154,8 +154,6 @@ func WalkNotEmpty(ctx context.Context, handler Handler, descs ...ocispec.Descrip
func Dispatch(ctx context.Context, handler Handler, limiter *semaphore.Weighted, descs ...ocispec.Descriptor) error {
eg, ctx2 := errgroup.WithContext(ctx)
for _, desc := range descs {
desc := desc
if limiter != nil {
if err := limiter.Acquire(ctx, 1); err != nil {
return err
vendor/github.com/containerd/containerd/v2/core/images/labels.go
Generated Vendored
-21
View File
@@ -1,21 +0,0 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package images
const (
ConvertedDockerSchema1LabelKey = "io.containerd.image/converted-docker-schema1"
)
vendor/github.com/containerd/containerd/v2/core/remotes/docker/auth/fetch.go
Generated Vendored
+2 -2
View File
@@ -78,10 +78,10 @@ type TokenOptions struct {
// FetchRefreshToken enables fetching a refresh token (aka "identity token", "offline token") along with the bearer token.
//
// For HTTP GET mode (FetchToken), FetchRefreshToken sets `offline_token=true` in the request.
// https://docs.docker.com/registry/spec/auth/token/#requesting-a-token
// https://distribution.github.io/distribution/spec/auth/token/#requesting-a-token
//
// For HTTP POST mode (FetchTokenWithOAuth), FetchRefreshToken sets `access_type=offline` in the request.
// https://docs.docker.com/registry/spec/auth/oauth/#getting-a-token
// https://distribution.github.io/distribution/spec/auth/oauth/#getting-a-token
FetchRefreshToken bool
}
vendor/github.com/containerd/containerd/v2/core/remotes/docker/authorizer.go
Generated Vendored
+12 -3
View File
@@ -70,9 +70,18 @@ func WithAuthCreds(creds func(string) (string, string, error)) AuthorizerOpt {
}
// WithAuthHeader provides HTTP headers for authorization
//
// We need to merge instead of replacing because header may be set by
// a per-host hosts.toml or/AND by a global header config (e.g., cri.config.headers)
func WithAuthHeader(hdr http.Header) AuthorizerOpt {
return func(opt *authorizerConfig) {
opt.header = hdr
if opt.header == nil {
opt.header = hdr.Clone()
} else {
for k, v := range hdr {
opt.header[k] = append(opt.header[k], v...)
}
}
}
}
@@ -88,7 +97,7 @@ func WithFetchRefreshToken(f OnFetchRefreshToken) AuthorizerOpt {
// NewDockerAuthorizer creates an authorizer using Docker's registry
// authentication spec.
// See https://docs.docker.com/registry/spec/auth/
// See https://distribution.github.io/distribution/spec/auth/
func NewDockerAuthorizer(opts ...AuthorizerOpt) Authorizer {
var ao authorizerConfig
for _, opt := range opts {
@@ -269,7 +278,7 @@ func (ah *authHandler) doBearerAuth(ctx context.Context) (token, refreshToken st
to.Scopes = GetTokenScopes(ctx, to.Scopes)
// Docs: https://docs.docker.com/registry/spec/auth/scope
// Docs: https://distribution.github.io/distribution/spec/auth/scope/
scoped := strings.Join(to.Scopes, " ")
// Keep track of the expiration time of cached bearer tokens so they can be
vendor/github.com/containerd/containerd/v2/core/remotes/docker/converter_fuzz.go
Generated Vendored
-54
View File
@@ -1,54 +0,0 @@
//go:build gofuzz
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package docker
import (
"context"
"os"
fuzz "github.com/AdaLogics/go-fuzz-headers"
"github.com/containerd/containerd/v2/plugins/content/local"
"github.com/containerd/log"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)
func FuzzConvertManifest(data []byte) int {
ctx := context.Background()
// Do not log the message below
// level=warning msg="do nothing for media type: ..."
log.G(ctx).Logger.SetLevel(log.PanicLevel)
f := fuzz.NewConsumer(data)
desc := ocispec.Descriptor{}
err := f.GenerateStruct(&desc)
if err != nil {
return 0
}
tmpdir, err := os.MkdirTemp("", "fuzzing-")
if err != nil {
return 0
}
cs, err := local.NewStore(tmpdir)
if err != nil {
return 0
}
_, _ = ConvertManifest(ctx, cs, desc)
return 1
}
vendor/github.com/containerd/containerd/v2/core/remotes/docker/fetcher.go
Generated Vendored
+326 -75
View File
@@ -17,26 +17,202 @@
package docker
import (
"bytes"
"compress/flate"
"compress/gzip"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"strings"
"sync"
"github.com/containerd/containerd/v2/core/images"
"github.com/containerd/containerd/v2/core/remotes"
"github.com/containerd/errdefs"
"github.com/containerd/log"
"github.com/klauspost/compress/zstd"
digest "github.com/opencontainers/go-digest"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/containerd/containerd/v2/core/images"
"github.com/containerd/containerd/v2/core/remotes"
)
type bufferPool struct {
pool *sync.Pool
}
func newbufferPool(bufCap int64) *bufferPool {
pool := &sync.Pool{
New: func() any {
return bytes.NewBuffer(make([]byte, 0, bufCap))
},
}
return &bufferPool{
pool: pool,
}
}
func (p *bufferPool) Get() *bytes.Buffer {
buf := p.pool.Get().(*bytes.Buffer)
return buf
}
func (p *bufferPool) Put(buffer *bytes.Buffer) {
p.pool.Put(buffer)
}
var ErrClosedPipe = errors.New("bufpipe: read/write on closed pipe")
// pipe implements an asynchronous buffered pipe designed for high-throughput
// I/O with configurable initial buffer sizes and buffer reuse. It decouples
// read/write operations, allowing writers to proceed without blocking (unless
// the pipe is closed) and readers to wait efficiently for incoming data.
//
// Key Characteristics:
// - Asynchronous Operation: Writers populate buffers independently of read
// timing, enabling continuous data flow without reader-writer synchronization.
// - Dynamic Buffering: Active buffer grows organically to handle large payloads, while
// the initial capacity (bufCap) balances memory pre-allocation and growth overhead.
// - Buffer Recycling: Retrieved from/pushed to a pool to minimize allocations, reducing
// garbage collection pressure in sustained I/O scenarios.
// - Error Semantics: Closes deterministically on first error (read or write), propagating
// errors atomically to both ends while draining buffered data.
//
// Difference with io.Pipe:
// - Unlike io.Pipe's strict synchronization (blocking write until read), this implementation
// allows writers to buffer data ahead of reads, improving throughput for bursty workloads.
//
// Synchronization & Internals:
// - Condition Variable (sync.Cond): Coordinates reader/writer, waking readers on new data
// or closure. Locking is centralized via the condition's mutex.
// - Buffer Lifecycle: Active buffer serves writes until read depletion, after which it's
// recycled to the pool. Pooled buffers retain their capacity across uses.
// - Error Handling: Write errors (werr) permanently fail writes; read errors (rerr) mark
// terminal read state after buffer exhaustion.
//
// Future Considerations:
// - Zero-copy reads/writes to avoid buffer copying overhead.
// - Memory-mapped file backing for multi-gigabyte payloads.
type pipe struct {
cond *sync.Cond // Coordinates read/write signaling via Lock+Wait/Signal
bufPool *bufferPool // Reusable buffers with initial capacity bufCap
buf *bytes.Buffer // Active data buffer (nil when empty/returned to pool)
rerr, werr error // Terminal read/write errors (sticky once set)
}
type pipeReader struct {
*pipe
}
type pipeWriter struct {
*pipe
}
func newPipeWriter(bufPool *bufferPool) (*pipeReader, *pipeWriter) {
p := &pipe{
cond: sync.NewCond(new(sync.Mutex)),
bufPool: bufPool,
buf: nil,
}
return &pipeReader{
pipe: p,
}, &pipeWriter{
pipe: p,
}
}
// Read implements the standard Read interface: it reads data from the pipe,
// reading from the internal buffer, otherwise blocking until a writer arrives
// or the write end is closed. If the write end is closed with an error, that
// error is returned as err; otherwise err is io.EOF.
func (r *pipeReader) Read(data []byte) (n int, err error) {
r.cond.L.Lock()
defer r.cond.L.Unlock()
if r.buf == nil {
r.buf = r.bufPool.Get()
}
for {
n, err = r.buf.Read(data)
// If not closed and no read, wait for writing.
if err == io.EOF && r.rerr == nil && n == 0 {
r.cond.Wait() // Wait for data to be written
continue
}
break
}
if err == io.EOF {
// Put buffer back to pool
r.bufPool.Put(r.buf)
r.buf = nil
return n, r.rerr
}
return n, err
}
// Close closes the reader; subsequent writes from the write half of the pipe
// will return error ErrClosedPipe.
func (r *pipeReader) Close() error {
return r.CloseWithError(nil)
}
// CloseWithError closes the reader; subsequent writes to the write half of the
// pipe will return the error err.
func (r *pipeReader) CloseWithError(err error) error {
r.cond.L.Lock()
defer r.cond.L.Unlock()
if err == nil {
err = ErrClosedPipe
}
r.werr = err
return nil
}
// Write implements the standard Write interface: it writes data to the internal
// buffer. If the read end is closed with an error, that err is returned as err;
// otherwise err is ErrClosedPipe.
func (w *pipeWriter) Write(data []byte) (int, error) {
w.cond.L.Lock()
defer w.cond.L.Unlock()
if w.werr != nil {
return 0, w.werr
}
if w.buf == nil {
w.buf = w.bufPool.Get()
}
n, err := w.buf.Write(data)
w.cond.Signal()
return n, err
}
// Close closes the writer; subsequent reads from the read half of the pipe will
// return io.EOF once the internal buffer get empty.
func (w *pipeWriter) Close() error {
return w.CloseWithError(nil)
}
// Close closes the writer; subsequent reads from the read half of the pipe will
// return err once the internal buffer get empty.
func (w *pipeWriter) CloseWithError(err error) error {
w.cond.L.Lock()
defer w.cond.L.Unlock()
if err == nil {
err = io.EOF
}
w.rerr = err
w.cond.Broadcast()
return nil
}
type dockerFetcher struct {
*dockerBase
}
@@ -84,7 +260,7 @@ func (r dockerFetcher) Fetch(ctx context.Context, desc ocispec.Descriptor) (io.R
req.path = req.path + "?" + u.RawQuery
}
rc, err := r.open(ctx, req, desc.MediaType, offset)
rc, err := r.open(ctx, req, desc.MediaType, offset, false)
if err != nil {
if errdefs.IsNotFound(err) {
continue // try one of the other urls.
@@ -97,17 +273,16 @@ func (r dockerFetcher) Fetch(ctx context.Context, desc ocispec.Descriptor) (io.R
}
// Try manifests endpoints for manifests types
if images.IsManifestType(desc.MediaType) || images.IsIndexType(desc.MediaType) ||
desc.MediaType == images.MediaTypeDockerSchema1Manifest {
if images.IsManifestType(desc.MediaType) || images.IsIndexType(desc.MediaType) {
var firstErr error
for _, host := range r.hosts {
for i, host := range r.hosts {
req := r.request(host, http.MethodGet, "manifests", desc.Digest.String())
if err := req.addNamespace(r.refspec.Hostname()); err != nil {
return nil, err
}
rc, err := r.open(ctx, req, desc.MediaType, offset)
rc, err := r.open(ctx, req, desc.MediaType, offset, i == len(r.hosts)-1)
if err != nil {
// Store the error for referencing later
if firstErr == nil {
@@ -124,13 +299,13 @@ func (r dockerFetcher) Fetch(ctx context.Context, desc ocispec.Descriptor) (io.R
// Finally use blobs endpoints
var firstErr error
for _, host := range r.hosts {
for i, host := range r.hosts {
req := r.request(host, http.MethodGet, "blobs", desc.Digest.String())
if err := req.addNamespace(r.refspec.Hostname()); err != nil {
return nil, err
}
rc, err := r.open(ctx, req, desc.MediaType, offset)
rc, err := r.open(ctx, req, desc.MediaType, offset, i == len(r.hosts)-1)
if err != nil {
// Store the error for referencing later
if firstErr == nil {
@@ -153,7 +328,7 @@ func (r dockerFetcher) Fetch(ctx context.Context, desc ocispec.Descriptor) (io.R
})
}
func (r dockerFetcher) createGetReq(ctx context.Context, host RegistryHost, mediatype string, ps ...string) (*request, int64, error) {
func (r dockerFetcher) createGetReq(ctx context.Context, host RegistryHost, lastHost bool, mediatype string, ps ...string) (*request, int64, error) {
headReq := r.request(host, http.MethodHead, ps...)
if err := headReq.addNamespace(r.refspec.Hostname()); err != nil {
return nil, 0, err
@@ -165,7 +340,7 @@ func (r dockerFetcher) createGetReq(ctx context.Context, host RegistryHost, medi
headReq.header.Set("Accept", strings.Join([]string{mediatype, `*/*`}, ", "))
}
headResp, err := headReq.doWithRetries(ctx, nil)
headResp, err := headReq.doWithRetries(ctx, lastHost)
if err != nil {
return nil, 0, err
}
@@ -209,8 +384,8 @@ func (r dockerFetcher) FetchByDigest(ctx context.Context, dgst digest.Digest, op
firstErr error
)
for _, host := range r.hosts {
getReq, sz, err = r.createGetReq(ctx, host, config.Mediatype, "blobs", dgst.String())
for i, host := range r.hosts {
getReq, sz, err = r.createGetReq(ctx, host, i == len(r.hosts)-1, config.Mediatype, "blobs", dgst.String())
if err == nil {
break
}
@@ -222,8 +397,8 @@ func (r dockerFetcher) FetchByDigest(ctx context.Context, dgst digest.Digest, op
if getReq == nil {
// Fall back to the "manifests" endpoint
for _, host := range r.hosts {
getReq, sz, err = r.createGetReq(ctx, host, config.Mediatype, "manifests", dgst.String())
for i, host := range r.hosts {
getReq, sz, err = r.createGetReq(ctx, host, i == len(r.hosts)-1, config.Mediatype, "manifests", dgst.String())
if err == nil {
break
}
@@ -245,7 +420,7 @@ func (r dockerFetcher) FetchByDigest(ctx context.Context, dgst digest.Digest, op
}
seeker, err := newHTTPReadSeeker(sz, func(offset int64) (io.ReadCloser, error) {
return r.open(ctx, getReq, config.Mediatype, offset)
return r.open(ctx, getReq, config.Mediatype, offset, true)
})
if err != nil {
return nil, desc, err
@@ -262,80 +437,142 @@ func (r dockerFetcher) FetchByDigest(ctx context.Context, dgst digest.Digest, op
return seeker, desc, nil
}
func (r dockerFetcher) open(ctx context.Context, req *request, mediatype string, offset int64) (_ io.ReadCloser, retErr error) {
if mediatype == "" {
req.header.Set("Accept", "*/*")
} else {
req.header.Set("Accept", strings.Join([]string{mediatype, `*/*`}, ", "))
func (r dockerFetcher) open(ctx context.Context, req *request, mediatype string, offset int64, lastHost bool) (_ io.ReadCloser, retErr error) {
const minChunkSize = 512
chunkSize := int64(r.performances.ConcurrentLayerFetchBuffer)
parallelism := int64(r.performances.MaxConcurrentDownloads)
if chunkSize < minChunkSize {
parallelism = 1
}
log.G(ctx).WithField("initial_parallelism", r.performances.MaxConcurrentDownloads).
WithField("parallelism", parallelism).
WithField("chunk_size", chunkSize).
WithField("offset", offset).
Debug("fetching layer")
req.setMediaType(mediatype)
req.header.Set("Accept-Encoding", "zstd;q=1.0, gzip;q=0.8, deflate;q=0.5")
req.setOffset(offset)
if offset > 0 {
// Note: "Accept-Ranges: bytes" cannot be trusted as some endpoints
// will return the header without supporting the range. The content
// range must always be checked.
req.header.Set("Range", fmt.Sprintf("bytes=%d-", offset))
}
resp, err := req.doWithRetries(ctx, nil)
if err != nil {
if err := r.Acquire(ctx, 1); err != nil {
return nil, err
}
defer func() {
if retErr != nil {
resp.Body.Close()
}
}()
if resp.StatusCode > 299 {
// TODO(stevvooe): When doing a offset specific request, we should
// really distinguish between a 206 and a 200. In the case of 200, we
// can discard the bytes, hiding the seek behavior from the
// implementation.
if resp.StatusCode == http.StatusNotFound {
return nil, fmt.Errorf("content at %v not found: %w", req.String(), errdefs.ErrNotFound)
}
var registryErr Errors
if err := json.NewDecoder(resp.Body).Decode(&registryErr); err != nil || registryErr.Len() < 1 {
return nil, fmt.Errorf("unexpected status code %v: %v", req.String(), resp.Status)
}
return nil, fmt.Errorf("unexpected status code %v: %s - Server message: %s", req.String(), resp.Status, registryErr.Error())
}
if offset > 0 {
cr := resp.Header.Get("content-range")
if cr != "" {
if !strings.HasPrefix(cr, fmt.Sprintf("bytes %d-", offset)) {
return nil, fmt.Errorf("unhandled content range in response: %v", cr)
}
} else {
// TODO: Should any cases where use of content range
// without the proper header be considered?
// 206 responses?
// Discard up to offset
// Could use buffer pool here but this case should be rare
n, err := io.Copy(io.Discard, io.LimitReader(resp.Body, offset))
if err != nil {
return nil, fmt.Errorf("failed to discard to offset: %w", err)
}
if n != offset {
return nil, errors.New("unable to discard to offset")
}
resp, err := req.doWithRetries(ctx, lastHost, withErrorCheck, withOffsetCheck(offset))
switch err {
case nil:
// all good
case errContentRangeIgnored:
if parallelism != 1 {
log.G(ctx).WithError(err).Info("remote host ignored content range, forcing parallelism to 1")
parallelism = 1
}
default:
log.G(ctx).WithError(err).Debug("fetch failed")
r.Release(1)
return nil, err
}
body := resp.Body
encoding := strings.FieldsFunc(resp.Header.Get("Content-Encoding"), func(r rune) bool {
return r == ' ' || r == '\t' || r == ','
})
remaining, _ := strconv.ParseInt(resp.Header.Get("Content-Length"), 10, 0)
if parallelism > 1 && req.body == nil {
// If we have a content length, we can use multiple requests to fetch
// the content in parallel. This will make download of bigger bodies
// faster, at the cost of parallelism more requests and max
// ~(max_parallelism * goroutine footprint) memory usage. The goroutine
// footprint should be: the goroutine stack + pipe buffer size
numChunks := remaining / chunkSize
if numChunks*chunkSize < remaining {
numChunks++
}
if numChunks < parallelism {
parallelism = numChunks
}
queue := make(chan int64, parallelism)
ctx, cancelCtx := context.WithCancel(ctx)
done := ctx.Done()
readers, writers := make([]io.Reader, numChunks), make([]*pipeWriter, numChunks)
bufPool := newbufferPool(chunkSize)
for i := range numChunks {
readers[i], writers[i] = newPipeWriter(bufPool)
}
go func() {
for i := range numChunks {
select {
case queue <- i:
case <-done:
return // avoid leaking a goroutine if we exit early.
}
}
close(queue)
}()
r.Release(1)
for range parallelism {
go func() {
for i := range queue { // first in first out
copy := func() error {
if err := r.Acquire(ctx, 1); err != nil {
return err
}
defer r.Release(1)
var body io.ReadCloser
if i == 0 {
body = resp.Body
} else {
reqClone := req.clone()
reqClone.setOffset(offset + i*chunkSize)
nresp, err := reqClone.doWithRetries(ctx, lastHost, withErrorCheck)
if err != nil {
_ = writers[i].CloseWithError(err)
select {
case <-done:
return ctx.Err()
default:
cancelCtx()
}
return err
}
body = nresp.Body
}
_, err := io.Copy(writers[i], io.LimitReader(body, chunkSize))
_ = body.Close()
_ = writers[i].CloseWithError(err)
if err != nil && err != io.EOF {
cancelCtx()
return err
}
return nil
}
if copy() != nil {
return
}
}
}()
}
body = &fnOnClose{
BeforeClose: func() {
cancelCtx()
},
ReadCloser: io.NopCloser(io.MultiReader(readers...)),
}
} else {
body = &fnOnClose{
BeforeClose: func() {
r.Release(1)
},
ReadCloser: body,
}
}
for i := len(encoding) - 1; i >= 0; i-- {
algorithm := strings.ToLower(encoding[i])
switch algorithm {
case "zstd":
r, err := zstd.NewReader(body)
r, err := zstd.NewReader(body,
zstd.WithDecoderLowmem(false),
)
if err != nil {
return nil, err
}
@@ -356,3 +593,17 @@ func (r dockerFetcher) open(ctx context.Context, req *request, mediatype string,
return body, nil
}
type fnOnClose struct {
BeforeClose func()
io.ReadCloser
}
// Close calls the BeforeClose function before closing the underlying
// ReadCloser.
func (f *fnOnClose) Close() error {
f.BeforeClose()
return f.ReadCloser.Close()
}
var _ io.ReadCloser = &fnOnClose{}
vendor/github.com/containerd/containerd/v2/core/remotes/docker/fetcher_fuzz.go
Generated Vendored
-75
View File
@@ -1,75 +0,0 @@
//go:build gofuzz
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package docker
import (
"context"
"fmt"
"io"
"net/http"
"net/http/httptest"
"net/url"
"strconv"
)
func FuzzFetcher(data []byte) int {
dataLen := len(data)
if dataLen == 0 {
return -1
}
s := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
rw.Header().Set("content-range", fmt.Sprintf("bytes %d-%d/%d", 0, dataLen-1, dataLen))
rw.Header().Set("content-length", strconv.Itoa(dataLen))
rw.Write(data)
}))
defer s.Close()
u, err := url.Parse(s.URL)
if err != nil {
return 0
}
f := dockerFetcher{&dockerBase{
repository: "nonempty",
}}
host := RegistryHost{
Client: s.Client(),
Host: u.Host,
Scheme: u.Scheme,
Path: u.Path,
}
ctx := context.Background()
req := f.request(host, http.MethodGet)
rc, err := f.open(ctx, req, "", 0)
if err != nil {
return 0
}
b, err := io.ReadAll(rc)
if err != nil {
return 0
}
expected := data
if len(b) != len(expected) {
panic("len of request is not equal to len of expected but should be")
}
return 1
}
vendor/github.com/containerd/containerd/v2/core/remotes/docker/pusher.go
Generated Vendored
+22 -17
View File
@@ -28,14 +28,15 @@ import (
"sync"
"time"
"github.com/containerd/containerd/v2/core/content"
"github.com/containerd/containerd/v2/core/images"
"github.com/containerd/containerd/v2/core/remotes"
remoteserrors "github.com/containerd/containerd/v2/core/remotes/errors"
"github.com/containerd/errdefs"
"github.com/containerd/log"
digest "github.com/opencontainers/go-digest"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/containerd/containerd/v2/core/content"
"github.com/containerd/containerd/v2/core/images"
"github.com/containerd/containerd/v2/core/remotes"
remoteserrors "github.com/containerd/containerd/v2/core/remotes/errors"
)
type dockerPusher struct {
@@ -115,7 +116,7 @@ func (p dockerPusher) push(ctx context.Context, desc ocispec.Descriptor, ref str
log.G(ctx).WithField("url", req.String()).Debugf("checking and pushing to")
resp, err := req.doWithRetries(ctx, nil)
resp, err := req.doWithRetries(ctx, true)
if err != nil {
if !errors.Is(err, ErrInvalidAuthorization) {
return nil, err
@@ -176,24 +177,28 @@ func (p dockerPusher) push(ctx context.Context, desc ocispec.Descriptor, ref str
//
// for the private repo, we should remove mount-from
// query and send the request again.
resp, err = preq.doWithRetries(pctx, nil)
resp, err = preq.doWithRetries(pctx, true)
if err != nil {
return nil, err
if !errors.Is(err, ErrInvalidAuthorization) {
return nil, fmt.Errorf("pushing with mount from %s: %w", fromRepo, err)
}
log.G(ctx).Debugf("failed to push with mount from repository %s: %v", fromRepo, err)
}
if resp != nil {
switch resp.StatusCode {
case http.StatusUnauthorized:
log.G(ctx).Debugf("failed to mount from repository %s, not authorized", fromRepo)
switch resp.StatusCode {
case http.StatusUnauthorized:
log.G(ctx).Debugf("failed to mount from repository %s", fromRepo)
resp.Body.Close()
resp = nil
case http.StatusCreated:
mountedFrom = path.Join(p.refspec.Hostname(), fromRepo)
resp.Body.Close()
resp = nil
case http.StatusCreated:
mountedFrom = path.Join(p.refspec.Hostname(), fromRepo)
}
}
}
if resp == nil {
resp, err = req.doWithRetries(ctx, nil)
resp, err = req.doWithRetries(ctx, true)
if err != nil {
if errors.Is(err, ErrInvalidAuthorization) {
return nil, fmt.Errorf("push access denied, repository does not exist or may require authorization: %w", err)
@@ -285,7 +290,7 @@ func (p dockerPusher) push(ctx context.Context, desc ocispec.Descriptor, ref str
req.size = desc.Size
go func() {
resp, err := req.doWithRetries(ctx, nil)
resp, err := req.doWithRetries(ctx, true)
if err != nil {
pushw.setError(err)
return
vendor/github.com/containerd/containerd/v2/core/remotes/docker/registry.go
Generated Vendored
+21 -1
View File
@@ -17,9 +17,11 @@
package docker
import (
"crypto/tls"
"errors"
"net"
"net/http"
"time"
)
// HostCapabilities represent the capabilities of the registry
@@ -170,7 +172,9 @@ func ConfigureDefaultRegistries(ropts ...RegistryOpt) RegistryHosts {
}
if config.Client == nil {
config.Client = http.DefaultClient
config.Client = &http.Client{
Transport: DefaultHTTPTransport(nil),
}
}
if opts.plainHTTP != nil {
@@ -242,3 +246,19 @@ func MatchLocalhost(host string) (bool, error) {
return ip.IsLoopback(), nil
}
func DefaultHTTPTransport(defaultTLSConfig *tls.Config) *http.Transport {
return &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
FallbackDelay: 300 * time.Millisecond,
}).DialContext,
MaxIdleConns: 10,
IdleConnTimeout: 30 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
TLSClientConfig: defaultTLSConfig,
ExpectContinueTimeout: 5 * time.Second,
}
}
vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go
Generated Vendored
+144 -27
View File
@@ -19,6 +19,7 @@ package docker
import (
"context"
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"io"
@@ -34,11 +35,12 @@ import (
"github.com/containerd/log"
"github.com/opencontainers/go-digest"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"golang.org/x/sync/semaphore"
"github.com/containerd/containerd/v2/core/images"
"github.com/containerd/containerd/v2/core/remotes"
"github.com/containerd/containerd/v2/core/remotes/docker/schema1" //nolint:staticcheck // Ignore SA1019. Need to keep deprecated package for compatibility.
remoteerrors "github.com/containerd/containerd/v2/core/remotes/errors"
"github.com/containerd/containerd/v2/core/transfer"
"github.com/containerd/containerd/v2/pkg/reference"
"github.com/containerd/containerd/v2/pkg/tracing"
"github.com/containerd/containerd/v2/version"
@@ -142,6 +144,7 @@ type dockerResolver struct {
header http.Header
resolveHeader http.Header
tracker StatusTracker
config transfer.ImageResolverOptions
}
// NewResolver returns a new resolver to a Docker registry
@@ -156,9 +159,6 @@ func NewResolver(options ResolverOptions) remotes.Resolver {
// make a copy of the headers to avoid race due to concurrent map write
options.Headers = options.Headers.Clone()
}
if _, ok := options.Headers["User-Agent"]; !ok {
options.Headers.Set("User-Agent", "containerd/"+version.Version)
}
resolveHeader := http.Header{}
if _, ok := options.Headers["Accept"]; !ok {
@@ -232,7 +232,7 @@ func (r *countingReader) Read(p []byte) (int, error) {
return n, err
}
var _ remotes.Resolver = &dockerResolver{}
var _ remotes.ResolverWithOptions = &dockerResolver{}
func (r *dockerResolver) Resolve(ctx context.Context, ref string) (string, ocispec.Descriptor, error) {
base, err := r.resolveDockerBase(ref)
@@ -307,7 +307,7 @@ func (r *dockerResolver) Resolve(ctx context.Context, ref string) (string, ocisp
}
log.G(ctx).Debug("resolving")
resp, err := req.doWithRetries(ctx, nil)
resp, err := req.doWithRetries(ctx, i == len(hosts)-1)
if err != nil {
if errors.Is(err, ErrInvalidAuthorization) {
err = fmt.Errorf("pull access denied, repository does not exist or may require authorization: %w", err)
@@ -371,7 +371,7 @@ func (r *dockerResolver) Resolve(ctx context.Context, ref string) (string, ocisp
req.header[key] = append(req.header[key], value...)
}
resp, err := req.doWithRetries(ctx, nil)
resp, err := req.doWithRetries(ctx, true)
if err != nil {
return "", ocispec.Descriptor{}, err
}
@@ -387,13 +387,8 @@ func (r *dockerResolver) Resolve(ctx context.Context, ref string) (string, ocisp
}
if contentType == images.MediaTypeDockerSchema1Manifest {
b, err := schema1.ReadStripSignature(&bodyReader)
if err != nil {
return err
}
dgst = digest.FromBytes(b)
return nil
return fmt.Errorf("%w: media type %q is no longer supported since containerd v2.0, please rebuild the image as %q or %q",
errdefs.ErrNotImplemented, images.MediaTypeDockerSchema1Manifest, images.MediaTypeDockerSchema2Manifest, ocispec.MediaTypeImageManifest)
}
dgst, err = digest.FromReader(&bodyReader)
@@ -433,6 +428,12 @@ func (r *dockerResolver) Resolve(ctx context.Context, ref string) (string, ocisp
return "", ocispec.Descriptor{}, firstErr
}
func (r *dockerResolver) SetOptions(options ...transfer.ImageResolverOption) {
for _, opt := range options {
opt(&r.config)
}
}
func (r *dockerResolver) Fetcher(ctx context.Context, ref string) (remotes.Fetcher, error) {
base, err := r.resolveDockerBase(ref)
if err != nil {
@@ -467,10 +468,25 @@ func (r *dockerResolver) resolveDockerBase(ref string) (*dockerBase, error) {
}
type dockerBase struct {
refspec reference.Spec
repository string
hosts []RegistryHost
header http.Header
refspec reference.Spec
repository string
hosts []RegistryHost
header http.Header
performances transfer.ImageResolverPerformanceSettings
limiter *semaphore.Weighted
}
func (r *dockerBase) Acquire(ctx context.Context, weight int64) error {
if r.limiter == nil {
return nil
}
return r.limiter.Acquire(ctx, weight)
}
func (r *dockerBase) Release(weight int64) {
if r.limiter != nil {
r.limiter.Release(weight)
}
}
func (r *dockerResolver) base(refspec reference.Spec) (*dockerBase, error) {
@@ -480,10 +496,12 @@ func (r *dockerResolver) base(refspec reference.Spec) (*dockerBase, error) {
return nil, err
}
return &dockerBase{
refspec: refspec,
repository: strings.TrimPrefix(refspec.Locator, host+"/"),
hosts: hosts,
header: r.header,
refspec: refspec,
repository: strings.TrimPrefix(refspec.Locator, host+"/"),
hosts: hosts,
header: r.header,
performances: r.config.Performances,
limiter: r.config.DownloadLimiter,
}, nil
}
@@ -505,6 +523,11 @@ func (r *dockerBase) request(host RegistryHost, method string, ps ...string) *re
for key, value := range host.Header {
header[key] = append(header[key], value...)
}
if len(header.Get("User-Agent")) == 0 {
header.Set("User-Agent", "containerd/"+version.Version)
}
parts := append([]string{"/", host.Path, r.repository}, ps...)
p := path.Join(parts...)
// Join strips trailing slash, re-add ending "/" if included
@@ -562,6 +585,12 @@ type request struct {
size int64
}
func (r *request) clone() *request {
res := *r
res.header = r.header.Clone()
return &res
}
func (r *request) do(ctx context.Context) (*http.Response, error) {
u := r.host.Scheme + "://" + r.host.Host + r.path
req, err := http.NewRequestWithContext(ctx, r.method, u, nil)
@@ -617,26 +646,94 @@ func (r *request) do(ctx context.Context) (*http.Response, error) {
return resp, nil
}
func (r *request) doWithRetries(ctx context.Context, responses []*http.Response) (*http.Response, error) {
type doChecks func(r *request, resp *http.Response) error
func withErrorCheck(r *request, resp *http.Response) error {
if resp.StatusCode > 299 {
if resp.StatusCode == http.StatusNotFound {
return fmt.Errorf("content at %v not found: %w", r.String(), errdefs.ErrNotFound)
}
var registryErr Errors
if err := json.NewDecoder(resp.Body).Decode(&registryErr); err != nil || registryErr.Len() < 1 {
return fmt.Errorf("unexpected status code %v: %v", r.String(), resp.Status)
}
return fmt.Errorf("unexpected status code %v: %s - Server message: %s", r.String(), resp.Status, registryErr.Error())
}
return nil
}
var errContentRangeIgnored = errors.New("content range requests ignored")
func withOffsetCheck(offset int64) doChecks {
return func(r *request, resp *http.Response) error {
if offset == 0 {
return nil
}
if resp.StatusCode == http.StatusPartialContent {
return nil
}
if cr := resp.Header.Get("Content-Range"); cr != "" {
if !strings.HasPrefix(cr, fmt.Sprintf("bytes %d-", offset)) {
return fmt.Errorf("unhandled content range in response: %v", cr)
}
return nil
}
// Discard up to offset
// Could use buffer pool here but this case should be rare
n, err := io.Copy(io.Discard, io.LimitReader(resp.Body, offset))
if err != nil {
return fmt.Errorf("failed to discard to offset: %w", err)
}
if n != offset {
return errors.New("unable to discard to offset")
}
// content range ignored, we can't do concurrent fetches here.
// return an error to be caught
return errContentRangeIgnored
}
}
func (r *request) doWithRetries(ctx context.Context, lastHost bool, checks ...doChecks) (resp *http.Response, err error) {
resp, err = r.doWithRetriesInner(ctx, nil, lastHost)
if err != nil {
return nil, err
}
defer func() {
if err != nil && err != errContentRangeIgnored {
resp.Body.Close()
}
}()
for _, check := range checks {
if err := check(r, resp); err != nil {
return resp, err
}
}
return resp, nil
}
func (r *request) doWithRetriesInner(ctx context.Context, responses []*http.Response, lastHost bool) (*http.Response, error) {
resp, err := r.do(ctx)
if err != nil {
return nil, err
}
responses = append(responses, resp)
retry, err := r.retryRequest(ctx, responses)
retry, err := r.retryRequest(ctx, responses, lastHost)
if err != nil {
resp.Body.Close()
return nil, err
}
if retry {
resp.Body.Close()
return r.doWithRetries(ctx, responses)
return r.doWithRetriesInner(ctx, responses, lastHost)
}
return resp, err
}
func (r *request) retryRequest(ctx context.Context, responses []*http.Response) (bool, error) {
func (r *request) retryRequest(ctx context.Context, responses []*http.Response, lastHost bool) (bool, error) {
if len(responses) > 5 {
return false, nil
}
@@ -662,9 +759,17 @@ func (r *request) retryRequest(ctx context.Context, responses []*http.Response)
}
case http.StatusRequestTimeout, http.StatusTooManyRequests:
return true, nil
case http.StatusServiceUnavailable, http.StatusGatewayTimeout, http.StatusInternalServerError:
// Do not retry if the same error was seen in the last request
if len(responses) > 1 && responses[len(responses)-2].StatusCode == last.StatusCode {
return false, nil
}
// Only retry if this is the last host that will be attempted
if lastHost {
return true, nil
}
}
// TODO: Handle 50x errors accounting for attempt history
return false, nil
}
@@ -672,6 +777,18 @@ func (r *request) String() string {
return r.host.Scheme + "://" + r.host.Host + r.path
}
func (r *request) setMediaType(mediatype string) {
if mediatype == "" {
r.header.Set("Accept", "*/*")
} else {
r.header.Set("Accept", strings.Join([]string{mediatype, `*/*`}, ", "))
}
}
func (r *request) setOffset(offset int64) {
r.header.Set("Range", fmt.Sprintf("bytes=%d-", offset))
}
func requestFields(req *http.Request) log.Fields {
fields := map[string]interface{}{
"request.method": req.Method,
vendor/github.com/containerd/containerd/v2/core/remotes/docker/schema1/converter.go
Generated Vendored
-626
View File
@@ -1,626 +0,0 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package schema1 provides a converter to fetch an image formatted in Docker Image Manifest v2, Schema 1.
//
// Deprecated: use images formatted in Docker Image Manifest v2, Schema 2, or OCI Image Spec v1.
package schema1
import (
"bytes"
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"strconv"
"strings"
"sync"
"time"
"github.com/containerd/containerd/v2/core/content"
"github.com/containerd/containerd/v2/core/images"
"github.com/containerd/containerd/v2/core/remotes"
"github.com/containerd/containerd/v2/pkg/archive/compression"
"github.com/containerd/containerd/v2/pkg/deprecation"
"github.com/containerd/containerd/v2/pkg/labels"
"github.com/containerd/errdefs"
"github.com/containerd/log"
digest "github.com/opencontainers/go-digest"
specs "github.com/opencontainers/image-spec/specs-go"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"golang.org/x/sync/errgroup"
)
const (
manifestSizeLimit = 8e6 // 8MB
labelDockerSchema1EmptyLayer = "containerd.io/docker.schema1.empty-layer"
)
type blobState struct {
diffID digest.Digest
empty bool
}
// Converter converts schema1 manifests to schema2 on fetch
type Converter struct {
contentStore content.Store
fetcher remotes.Fetcher
pulledManifest *manifest
mu sync.Mutex
blobMap map[digest.Digest]blobState
layerBlobs map[digest.Digest]ocispec.Descriptor
}
var ErrDisabled = fmt.Errorf("Pulling Schema 1 images have been deprecated and disabled by default since containerd v2.0. "+
"As a workaround you may set an environment variable `%s=1`, but this will be completely removed in containerd v2.1.",
deprecation.EnvPullSchema1Image)
// NewConverter returns a new converter
func NewConverter(contentStore content.Store, fetcher remotes.Fetcher) (*Converter, error) {
s := os.Getenv(deprecation.EnvPullSchema1Image)
if s == "" {
return nil, ErrDisabled
}
enable, err := strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("failed to parse `%s=%s`: %w", deprecation.EnvPullSchema1Image, s, err)
}
if !enable {
return nil, ErrDisabled
}
log.L.Warn(ErrDisabled)
return &Converter{
contentStore: contentStore,
fetcher: fetcher,
blobMap: map[digest.Digest]blobState{},
layerBlobs: map[digest.Digest]ocispec.Descriptor{},
}, nil
}
// Handle fetching descriptors for a docker media type
func (c *Converter) Handle(ctx context.Context, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
switch desc.MediaType {
case images.MediaTypeDockerSchema1Manifest:
if err := c.fetchManifest(ctx, desc); err != nil {
return nil, err
}
m := c.pulledManifest
if len(m.FSLayers) != len(m.History) {
return nil, errors.New("invalid schema 1 manifest, history and layer mismatch")
}
descs := make([]ocispec.Descriptor, 0, len(c.pulledManifest.FSLayers))
for i := range m.FSLayers {
if _, ok := c.blobMap[c.pulledManifest.FSLayers[i].BlobSum]; !ok {
empty, err := isEmptyLayer([]byte(m.History[i].V1Compatibility))
if err != nil {
return nil, err
}
// Do no attempt to download a known empty blob
if !empty {
descs = append([]ocispec.Descriptor{
{
MediaType: images.MediaTypeDockerSchema2LayerGzip,
Digest: c.pulledManifest.FSLayers[i].BlobSum,
Size: -1,
},
}, descs...)
}
c.blobMap[c.pulledManifest.FSLayers[i].BlobSum] = blobState{
empty: empty,
}
}
}
return descs, nil
case images.MediaTypeDockerSchema2LayerGzip:
if c.pulledManifest == nil {
return nil, errors.New("manifest required for schema 1 blob pull")
}
return nil, c.fetchBlob(ctx, desc)
default:
return nil, fmt.Errorf("%v not support for schema 1 manifests", desc.MediaType)
}
}
// ConvertOptions provides options on converting a docker schema1 manifest.
type ConvertOptions struct {
// ManifestMediaType specifies the media type of the manifest OCI descriptor.
ManifestMediaType string
// ConfigMediaType specifies the media type of the manifest config OCI
// descriptor.
ConfigMediaType string
}
// ConvertOpt allows configuring a convert operation.
type ConvertOpt func(context.Context, *ConvertOptions) error
// UseDockerSchema2 is used to indicate that a schema1 manifest should be
// converted into the media types for a docker schema2 manifest.
func UseDockerSchema2() ConvertOpt {
return func(ctx context.Context, o *ConvertOptions) error {
o.ManifestMediaType = images.MediaTypeDockerSchema2Manifest
o.ConfigMediaType = images.MediaTypeDockerSchema2Config
return nil
}
}
// Convert a docker manifest to an OCI descriptor
func (c *Converter) Convert(ctx context.Context, opts ...ConvertOpt) (ocispec.Descriptor, error) {
co := ConvertOptions{
ManifestMediaType: ocispec.MediaTypeImageManifest,
ConfigMediaType: ocispec.MediaTypeImageConfig,
}
for _, opt := range opts {
if err := opt(ctx, &co); err != nil {
return ocispec.Descriptor{}, err
}
}
history, diffIDs, err := c.schema1ManifestHistory()
if err != nil {
return ocispec.Descriptor{}, fmt.Errorf("schema 1 conversion failed: %w", err)
}
var img ocispec.Image
if err := json.Unmarshal([]byte(c.pulledManifest.History[0].V1Compatibility), &img); err != nil {
return ocispec.Descriptor{}, fmt.Errorf("failed to unmarshal image from schema 1 history: %w", err)
}
img.History = history
img.RootFS = ocispec.RootFS{
Type: "layers",
DiffIDs: diffIDs,
}
b, err := json.MarshalIndent(img, "", " ")
if err != nil {
return ocispec.Descriptor{}, fmt.Errorf("failed to marshal image: %w", err)
}
config := ocispec.Descriptor{
MediaType: co.ConfigMediaType,
Digest: digest.Canonical.FromBytes(b),
Size: int64(len(b)),
}
layers := make([]ocispec.Descriptor, len(diffIDs))
for i, diffID := range diffIDs {
layers[i] = c.layerBlobs[diffID]
}
manifest := ocispec.Manifest{
Versioned: specs.Versioned{
SchemaVersion: 2,
},
Config: config,
Layers: layers,
}
mb, err := json.MarshalIndent(manifest, "", " ")
if err != nil {
return ocispec.Descriptor{}, fmt.Errorf("failed to marshal image: %w", err)
}
desc := ocispec.Descriptor{
MediaType: co.ManifestMediaType,
Digest: digest.Canonical.FromBytes(mb),
Size: int64(len(mb)),
}
labels := map[string]string{}
labels["containerd.io/gc.ref.content.0"] = manifest.Config.Digest.String()
for i, ch := range manifest.Layers {
labels[fmt.Sprintf("containerd.io/gc.ref.content.%d", i+1)] = ch.Digest.String()
}
ref := remotes.MakeRefKey(ctx, desc)
if err := content.WriteBlob(ctx, c.contentStore, ref, bytes.NewReader(mb), desc, content.WithLabels(labels)); err != nil {
return ocispec.Descriptor{}, fmt.Errorf("failed to write image manifest: %w", err)
}
ref = remotes.MakeRefKey(ctx, config)
if err := content.WriteBlob(ctx, c.contentStore, ref, bytes.NewReader(b), config); err != nil {
return ocispec.Descriptor{}, fmt.Errorf("failed to write image config: %w", err)
}
return desc, nil
}
// ReadStripSignature reads in a schema1 manifest and returns a byte array
// with the "signatures" field stripped
func ReadStripSignature(schema1Blob io.Reader) ([]byte, error) {
b, err := io.ReadAll(io.LimitReader(schema1Blob, manifestSizeLimit)) // limit to 8MB
if err != nil {
return nil, err
}
return stripSignature(b)
}
func (c *Converter) fetchManifest(ctx context.Context, desc ocispec.Descriptor) error {
log.G(ctx).Debug("fetch schema 1")
rc, err := c.fetcher.Fetch(ctx, desc)
if err != nil {
return err
}
b, err := ReadStripSignature(rc)
rc.Close()
if err != nil {
return err
}
var m manifest
if err := json.Unmarshal(b, &m); err != nil {
return err
}
if len(m.Manifests) != 0 || len(m.Layers) != 0 {
return errors.New("converter: expected schema1 document but found extra keys")
}
c.pulledManifest = &m
return nil
}
func (c *Converter) fetchBlob(ctx context.Context, desc ocispec.Descriptor) error {
log.G(ctx).Debug("fetch blob")
var (
ref = remotes.MakeRefKey(ctx, desc)
calc = newBlobStateCalculator()
compressMethod = compression.Gzip
)
// size may be unknown, set to zero for content ingest
ingestDesc := desc
if ingestDesc.Size == -1 {
ingestDesc.Size = 0
}
cw, err := content.OpenWriter(ctx, c.contentStore, content.WithRef(ref), content.WithDescriptor(ingestDesc))
if err != nil {
if !errdefs.IsAlreadyExists(err) {
return err
}
reuse, err := c.reuseLabelBlobState(ctx, desc)
if err != nil {
return err
}
if reuse {
return nil
}
ra, err := c.contentStore.ReaderAt(ctx, desc)
if err != nil {
return err
}
defer ra.Close()
r, err := compression.DecompressStream(content.NewReader(ra))
if err != nil {
return err
}
compressMethod = r.GetCompression()
_, err = io.Copy(calc, r)
r.Close()
if err != nil {
return err
}
} else {
defer cw.Close()
rc, err := c.fetcher.Fetch(ctx, desc)
if err != nil {
return err
}
defer rc.Close()
eg, _ := errgroup.WithContext(ctx)
pr, pw := io.Pipe()
eg.Go(func() error {
r, err := compression.DecompressStream(pr)
if err != nil {
return err
}
compressMethod = r.GetCompression()
_, err = io.Copy(calc, r)
r.Close()
pr.CloseWithError(err)
return err
})
eg.Go(func() error {
defer pw.Close()
return content.Copy(ctx, cw, io.TeeReader(rc, pw), ingestDesc.Size, ingestDesc.Digest)
})
if err := eg.Wait(); err != nil {
return err
}
}
if desc.Size == -1 {
info, err := c.contentStore.Info(ctx, desc.Digest)
if err != nil {
return fmt.Errorf("failed to get blob info: %w", err)
}
desc.Size = info.Size
}
if compressMethod == compression.Uncompressed {
log.G(ctx).WithField("id", desc.Digest).Debugf("changed media type for uncompressed schema1 layer blob")
desc.MediaType = images.MediaTypeDockerSchema2Layer
}
state := calc.State()
cinfo := content.Info{
Digest: desc.Digest,
Labels: map[string]string{
labels.LabelUncompressed: state.diffID.String(),
labelDockerSchema1EmptyLayer: strconv.FormatBool(state.empty),
},
}
if _, err := c.contentStore.Update(ctx, cinfo, "labels."+labels.LabelUncompressed, fmt.Sprintf("labels.%s", labelDockerSchema1EmptyLayer)); err != nil {
return fmt.Errorf("failed to update uncompressed label: %w", err)
}
c.mu.Lock()
c.blobMap[desc.Digest] = state
c.layerBlobs[state.diffID] = desc
c.mu.Unlock()
return nil
}
func (c *Converter) reuseLabelBlobState(ctx context.Context, desc ocispec.Descriptor) (bool, error) {
cinfo, err := c.contentStore.Info(ctx, desc.Digest)
if err != nil {
return false, fmt.Errorf("failed to get blob info: %w", err)
}
desc.Size = cinfo.Size
diffID, ok := cinfo.Labels[labels.LabelUncompressed]
if !ok {
return false, nil
}
emptyVal, ok := cinfo.Labels[labelDockerSchema1EmptyLayer]
if !ok {
return false, nil
}
isEmpty, err := strconv.ParseBool(emptyVal)
if err != nil {
log.G(ctx).WithField("id", desc.Digest).Warnf("failed to parse bool from label %s: %v", labelDockerSchema1EmptyLayer, isEmpty)
return false, nil
}
bState := blobState{empty: isEmpty}
if bState.diffID, err = digest.Parse(diffID); err != nil {
log.G(ctx).WithField("id", desc.Digest).Warnf("failed to parse digest from label %s: %v", labels.LabelUncompressed, diffID)
return false, nil
}
// NOTE: there is no need to read header to get compression method
// because there are only two kinds of methods.
if bState.diffID == desc.Digest {
desc.MediaType = images.MediaTypeDockerSchema2Layer
} else {
desc.MediaType = images.MediaTypeDockerSchema2LayerGzip
}
c.mu.Lock()
c.blobMap[desc.Digest] = bState
c.layerBlobs[bState.diffID] = desc
c.mu.Unlock()
return true, nil
}
func (c *Converter) schema1ManifestHistory() ([]ocispec.History, []digest.Digest, error) {
if c.pulledManifest == nil {
return nil, nil, errors.New("missing schema 1 manifest for conversion")
}
m := *c.pulledManifest
if len(m.History) == 0 {
return nil, nil, errors.New("no history")
}
history := make([]ocispec.History, len(m.History))
diffIDs := []digest.Digest{}
for i := range m.History {
var h v1History
if err := json.Unmarshal([]byte(m.History[i].V1Compatibility), &h); err != nil {
return nil, nil, fmt.Errorf("failed to unmarshal history: %w", err)
}
blobSum := m.FSLayers[i].BlobSum
state := c.blobMap[blobSum]
history[len(history)-i-1] = ocispec.History{
Author: h.Author,
Comment: h.Comment,
Created: &h.Created,
CreatedBy: strings.Join(h.ContainerConfig.Cmd, " "),
EmptyLayer: state.empty,
}
if !state.empty {
diffIDs = append([]digest.Digest{state.diffID}, diffIDs...)
}
}
return history, diffIDs, nil
}
type fsLayer struct {
BlobSum digest.Digest `json:"blobSum"`
}
type history struct {
V1Compatibility string `json:"v1Compatibility"`
}
type manifest struct {
FSLayers []fsLayer `json:"fsLayers"`
History []history `json:"history"`
Layers json.RawMessage `json:"layers,omitempty"` // OCI manifest
Manifests json.RawMessage `json:"manifests,omitempty"` // OCI index
}
type v1History struct {
Author string `json:"author,omitempty"`
Created time.Time `json:"created"`
Comment string `json:"comment,omitempty"`
ThrowAway *bool `json:"throwaway,omitempty"`
Size *int `json:"Size,omitempty"` // used before ThrowAway field
ContainerConfig struct {
Cmd []string `json:"Cmd,omitempty"`
} `json:"container_config,omitempty"`
}
// isEmptyLayer returns whether the v1 compatibility history describes an
// empty layer. A return value of true indicates the layer is empty,
// however false does not indicate non-empty.
func isEmptyLayer(compatHistory []byte) (bool, error) {
var h v1History
if err := json.Unmarshal(compatHistory, &h); err != nil {
return false, err
}
if h.ThrowAway != nil {
return *h.ThrowAway, nil
}
if h.Size != nil {
return *h.Size == 0, nil
}
// If no `Size` or `throwaway` field is given, then
// it cannot be determined whether the layer is empty
// from the history, return false
return false, nil
}
type signature struct {
Signatures []jsParsedSignature `json:"signatures"`
}
type jsParsedSignature struct {
Protected string `json:"protected"`
}
type protectedBlock struct {
Length int `json:"formatLength"`
Tail string `json:"formatTail"`
}
// joseBase64UrlDecode decodes the given string using the standard base64 url
// decoder but first adds the appropriate number of trailing '=' characters in
// accordance with the jose specification.
// http://tools.ietf.org/html/draft-ietf-jose-json-web-signature-31#section-2
func joseBase64UrlDecode(s string) ([]byte, error) {
switch len(s) % 4 {
case 0:
case 2:
s += "=="
case 3:
s += "="
default:
return nil, errors.New("illegal base64url string")
}
return base64.URLEncoding.DecodeString(s)
}
func stripSignature(b []byte) ([]byte, error) {
var sig signature
if err := json.Unmarshal(b, &sig); err != nil {
return nil, err
}
if len(sig.Signatures) == 0 {
return nil, errors.New("no signatures")
}
pb, err := joseBase64UrlDecode(sig.Signatures[0].Protected)
if err != nil {
return nil, fmt.Errorf("could not decode %s: %w", sig.Signatures[0].Protected, err)
}
var protected protectedBlock
if err := json.Unmarshal(pb, &protected); err != nil {
return nil, err
}
if protected.Length > len(b) {
return nil, errors.New("invalid protected length block")
}
tail, err := joseBase64UrlDecode(protected.Tail)
if err != nil {
return nil, fmt.Errorf("invalid tail base 64 value: %w", err)
}
return append(b[:protected.Length], tail...), nil
}
type blobStateCalculator struct {
empty bool
digester digest.Digester
}
func newBlobStateCalculator() *blobStateCalculator {
return &blobStateCalculator{
empty: true,
digester: digest.Canonical.Digester(),
}
}
func (c *blobStateCalculator) Write(p []byte) (int, error) {
if c.empty {
for _, b := range p {
if b != 0x00 {
c.empty = false
break
}
}
}
return c.digester.Hash().Write(p)
}
func (c *blobStateCalculator) State() blobState {
return blobState{
empty: c.empty,
diffID: c.digester.Digest(),
}
}
vendor/github.com/containerd/containerd/v2/core/remotes/docker/scope.go
Generated Vendored
+1 -1
View File
@@ -88,7 +88,7 @@ func GetTokenScopes(ctx context.Context, common []string) []string {
l := 0
for idx := 1; idx < len(scopes); idx++ {
// Note: this comparison is unaware of the scope grammar (https://docs.docker.com/registry/spec/auth/scope/)
// Note: this comparison is unaware of the scope grammar (https://distribution.github.io/distribution/spec/auth/scope/)
// So, "repository:foo/bar:pull,push" != "repository:foo/bar:push,pull", although semantically they are equal.
if scopes[l] == scopes[idx] {
continue
vendor/github.com/containerd/containerd/v2/core/remotes/resolver.go
Generated Vendored
+7
View File
@@ -21,6 +21,7 @@ import (
"io"
"github.com/containerd/containerd/v2/core/content"
"github.com/containerd/containerd/v2/core/transfer"
"github.com/opencontainers/go-digest"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)
@@ -51,6 +52,12 @@ type Resolver interface {
Pusher(ctx context.Context, ref string) (Pusher, error)
}
// ResolverWithOptions is a Resolver that also supports setting options.
type ResolverWithOptions interface {
Resolver
SetOptions(options ...transfer.ImageResolverOption)
}
// Fetcher fetches content.
// A fetcher implementation may implement the FetcherByDigest interface too.
type Fetcher interface {
vendor/github.com/containerd/containerd/v2/core/transfer/transfer.go
Generated Vendored
+178
View File
@@ -0,0 +1,178 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package transfer
import (
"context"
"io"
"golang.org/x/sync/semaphore"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/containerd/containerd/v2/core/content"
"github.com/containerd/containerd/v2/core/images"
)
type Transferrer interface {
Transfer(ctx context.Context, source interface{}, destination interface{}, opts ...Opt) error
}
type ImageResolver interface {
Resolve(ctx context.Context) (name string, desc ocispec.Descriptor, err error)
}
type ImageResolverOptionSetter interface {
ImageResolver
SetResolverOptions(opts ...ImageResolverOption)
}
type ImageResolverOption func(*ImageResolverOptions)
type ImageResolverOptions struct {
DownloadLimiter *semaphore.Weighted
Performances ImageResolverPerformanceSettings
}
type ImageResolverPerformanceSettings struct {
MaxConcurrentDownloads int
ConcurrentLayerFetchBuffer int
}
func WithDownloadLimiter(limiter *semaphore.Weighted) ImageResolverOption {
return func(opts *ImageResolverOptions) {
opts.DownloadLimiter = limiter
}
}
func WithMaxConcurrentDownloads(maxConcurrentDownloads int) ImageResolverOption {
return func(opts *ImageResolverOptions) {
opts.Performances.MaxConcurrentDownloads = maxConcurrentDownloads
}
}
func WithConcurrentLayerFetchBuffer(ConcurrentLayerFetchBuffer int) ImageResolverOption {
return func(opts *ImageResolverOptions) {
opts.Performances.ConcurrentLayerFetchBuffer = ConcurrentLayerFetchBuffer
}
}
type ImageFetcher interface {
ImageResolver
Fetcher(ctx context.Context, ref string) (Fetcher, error)
}
type ImagePusher interface {
Pusher(context.Context, ocispec.Descriptor) (Pusher, error)
}
type Fetcher interface {
Fetch(context.Context, ocispec.Descriptor) (io.ReadCloser, error)
}
type Pusher interface {
Push(context.Context, ocispec.Descriptor) (content.Writer, error)
}
// ImageFilterer is used to filter out child objects of an image
type ImageFilterer interface {
ImageFilter(images.HandlerFunc, content.Store) images.HandlerFunc
}
// ImageStorer is a type which is capable of storing images for
// the provided descriptor. The descriptor may be any type of manifest
// including an index with multiple image references.
type ImageStorer interface {
Store(context.Context, ocispec.Descriptor, images.Store) ([]images.Image, error)
}
// ImageGetter is type which returns an image from an image store
type ImageGetter interface {
Get(context.Context, images.Store) (images.Image, error)
}
// ImageLookup is a type which returns images from an image store
// based on names or prefixes
type ImageLookup interface {
Lookup(context.Context, images.Store) ([]images.Image, error)
}
// ImageExporter exports images to a writer
type ImageExporter interface {
Export(context.Context, content.Store, []images.Image) error
}
// ImageImporter imports an image into a content store
type ImageImporter interface {
Import(context.Context, content.Store) (ocispec.Descriptor, error)
}
// ImageImportStreamer returns an import streamer based on OCI or
// Docker image tar archives. The stream should be a raw tar stream
// and without compression.
type ImageImportStreamer interface {
ImportStream(context.Context) (io.Reader, string, error)
}
type ImageExportStreamer interface {
ExportStream(context.Context) (io.WriteCloser, string, error)
}
type ImageUnpacker interface {
UnpackPlatforms() []UnpackConfiguration
}
// ImagePlatformsGetter is type which returns configured platforms.
type ImagePlatformsGetter interface {
Platforms() []ocispec.Platform
}
// UnpackConfiguration specifies the platform and snapshotter to use for resolving
// the unpack Platform, if snapshotter is not specified the platform default will
// be used.
type UnpackConfiguration struct {
Platform ocispec.Platform
Snapshotter string
}
type ProgressFunc func(Progress)
type Config struct {
Progress ProgressFunc
}
type Opt func(*Config)
func WithProgress(f ProgressFunc) Opt {
return func(opts *Config) {
opts.Progress = f
}
}
// Progress is used to represent a particular progress event or incremental
// update for the provided named object. The parents represent the names of
// the objects which initiated the progress for the provided named object.
// The name and what object it represents is determined by the implementation.
type Progress struct {
Event string
Name string
Parents []string
Progress int64
Total int64
Desc *ocispec.Descriptor // since containerd v2.0
}
vendor/github.com/containerd/containerd/v2/defaults/defaults_darwin.go
Generated Vendored
-15
View File
@@ -17,21 +17,6 @@
package defaults
const (
// DefaultRootDir is the default location used by containerd to store
// persistent data
DefaultRootDir = "/var/lib/containerd"
// DefaultStateDir is the default location used by containerd to store
// transient data
DefaultStateDir = "/var/run/containerd"
// DefaultAddress is the default unix socket address
DefaultAddress = "/var/run/containerd/containerd.sock"
// DefaultDebugAddress is the default unix socket address for pprof data
DefaultDebugAddress = "/var/run/containerd/debug.sock"
// DefaultFIFODir is the default location used by client-side cio library
// to store FIFOs.
DefaultFIFODir = "/var/run/containerd/fifo"
// DefaultRuntime would be a multiple of choices, thus empty
DefaultRuntime = ""
// DefaultConfigDir is the default location for config files.
DefaultConfigDir = "/etc/containerd"
)
vendor/github.com/containerd/containerd/v2/defaults/defaults_differ_windows.go
Generated Vendored
-23
View File
@@ -1,23 +0,0 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package defaults
const (
// DefaultDiffer will set the default differ for the platform.
// This differ should be compatible with the windows snapshotter.
DefaultDiffer = "windows"
)
vendor/github.com/containerd/containerd/v2/defaults/defaults_linux.go
Generated Vendored
+16
View File
@@ -17,6 +17,22 @@
package defaults
const (
// DefaultAddress is the default unix socket address
DefaultAddress = "/run/containerd/containerd.sock"
// DefaultDebugAddress is the default unix socket address for pprof data
DefaultDebugAddress = "/run/containerd/debug.sock"
// DefaultFIFODir is the default location used by client-side cio library
// to store FIFOs.
DefaultFIFODir = "/run/containerd/fifo"
// DefaultRuntime is the default linux runtime
DefaultRuntime = "io.containerd.runc.v2"
// DefaultSnapshotter will set the default snapshotter for the platform.
// This will be based on the client compilation target, so take that into
// account when choosing this value.
DefaultSnapshotter = "overlayfs"
// DefaultStateDir is the default location used by containerd to store
// transient data
DefaultStateDir = "/run/containerd"
// DefaultDiffer will set the default differ for the platform.
DefaultDiffer = "walking"
)
vendor/github.com/containerd/containerd/v2/defaults/defaults_snapshotter_linux.go
Generated Vendored
-26
View File
@@ -1,26 +0,0 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package defaults
const (
// DefaultSnapshotter will set the default snapshotter for the platform.
// This will be based on the client compilation target, so take that into
// account when choosing this value.
DefaultSnapshotter = "overlayfs"
// DefaultDiffer will set the default differ for the platform.
DefaultDiffer = "walking"
)
vendor/github.com/containerd/containerd/v2/defaults/defaults_snapshotter_windows.go
Generated Vendored
-24
View File
@@ -1,24 +0,0 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package defaults
const (
// DefaultSnapshotter will set the default snapshotter for the platform.
// This will be based on the client compilation target, so take that into
// account when choosing this value.
DefaultSnapshotter = "windows"
)
vendor/github.com/containerd/containerd/v2/defaults/defaults_unix.go
Generated Vendored
+3 -13
View File
@@ -1,4 +1,4 @@
//go:build !windows && !darwin
//go:build unix
/*
Copyright The containerd Authors.
@@ -19,19 +19,9 @@
package defaults
const (
// DefaultConfigDir is the default location for config files.
DefaultConfigDir = "/etc/containerd"
// DefaultRootDir is the default location used by containerd to store
// persistent data
DefaultRootDir = "/var/lib/containerd"
// DefaultStateDir is the default location used by containerd to store
// transient data
DefaultStateDir = "/run/containerd"
// DefaultAddress is the default unix socket address
DefaultAddress = "/run/containerd/containerd.sock"
// DefaultDebugAddress is the default unix socket address for pprof data
DefaultDebugAddress = "/run/containerd/debug.sock"
// DefaultFIFODir is the default location used by client-side cio library
// to store FIFOs.
DefaultFIFODir = "/run/containerd/fifo"
// DefaultConfigDir is the default location for config files.
DefaultConfigDir = "/etc/containerd"
)
vendor/github.com/containerd/containerd/v2/defaults/defaults_snapshotter_unix.go → vendor/github.com/containerd/containerd/v2/defaults/defaults_unix_nolinux.go
Generated Vendored
+11 -1
View File
@@ -1,4 +1,4 @@
//go:build darwin || freebsd || solaris
//go:build unix && !linux
/*
Copyright The containerd Authors.
@@ -19,10 +19,20 @@
package defaults
const (
// DefaultAddress is the default unix socket address
DefaultAddress = "/var/run/containerd/containerd.sock"
// DefaultDebugAddress is the default unix socket address for pprof data
DefaultDebugAddress = "/var/run/containerd/debug.sock"
// DefaultFIFODir is the default location used by client-side cio library
// to store FIFOs.
DefaultFIFODir = "/var/run/containerd/fifo"
// DefaultSnapshotter will set the default snapshotter for the platform.
// This will be based on the client compilation target, so take that into
// account when choosing this value.
DefaultSnapshotter = "native"
// DefaultStateDir is the default location used by containerd to store
// transient data
DefaultStateDir = "/var/run/containerd"
// DefaultDiffer will set the default differ for the platform.
DefaultDiffer = "walking"
)
vendor/github.com/containerd/containerd/v2/defaults/defaults_windows.go
Generated Vendored
+7
View File
@@ -38,9 +38,16 @@ const (
DefaultAddress = `\\.\pipe\containerd-containerd`
// DefaultDebugAddress is the default winpipe address for pprof data
DefaultDebugAddress = `\\.\pipe\containerd-debug`
// DefaultDiffer will set the default differ for the platform.
// This differ should be compatible with the windows snapshotter.
DefaultDiffer = "windows"
// DefaultFIFODir is the default location used by client-side cio library
// to store FIFOs. Unused on Windows.
DefaultFIFODir = ""
// DefaultRuntime is the default windows runtime
DefaultRuntime = "io.containerd.runhcs.v1"
// DefaultSnapshotter will set the default snapshotter for the platform.
// This will be based on the client compilation target, so take that into
// account when choosing this value.
DefaultSnapshotter = "windows"
)
vendor/github.com/containerd/containerd/v2/internal/lazyregexp/lazyregexp.go
Generated Vendored
+86
View File
@@ -0,0 +1,86 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Copyright 2018 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Code below was largely copied from golang.org/x/mod@v0.22;
// https://github.com/golang/mod/blob/v0.22.0/internal/lazyregexp/lazyre.go
// with some additional methods added.
// Package lazyregexp is a thin wrapper over regexp, allowing the use of global
// regexp variables without forcing them to be compiled at init.
package lazyregexp
import (
"os"
"regexp"
"strings"
"sync"
)
// Regexp is a wrapper around [regexp.Regexp], where the underlying regexp will be
// compiled the first time it is needed.
type Regexp struct {
str string
once sync.Once
rx *regexp.Regexp
}
func (re *Regexp) re() *regexp.Regexp {
re.once.Do(re.build)
return re.rx
}
func (re *Regexp) build() {
re.rx = regexp.MustCompile(re.str)
re.str = ""
}
func (re *Regexp) FindStringIndex(s string) (loc []int) {
return re.re().FindStringIndex(s)
}
func (re *Regexp) FindStringSubmatch(s string) []string {
return re.re().FindStringSubmatch(s)
}
func (re *Regexp) MatchString(s string) bool {
return re.re().MatchString(s)
}
func (re *Regexp) ReplaceAll(src, repl []byte) []byte {
return re.re().ReplaceAll(src, repl)
}
func (re *Regexp) String() string {
return re.re().String()
}
var inTest = len(os.Args) > 0 && strings.HasSuffix(strings.TrimSuffix(os.Args[0], ".exe"), ".test")
// New creates a new lazy regexp, delaying the compiling work until it is first
// needed. If the code is being run as part of tests, the regexp compiling will
// happen immediately.
func New(str string) *Regexp {
lr := &Regexp{str: str}
if inTest {
// In tests, always compile the regexps early.
lr.re()
}
return lr
}
vendor/github.com/containerd/containerd/v2/pkg/archive/compression/compression.go
Generated Vendored
+3 -1
View File
@@ -220,7 +220,9 @@ func DecompressStream(archive io.Reader) (DecompressReadCloser, error) {
},
}, nil
case Zstd:
zstdReader, err := zstd.NewReader(buf)
zstdReader, err := zstd.NewReader(buf,
zstd.WithDecoderLowmem(false),
)
if err != nil {
return nil, err
}
vendor/github.com/containerd/containerd/v2/pkg/archive/compression/compression_fuzzer.go
Generated Vendored
-28
View File
@@ -1,28 +0,0 @@
//go:build gofuzz
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package compression
import (
"bytes"
)
func FuzzDecompressStream(data []byte) int {
_, _ = DecompressStream(bytes.NewReader(data))
return 1
}
vendor/github.com/containerd/containerd/v2/pkg/deprecation/deprecation.go
Generated Vendored
-72
View File
@@ -1,72 +0,0 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package deprecation
type Warning string
const (
// Prefix is a standard prefix for all Warnings, used for filtering plugin Exports
Prefix = "io.containerd.deprecation/"
// PullSchema1Image is a warning for the use of schema 1 images
PullSchema1Image Warning = Prefix + "pull-schema-1-image"
// GoPluginLibrary is a warning for the use of dynamic library Go plugins
GoPluginLibrary Warning = Prefix + "go-plugin-library"
// CRIRegistryMirrors is a warning for the use of the `mirrors` property
CRIRegistryMirrors Warning = Prefix + "cri-registry-mirrors"
// CRIRegistryAuths is a warning for the use of the `auths` property
CRIRegistryAuths Warning = Prefix + "cri-registry-auths"
// CRIRegistryConfigs is a warning for the use of the `configs` property
CRIRegistryConfigs Warning = Prefix + "cri-registry-configs"
// OTLPTracingConfig is a warning for the use of the `otlp` property
TracingOTLPConfig Warning = Prefix + "tracing-processor-config"
// TracingServiceConfig is a warning for the use of the `tracing` property
TracingServiceConfig Warning = Prefix + "tracing-service-config"
)
const (
EnvPrefix = "CONTAINERD_ENABLE_DEPRECATED_"
EnvPullSchema1Image = EnvPrefix + "PULL_SCHEMA_1_IMAGE"
)
var messages = map[Warning]string{
PullSchema1Image: "Schema 1 images are deprecated since containerd v1.7, disabled in containerd v2.0, and will be removed in containerd v2.1. " +
`Since containerd v1.7.8, schema 1 images are identified by the "io.containerd.image/converted-docker-schema1" label.`,
GoPluginLibrary: "Dynamically-linked Go plugins as containerd runtimes are deprecated since containerd v2.0 and removed in containerd v2.1.",
CRIRegistryMirrors: "The `mirrors` property of `[plugins.\"io.containerd.grpc.v1.cri\".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.1." +
"Use `config_path` instead.",
CRIRegistryAuths: "The `auths` property of `[plugins.\"io.containerd.grpc.v1.cri\".registry]` is deprecated since containerd v1.3 and will be removed in containerd v2.1." +
"Use `ImagePullSecrets` instead.",
CRIRegistryConfigs: "The `configs` property of `[plugins.\"io.containerd.grpc.v1.cri\".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.1." +
"Use `config_path` instead.",
TracingOTLPConfig: "The `otlp` property of `[plugins.\"io.containerd.tracing.processor.v1\".otlp]` is deprecated since containerd v1.6 and will be removed in containerd v2.0." +
"Use OTLP environment variables instead: https://opentelemetry.io/docs/specs/otel/protocol/exporter/",
TracingServiceConfig: "The `tracing` property of `[plugins.\"io.containerd.internal.v1\".tracing]` is deprecated since containerd v1.6 and will be removed in containerd v2.0." +
"Use OTEL environment variables instead: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/",
}
// Valid checks whether a given Warning is valid
func Valid(id Warning) bool {
_, ok := messages[id]
return ok
}
// Message returns the human-readable message for a given Warning
func Message(id Warning) (string, bool) {
msg, ok := messages[id]
return msg, ok
}
vendor/github.com/containerd/containerd/v2/pkg/identifiers/validate.go
Generated Vendored
+2 -2
View File
@@ -26,8 +26,8 @@ package identifiers
import (
"fmt"
"regexp"
"github.com/containerd/containerd/v2/internal/lazyregexp"
"github.com/containerd/errdefs"
)
@@ -39,7 +39,7 @@ const (
var (
// identifierRe defines the pattern for valid identifiers.
identifierRe = regexp.MustCompile(reAnchor(alphanum + reGroup(separators+reGroup(alphanum)) + "*"))
identifierRe = lazyregexp.New(reAnchor(alphanum + reGroup(separators+reGroup(alphanum)) + "*"))
)
// Validate returns nil if the string s is a valid identifier.
vendor/github.com/containerd/containerd/v2/pkg/namespaces/ttrpc.go
Generated Vendored
+1 -9
View File
@@ -27,20 +27,12 @@ const (
TTRPCHeader = "containerd-namespace-ttrpc"
)
func copyMetadata(src ttrpc.MD) ttrpc.MD {
md := ttrpc.MD{}
for k, v := range src {
md[k] = append(md[k], v...)
}
return md
}
func withTTRPCNamespaceHeader(ctx context.Context, namespace string) context.Context {
md, ok := ttrpc.GetMetadata(ctx)
if !ok {
md = ttrpc.MD{}
} else {
md = copyMetadata(md)
md = md.Clone()
}
md.Set(TTRPCHeader, namespace)
return ttrpc.WithMetadata(ctx, md)
vendor/github.com/containerd/containerd/v2/pkg/reference/reference.go
Generated Vendored
+2 -2
View File
@@ -20,9 +20,9 @@ import (
"errors"
"net/url"
"path"
"regexp"
"strings"
"github.com/containerd/containerd/v2/internal/lazyregexp"
digest "github.com/opencontainers/go-digest"
)
@@ -80,7 +80,7 @@ type Spec struct {
Object string
}
var splitRe = regexp.MustCompile(`[:@]`)
var splitRe = lazyregexp.New(`[:@]`)
// Parse parses the string into a structured ref.
func Parse(s string) (Spec, error) {
vendor/github.com/containerd/containerd/v2/plugins/content/local/content_local_fuzzer.go
Generated Vendored
-76
View File
@@ -1,76 +0,0 @@
//go:build gofuzz
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package local
import (
"bufio"
"bytes"
"context"
_ "crypto/sha256"
"io"
"testing"
"github.com/opencontainers/go-digest"
"github.com/containerd/containerd/v2/core/content"
)
func FuzzContentStoreWriter(data []byte) int {
t := &testing.T{}
ctx := context.Background()
ctx, _, cs, cleanup := contentStoreEnv(t)
defer cleanup()
cw, err := cs.Writer(ctx, content.WithRef("myref"))
if err != nil {
return 0
}
if err := cw.Close(); err != nil {
return 0
}
// reopen, so we can test things
cw, err = cs.Writer(ctx, content.WithRef("myref"))
if err != nil {
return 0
}
err = checkCopyFuzz(int64(len(data)), cw, bufio.NewReader(io.NopCloser(bytes.NewReader(data))))
if err != nil {
return 0
}
expected := digest.FromBytes(data)
if err = cw.Commit(ctx, int64(len(data)), expected); err != nil {
return 0
}
return 1
}
func checkCopyFuzz(size int64, dst io.Writer, src io.Reader) error {
nn, err := io.Copy(dst, src)
if err != nil {
return err
}
if nn != size {
return err
}
return nil
}
vendor/github.com/containerd/containerd/v2/plugins/content/local/test_helper.go
Generated Vendored
-38
View File
@@ -1,38 +0,0 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package local
import (
"context"
"testing"
"github.com/containerd/containerd/v2/core/content"
)
func contentStoreEnv(t testing.TB) (context.Context, string, content.Store, func()) {
tmpdir := t.TempDir()
cs, err := NewStore(tmpdir)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
return ctx, tmpdir, cs, func() {
cancel()
}
}
vendor/github.com/containerd/containerd/v2/version/version.go
Generated Vendored
+1 -1
View File
@@ -24,7 +24,7 @@ var (
Package = "github.com/containerd/containerd/v2"
// Version holds the complete version number. Filled in at linking time.
Version = "2.0.5+unknown"
Version = "2.1.1+unknown"
// Revision is filled with the VCS (e.g. git) revision being used to build
// the program at linking time.
vendor/github.com/go-openapi/jsonpointer/.golangci.yml
Generated Vendored
+61
View File
@@ -0,0 +1,61 @@
linters-settings:
govet:
check-shadowing: true
golint:
min-confidence: 0
gocyclo:
min-complexity: 45
maligned:
suggest-new: true
dupl:
threshold: 200
goconst:
min-len: 2
min-occurrences: 3
linters:
enable-all: true
disable:
- maligned
- unparam
- lll
- gochecknoinits
- gochecknoglobals
- funlen
- godox
- gocognit
- whitespace
- wsl
- wrapcheck
- testpackage
- nlreturn
- gomnd
- exhaustivestruct
- goerr113
- errorlint
- nestif
- godot
- gofumpt
- paralleltest
- tparallel
- thelper
- ifshort
- exhaustruct
- varnamelen
- gci
- depguard
- errchkjson
- inamedparam
- nonamedreturns
- musttag
- ireturn
- forcetypeassert
- cyclop
# deprecated linters
- deadcode
- interfacer
- scopelint
- varcheck
- structcheck
- golint
- nosnakecase
vendor/github.com/go-openapi/jsonpointer/README.md
Generated Vendored
+6 -2
View File
@@ -1,6 +1,10 @@
# gojsonpointer [![Build Status](https://travis-ci.org/go-openapi/jsonpointer.svg?branch=master)](https://travis-ci.org/go-openapi/jsonpointer) [![codecov](https://codecov.io/gh/go-openapi/jsonpointer/branch/master/graph/badge.svg)](https://codecov.io/gh/go-openapi/jsonpointer) [![Slack Status](https://slackin.goswagger.io/badge.svg)](https://slackin.goswagger.io)
# gojsonpointer [![Build Status](https://github.com/go-openapi/jsonpointer/actions/workflows/go-test.yml/badge.svg)](https://github.com/go-openapi/jsonpointer/actions?query=workflow%3A"go+test") [![codecov](https://codecov.io/gh/go-openapi/jsonpointer/branch/master/graph/badge.svg)](https://codecov.io/gh/go-openapi/jsonpointer)
[![Slack Status](https://slackin.goswagger.io/badge.svg)](https://slackin.goswagger.io)
[![license](http://img.shields.io/badge/license-Apache%20v2-orange.svg)](https://raw.githubusercontent.com/go-openapi/jsonpointer/master/LICENSE)
[![Go Reference](https://pkg.go.dev/badge/github.com/go-openapi/jsonpointer.svg)](https://pkg.go.dev/github.com/go-openapi/jsonpointer)
[![Go Report Card](https://goreportcard.com/badge/github.com/go-openapi/jsonpointer)](https://goreportcard.com/report/github.com/go-openapi/jsonpointer)
[![license](http://img.shields.io/badge/license-Apache%20v2-orange.svg)](https://raw.githubusercontent.com/go-openapi/jsonpointer/master/LICENSE) [![GoDoc](https://godoc.org/github.com/go-openapi/jsonpointer?status.svg)](http://godoc.org/github.com/go-openapi/jsonpointer)
An implementation of JSON Pointer - Go language
## Status
vendor/github.com/go-openapi/jsonpointer/pointer.go
Generated Vendored
+166 -25
View File
@@ -26,6 +26,7 @@
package jsonpointer
import (
"encoding/json"
"errors"
"fmt"
"reflect"
@@ -40,6 +41,7 @@ const (
pointerSeparator = `/`
invalidStart = `JSON pointer must be empty or start with a "` + pointerSeparator
notFound = `Can't find the pointer in the document`
)
var jsonPointableType = reflect.TypeOf(new(JSONPointable)).Elem()
@@ -48,13 +50,13 @@ var jsonSetableType = reflect.TypeOf(new(JSONSetable)).Elem()
// JSONPointable is an interface for structs to implement when they need to customize the
// json pointer process
type JSONPointable interface {
JSONLookup(string) (interface{}, error)
JSONLookup(string) (any, error)
}
// JSONSetable is an interface for structs to implement when they need to customize the
// json pointer process
type JSONSetable interface {
JSONSet(string, interface{}) error
JSONSet(string, any) error
}
// New creates a new json pointer for the given string
@@ -81,9 +83,7 @@ func (p *Pointer) parse(jsonPointerString string) error {
err = errors.New(invalidStart)
} else {
referenceTokens := strings.Split(jsonPointerString, pointerSeparator)
for _, referenceToken := range referenceTokens[1:] {
p.referenceTokens = append(p.referenceTokens, referenceToken)
}
p.referenceTokens = append(p.referenceTokens, referenceTokens[1:]...)
}
}
@@ -91,38 +91,58 @@ func (p *Pointer) parse(jsonPointerString string) error {
}
// Get uses the pointer to retrieve a value from a JSON document
func (p *Pointer) Get(document interface{}) (interface{}, reflect.Kind, error) {
func (p *Pointer) Get(document any) (any, reflect.Kind, error) {
return p.get(document, swag.DefaultJSONNameProvider)
}
// Set uses the pointer to set a value from a JSON document
func (p *Pointer) Set(document interface{}, value interface{}) (interface{}, error) {
func (p *Pointer) Set(document any, value any) (any, error) {
return document, p.set(document, value, swag.DefaultJSONNameProvider)
}
// GetForToken gets a value for a json pointer token 1 level deep
func GetForToken(document interface{}, decodedToken string) (interface{}, reflect.Kind, error) {
func GetForToken(document any, decodedToken string) (any, reflect.Kind, error) {
return getSingleImpl(document, decodedToken, swag.DefaultJSONNameProvider)
}
// SetForToken gets a value for a json pointer token 1 level deep
func SetForToken(document interface{}, decodedToken string, value interface{}) (interface{}, error) {
func SetForToken(document any, decodedToken string, value any) (any, error) {
return document, setSingleImpl(document, value, decodedToken, swag.DefaultJSONNameProvider)
}
func getSingleImpl(node interface{}, decodedToken string, nameProvider *swag.NameProvider) (interface{}, reflect.Kind, error) {
func isNil(input any) bool {
if input == nil {
return true
}
kind := reflect.TypeOf(input).Kind()
switch kind { //nolint:exhaustive
case reflect.Ptr, reflect.Map, reflect.Slice, reflect.Chan:
return reflect.ValueOf(input).IsNil()
default:
return false
}
}
func getSingleImpl(node any, decodedToken string, nameProvider *swag.NameProvider) (any, reflect.Kind, error) {
rValue := reflect.Indirect(reflect.ValueOf(node))
kind := rValue.Kind()
if isNil(node) {
return nil, kind, fmt.Errorf("nil value has not field %q", decodedToken)
}
if rValue.Type().Implements(jsonPointableType) {
r, err := node.(JSONPointable).JSONLookup(decodedToken)
switch typed := node.(type) {
case JSONPointable:
r, err := typed.JSONLookup(decodedToken)
if err != nil {
return nil, kind, err
}
return r, kind, nil
case *any: // case of a pointer to interface, that is not resolved by reflect.Indirect
return getSingleImpl(*typed, decodedToken, nameProvider)
}
switch kind {
switch kind { //nolint:exhaustive
case reflect.Struct:
nm, ok := nameProvider.GetGoNameForType(rValue.Type(), decodedToken)
if !ok {
@@ -159,7 +179,7 @@ func getSingleImpl(node interface{}, decodedToken string, nameProvider *swag.Nam
}
func setSingleImpl(node, data interface{}, decodedToken string, nameProvider *swag.NameProvider) error {
func setSingleImpl(node, data any, decodedToken string, nameProvider *swag.NameProvider) error {
rValue := reflect.Indirect(reflect.ValueOf(node))
if ns, ok := node.(JSONSetable); ok { // pointer impl
@@ -170,7 +190,7 @@ func setSingleImpl(node, data interface{}, decodedToken string, nameProvider *sw
return node.(JSONSetable).JSONSet(decodedToken, data)
}
switch rValue.Kind() {
switch rValue.Kind() { //nolint:exhaustive
case reflect.Struct:
nm, ok := nameProvider.GetGoNameForType(rValue.Type(), decodedToken)
if !ok {
@@ -210,7 +230,7 @@ func setSingleImpl(node, data interface{}, decodedToken string, nameProvider *sw
}
func (p *Pointer) get(node interface{}, nameProvider *swag.NameProvider) (interface{}, reflect.Kind, error) {
func (p *Pointer) get(node any, nameProvider *swag.NameProvider) (any, reflect.Kind, error) {
if nameProvider == nil {
nameProvider = swag.DefaultJSONNameProvider
@@ -231,8 +251,7 @@ func (p *Pointer) get(node interface{}, nameProvider *swag.NameProvider) (interf
if err != nil {
return nil, knd, err
}
node, kind = r, knd
node = r
}
rValue := reflect.ValueOf(node)
@@ -241,11 +260,11 @@ func (p *Pointer) get(node interface{}, nameProvider *swag.NameProvider) (interf
return node, kind, nil
}
func (p *Pointer) set(node, data interface{}, nameProvider *swag.NameProvider) error {
func (p *Pointer) set(node, data any, nameProvider *swag.NameProvider) error {
knd := reflect.ValueOf(node).Kind()
if knd != reflect.Ptr && knd != reflect.Struct && knd != reflect.Map && knd != reflect.Slice && knd != reflect.Array {
return fmt.Errorf("only structs, pointers, maps and slices are supported for setting values")
return errors.New("only structs, pointers, maps and slices are supported for setting values")
}
if nameProvider == nil {
@@ -284,7 +303,7 @@ func (p *Pointer) set(node, data interface{}, nameProvider *swag.NameProvider) e
continue
}
switch kind {
switch kind { //nolint:exhaustive
case reflect.Struct:
nm, ok := nameProvider.GetGoNameForType(rValue.Type(), decodedToken)
if !ok {
@@ -363,6 +382,128 @@ func (p *Pointer) String() string {
return pointerString
}
func (p *Pointer) Offset(document string) (int64, error) {
dec := json.NewDecoder(strings.NewReader(document))
var offset int64
for _, ttk := range p.DecodedTokens() {
tk, err := dec.Token()
if err != nil {
return 0, err
}
switch tk := tk.(type) {
case json.Delim:
switch tk {
case '{':
offset, err = offsetSingleObject(dec, ttk)
if err != nil {
return 0, err
}
case '[':
offset, err = offsetSingleArray(dec, ttk)
if err != nil {
return 0, err
}
default:
return 0, fmt.Errorf("invalid token %#v", tk)
}
default:
return 0, fmt.Errorf("invalid token %#v", tk)
}
}
return offset, nil
}
func offsetSingleObject(dec *json.Decoder, decodedToken string) (int64, error) {
for dec.More() {
offset := dec.InputOffset()
tk, err := dec.Token()
if err != nil {
return 0, err
}
switch tk := tk.(type) {
case json.Delim:
switch tk {
case '{':
if err = drainSingle(dec); err != nil {
return 0, err
}
case '[':
if err = drainSingle(dec); err != nil {
return 0, err
}
}
case string:
if tk == decodedToken {
return offset, nil
}
default:
return 0, fmt.Errorf("invalid token %#v", tk)
}
}
return 0, fmt.Errorf("token reference %q not found", decodedToken)
}
func offsetSingleArray(dec *json.Decoder, decodedToken string) (int64, error) {
idx, err := strconv.Atoi(decodedToken)
if err != nil {
return 0, fmt.Errorf("token reference %q is not a number: %v", decodedToken, err)
}
var i int
for i = 0; i < idx && dec.More(); i++ {
tk, err := dec.Token()
if err != nil {
return 0, err
}
if delim, isDelim := tk.(json.Delim); isDelim {
switch delim {
case '{':
if err = drainSingle(dec); err != nil {
return 0, err
}
case '[':
if err = drainSingle(dec); err != nil {
return 0, err
}
}
}
}
if !dec.More() {
return 0, fmt.Errorf("token reference %q not found", decodedToken)
}
return dec.InputOffset(), nil
}
// drainSingle drains a single level of object or array.
// The decoder has to guarantee the beginning delim (i.e. '{' or '[') has been consumed.
func drainSingle(dec *json.Decoder) error {
for dec.More() {
tk, err := dec.Token()
if err != nil {
return err
}
if delim, isDelim := tk.(json.Delim); isDelim {
switch delim {
case '{':
if err = drainSingle(dec); err != nil {
return err
}
case '[':
if err = drainSingle(dec); err != nil {
return err
}
}
}
}
// Consumes the ending delim
if _, err := dec.Token(); err != nil {
return err
}
return nil
}
// Specific JSON pointer encoding here
// ~0 => ~
// ~1 => /
@@ -377,14 +518,14 @@ const (
// Unescape unescapes a json pointer reference token string to the original representation
func Unescape(token string) string {
step1 := strings.Replace(token, encRefTok1, decRefTok1, -1)
step2 := strings.Replace(step1, encRefTok0, decRefTok0, -1)
step1 := strings.ReplaceAll(token, encRefTok1, decRefTok1)
step2 := strings.ReplaceAll(step1, encRefTok0, decRefTok0)
return step2
}
// Escape escapes a pointer reference token string
func Escape(token string) string {
step1 := strings.Replace(token, decRefTok0, encRefTok0, -1)
step2 := strings.Replace(step1, decRefTok1, encRefTok1, -1)
step1 := strings.ReplaceAll(token, decRefTok0, encRefTok0)
step2 := strings.ReplaceAll(step1, decRefTok1, encRefTok1)
return step2
}
vendor/github.com/go-openapi/swag/.gitignore
Generated
+1
View File
@@ -2,3 +2,4 @@ secrets.yml
vendor
Godeps
.idea
*.out
vendor/github.com/go-openapi/swag/.golangci.yml
Generated Vendored
+30 -24
View File
@@ -4,14 +4,14 @@ linters-settings:
golint:
min-confidence: 0
gocyclo:
min-complexity: 25
min-complexity: 45
maligned:
suggest-new: true
dupl:
threshold: 100
threshold: 200
goconst:
min-len: 3
min-occurrences: 2
min-occurrences: 3
linters:
enable-all: true
@@ -20,35 +20,41 @@ linters:
- lll
- gochecknoinits
- gochecknoglobals
- nlreturn
- testpackage
- funlen
- godox
- gocognit
- whitespace
- wsl
- wrapcheck
- testpackage
- nlreturn
- gomnd
- exhaustive
- exhaustivestruct
- goerr113
- wsl
- whitespace
- gofumpt
- godot
- errorlint
- nestif
- godox
- funlen
- gci
- gocognit
- godot
- gofumpt
- paralleltest
- tparallel
- thelper
- ifshort
- gomoddirectives
- cyclop
- forcetypeassert
- ireturn
- tagliatelle
- varnamelen
- goimports
- tenv
- golint
- exhaustruct
- nilnil
- varnamelen
- gci
- depguard
- errchkjson
- inamedparam
- nonamedreturns
- musttag
- ireturn
- forcetypeassert
- cyclop
# deprecated linters
- deadcode
- interfacer
- scopelint
- varcheck
- structcheck
- golint
- nosnakecase
vendor/github.com/go-openapi/swag/BENCHMARK.md
Generated Vendored
+52
View File
@@ -0,0 +1,52 @@
# Benchmarks
## Name mangling utilities
```bash
go test -bench XXX -run XXX -benchtime 30s
```
### Benchmarks at b3e7a5386f996177e4808f11acb2aa93a0f660df
```
goos: linux
goarch: amd64
pkg: github.com/go-openapi/swag
cpu: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
BenchmarkToXXXName/ToGoName-4 862623 44101 ns/op 10450 B/op 732 allocs/op
BenchmarkToXXXName/ToVarName-4 853656 40728 ns/op 10468 B/op 734 allocs/op
BenchmarkToXXXName/ToFileName-4 1268312 27813 ns/op 9785 B/op 617 allocs/op
BenchmarkToXXXName/ToCommandName-4 1276322 27903 ns/op 9785 B/op 617 allocs/op
BenchmarkToXXXName/ToHumanNameLower-4 895334 40354 ns/op 10472 B/op 731 allocs/op
BenchmarkToXXXName/ToHumanNameTitle-4 882441 40678 ns/op 10566 B/op 749 allocs/op
```
### Benchmarks after PR #79
~ x10 performance improvement and ~ /100 memory allocations.
```
goos: linux
goarch: amd64
pkg: github.com/go-openapi/swag
cpu: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
BenchmarkToXXXName/ToGoName-4 9595830 3991 ns/op 42 B/op 5 allocs/op
BenchmarkToXXXName/ToVarName-4 9194276 3984 ns/op 62 B/op 7 allocs/op
BenchmarkToXXXName/ToFileName-4 17002711 2123 ns/op 147 B/op 7 allocs/op
BenchmarkToXXXName/ToCommandName-4 16772926 2111 ns/op 147 B/op 7 allocs/op
BenchmarkToXXXName/ToHumanNameLower-4 9788331 3749 ns/op 92 B/op 6 allocs/op
BenchmarkToXXXName/ToHumanNameTitle-4 9188260 3941 ns/op 104 B/op 6 allocs/op
```
```
goos: linux
goarch: amd64
pkg: github.com/go-openapi/swag
cpu: AMD Ryzen 7 5800X 8-Core Processor
BenchmarkToXXXName/ToGoName-16 18527378 1972 ns/op 42 B/op 5 allocs/op
BenchmarkToXXXName/ToVarName-16 15552692 2093 ns/op 62 B/op 7 allocs/op
BenchmarkToXXXName/ToFileName-16 32161176 1117 ns/op 147 B/op 7 allocs/op
BenchmarkToXXXName/ToCommandName-16 32256634 1137 ns/op 147 B/op 7 allocs/op
BenchmarkToXXXName/ToHumanNameLower-16 18599661 1946 ns/op 92 B/op 6 allocs/op
BenchmarkToXXXName/ToHumanNameTitle-16 17581353 2054 ns/op 105 B/op 6 allocs/op
```
vendor/github.com/go-openapi/swag/README.md
Generated Vendored
+5 -3
View File
@@ -1,7 +1,8 @@
# Swag [![Build Status](https://travis-ci.org/go-openapi/swag.svg?branch=master)](https://travis-ci.org/go-openapi/swag) [![codecov](https://codecov.io/gh/go-openapi/swag/branch/master/graph/badge.svg)](https://codecov.io/gh/go-openapi/swag) [![Slack Status](https://slackin.goswagger.io/badge.svg)](https://slackin.goswagger.io)
# Swag [![Build Status](https://github.com/go-openapi/swag/actions/workflows/go-test.yml/badge.svg)](https://github.com/go-openapi/swag/actions?query=workflow%3A"go+test") [![codecov](https://codecov.io/gh/go-openapi/swag/branch/master/graph/badge.svg)](https://codecov.io/gh/go-openapi/swag)
[![Slack Status](https://slackin.goswagger.io/badge.svg)](https://slackin.goswagger.io)
[![license](http://img.shields.io/badge/license-Apache%20v2-orange.svg)](https://raw.githubusercontent.com/go-openapi/swag/master/LICENSE)
[![GoDoc](https://godoc.org/github.com/go-openapi/swag?status.svg)](http://godoc.org/github.com/go-openapi/swag)
[![Go Reference](https://pkg.go.dev/badge/github.com/go-openapi/swag.svg)](https://pkg.go.dev/github.com/go-openapi/swag)
[![Go Report Card](https://goreportcard.com/badge/github.com/go-openapi/swag)](https://goreportcard.com/report/github.com/go-openapi/swag)
Contains a bunch of helper functions for go-openapi and go-swagger projects.
@@ -18,4 +19,5 @@ You may also use it standalone for your projects.
This repo has only few dependencies outside of the standard library:
* YAML utilities depend on gopkg.in/yaml.v2
* YAML utilities depend on `gopkg.in/yaml.v3`
* `github.com/mailru/easyjson v0.7.7`
vendor/github.com/go-openapi/swag/initialism_index.go
Generated Vendored
+202
View File
@@ -0,0 +1,202 @@
// Copyright 2015 go-swagger maintainers
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package swag
import (
"sort"
"strings"
"sync"
)
var (
// commonInitialisms are common acronyms that are kept as whole uppercased words.
commonInitialisms *indexOfInitialisms
// initialisms is a slice of sorted initialisms
initialisms []string
// a copy of initialisms pre-baked as []rune
initialismsRunes [][]rune
initialismsUpperCased [][]rune
isInitialism func(string) bool
maxAllocMatches int
)
func init() {
// Taken from https://github.com/golang/lint/blob/3390df4df2787994aea98de825b964ac7944b817/lint.go#L732-L769
configuredInitialisms := map[string]bool{
"ACL": true,
"API": true,
"ASCII": true,
"CPU": true,
"CSS": true,
"DNS": true,
"EOF": true,
"GUID": true,
"HTML": true,
"HTTPS": true,
"HTTP": true,
"ID": true,
"IP": true,
"IPv4": true,
"IPv6": true,
"JSON": true,
"LHS": true,
"OAI": true,
"QPS": true,
"RAM": true,
"RHS": true,
"RPC": true,
"SLA": true,
"SMTP": true,
"SQL": true,
"SSH": true,
"TCP": true,
"TLS": true,
"TTL": true,
"UDP": true,
"UI": true,
"UID": true,
"UUID": true,
"URI": true,
"URL": true,
"UTF8": true,
"VM": true,
"XML": true,
"XMPP": true,
"XSRF": true,
"XSS": true,
}
// a thread-safe index of initialisms
commonInitialisms = newIndexOfInitialisms().load(configuredInitialisms)
initialisms = commonInitialisms.sorted()
initialismsRunes = asRunes(initialisms)
initialismsUpperCased = asUpperCased(initialisms)
maxAllocMatches = maxAllocHeuristic(initialismsRunes)
// a test function
isInitialism = commonInitialisms.isInitialism
}
func asRunes(in []string) [][]rune {
out := make([][]rune, len(in))
for i, initialism := range in {
out[i] = []rune(initialism)
}
return out
}
func asUpperCased(in []string) [][]rune {
out := make([][]rune, len(in))
for i, initialism := range in {
out[i] = []rune(upper(trim(initialism)))
}
return out
}
func maxAllocHeuristic(in [][]rune) int {
heuristic := make(map[rune]int)
for _, initialism := range in {
heuristic[initialism[0]]++
}
var maxAlloc int
for _, val := range heuristic {
if val > maxAlloc {
maxAlloc = val
}
}
return maxAlloc
}
// AddInitialisms add additional initialisms
func AddInitialisms(words ...string) {
for _, word := range words {
// commonInitialisms[upper(word)] = true
commonInitialisms.add(upper(word))
}
// sort again
initialisms = commonInitialisms.sorted()
initialismsRunes = asRunes(initialisms)
initialismsUpperCased = asUpperCased(initialisms)
}
// indexOfInitialisms is a thread-safe implementation of the sorted index of initialisms.
// Since go1.9, this may be implemented with sync.Map.
type indexOfInitialisms struct {
sortMutex *sync.Mutex
index *sync.Map
}
func newIndexOfInitialisms() *indexOfInitialisms {
return &indexOfInitialisms{
sortMutex: new(sync.Mutex),
index: new(sync.Map),
}
}
func (m *indexOfInitialisms) load(initial map[string]bool) *indexOfInitialisms {
m.sortMutex.Lock()
defer m.sortMutex.Unlock()
for k, v := range initial {
m.index.Store(k, v)
}
return m
}
func (m *indexOfInitialisms) isInitialism(key string) bool {
_, ok := m.index.Load(key)
return ok
}
func (m *indexOfInitialisms) add(key string) *indexOfInitialisms {
m.index.Store(key, true)
return m
}
func (m *indexOfInitialisms) sorted() (result []string) {
m.sortMutex.Lock()
defer m.sortMutex.Unlock()
m.index.Range(func(key, _ interface{}) bool {
k := key.(string)
result = append(result, k)
return true
})
sort.Sort(sort.Reverse(byInitialism(result)))
return
}
type byInitialism []string
func (s byInitialism) Len() int {
return len(s)
}
func (s byInitialism) Swap(i, j int) {
s[i], s[j] = s[j], s[i]
}
func (s byInitialism) Less(i, j int) bool {
if len(s[i]) != len(s[j]) {
return len(s[i]) < len(s[j])
}
return strings.Compare(s[i], s[j]) > 0
}
vendor/github.com/go-openapi/swag/loading.go
Generated Vendored
+80 -25
View File
@@ -21,6 +21,7 @@ import (
"net/http"
"net/url"
"os"
"path"
"path/filepath"
"runtime"
"strings"
@@ -40,43 +41,97 @@ var LoadHTTPBasicAuthPassword = ""
var LoadHTTPCustomHeaders = map[string]string{}
// LoadFromFileOrHTTP loads the bytes from a file or a remote http server based on the path passed in
func LoadFromFileOrHTTP(path string) ([]byte, error) {
return LoadStrategy(path, os.ReadFile, loadHTTPBytes(LoadHTTPTimeout))(path)
func LoadFromFileOrHTTP(pth string) ([]byte, error) {
return LoadStrategy(pth, os.ReadFile, loadHTTPBytes(LoadHTTPTimeout))(pth)
}
// LoadFromFileOrHTTPWithTimeout loads the bytes from a file or a remote http server based on the path passed in
// timeout arg allows for per request overriding of the request timeout
func LoadFromFileOrHTTPWithTimeout(path string, timeout time.Duration) ([]byte, error) {
return LoadStrategy(path, os.ReadFile, loadHTTPBytes(timeout))(path)
func LoadFromFileOrHTTPWithTimeout(pth string, timeout time.Duration) ([]byte, error) {
return LoadStrategy(pth, os.ReadFile, loadHTTPBytes(timeout))(pth)
}
// LoadStrategy returns a loader function for a given path or uri
func LoadStrategy(path string, local, remote func(string) ([]byte, error)) func(string) ([]byte, error) {
if strings.HasPrefix(path, "http") {
// LoadStrategy returns a loader function for a given path or URI.
//
// The load strategy returns the remote load for any path starting with `http`.
// So this works for any URI with a scheme `http` or `https`.
//
// The fallback strategy is to call the local loader.
//
// The local loader takes a local file system path (absolute or relative) as argument,
// or alternatively a `file://...` URI, **without host** (see also below for windows).
//
// There are a few liberalities, initially intended to be tolerant regarding the URI syntax,
// especially on windows.
//
// Before the local loader is called, the given path is transformed:
// - percent-encoded characters are unescaped
// - simple paths (e.g. `./folder/file`) are passed as-is
// - on windows, occurrences of `/` are replaced by `\`, so providing a relative path such a `folder/file` works too.
//
// For paths provided as URIs with the "file" scheme, please note that:
// - `file://` is simply stripped.
// This means that the host part of the URI is not parsed at all.
// For example, `file:///folder/file" becomes "/folder/file`,
// but `file://localhost/folder/file` becomes `localhost/folder/file` on unix systems.
// Similarly, `file://./folder/file` yields `./folder/file`.
// - on windows, `file://...` can take a host so as to specify an UNC share location.
//
// Reminder about windows-specifics:
// - `file://host/folder/file` becomes an UNC path like `\\host\folder\file` (no port specification is supported)
// - `file:///c:/folder/file` becomes `C:\folder\file`
// - `file://c:/folder/file` is tolerated (without leading `/`) and becomes `c:\folder\file`
func LoadStrategy(pth string, local, remote func(string) ([]byte, error)) func(string) ([]byte, error) {
if strings.HasPrefix(pth, "http") {
return remote
}
return func(pth string) ([]byte, error) {
upth, err := pathUnescape(pth)
return func(p string) ([]byte, error) {
upth, err := url.PathUnescape(p)
if err != nil {
return nil, err
}
if strings.HasPrefix(pth, `file://`) {
if runtime.GOOS == "windows" {
// support for canonical file URIs on windows.
// Zero tolerance here for dodgy URIs.
u, _ := url.Parse(upth)
if u.Host != "" {
// assume UNC name (volume share)
// file://host/share/folder\... ==> \\host\share\path\folder
// NOTE: UNC port not yet supported
upth = strings.Join([]string{`\`, u.Host, u.Path}, `\`)
} else {
// file:///c:/folder/... ==> just remove the leading slash
upth = strings.TrimPrefix(upth, `file:///`)
}
} else {
upth = strings.TrimPrefix(upth, `file://`)
if !strings.HasPrefix(p, `file://`) {
// regular file path provided: just normalize slashes
return local(filepath.FromSlash(upth))
}
if runtime.GOOS != "windows" {
// crude processing: this leaves full URIs with a host with a (mostly) unexpected result
upth = strings.TrimPrefix(upth, `file://`)
return local(filepath.FromSlash(upth))
}
// windows-only pre-processing of file://... URIs
// support for canonical file URIs on windows.
u, err := url.Parse(filepath.ToSlash(upth))
if err != nil {
return nil, err
}
if u.Host != "" {
// assume UNC name (volume share)
// NOTE: UNC port not yet supported
// when the "host" segment is a drive letter:
// file://C:/folder/... => C:\folder
upth = path.Clean(strings.Join([]string{u.Host, u.Path}, `/`))
if !strings.HasSuffix(u.Host, ":") && u.Host[0] != '.' {
// tolerance: if we have a leading dot, this can't be a host
// file://host/share/folder\... ==> \\host\share\path\folder
upth = "//" + upth
}
} else {
// no host, let's figure out if this is a drive letter
upth = strings.TrimPrefix(upth, `file://`)
first, _, _ := strings.Cut(strings.TrimPrefix(u.Path, "/"), "/")
if strings.HasSuffix(first, ":") {
// drive letter in the first segment:
// file:///c:/folder/... ==> strip the leading slash
upth = strings.TrimPrefix(upth, `/`)
}
}
vendor/github.com/go-openapi/swag/name_lexem.go
Generated Vendored
+39 -33
View File
@@ -14,74 +14,80 @@
package swag
import "unicode"
import (
"unicode"
"unicode/utf8"
)
type (
nameLexem interface {
GetUnsafeGoName() string
GetOriginal() string
IsInitialism() bool
}
lexemKind uint8
initialismNameLexem struct {
nameLexem struct {
original string
matchedInitialism string
}
casualNameLexem struct {
original string
kind lexemKind
}
)
func newInitialismNameLexem(original, matchedInitialism string) *initialismNameLexem {
return &initialismNameLexem{
const (
lexemKindCasualName lexemKind = iota
lexemKindInitialismName
)
func newInitialismNameLexem(original, matchedInitialism string) nameLexem {
return nameLexem{
kind: lexemKindInitialismName,
original: original,
matchedInitialism: matchedInitialism,
}
}
func newCasualNameLexem(original string) *casualNameLexem {
return &casualNameLexem{
func newCasualNameLexem(original string) nameLexem {
return nameLexem{
kind: lexemKindCasualName,
original: original,
}
}
func (l *initialismNameLexem) GetUnsafeGoName() string {
return l.matchedInitialism
}
func (l nameLexem) GetUnsafeGoName() string {
if l.kind == lexemKindInitialismName {
return l.matchedInitialism
}
var (
first rune
rest string
)
func (l *casualNameLexem) GetUnsafeGoName() string {
var first rune
var rest string
for i, orig := range l.original {
if i == 0 {
first = orig
continue
}
if i > 0 {
rest = l.original[i:]
break
}
}
if len(l.original) > 1 {
return string(unicode.ToUpper(first)) + lower(rest)
b := poolOfBuffers.BorrowBuffer(utf8.UTFMax + len(rest))
defer func() {
poolOfBuffers.RedeemBuffer(b)
}()
b.WriteRune(unicode.ToUpper(first))
b.WriteString(lower(rest))
return b.String()
}
return l.original
}
func (l *initialismNameLexem) GetOriginal() string {
func (l nameLexem) GetOriginal() string {
return l.original
}
func (l *casualNameLexem) GetOriginal() string {
return l.original
}
func (l *initialismNameLexem) IsInitialism() bool {
return true
}
func (l *casualNameLexem) IsInitialism() bool {
return false
func (l nameLexem) IsInitialism() bool {
return l.kind == lexemKindInitialismName
}
vendor/github.com/go-openapi/swag/post_go18.go
Generated Vendored
-24
View File
@@ -1,24 +0,0 @@
// Copyright 2015 go-swagger maintainers
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//go:build go1.8
// +build go1.8
package swag
import "net/url"
func pathUnescape(path string) (string, error) {
return url.PathUnescape(path)
}
vendor/github.com/go-openapi/swag/post_go19.go
Generated Vendored
-68
View File
@@ -1,68 +0,0 @@
// Copyright 2015 go-swagger maintainers
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//go:build go1.9
// +build go1.9
package swag
import (
"sort"
"sync"
)
// indexOfInitialisms is a thread-safe implementation of the sorted index of initialisms.
// Since go1.9, this may be implemented with sync.Map.
type indexOfInitialisms struct {
sortMutex *sync.Mutex
index *sync.Map
}
func newIndexOfInitialisms() *indexOfInitialisms {
return &indexOfInitialisms{
sortMutex: new(sync.Mutex),
index: new(sync.Map),
}
}
func (m *indexOfInitialisms) load(initial map[string]bool) *indexOfInitialisms {
m.sortMutex.Lock()
defer m.sortMutex.Unlock()
for k, v := range initial {
m.index.Store(k, v)
}
return m
}
func (m *indexOfInitialisms) isInitialism(key string) bool {
_, ok := m.index.Load(key)
return ok
}
func (m *indexOfInitialisms) add(key string) *indexOfInitialisms {
m.index.Store(key, true)
return m
}
func (m *indexOfInitialisms) sorted() (result []string) {
m.sortMutex.Lock()
defer m.sortMutex.Unlock()
m.index.Range(func(key, value interface{}) bool {
k := key.(string)
result = append(result, k)
return true
})
sort.Sort(sort.Reverse(byInitialism(result)))
return
}
vendor/github.com/go-openapi/swag/pre_go18.go
Generated Vendored
-24
View File
@@ -1,24 +0,0 @@
// Copyright 2015 go-swagger maintainers
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//go:build !go1.8
// +build !go1.8
package swag
import "net/url"
func pathUnescape(path string) (string, error) {
return url.QueryUnescape(path)
}
vendor/github.com/go-openapi/swag/pre_go19.go
Generated Vendored
-70
View File
@@ -1,70 +0,0 @@
// Copyright 2015 go-swagger maintainers
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//go:build !go1.9
// +build !go1.9
package swag
import (
"sort"
"sync"
)
// indexOfInitialisms is a thread-safe implementation of the sorted index of initialisms.
// Before go1.9, this may be implemented with a mutex on the map.
type indexOfInitialisms struct {
getMutex *sync.Mutex
index map[string]bool
}
func newIndexOfInitialisms() *indexOfInitialisms {
return &indexOfInitialisms{
getMutex: new(sync.Mutex),
index: make(map[string]bool, 50),
}
}
func (m *indexOfInitialisms) load(initial map[string]bool) *indexOfInitialisms {
m.getMutex.Lock()
defer m.getMutex.Unlock()
for k, v := range initial {
m.index[k] = v
}
return m
}
func (m *indexOfInitialisms) isInitialism(key string) bool {
m.getMutex.Lock()
defer m.getMutex.Unlock()
_, ok := m.index[key]
return ok
}
func (m *indexOfInitialisms) add(key string) *indexOfInitialisms {
m.getMutex.Lock()
defer m.getMutex.Unlock()
m.index[key] = true
return m
}
func (m *indexOfInitialisms) sorted() (result []string) {
m.getMutex.Lock()
defer m.getMutex.Unlock()
for k := range m.index {
result = append(result, k)
}
sort.Sort(sort.Reverse(byInitialism(result)))
return
}
vendor/github.com/go-openapi/swag/split.go
Generated Vendored
+367 -121
View File
@@ -15,124 +15,269 @@
package swag
import (
"bytes"
"sync"
"unicode"
"unicode/utf8"
)
var nameReplaceTable = map[rune]string{
'@': "At ",
'&': "And ",
'|': "Pipe ",
'$': "Dollar ",
'!': "Bang ",
'-': "",
'_': "",
}
type (
splitter struct {
postSplitInitialismCheck bool
initialisms []string
initialismsRunes [][]rune
initialismsUpperCased [][]rune // initialisms cached in their trimmed, upper-cased version
postSplitInitialismCheck bool
}
splitterOption func(*splitter) *splitter
splitterOption func(*splitter)
initialismMatch struct {
body []rune
start, end int
complete bool
}
initialismMatches []initialismMatch
)
// split calls the splitter; splitter provides more control and post options
func split(str string) []string {
lexems := newSplitter().split(str)
result := make([]string, 0, len(lexems))
type (
// memory pools of temporary objects.
//
// These are used to recycle temporarily allocated objects
// and relieve the GC from undue pressure.
for _, lexem := range lexems {
matchesPool struct {
*sync.Pool
}
buffersPool struct {
*sync.Pool
}
lexemsPool struct {
*sync.Pool
}
splittersPool struct {
*sync.Pool
}
)
var (
// poolOfMatches holds temporary slices for recycling during the initialism match process
poolOfMatches = matchesPool{
Pool: &sync.Pool{
New: func() any {
s := make(initialismMatches, 0, maxAllocMatches)
return &s
},
},
}
poolOfBuffers = buffersPool{
Pool: &sync.Pool{
New: func() any {
return new(bytes.Buffer)
},
},
}
poolOfLexems = lexemsPool{
Pool: &sync.Pool{
New: func() any {
s := make([]nameLexem, 0, maxAllocMatches)
return &s
},
},
}
poolOfSplitters = splittersPool{
Pool: &sync.Pool{
New: func() any {
s := newSplitter()
return &s
},
},
}
)
// nameReplaceTable finds a word representation for special characters.
func nameReplaceTable(r rune) (string, bool) {
switch r {
case '@':
return "At ", true
case '&':
return "And ", true
case '|':
return "Pipe ", true
case '$':
return "Dollar ", true
case '!':
return "Bang ", true
case '-':
return "", true
case '_':
return "", true
default:
return "", false
}
}
// split calls the splitter.
//
// Use newSplitter for more control and options
func split(str string) []string {
s := poolOfSplitters.BorrowSplitter()
lexems := s.split(str)
result := make([]string, 0, len(*lexems))
for _, lexem := range *lexems {
result = append(result, lexem.GetOriginal())
}
poolOfLexems.RedeemLexems(lexems)
poolOfSplitters.RedeemSplitter(s)
return result
}
func (s *splitter) split(str string) []nameLexem {
return s.toNameLexems(str)
}
func newSplitter(options ...splitterOption) *splitter {
splitter := &splitter{
func newSplitter(options ...splitterOption) splitter {
s := splitter{
postSplitInitialismCheck: false,
initialisms: initialisms,
initialismsRunes: initialismsRunes,
initialismsUpperCased: initialismsUpperCased,
}
for _, option := range options {
splitter = option(splitter)
option(&s)
}
return splitter
}
// withPostSplitInitialismCheck allows to catch initialisms after main split process
func withPostSplitInitialismCheck(s *splitter) *splitter {
s.postSplitInitialismCheck = true
return s
}
type (
initialismMatch struct {
start, end int
body []rune
complete bool
}
initialismMatches []*initialismMatch
)
// withPostSplitInitialismCheck allows to catch initialisms after main split process
func withPostSplitInitialismCheck(s *splitter) {
s.postSplitInitialismCheck = true
}
func (s *splitter) toNameLexems(name string) []nameLexem {
func (p matchesPool) BorrowMatches() *initialismMatches {
s := p.Get().(*initialismMatches)
*s = (*s)[:0] // reset slice, keep allocated capacity
return s
}
func (p buffersPool) BorrowBuffer(size int) *bytes.Buffer {
s := p.Get().(*bytes.Buffer)
s.Reset()
if s.Cap() < size {
s.Grow(size)
}
return s
}
func (p lexemsPool) BorrowLexems() *[]nameLexem {
s := p.Get().(*[]nameLexem)
*s = (*s)[:0] // reset slice, keep allocated capacity
return s
}
func (p splittersPool) BorrowSplitter(options ...splitterOption) *splitter {
s := p.Get().(*splitter)
s.postSplitInitialismCheck = false // reset options
for _, apply := range options {
apply(s)
}
return s
}
func (p matchesPool) RedeemMatches(s *initialismMatches) {
p.Put(s)
}
func (p buffersPool) RedeemBuffer(s *bytes.Buffer) {
p.Put(s)
}
func (p lexemsPool) RedeemLexems(s *[]nameLexem) {
p.Put(s)
}
func (p splittersPool) RedeemSplitter(s *splitter) {
p.Put(s)
}
func (m initialismMatch) isZero() bool {
return m.start == 0 && m.end == 0
}
func (s splitter) split(name string) *[]nameLexem {
nameRunes := []rune(name)
matches := s.gatherInitialismMatches(nameRunes)
if matches == nil {
return poolOfLexems.BorrowLexems()
}
return s.mapMatchesToNameLexems(nameRunes, matches)
}
func (s *splitter) gatherInitialismMatches(nameRunes []rune) initialismMatches {
matches := make(initialismMatches, 0)
func (s splitter) gatherInitialismMatches(nameRunes []rune) *initialismMatches {
var matches *initialismMatches
for currentRunePosition, currentRune := range nameRunes {
newMatches := make(initialismMatches, 0, len(matches))
// recycle these allocations as we loop over runes
// with such recycling, only 2 slices should be allocated per call
// instead of o(n).
newMatches := poolOfMatches.BorrowMatches()
// check current initialism matches
for _, match := range matches {
if keepCompleteMatch := match.complete; keepCompleteMatch {
newMatches = append(newMatches, match)
continue
}
// drop failed match
currentMatchRune := match.body[currentRunePosition-match.start]
if !s.initialismRuneEqual(currentMatchRune, currentRune) {
continue
}
// try to complete ongoing match
if currentRunePosition-match.start == len(match.body)-1 {
// we are close; the next step is to check the symbol ahead
// if it is a small letter, then it is not the end of match
// but beginning of the next word
if currentRunePosition < len(nameRunes)-1 {
nextRune := nameRunes[currentRunePosition+1]
if newWord := unicode.IsLower(nextRune); newWord {
// oh ok, it was the start of a new word
continue
}
if matches != nil { // skip first iteration
for _, match := range *matches {
if keepCompleteMatch := match.complete; keepCompleteMatch {
*newMatches = append(*newMatches, match)
continue
}
match.complete = true
match.end = currentRunePosition
}
// drop failed match
currentMatchRune := match.body[currentRunePosition-match.start]
if currentMatchRune != currentRune {
continue
}
newMatches = append(newMatches, match)
// try to complete ongoing match
if currentRunePosition-match.start == len(match.body)-1 {
// we are close; the next step is to check the symbol ahead
// if it is a small letter, then it is not the end of match
// but beginning of the next word
if currentRunePosition < len(nameRunes)-1 {
nextRune := nameRunes[currentRunePosition+1]
if newWord := unicode.IsLower(nextRune); newWord {
// oh ok, it was the start of a new word
continue
}
}
match.complete = true
match.end = currentRunePosition
}
*newMatches = append(*newMatches, match)
}
}
// check for new initialism matches
for _, initialism := range s.initialisms {
initialismRunes := []rune(initialism)
if s.initialismRuneEqual(initialismRunes[0], currentRune) {
newMatches = append(newMatches, &initialismMatch{
for i := range s.initialisms {
initialismRunes := s.initialismsRunes[i]
if initialismRunes[0] == currentRune {
*newMatches = append(*newMatches, initialismMatch{
start: currentRunePosition,
body: initialismRunes,
complete: false,
@@ -140,24 +285,28 @@ func (s *splitter) gatherInitialismMatches(nameRunes []rune) initialismMatches {
}
}
if matches != nil {
poolOfMatches.RedeemMatches(matches)
}
matches = newMatches
}
// up to the caller to redeem this last slice
return matches
}
func (s *splitter) mapMatchesToNameLexems(nameRunes []rune, matches initialismMatches) []nameLexem {
nameLexems := make([]nameLexem, 0)
func (s splitter) mapMatchesToNameLexems(nameRunes []rune, matches *initialismMatches) *[]nameLexem {
nameLexems := poolOfLexems.BorrowLexems()
var lastAcceptedMatch *initialismMatch
for _, match := range matches {
var lastAcceptedMatch initialismMatch
for _, match := range *matches {
if !match.complete {
continue
}
if firstMatch := lastAcceptedMatch == nil; firstMatch {
nameLexems = append(nameLexems, s.breakCasualString(nameRunes[:match.start])...)
nameLexems = append(nameLexems, s.breakInitialism(string(match.body)))
if firstMatch := lastAcceptedMatch.isZero(); firstMatch {
s.appendBrokenDownCasualString(nameLexems, nameRunes[:match.start])
*nameLexems = append(*nameLexems, s.breakInitialism(string(match.body)))
lastAcceptedMatch = match
@@ -169,63 +318,66 @@ func (s *splitter) mapMatchesToNameLexems(nameRunes []rune, matches initialismMa
}
middle := nameRunes[lastAcceptedMatch.end+1 : match.start]
nameLexems = append(nameLexems, s.breakCasualString(middle)...)
nameLexems = append(nameLexems, s.breakInitialism(string(match.body)))
s.appendBrokenDownCasualString(nameLexems, middle)
*nameLexems = append(*nameLexems, s.breakInitialism(string(match.body)))
lastAcceptedMatch = match
}
// we have not found any accepted matches
if lastAcceptedMatch == nil {
return s.breakCasualString(nameRunes)
if lastAcceptedMatch.isZero() {
*nameLexems = (*nameLexems)[:0]
s.appendBrokenDownCasualString(nameLexems, nameRunes)
} else if lastAcceptedMatch.end+1 != len(nameRunes) {
rest := nameRunes[lastAcceptedMatch.end+1:]
s.appendBrokenDownCasualString(nameLexems, rest)
}
if lastAcceptedMatch.end+1 != len(nameRunes) {
rest := nameRunes[lastAcceptedMatch.end+1:]
nameLexems = append(nameLexems, s.breakCasualString(rest)...)
}
poolOfMatches.RedeemMatches(matches)
return nameLexems
}
func (s *splitter) initialismRuneEqual(a, b rune) bool {
return a == b
}
func (s *splitter) breakInitialism(original string) nameLexem {
func (s splitter) breakInitialism(original string) nameLexem {
return newInitialismNameLexem(original, original)
}
func (s *splitter) breakCasualString(str []rune) []nameLexem {
segments := make([]nameLexem, 0)
currentSegment := ""
func (s splitter) appendBrokenDownCasualString(segments *[]nameLexem, str []rune) {
currentSegment := poolOfBuffers.BorrowBuffer(len(str)) // unlike strings.Builder, bytes.Buffer initial storage can reused
defer func() {
poolOfBuffers.RedeemBuffer(currentSegment)
}()
addCasualNameLexem := func(original string) {
segments = append(segments, newCasualNameLexem(original))
*segments = append(*segments, newCasualNameLexem(original))
}
addInitialismNameLexem := func(original, match string) {
segments = append(segments, newInitialismNameLexem(original, match))
*segments = append(*segments, newInitialismNameLexem(original, match))
}
addNameLexem := func(original string) {
if s.postSplitInitialismCheck {
for _, initialism := range s.initialisms {
if upper(initialism) == upper(original) {
addInitialismNameLexem(original, initialism)
var addNameLexem func(string)
if s.postSplitInitialismCheck {
addNameLexem = func(original string) {
for i := range s.initialisms {
if isEqualFoldIgnoreSpace(s.initialismsUpperCased[i], original) {
addInitialismNameLexem(original, s.initialisms[i])
return
}
}
}
addCasualNameLexem(original)
addCasualNameLexem(original)
}
} else {
addNameLexem = addCasualNameLexem
}
for _, rn := range string(str) {
if replace, found := nameReplaceTable[rn]; found {
if currentSegment != "" {
addNameLexem(currentSegment)
currentSegment = ""
for _, rn := range str {
if replace, found := nameReplaceTable(rn); found {
if currentSegment.Len() > 0 {
addNameLexem(currentSegment.String())
currentSegment.Reset()
}
if replace != "" {
@@ -236,27 +388,121 @@ func (s *splitter) breakCasualString(str []rune) []nameLexem {
}
if !unicode.In(rn, unicode.L, unicode.M, unicode.N, unicode.Pc) {
if currentSegment != "" {
addNameLexem(currentSegment)
currentSegment = ""
if currentSegment.Len() > 0 {
addNameLexem(currentSegment.String())
currentSegment.Reset()
}
continue
}
if unicode.IsUpper(rn) {
if currentSegment != "" {
addNameLexem(currentSegment)
if currentSegment.Len() > 0 {
addNameLexem(currentSegment.String())
}
currentSegment = ""
currentSegment.Reset()
}
currentSegment += string(rn)
currentSegment.WriteRune(rn)
}
if currentSegment != "" {
addNameLexem(currentSegment)
if currentSegment.Len() > 0 {
addNameLexem(currentSegment.String())
}
}
return segments
// isEqualFoldIgnoreSpace is the same as strings.EqualFold, but
// it ignores leading and trailing blank spaces in the compared
// string.
//
// base is assumed to be composed of upper-cased runes, and be already
// trimmed.
//
// This code is heavily inspired from strings.EqualFold.
func isEqualFoldIgnoreSpace(base []rune, str string) bool {
var i, baseIndex int
// equivalent to b := []byte(str), but without data copy
b := hackStringBytes(str)
for i < len(b) {
if c := b[i]; c < utf8.RuneSelf {
// fast path for ASCII
if c != ' ' && c != '\t' {
break
}
i++
continue
}
// unicode case
r, size := utf8.DecodeRune(b[i:])
if !unicode.IsSpace(r) {
break
}
i += size
}
if i >= len(b) {
return len(base) == 0
}
for _, baseRune := range base {
if i >= len(b) {
break
}
if c := b[i]; c < utf8.RuneSelf {
// single byte rune case (ASCII)
if baseRune >= utf8.RuneSelf {
return false
}
baseChar := byte(baseRune)
if c != baseChar &&
!('a' <= c && c <= 'z' && c-'a'+'A' == baseChar) {
return false
}
baseIndex++
i++
continue
}
// unicode case
r, size := utf8.DecodeRune(b[i:])
if unicode.ToUpper(r) != baseRune {
return false
}
baseIndex++
i += size
}
if baseIndex != len(base) {
return false
}
// all passed: now we should only have blanks
for i < len(b) {
if c := b[i]; c < utf8.RuneSelf {
// fast path for ASCII
if c != ' ' && c != '\t' {
return false
}
i++
continue
}
// unicode case
r, size := utf8.DecodeRune(b[i:])
if !unicode.IsSpace(r) {
return false
}
i += size
}
return true
}
vendor/github.com/go-openapi/swag/string_bytes.go
Generated Vendored
+8
View File
@@ -0,0 +1,8 @@
package swag
import "unsafe"
// hackStringBytes returns the (unsafe) underlying bytes slice of a string.
func hackStringBytes(str string) []byte {
return unsafe.Slice(unsafe.StringData(str), len(str))
}
vendor/github.com/go-openapi/swag/util.go
Generated Vendored
+90 -120
View File
@@ -18,76 +18,25 @@ import (
"reflect"
"strings"
"unicode"
"unicode/utf8"
)
// commonInitialisms are common acronyms that are kept as whole uppercased words.
var commonInitialisms *indexOfInitialisms
// initialisms is a slice of sorted initialisms
var initialisms []string
var isInitialism func(string) bool
// GoNamePrefixFunc sets an optional rule to prefix go names
// which do not start with a letter.
//
// The prefix function is assumed to return a string that starts with an upper case letter.
//
// e.g. to help convert "123" into "{prefix}123"
//
// The default is to prefix with "X"
var GoNamePrefixFunc func(string) string
func init() {
// Taken from https://github.com/golang/lint/blob/3390df4df2787994aea98de825b964ac7944b817/lint.go#L732-L769
var configuredInitialisms = map[string]bool{
"ACL": true,
"API": true,
"ASCII": true,
"CPU": true,
"CSS": true,
"DNS": true,
"EOF": true,
"GUID": true,
"HTML": true,
"HTTPS": true,
"HTTP": true,
"ID": true,
"IP": true,
"IPv4": true,
"IPv6": true,
"JSON": true,
"LHS": true,
"OAI": true,
"QPS": true,
"RAM": true,
"RHS": true,
"RPC": true,
"SLA": true,
"SMTP": true,
"SQL": true,
"SSH": true,
"TCP": true,
"TLS": true,
"TTL": true,
"UDP": true,
"UI": true,
"UID": true,
"UUID": true,
"URI": true,
"URL": true,
"UTF8": true,
"VM": true,
"XML": true,
"XMPP": true,
"XSRF": true,
"XSS": true,
func prefixFunc(name, in string) string {
if GoNamePrefixFunc == nil {
return "X" + in
}
// a thread-safe index of initialisms
commonInitialisms = newIndexOfInitialisms().load(configuredInitialisms)
initialisms = commonInitialisms.sorted()
// a test function
isInitialism = commonInitialisms.isInitialism
return GoNamePrefixFunc(name) + in
}
const (
@@ -156,25 +105,9 @@ func SplitByFormat(data, format string) []string {
return result
}
type byInitialism []string
func (s byInitialism) Len() int {
return len(s)
}
func (s byInitialism) Swap(i, j int) {
s[i], s[j] = s[j], s[i]
}
func (s byInitialism) Less(i, j int) bool {
if len(s[i]) != len(s[j]) {
return len(s[i]) < len(s[j])
}
return strings.Compare(s[i], s[j]) > 0
}
// Removes leading whitespaces
func trim(str string) string {
return strings.Trim(str, " ")
return strings.TrimSpace(str)
}
// Shortcut to strings.ToUpper()
@@ -188,15 +121,20 @@ func lower(str string) string {
}
// Camelize an uppercased word
func Camelize(word string) (camelized string) {
func Camelize(word string) string {
camelized := poolOfBuffers.BorrowBuffer(len(word))
defer func() {
poolOfBuffers.RedeemBuffer(camelized)
}()
for pos, ru := range []rune(word) {
if pos > 0 {
camelized += string(unicode.ToLower(ru))
camelized.WriteRune(unicode.ToLower(ru))
} else {
camelized += string(unicode.ToUpper(ru))
camelized.WriteRune(unicode.ToUpper(ru))
}
}
return
return camelized.String()
}
// ToFileName lowercases and underscores a go type name
@@ -224,33 +162,40 @@ func ToCommandName(name string) string {
// ToHumanNameLower represents a code name as a human series of words
func ToHumanNameLower(name string) string {
in := newSplitter(withPostSplitInitialismCheck).split(name)
out := make([]string, 0, len(in))
s := poolOfSplitters.BorrowSplitter(withPostSplitInitialismCheck)
in := s.split(name)
poolOfSplitters.RedeemSplitter(s)
out := make([]string, 0, len(*in))
for _, w := range in {
for _, w := range *in {
if !w.IsInitialism() {
out = append(out, lower(w.GetOriginal()))
} else {
out = append(out, w.GetOriginal())
out = append(out, trim(w.GetOriginal()))
}
}
poolOfLexems.RedeemLexems(in)
return strings.Join(out, " ")
}
// ToHumanNameTitle represents a code name as a human series of words with the first letters titleized
func ToHumanNameTitle(name string) string {
in := newSplitter(withPostSplitInitialismCheck).split(name)
s := poolOfSplitters.BorrowSplitter(withPostSplitInitialismCheck)
in := s.split(name)
poolOfSplitters.RedeemSplitter(s)
out := make([]string, 0, len(in))
for _, w := range in {
original := w.GetOriginal()
out := make([]string, 0, len(*in))
for _, w := range *in {
original := trim(w.GetOriginal())
if !w.IsInitialism() {
out = append(out, Camelize(original))
} else {
out = append(out, original)
}
}
poolOfLexems.RedeemLexems(in)
return strings.Join(out, " ")
}
@@ -264,7 +209,7 @@ func ToJSONName(name string) string {
out = append(out, lower(w))
continue
}
out = append(out, Camelize(w))
out = append(out, Camelize(trim(w)))
}
return strings.Join(out, "")
}
@@ -283,35 +228,70 @@ func ToVarName(name string) string {
// ToGoName translates a swagger name which can be underscored or camel cased to a name that golint likes
func ToGoName(name string) string {
lexems := newSplitter(withPostSplitInitialismCheck).split(name)
s := poolOfSplitters.BorrowSplitter(withPostSplitInitialismCheck)
lexems := s.split(name)
poolOfSplitters.RedeemSplitter(s)
defer func() {
poolOfLexems.RedeemLexems(lexems)
}()
lexemes := *lexems
result := ""
for _, lexem := range lexems {
if len(lexemes) == 0 {
return ""
}
result := poolOfBuffers.BorrowBuffer(len(name))
defer func() {
poolOfBuffers.RedeemBuffer(result)
}()
// check if not starting with a letter, upper case
firstPart := lexemes[0].GetUnsafeGoName()
if lexemes[0].IsInitialism() {
firstPart = upper(firstPart)
}
if c := firstPart[0]; c < utf8.RuneSelf {
// ASCII
switch {
case 'A' <= c && c <= 'Z':
result.WriteString(firstPart)
case 'a' <= c && c <= 'z':
result.WriteByte(c - 'a' + 'A')
result.WriteString(firstPart[1:])
default:
result.WriteString(prefixFunc(name, firstPart))
// NOTE: no longer check if prefixFunc returns a string that starts with uppercase:
// assume this is always the case
}
} else {
// unicode
firstRune, _ := utf8.DecodeRuneInString(firstPart)
switch {
case !unicode.IsLetter(firstRune):
result.WriteString(prefixFunc(name, firstPart))
case !unicode.IsUpper(firstRune):
result.WriteString(prefixFunc(name, firstPart))
/*
result.WriteRune(unicode.ToUpper(firstRune))
result.WriteString(firstPart[offset:])
*/
default:
result.WriteString(firstPart)
}
}
for _, lexem := range lexemes[1:] {
goName := lexem.GetUnsafeGoName()
// to support old behavior
if lexem.IsInitialism() {
goName = upper(goName)
}
result += goName
result.WriteString(goName)
}
if len(result) > 0 {
// Only prefix with X when the first character isn't an ascii letter
first := []rune(result)[0]
if !unicode.IsLetter(first) || (first > unicode.MaxASCII && !unicode.IsUpper(first)) {
if GoNamePrefixFunc == nil {
return "X" + result
}
result = GoNamePrefixFunc(name) + result
}
first = []rune(result)[0]
if unicode.IsLetter(first) && !unicode.IsUpper(first) {
result = string(append([]rune{unicode.ToUpper(first)}, []rune(result)[1:]...))
}
}
return result
return result.String()
}
// ContainsStrings searches a slice of strings for a case-sensitive match
@@ -343,7 +323,7 @@ type zeroable interface {
func IsZero(data interface{}) bool {
v := reflect.ValueOf(data)
// check for nil data
switch v.Kind() {
switch v.Kind() { //nolint:exhaustive
case reflect.Interface, reflect.Map, reflect.Ptr, reflect.Slice:
if v.IsNil() {
return true
@@ -356,7 +336,7 @@ func IsZero(data interface{}) bool {
}
// continue with slightly more complex reflection
switch v.Kind() {
switch v.Kind() { //nolint:exhaustive
case reflect.String:
return v.Len() == 0
case reflect.Bool:
@@ -376,16 +356,6 @@ func IsZero(data interface{}) bool {
}
}
// AddInitialisms add additional initialisms
func AddInitialisms(words ...string) {
for _, word := range words {
// commonInitialisms[upper(word)] = true
commonInitialisms.add(upper(word))
}
// sort again
initialisms = commonInitialisms.sorted()
}
// CommandLineOptionsGroup represents a group of user-defined command line options
type CommandLineOptionsGroup struct {
ShortDescription string
vendor/github.com/go-openapi/swag/yaml.go
Generated Vendored
+35 -4
View File
@@ -16,8 +16,11 @@ package swag
import (
"encoding/json"
"errors"
"fmt"
"path/filepath"
"reflect"
"sort"
"strconv"
"github.com/mailru/easyjson/jlexer"
@@ -48,7 +51,7 @@ func BytesToYAMLDoc(data []byte) (interface{}, error) {
return nil, err
}
if document.Kind != yaml.DocumentNode || len(document.Content) != 1 || document.Content[0].Kind != yaml.MappingNode {
return nil, fmt.Errorf("only YAML documents that are objects are supported")
return nil, errors.New("only YAML documents that are objects are supported")
}
return &document, nil
}
@@ -147,7 +150,7 @@ func yamlScalar(node *yaml.Node) (interface{}, error) {
case yamlTimestamp:
return node.Value, nil
case yamlNull:
return nil, nil
return nil, nil //nolint:nilnil
default:
return nil, fmt.Errorf("YAML tag %q is not supported", node.LongTag())
}
@@ -245,7 +248,27 @@ func (s JSONMapSlice) MarshalYAML() (interface{}, error) {
return yaml.Marshal(&n)
}
func isNil(input interface{}) bool {
if input == nil {
return true
}
kind := reflect.TypeOf(input).Kind()
switch kind { //nolint:exhaustive
case reflect.Ptr, reflect.Map, reflect.Slice, reflect.Chan:
return reflect.ValueOf(input).IsNil()
default:
return false
}
}
func json2yaml(item interface{}) (*yaml.Node, error) {
if isNil(item) {
return &yaml.Node{
Kind: yaml.ScalarNode,
Value: "null",
}, nil
}
switch val := item.(type) {
case JSONMapSlice:
var n yaml.Node
@@ -265,7 +288,14 @@ func json2yaml(item interface{}) (*yaml.Node, error) {
case map[string]interface{}:
var n yaml.Node
n.Kind = yaml.MappingNode
for k, v := range val {
keys := make([]string, 0, len(val))
for k := range val {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
v := val[k]
childNode, err := json2yaml(v)
if err != nil {
return nil, err
@@ -318,8 +348,9 @@ func json2yaml(item interface{}) (*yaml.Node, error) {
Tag: yamlBoolScalar,
Value: strconv.FormatBool(val),
}, nil
default:
return nil, fmt.Errorf("unhandled type: %T", val)
}
return nil, nil
}
// JSONMapItem represents the value of a key in a JSON object held by JSONMapSlice
vendor/github.com/grpc-ecosystem/grpc-gateway/v2/runtime/convert.go
Generated Vendored
+3 -3
View File
@@ -94,7 +94,7 @@ func Int64(val string) (int64, error) {
}
// Int64Slice converts 'val' where individual integers are separated by
// 'sep' into a int64 slice.
// 'sep' into an int64 slice.
func Int64Slice(val, sep string) ([]int64, error) {
s := strings.Split(val, sep)
values := make([]int64, len(s))
@@ -118,7 +118,7 @@ func Int32(val string) (int32, error) {
}
// Int32Slice converts 'val' where individual integers are separated by
// 'sep' into a int32 slice.
// 'sep' into an int32 slice.
func Int32Slice(val, sep string) ([]int32, error) {
s := strings.Split(val, sep)
values := make([]int32, len(s))
@@ -190,7 +190,7 @@ func Bytes(val string) ([]byte, error) {
}
// BytesSlice converts 'val' where individual bytes sequences, encoded in URL-safe
// base64 without padding, are separated by 'sep' into a slice of bytes slices slice.
// base64 without padding, are separated by 'sep' into a slice of byte slices.
func BytesSlice(val, sep string) ([][]byte, error) {
s := strings.Split(val, sep)
values := make([][]byte, len(s))
vendor/github.com/grpc-ecosystem/grpc-gateway/v2/runtime/errors.go
Generated Vendored
+15
View File
@@ -81,6 +81,21 @@ func HTTPError(ctx context.Context, mux *ServeMux, marshaler Marshaler, w http.R
mux.errorHandler(ctx, mux, marshaler, w, r, err)
}
// HTTPStreamError uses the mux-configured stream error handler to notify error to the client without closing the connection.
func HTTPStreamError(ctx context.Context, mux *ServeMux, marshaler Marshaler, w http.ResponseWriter, r *http.Request, err error) {
st := mux.streamErrorHandler(ctx, err)
msg := errorChunk(st)
buf, err := marshaler.Marshal(msg)
if err != nil {
grpclog.Errorf("Failed to marshal an error: %v", err)
return
}
if _, err := w.Write(buf); err != nil {
grpclog.Errorf("Failed to notify error to client: %v", err)
return
}
}
// DefaultHTTPErrorHandler is the default error handler.
// If "err" is a gRPC Status, the function replies with the status code mapped by HTTPStatusFromCode.
// If "err" is a HTTPStatusError, the function replies with the status code provide by that struct. This is
vendor/github.com/grpc-ecosystem/grpc-gateway/v2/runtime/fieldmask.go
Generated Vendored
+1 -1
View File
@@ -155,7 +155,7 @@ func buildPathsBlindly(name string, in interface{}) []string {
return paths
}
// fieldMaskPathItem stores a in-progress deconstruction of a path for a fieldmask
// fieldMaskPathItem stores an in-progress deconstruction of a path for a fieldmask
type fieldMaskPathItem struct {
// the list of prior fields leading up to node connected by dots
path string
vendor/github.com/grpc-ecosystem/grpc-gateway/v2/runtime/handler.go
Generated Vendored
+9 -3
View File
@@ -64,7 +64,13 @@ func ForwardResponseStream(ctx context.Context, mux *ServeMux, marshaler Marshal
}
if !wroteHeader {
w.Header().Set("Content-Type", marshaler.ContentType(respRw))
var contentType string
if sct, ok := marshaler.(StreamContentType); ok {
contentType = sct.StreamContentType(respRw)
} else {
contentType = marshaler.ContentType(respRw)
}
w.Header().Set("Content-Type", contentType)
}
var buf []byte
@@ -190,11 +196,11 @@ func ForwardResponseMessage(ctx context.Context, mux *ServeMux, marshaler Marsha
return
}
if !doForwardTrailers {
if !doForwardTrailers && mux.writeContentLength {
w.Header().Set("Content-Length", strconv.Itoa(len(buf)))
}
if _, err = w.Write(buf); err != nil {
if _, err = w.Write(buf); err != nil && !errors.Is(err, http.ErrBodyNotAllowed) {
grpclog.Errorf("Failed to write response: %v", err)
}
vendor/github.com/grpc-ecosystem/grpc-gateway/v2/runtime/marshaler.go
Generated Vendored
+8
View File
@@ -48,3 +48,11 @@ type Delimited interface {
// Delimiter returns the record separator for the stream.
Delimiter() []byte
}
// StreamContentType defines the streaming content type.
type StreamContentType interface {
// StreamContentType returns the content type for a stream. This shares the
// same behaviour as for `Marshaler.ContentType`, but is called, if present,
// in the case of a streamed response.
StreamContentType(v interface{}) string
}
vendor/github.com/grpc-ecosystem/grpc-gateway/v2/runtime/marshaler_registry.go
Generated Vendored
+2 -2
View File
@@ -86,8 +86,8 @@ func (m marshalerRegistry) add(mime string, marshaler Marshaler) error {
// It allows for a mapping of case-sensitive Content-Type MIME type string to runtime.Marshaler interfaces.
//
// For example, you could allow the client to specify the use of the runtime.JSONPb marshaler
// with a "application/jsonpb" Content-Type and the use of the runtime.JSONBuiltin marshaler
// with a "application/json" Content-Type.
// with an "application/jsonpb" Content-Type and the use of the runtime.JSONBuiltin marshaler
// with an "application/json" Content-Type.
// "*" can be used to match any Content-Type.
// This can be attached to a ServerMux with the marshaler option.
func makeMarshalerMIMERegistry() marshalerRegistry {
vendor/github.com/grpc-ecosystem/grpc-gateway/v2/runtime/mux.go
Generated Vendored
+8
View File
@@ -71,6 +71,7 @@ type ServeMux struct {
routingErrorHandler RoutingErrorHandlerFunc
disablePathLengthFallback bool
unescapingMode UnescapingMode
writeContentLength bool
}
// ServeMuxOption is an option that can be given to a ServeMux on construction.
@@ -258,6 +259,13 @@ func WithDisablePathLengthFallback() ServeMuxOption {
}
}
// WithWriteContentLength returns a ServeMuxOption to enable writing content length on non-streaming responses
func WithWriteContentLength() ServeMuxOption {
return func(serveMux *ServeMux) {
serveMux.writeContentLength = true
}
}
// WithHealthEndpointAt returns a ServeMuxOption that will add an endpoint to the created ServeMux at the path specified by endpointPath.
// When called the handler will forward the request to the upstream grpc service health check (defined in the
// gRPC Health Checking Protocol).
vendor/github.com/grpc-ecosystem/grpc-gateway/v2/runtime/proto2_convert.go
Generated Vendored
+2 -2
View File
@@ -40,7 +40,7 @@ func Float32P(val string) (*float32, error) {
}
// Int64P parses the given string representation of an integer
// and returns a pointer to a int64 whose value is same as the parsed integer.
// and returns a pointer to an int64 whose value is same as the parsed integer.
func Int64P(val string) (*int64, error) {
i, err := Int64(val)
if err != nil {
@@ -50,7 +50,7 @@ func Int64P(val string) (*int64, error) {
}
// Int32P parses the given string representation of an integer
// and returns a pointer to a int32 whose value is same as the parsed integer.
// and returns a pointer to an int32 whose value is same as the parsed integer.
func Int32P(val string) (*int32, error) {
i, err := Int32(val)
if err != nil {
vendor/github.com/grpc-ecosystem/grpc-gateway/v2/runtime/query.go
Generated Vendored
+14 -8
View File
@@ -126,6 +126,15 @@ func populateFieldValueFromPath(msgValue protoreflect.Message, fieldPath []strin
}
}
// Check if oneof already set
if of := fieldDescriptor.ContainingOneof(); of != nil && !of.IsSynthetic() {
if f := msgValue.WhichOneof(of); f != nil {
if fieldDescriptor.Message() == nil || fieldDescriptor.FullName() != f.FullName() {
return fmt.Errorf("field already set for oneof %q", of.FullName().Name())
}
}
}
// If this is the last element, we're done
if i == len(fieldPath)-1 {
break
@@ -140,13 +149,6 @@ func populateFieldValueFromPath(msgValue protoreflect.Message, fieldPath []strin
msgValue = msgValue.Mutable(fieldDescriptor).Message()
}
// Check if oneof already set
if of := fieldDescriptor.ContainingOneof(); of != nil {
if f := msgValue.WhichOneof(of); f != nil {
return fmt.Errorf("field already set for oneof %q", of.FullName().Name())
}
}
switch {
case fieldDescriptor.IsList():
return populateRepeatedField(fieldDescriptor, msgValue.Mutable(fieldDescriptor).List(), values)
@@ -291,7 +293,11 @@ func parseMessage(msgDescriptor protoreflect.MessageDescriptor, value string) (p
if err != nil {
return protoreflect.Value{}, err
}
msg = timestamppb.New(t)
timestamp := timestamppb.New(t)
if ok := timestamp.IsValid(); !ok {
return protoreflect.Value{}, fmt.Errorf("%s before 0001-01-01", value)
}
msg = timestamp
case "google.protobuf.Duration":
d, err := time.ParseDuration(value)
if err != nil {
vendor/github.com/grpc-ecosystem/grpc-gateway/v2/utilities/pattern.go
Generated Vendored
+1 -1
View File
@@ -1,6 +1,6 @@
package utilities
// An OpCode is a opcode of compiled path patterns.
// OpCode is an opcode of compiled path patterns.
type OpCode int
// These constants are the valid values of OpCode.
vendor/github.com/grpc-ecosystem/grpc-gateway/v2/utilities/string_array_flag.go
Generated Vendored
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"strings"
)
// flagInterface is an cut down interface to `flag`
// flagInterface is a cut down interface to `flag`
type flagInterface interface {
Var(value flag.Value, name string, usage string)
}
vendor/github.com/imdario/mergo/.deepsource.toml
Generated Vendored
-12
View File
@@ -1,12 +0,0 @@
version = 1
test_patterns = [
"*_test.go"
]
[[analyzers]]
name = "go"
enabled = true
[analyzers.meta]
import_path = "github.com/imdario/mergo"
vendor/github.com/imdario/mergo/.gitignore
Generated
-33
View File
@@ -1,33 +0,0 @@
#### joe made this: http://goel.io/joe
#### go ####
# Binaries for programs and plugins
*.exe
*.dll
*.so
*.dylib
# Test binary, build with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
# Project-local glide cache, RE: https://github.com/Masterminds/glide/issues/736
.glide/
#### vim ####
# Swap
[._]*.s[a-v][a-z]
[._]*.sw[a-p]
[._]s[a-v][a-z]
[._]sw[a-p]
# Session
Session.vim
# Temporary
.netrwhist
*~
# Auto-generated tag files
tags
vendor/github.com/imdario/mergo/.travis.yml
Generated Vendored
-12
View File
@@ -1,12 +0,0 @@
language: go
arch:
- amd64
- ppc64le
install:
- go get -t
- go get golang.org/x/tools/cmd/cover
- go get github.com/mattn/goveralls
script:
- go test -race -v ./...
after_script:
- $HOME/gopath/bin/goveralls -service=travis-ci -repotoken $COVERALLS_TOKEN
vendor/github.com/imdario/mergo/CODE_OF_CONDUCT.md
Generated Vendored
-46
View File
@@ -1,46 +0,0 @@
# Contributor Covenant Code of Conduct
## Our Pledge
In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
## Scope
This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at i@dario.im. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
[homepage]: http://contributor-covenant.org
[version]: http://contributor-covenant.org/version/1/4/
vendor/github.com/imdario/mergo/CONTRIBUTING.md
Generated Vendored
-112
View File
@@ -1,112 +0,0 @@
<!-- omit in toc -->
# Contributing to mergo
First off, thanks for taking the time to contribute! ❤️
All types of contributions are encouraged and valued. See the [Table of Contents](#table-of-contents) for different ways to help and details about how this project handles them. Please make sure to read the relevant section before making your contribution. It will make it a lot easier for us maintainers and smooth out the experience for all involved. The community looks forward to your contributions. 🎉
> And if you like the project, but just don't have time to contribute, that's fine. There are other easy ways to support the project and show your appreciation, which we would also be very happy about:
> - Star the project
> - Tweet about it
> - Refer this project in your project's readme
> - Mention the project at local meetups and tell your friends/colleagues
<!-- omit in toc -->
## Table of Contents
- [Code of Conduct](#code-of-conduct)
- [I Have a Question](#i-have-a-question)
- [I Want To Contribute](#i-want-to-contribute)
- [Reporting Bugs](#reporting-bugs)
- [Suggesting Enhancements](#suggesting-enhancements)
## Code of Conduct
This project and everyone participating in it is governed by the
[mergo Code of Conduct](https://github.com/imdario/mergoblob/master/CODE_OF_CONDUCT.md).
By participating, you are expected to uphold this code. Please report unacceptable behavior
to <>.
## I Have a Question
> If you want to ask a question, we assume that you have read the available [Documentation](https://pkg.go.dev/github.com/imdario/mergo).
Before you ask a question, it is best to search for existing [Issues](https://github.com/imdario/mergo/issues) that might help you. In case you have found a suitable issue and still need clarification, you can write your question in this issue. It is also advisable to search the internet for answers first.
If you then still feel the need to ask a question and need clarification, we recommend the following:
- Open an [Issue](https://github.com/imdario/mergo/issues/new).
- Provide as much context as you can about what you're running into.
- Provide project and platform versions (nodejs, npm, etc), depending on what seems relevant.
We will then take care of the issue as soon as possible.
## I Want To Contribute
> ### Legal Notice <!-- omit in toc -->
> When contributing to this project, you must agree that you have authored 100% of the content, that you have the necessary rights to the content and that the content you contribute may be provided under the project license.
### Reporting Bugs
<!-- omit in toc -->
#### Before Submitting a Bug Report
A good bug report shouldn't leave others needing to chase you up for more information. Therefore, we ask you to investigate carefully, collect information and describe the issue in detail in your report. Please complete the following steps in advance to help us fix any potential bug as fast as possible.
- Make sure that you are using the latest version.
- Determine if your bug is really a bug and not an error on your side e.g. using incompatible environment components/versions (Make sure that you have read the [documentation](). If you are looking for support, you might want to check [this section](#i-have-a-question)).
- To see if other users have experienced (and potentially already solved) the same issue you are having, check if there is not already a bug report existing for your bug or error in the [bug tracker](https://github.com/imdario/mergoissues?q=label%3Abug).
- Also make sure to search the internet (including Stack Overflow) to see if users outside of the GitHub community have discussed the issue.
- Collect information about the bug:
- Stack trace (Traceback)
- OS, Platform and Version (Windows, Linux, macOS, x86, ARM)
- Version of the interpreter, compiler, SDK, runtime environment, package manager, depending on what seems relevant.
- Possibly your input and the output
- Can you reliably reproduce the issue? And can you also reproduce it with older versions?
<!-- omit in toc -->
#### How Do I Submit a Good Bug Report?
> You must never report security related issues, vulnerabilities or bugs including sensitive information to the issue tracker, or elsewhere in public. Instead sensitive bugs must be sent by email to .
<!-- You may add a PGP key to allow the messages to be sent encrypted as well. -->
We use GitHub issues to track bugs and errors. If you run into an issue with the project:
- Open an [Issue](https://github.com/imdario/mergo/issues/new). (Since we can't be sure at this point whether it is a bug or not, we ask you not to talk about a bug yet and not to label the issue.)
- Explain the behavior you would expect and the actual behavior.
- Please provide as much context as possible and describe the *reproduction steps* that someone else can follow to recreate the issue on their own. This usually includes your code. For good bug reports you should isolate the problem and create a reduced test case.
- Provide the information you collected in the previous section.
Once it's filed:
- The project team will label the issue accordingly.
- A team member will try to reproduce the issue with your provided steps. If there are no reproduction steps or no obvious way to reproduce the issue, the team will ask you for those steps and mark the issue as `needs-repro`. Bugs with the `needs-repro` tag will not be addressed until they are reproduced.
- If the team is able to reproduce the issue, it will be marked `needs-fix`, as well as possibly other tags (such as `critical`), and the issue will be left to be implemented by someone.
### Suggesting Enhancements
This section guides you through submitting an enhancement suggestion for mergo, **including completely new features and minor improvements to existing functionality**. Following these guidelines will help maintainers and the community to understand your suggestion and find related suggestions.
<!-- omit in toc -->
#### Before Submitting an Enhancement
- Make sure that you are using the latest version.
- Read the [documentation]() carefully and find out if the functionality is already covered, maybe by an individual configuration.
- Perform a [search](https://github.com/imdario/mergo/issues) to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one.
- Find out whether your idea fits with the scope and aims of the project. It's up to you to make a strong case to convince the project's developers of the merits of this feature. Keep in mind that we want features that will be useful to the majority of our users and not just a small subset. If you're just targeting a minority of users, consider writing an add-on/plugin library.
<!-- omit in toc -->
#### How Do I Submit a Good Enhancement Suggestion?
Enhancement suggestions are tracked as [GitHub issues](https://github.com/imdario/mergo/issues).
- Use a **clear and descriptive title** for the issue to identify the suggestion.
- Provide a **step-by-step description of the suggested enhancement** in as many details as possible.
- **Describe the current behavior** and **explain which behavior you expected to see instead** and why. At this point you can also tell which alternatives do not work for you.
- You may want to **include screenshots and animated GIFs** which help you demonstrate the steps or point out the part which the suggestion is related to. You can use [this tool](https://www.cockos.com/licecap/) to record GIFs on macOS and Windows, and [this tool](https://github.com/colinkeenan/silentcast) or [this tool](https://github.com/GNOME/byzanz) on Linux. <!-- this should only be included if the project has a GUI -->
- **Explain why this enhancement would be useful** to most mergo users. You may also want to point out the other projects that solved it better and which could serve as inspiration.
<!-- omit in toc -->
## Attribution
This guide is based on the **contributing-gen**. [Make your own](https://github.com/bttger/contributing-gen)!
vendor/github.com/imdario/mergo/LICENSE
Generated Vendored
-28
View File
@@ -1,28 +0,0 @@
Copyright (c) 2013 Dario Castañé. All rights reserved.
Copyright (c) 2012 The Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
vendor/github.com/imdario/mergo/README.md
Generated Vendored
-242
View File
@@ -1,242 +0,0 @@
# Mergo
[![GitHub release][5]][6]
[![GoCard][7]][8]
[![Test status][1]][2]
[![OpenSSF Scorecard][21]][22]
[![OpenSSF Best Practices][19]][20]
[![Coverage status][9]][10]
[![Sourcegraph][11]][12]
[![FOSSA status][13]][14]
[![GoDoc][3]][4]
[![Become my sponsor][15]][16]
[![Tidelift][17]][18]
[1]: https://github.com/imdario/mergo/workflows/tests/badge.svg?branch=master
[2]: https://github.com/imdario/mergo/actions/workflows/tests.yml
[3]: https://godoc.org/github.com/imdario/mergo?status.svg
[4]: https://godoc.org/github.com/imdario/mergo
[5]: https://img.shields.io/github/release/imdario/mergo.svg
[6]: https://github.com/imdario/mergo/releases
[7]: https://goreportcard.com/badge/imdario/mergo
[8]: https://goreportcard.com/report/github.com/imdario/mergo
[9]: https://coveralls.io/repos/github/imdario/mergo/badge.svg?branch=master
[10]: https://coveralls.io/github/imdario/mergo?branch=master
[11]: https://sourcegraph.com/github.com/imdario/mergo/-/badge.svg
[12]: https://sourcegraph.com/github.com/imdario/mergo?badge
[13]: https://app.fossa.io/api/projects/git%2Bgithub.com%2Fimdario%2Fmergo.svg?type=shield
[14]: https://app.fossa.io/projects/git%2Bgithub.com%2Fimdario%2Fmergo?ref=badge_shield
[15]: https://img.shields.io/github/sponsors/imdario
[16]: https://github.com/sponsors/imdario
[17]: https://tidelift.com/badges/package/go/github.com%2Fimdario%2Fmergo
[18]: https://tidelift.com/subscription/pkg/go-github.com-imdario-mergo
[19]: https://bestpractices.coreinfrastructure.org/projects/7177/badge
[20]: https://bestpractices.coreinfrastructure.org/projects/7177
[21]: https://api.securityscorecards.dev/projects/github.com/imdario/mergo/badge
[22]: https://api.securityscorecards.dev/projects/github.com/imdario/mergo
A helper to merge structs and maps in Golang. Useful for configuration default values, avoiding messy if-statements.
Mergo merges same-type structs and maps by setting default values in zero-value fields. Mergo won't merge unexported (private) fields. It will do recursively any exported one. It also won't merge structs inside maps (because they are not addressable using Go reflection).
Also a lovely [comune](http://en.wikipedia.org/wiki/Mergo) (municipality) in the Province of Ancona in the Italian region of Marche.
## Status
It is ready for production use. [It is used in several projects by Docker, Google, The Linux Foundation, VMWare, Shopify, Microsoft, etc](https://github.com/imdario/mergo#mergo-in-the-wild).
### Important note
Please keep in mind that a problematic PR broke [0.3.9](//github.com/imdario/mergo/releases/tag/0.3.9). I reverted it in [0.3.10](//github.com/imdario/mergo/releases/tag/0.3.10), and I consider it stable but not bug-free. Also, this version adds support for go modules.
Keep in mind that in [0.3.2](//github.com/imdario/mergo/releases/tag/0.3.2), Mergo changed `Merge()`and `Map()` signatures to support [transformers](#transformers). I added an optional/variadic argument so that it won't break the existing code.
If you were using Mergo before April 6th, 2015, please check your project works as intended after updating your local copy with ```go get -u github.com/imdario/mergo```. I apologize for any issue caused by its previous behavior and any future bug that Mergo could cause in existing projects after the change (release 0.2.0).
### Donations
If Mergo is useful to you, consider buying me a coffee, a beer, or making a monthly donation to allow me to keep building great free software. :heart_eyes:
<a href='https://ko-fi.com/B0B58839' target='_blank'><img height='36' style='border:0px;height:36px;' src='https://az743702.vo.msecnd.net/cdn/kofi1.png?v=0' border='0' alt='Buy Me a Coffee at ko-fi.com' /></a>
<a href="https://liberapay.com/dario/donate"><img alt="Donate using Liberapay" src="https://liberapay.com/assets/widgets/donate.svg"></a>
<a href='https://github.com/sponsors/imdario' target='_blank'><img alt="Become my sponsor" src="https://img.shields.io/github/sponsors/imdario?style=for-the-badge" /></a>
### Mergo in the wild
- [moby/moby](https://github.com/moby/moby)
- [kubernetes/kubernetes](https://github.com/kubernetes/kubernetes)
- [vmware/dispatch](https://github.com/vmware/dispatch)
- [Shopify/themekit](https://github.com/Shopify/themekit)
- [imdario/zas](https://github.com/imdario/zas)
- [matcornic/hermes](https://github.com/matcornic/hermes)
- [OpenBazaar/openbazaar-go](https://github.com/OpenBazaar/openbazaar-go)
- [kataras/iris](https://github.com/kataras/iris)
- [michaelsauter/crane](https://github.com/michaelsauter/crane)
- [go-task/task](https://github.com/go-task/task)
- [sensu/uchiwa](https://github.com/sensu/uchiwa)
- [ory/hydra](https://github.com/ory/hydra)
- [sisatech/vcli](https://github.com/sisatech/vcli)
- [dairycart/dairycart](https://github.com/dairycart/dairycart)
- [projectcalico/felix](https://github.com/projectcalico/felix)
- [resin-os/balena](https://github.com/resin-os/balena)
- [go-kivik/kivik](https://github.com/go-kivik/kivik)
- [Telefonica/govice](https://github.com/Telefonica/govice)
- [supergiant/supergiant](supergiant/supergiant)
- [SergeyTsalkov/brooce](https://github.com/SergeyTsalkov/brooce)
- [soniah/dnsmadeeasy](https://github.com/soniah/dnsmadeeasy)
- [ohsu-comp-bio/funnel](https://github.com/ohsu-comp-bio/funnel)
- [EagerIO/Stout](https://github.com/EagerIO/Stout)
- [lynndylanhurley/defsynth-api](https://github.com/lynndylanhurley/defsynth-api)
- [russross/canvasassignments](https://github.com/russross/canvasassignments)
- [rdegges/cryptly-api](https://github.com/rdegges/cryptly-api)
- [casualjim/exeggutor](https://github.com/casualjim/exeggutor)
- [divshot/gitling](https://github.com/divshot/gitling)
- [RWJMurphy/gorl](https://github.com/RWJMurphy/gorl)
- [andrerocker/deploy42](https://github.com/andrerocker/deploy42)
- [elwinar/rambler](https://github.com/elwinar/rambler)
- [tmaiaroto/gopartman](https://github.com/tmaiaroto/gopartman)
- [jfbus/impressionist](https://github.com/jfbus/impressionist)
- [Jmeyering/zealot](https://github.com/Jmeyering/zealot)
- [godep-migrator/rigger-host](https://github.com/godep-migrator/rigger-host)
- [Dronevery/MultiwaySwitch-Go](https://github.com/Dronevery/MultiwaySwitch-Go)
- [thoas/picfit](https://github.com/thoas/picfit)
- [mantasmatelis/whooplist-server](https://github.com/mantasmatelis/whooplist-server)
- [jnuthong/item_search](https://github.com/jnuthong/item_search)
- [bukalapak/snowboard](https://github.com/bukalapak/snowboard)
- [containerssh/containerssh](https://github.com/containerssh/containerssh)
- [goreleaser/goreleaser](https://github.com/goreleaser/goreleaser)
- [tjpnz/structbot](https://github.com/tjpnz/structbot)
## Install
go get github.com/imdario/mergo
// use in your .go code
import (
"github.com/imdario/mergo"
)
## Usage
You can only merge same-type structs with exported fields initialized as zero value of their type and same-types maps. Mergo won't merge unexported (private) fields but will do recursively any exported one. It won't merge empty structs value as [they are zero values](https://golang.org/ref/spec#The_zero_value) too. Also, maps will be merged recursively except for structs inside maps (because they are not addressable using Go reflection).
```go
if err := mergo.Merge(&dst, src); err != nil {
// ...
}
```
Also, you can merge overwriting values using the transformer `WithOverride`.
```go
if err := mergo.Merge(&dst, src, mergo.WithOverride); err != nil {
// ...
}
```
Additionally, you can map a `map[string]interface{}` to a struct (and otherwise, from struct to map), following the same restrictions as in `Merge()`. Keys are capitalized to find each corresponding exported field.
```go
if err := mergo.Map(&dst, srcMap); err != nil {
// ...
}
```
Warning: if you map a struct to map, it won't do it recursively. Don't expect Mergo to map struct members of your struct as `map[string]interface{}`. They will be just assigned as values.
Here is a nice example:
```go
package main
import (
"fmt"
"github.com/imdario/mergo"
)
type Foo struct {
A string
B int64
}
func main() {
src := Foo{
A: "one",
B: 2,
}
dest := Foo{
A: "two",
}
mergo.Merge(&dest, src)
fmt.Println(dest)
// Will print
// {two 2}
}
```
Note: if test are failing due missing package, please execute:
go get gopkg.in/yaml.v3
### Transformers
Transformers allow to merge specific types differently than in the default behavior. In other words, now you can customize how some types are merged. For example, `time.Time` is a struct; it doesn't have zero value but IsZero can return true because it has fields with zero value. How can we merge a non-zero `time.Time`?
```go
package main
import (
"fmt"
"github.com/imdario/mergo"
"reflect"
"time"
)
type timeTransformer struct {
}
func (t timeTransformer) Transformer(typ reflect.Type) func(dst, src reflect.Value) error {
if typ == reflect.TypeOf(time.Time{}) {
return func(dst, src reflect.Value) error {
if dst.CanSet() {
isZero := dst.MethodByName("IsZero")
result := isZero.Call([]reflect.Value{})
if result[0].Bool() {
dst.Set(src)
}
}
return nil
}
}
return nil
}
type Snapshot struct {
Time time.Time
// ...
}
func main() {
src := Snapshot{time.Now()}
dest := Snapshot{}
mergo.Merge(&dest, src, mergo.WithTransformers(timeTransformer{}))
fmt.Println(dest)
// Will print
// { 2018-01-12 01:15:00 +0000 UTC m=+0.000000001 }
}
```
## Contact me
If I can help you, you have an idea or you are using Mergo in your projects, don't hesitate to drop me a line (or a pull request): [@im_dario](https://twitter.com/im_dario)
## About
Written by [Dario Castañé](http://dario.im).
## License
[BSD 3-Clause](http://opensource.org/licenses/BSD-3-Clause) license, as [Go language](http://golang.org/LICENSE).
[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fimdario%2Fmergo.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fimdario%2Fmergo?ref=badge_large)
vendor/github.com/imdario/mergo/SECURITY.md
Generated Vendored
-14
View File
@@ -1,14 +0,0 @@
# Security Policy
## Supported Versions
| Version | Supported |
| ------- | ------------------ |
| 0.3.x | :white_check_mark: |
| < 0.3 | :x: |
## Security contact information
To report a security vulnerability, please use the
[Tidelift security contact](https://tidelift.com/security).
Tidelift will coordinate the fix and disclosure.
vendor/github.com/imdario/mergo/doc.go
Generated Vendored
-143
View File
@@ -1,143 +0,0 @@
// Copyright 2013 Dario Castañé. All rights reserved.
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
/*
A helper to merge structs and maps in Golang. Useful for configuration default values, avoiding messy if-statements.
Mergo merges same-type structs and maps by setting default values in zero-value fields. Mergo won't merge unexported (private) fields. It will do recursively any exported one. It also won't merge structs inside maps (because they are not addressable using Go reflection).
Status
It is ready for production use. It is used in several projects by Docker, Google, The Linux Foundation, VMWare, Shopify, etc.
Important note
Please keep in mind that a problematic PR broke 0.3.9. We reverted it in 0.3.10. We consider 0.3.10 as stable but not bug-free. . Also, this version adds suppot for go modules.
Keep in mind that in 0.3.2, Mergo changed Merge() and Map() signatures to support transformers. We added an optional/variadic argument so that it won't break the existing code.
If you were using Mergo before April 6th, 2015, please check your project works as intended after updating your local copy with go get -u github.com/imdario/mergo. I apologize for any issue caused by its previous behavior and any future bug that Mergo could cause in existing projects after the change (release 0.2.0).
Install
Do your usual installation procedure:
go get github.com/imdario/mergo
// use in your .go code
import (
"github.com/imdario/mergo"
)
Usage
You can only merge same-type structs with exported fields initialized as zero value of their type and same-types maps. Mergo won't merge unexported (private) fields but will do recursively any exported one. It won't merge empty structs value as they are zero values too. Also, maps will be merged recursively except for structs inside maps (because they are not addressable using Go reflection).
if err := mergo.Merge(&dst, src); err != nil {
// ...
}
Also, you can merge overwriting values using the transformer WithOverride.
if err := mergo.Merge(&dst, src, mergo.WithOverride); err != nil {
// ...
}
Additionally, you can map a map[string]interface{} to a struct (and otherwise, from struct to map), following the same restrictions as in Merge(). Keys are capitalized to find each corresponding exported field.
if err := mergo.Map(&dst, srcMap); err != nil {
// ...
}
Warning: if you map a struct to map, it won't do it recursively. Don't expect Mergo to map struct members of your struct as map[string]interface{}. They will be just assigned as values.
Here is a nice example:
package main
import (
"fmt"
"github.com/imdario/mergo"
)
type Foo struct {
A string
B int64
}
func main() {
src := Foo{
A: "one",
B: 2,
}
dest := Foo{
A: "two",
}
mergo.Merge(&dest, src)
fmt.Println(dest)
// Will print
// {two 2}
}
Transformers
Transformers allow to merge specific types differently than in the default behavior. In other words, now you can customize how some types are merged. For example, time.Time is a struct; it doesn't have zero value but IsZero can return true because it has fields with zero value. How can we merge a non-zero time.Time?
package main
import (
"fmt"
"github.com/imdario/mergo"
"reflect"
"time"
)
type timeTransformer struct {
}
func (t timeTransformer) Transformer(typ reflect.Type) func(dst, src reflect.Value) error {
if typ == reflect.TypeOf(time.Time{}) {
return func(dst, src reflect.Value) error {
if dst.CanSet() {
isZero := dst.MethodByName("IsZero")
result := isZero.Call([]reflect.Value{})
if result[0].Bool() {
dst.Set(src)
}
}
return nil
}
}
return nil
}
type Snapshot struct {
Time time.Time
// ...
}
func main() {
src := Snapshot{time.Now()}
dest := Snapshot{}
mergo.Merge(&dest, src, mergo.WithTransformers(timeTransformer{}))
fmt.Println(dest)
// Will print
// { 2018-01-12 01:15:00 +0000 UTC m=+0.000000001 }
}
Contact me
If I can help you, you have an idea or you are using Mergo in your projects, don't hesitate to drop me a line (or a pull request): https://twitter.com/im_dario
About
Written by Dario Castañé: https://da.rio.hn
License
BSD 3-Clause license, as Go language.
*/
package mergo
vendor/github.com/imdario/mergo/map.go
Generated Vendored
-178
View File
@@ -1,178 +0,0 @@
// Copyright 2014 Dario Castañé. All rights reserved.
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Based on src/pkg/reflect/deepequal.go from official
// golang's stdlib.
package mergo
import (
"fmt"
"reflect"
"unicode"
"unicode/utf8"
)
func changeInitialCase(s string, mapper func(rune) rune) string {
if s == "" {
return s
}
r, n := utf8.DecodeRuneInString(s)
return string(mapper(r)) + s[n:]
}
func isExported(field reflect.StructField) bool {
r, _ := utf8.DecodeRuneInString(field.Name)
return r >= 'A' && r <= 'Z'
}
// Traverses recursively both values, assigning src's fields values to dst.
// The map argument tracks comparisons that have already been seen, which allows
// short circuiting on recursive types.
func deepMap(dst, src reflect.Value, visited map[uintptr]*visit, depth int, config *Config) (err error) {
overwrite := config.Overwrite
if dst.CanAddr() {
addr := dst.UnsafeAddr()
h := 17 * addr
seen := visited[h]
typ := dst.Type()
for p := seen; p != nil; p = p.next {
if p.ptr == addr && p.typ == typ {
return nil
}
}
// Remember, remember...
visited[h] = &visit{typ, seen, addr}
}
zeroValue := reflect.Value{}
switch dst.Kind() {
case reflect.Map:
dstMap := dst.Interface().(map[string]interface{})
for i, n := 0, src.NumField(); i < n; i++ {
srcType := src.Type()
field := srcType.Field(i)
if !isExported(field) {
continue
}
fieldName := field.Name
fieldName = changeInitialCase(fieldName, unicode.ToLower)
if v, ok := dstMap[fieldName]; !ok || (isEmptyValue(reflect.ValueOf(v), !config.ShouldNotDereference) || overwrite) {
dstMap[fieldName] = src.Field(i).Interface()
}
}
case reflect.Ptr:
if dst.IsNil() {
v := reflect.New(dst.Type().Elem())
dst.Set(v)
}
dst = dst.Elem()
fallthrough
case reflect.Struct:
srcMap := src.Interface().(map[string]interface{})
for key := range srcMap {
config.overwriteWithEmptyValue = true
srcValue := srcMap[key]
fieldName := changeInitialCase(key, unicode.ToUpper)
dstElement := dst.FieldByName(fieldName)
if dstElement == zeroValue {
// We discard it because the field doesn't exist.
continue
}
srcElement := reflect.ValueOf(srcValue)
dstKind := dstElement.Kind()
srcKind := srcElement.Kind()
if srcKind == reflect.Ptr && dstKind != reflect.Ptr {
srcElement = srcElement.Elem()
srcKind = reflect.TypeOf(srcElement.Interface()).Kind()
} else if dstKind == reflect.Ptr {
// Can this work? I guess it can't.
if srcKind != reflect.Ptr && srcElement.CanAddr() {
srcPtr := srcElement.Addr()
srcElement = reflect.ValueOf(srcPtr)
srcKind = reflect.Ptr
}
}
if !srcElement.IsValid() {
continue
}
if srcKind == dstKind {
if err = deepMerge(dstElement, srcElement, visited, depth+1, config); err != nil {
return
}
} else if dstKind == reflect.Interface && dstElement.Kind() == reflect.Interface {
if err = deepMerge(dstElement, srcElement, visited, depth+1, config); err != nil {
return
}
} else if srcKind == reflect.Map {
if err = deepMap(dstElement, srcElement, visited, depth+1, config); err != nil {
return
}
} else {
return fmt.Errorf("type mismatch on %s field: found %v, expected %v", fieldName, srcKind, dstKind)
}
}
}
return
}
// Map sets fields' values in dst from src.
// src can be a map with string keys or a struct. dst must be the opposite:
// if src is a map, dst must be a valid pointer to struct. If src is a struct,
// dst must be map[string]interface{}.
// It won't merge unexported (private) fields and will do recursively
// any exported field.
// If dst is a map, keys will be src fields' names in lower camel case.
// Missing key in src that doesn't match a field in dst will be skipped. This
// doesn't apply if dst is a map.
// This is separated method from Merge because it is cleaner and it keeps sane
// semantics: merging equal types, mapping different (restricted) types.
func Map(dst, src interface{}, opts ...func(*Config)) error {
return _map(dst, src, opts...)
}
// MapWithOverwrite will do the same as Map except that non-empty dst attributes will be overridden by
// non-empty src attribute values.
// Deprecated: Use Map(…) with WithOverride
func MapWithOverwrite(dst, src interface{}, opts ...func(*Config)) error {
return _map(dst, src, append(opts, WithOverride)...)
}
func _map(dst, src interface{}, opts ...func(*Config)) error {
if dst != nil && reflect.ValueOf(dst).Kind() != reflect.Ptr {
return ErrNonPointerArgument
}
var (
vDst, vSrc reflect.Value
err error
)
config := &Config{}
for _, opt := range opts {
opt(config)
}
if vDst, vSrc, err = resolveValues(dst, src); err != nil {
return err
}
// To be friction-less, we redirect equal-type arguments
// to deepMerge. Only because arguments can be anything.
if vSrc.Kind() == vDst.Kind() {
return deepMerge(vDst, vSrc, make(map[uintptr]*visit), 0, config)
}
switch vSrc.Kind() {
case reflect.Struct:
if vDst.Kind() != reflect.Map {
return ErrExpectedMapAsDestination
}
case reflect.Map:
if vDst.Kind() != reflect.Struct {
return ErrExpectedStructAsDestination
}
default:
return ErrNotSupported
}
return deepMap(vDst, vSrc, make(map[uintptr]*visit), 0, config)
}
vendor/github.com/imdario/mergo/merge.go
Generated Vendored
-409
View File
@@ -1,409 +0,0 @@
// Copyright 2013 Dario Castañé. All rights reserved.
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Based on src/pkg/reflect/deepequal.go from official
// golang's stdlib.
package mergo
import (
"fmt"
"reflect"
)
func hasMergeableFields(dst reflect.Value) (exported bool) {
for i, n := 0, dst.NumField(); i < n; i++ {
field := dst.Type().Field(i)
if field.Anonymous && dst.Field(i).Kind() == reflect.Struct {
exported = exported || hasMergeableFields(dst.Field(i))
} else if isExportedComponent(&field) {
exported = exported || len(field.PkgPath) == 0
}
}
return
}
func isExportedComponent(field *reflect.StructField) bool {
pkgPath := field.PkgPath
if len(pkgPath) > 0 {
return false
}
c := field.Name[0]
if 'a' <= c && c <= 'z' || c == '_' {
return false
}
return true
}
type Config struct {
Transformers Transformers
Overwrite bool
ShouldNotDereference bool
AppendSlice bool
TypeCheck bool
overwriteWithEmptyValue bool
overwriteSliceWithEmptyValue bool
sliceDeepCopy bool
debug bool
}
type Transformers interface {
Transformer(reflect.Type) func(dst, src reflect.Value) error
}
// Traverses recursively both values, assigning src's fields values to dst.
// The map argument tracks comparisons that have already been seen, which allows
// short circuiting on recursive types.
func deepMerge(dst, src reflect.Value, visited map[uintptr]*visit, depth int, config *Config) (err error) {
overwrite := config.Overwrite
typeCheck := config.TypeCheck
overwriteWithEmptySrc := config.overwriteWithEmptyValue
overwriteSliceWithEmptySrc := config.overwriteSliceWithEmptyValue
sliceDeepCopy := config.sliceDeepCopy
if !src.IsValid() {
return
}
if dst.CanAddr() {
addr := dst.UnsafeAddr()
h := 17 * addr
seen := visited[h]
typ := dst.Type()
for p := seen; p != nil; p = p.next {
if p.ptr == addr && p.typ == typ {
return nil
}
}
// Remember, remember...
visited[h] = &visit{typ, seen, addr}
}
if config.Transformers != nil && !isReflectNil(dst) && dst.IsValid() {
if fn := config.Transformers.Transformer(dst.Type()); fn != nil {
err = fn(dst, src)
return
}
}
switch dst.Kind() {
case reflect.Struct:
if hasMergeableFields(dst) {
for i, n := 0, dst.NumField(); i < n; i++ {
if err = deepMerge(dst.Field(i), src.Field(i), visited, depth+1, config); err != nil {
return
}
}
} else {
if dst.CanSet() && (isReflectNil(dst) || overwrite) && (!isEmptyValue(src, !config.ShouldNotDereference) || overwriteWithEmptySrc) {
dst.Set(src)
}
}
case reflect.Map:
if dst.IsNil() && !src.IsNil() {
if dst.CanSet() {
dst.Set(reflect.MakeMap(dst.Type()))
} else {
dst = src
return
}
}
if src.Kind() != reflect.Map {
if overwrite && dst.CanSet() {
dst.Set(src)
}
return
}
for _, key := range src.MapKeys() {
srcElement := src.MapIndex(key)
if !srcElement.IsValid() {
continue
}
dstElement := dst.MapIndex(key)
switch srcElement.Kind() {
case reflect.Chan, reflect.Func, reflect.Map, reflect.Interface, reflect.Slice:
if srcElement.IsNil() {
if overwrite {
dst.SetMapIndex(key, srcElement)
}
continue
}
fallthrough
default:
if !srcElement.CanInterface() {
continue
}
switch reflect.TypeOf(srcElement.Interface()).Kind() {
case reflect.Struct:
fallthrough
case reflect.Ptr:
fallthrough
case reflect.Map:
srcMapElm := srcElement
dstMapElm := dstElement
if srcMapElm.CanInterface() {
srcMapElm = reflect.ValueOf(srcMapElm.Interface())
if dstMapElm.IsValid() {
dstMapElm = reflect.ValueOf(dstMapElm.Interface())
}
}
if err = deepMerge(dstMapElm, srcMapElm, visited, depth+1, config); err != nil {
return
}
case reflect.Slice:
srcSlice := reflect.ValueOf(srcElement.Interface())
var dstSlice reflect.Value
if !dstElement.IsValid() || dstElement.IsNil() {
dstSlice = reflect.MakeSlice(srcSlice.Type(), 0, srcSlice.Len())
} else {
dstSlice = reflect.ValueOf(dstElement.Interface())
}
if (!isEmptyValue(src, !config.ShouldNotDereference) || overwriteWithEmptySrc || overwriteSliceWithEmptySrc) && (overwrite || isEmptyValue(dst, !config.ShouldNotDereference)) && !config.AppendSlice && !sliceDeepCopy {
if typeCheck && srcSlice.Type() != dstSlice.Type() {
return fmt.Errorf("cannot override two slices with different type (%s, %s)", srcSlice.Type(), dstSlice.Type())
}
dstSlice = srcSlice
} else if config.AppendSlice {
if srcSlice.Type() != dstSlice.Type() {
return fmt.Errorf("cannot append two slices with different type (%s, %s)", srcSlice.Type(), dstSlice.Type())
}
dstSlice = reflect.AppendSlice(dstSlice, srcSlice)
} else if sliceDeepCopy {
i := 0
for ; i < srcSlice.Len() && i < dstSlice.Len(); i++ {
srcElement := srcSlice.Index(i)
dstElement := dstSlice.Index(i)
if srcElement.CanInterface() {
srcElement = reflect.ValueOf(srcElement.Interface())
}
if dstElement.CanInterface() {
dstElement = reflect.ValueOf(dstElement.Interface())
}
if err = deepMerge(dstElement, srcElement, visited, depth+1, config); err != nil {
return
}
}
}
dst.SetMapIndex(key, dstSlice)
}
}
if dstElement.IsValid() && !isEmptyValue(dstElement, !config.ShouldNotDereference) {
if reflect.TypeOf(srcElement.Interface()).Kind() == reflect.Slice {
continue
}
if reflect.TypeOf(srcElement.Interface()).Kind() == reflect.Map && reflect.TypeOf(dstElement.Interface()).Kind() == reflect.Map {
continue
}
}
if srcElement.IsValid() && ((srcElement.Kind() != reflect.Ptr && overwrite) || !dstElement.IsValid() || isEmptyValue(dstElement, !config.ShouldNotDereference)) {
if dst.IsNil() {
dst.Set(reflect.MakeMap(dst.Type()))
}
dst.SetMapIndex(key, srcElement)
}
}
// Ensure that all keys in dst are deleted if they are not in src.
if overwriteWithEmptySrc {
for _, key := range dst.MapKeys() {
srcElement := src.MapIndex(key)
if !srcElement.IsValid() {
dst.SetMapIndex(key, reflect.Value{})
}
}
}
case reflect.Slice:
if !dst.CanSet() {
break
}
if (!isEmptyValue(src, !config.ShouldNotDereference) || overwriteWithEmptySrc || overwriteSliceWithEmptySrc) && (overwrite || isEmptyValue(dst, !config.ShouldNotDereference)) && !config.AppendSlice && !sliceDeepCopy {
dst.Set(src)
} else if config.AppendSlice {
if src.Type() != dst.Type() {
return fmt.Errorf("cannot append two slice with different type (%s, %s)", src.Type(), dst.Type())
}
dst.Set(reflect.AppendSlice(dst, src))
} else if sliceDeepCopy {
for i := 0; i < src.Len() && i < dst.Len(); i++ {
srcElement := src.Index(i)
dstElement := dst.Index(i)
if srcElement.CanInterface() {
srcElement = reflect.ValueOf(srcElement.Interface())
}
if dstElement.CanInterface() {
dstElement = reflect.ValueOf(dstElement.Interface())
}
if err = deepMerge(dstElement, srcElement, visited, depth+1, config); err != nil {
return
}
}
}
case reflect.Ptr:
fallthrough
case reflect.Interface:
if isReflectNil(src) {
if overwriteWithEmptySrc && dst.CanSet() && src.Type().AssignableTo(dst.Type()) {
dst.Set(src)
}
break
}
if src.Kind() != reflect.Interface {
if dst.IsNil() || (src.Kind() != reflect.Ptr && overwrite) {
if dst.CanSet() && (overwrite || isEmptyValue(dst, !config.ShouldNotDereference)) {
dst.Set(src)
}
} else if src.Kind() == reflect.Ptr {
if !config.ShouldNotDereference {
if err = deepMerge(dst.Elem(), src.Elem(), visited, depth+1, config); err != nil {
return
}
} else {
if overwriteWithEmptySrc || (overwrite && !src.IsNil()) || dst.IsNil() {
dst.Set(src)
}
}
} else if dst.Elem().Type() == src.Type() {
if err = deepMerge(dst.Elem(), src, visited, depth+1, config); err != nil {
return
}
} else {
return ErrDifferentArgumentsTypes
}
break
}
if dst.IsNil() || overwrite {
if dst.CanSet() && (overwrite || isEmptyValue(dst, !config.ShouldNotDereference)) {
dst.Set(src)
}
break
}
if dst.Elem().Kind() == src.Elem().Kind() {
if err = deepMerge(dst.Elem(), src.Elem(), visited, depth+1, config); err != nil {
return
}
break
}
default:
mustSet := (isEmptyValue(dst, !config.ShouldNotDereference) || overwrite) && (!isEmptyValue(src, !config.ShouldNotDereference) || overwriteWithEmptySrc)
if mustSet {
if dst.CanSet() {
dst.Set(src)
} else {
dst = src
}
}
}
return
}
// Merge will fill any empty for value type attributes on the dst struct using corresponding
// src attributes if they themselves are not empty. dst and src must be valid same-type structs
// and dst must be a pointer to struct.
// It won't merge unexported (private) fields and will do recursively any exported field.
func Merge(dst, src interface{}, opts ...func(*Config)) error {
return merge(dst, src, opts...)
}
// MergeWithOverwrite will do the same as Merge except that non-empty dst attributes will be overridden by
// non-empty src attribute values.
// Deprecated: use Merge(…) with WithOverride
func MergeWithOverwrite(dst, src interface{}, opts ...func(*Config)) error {
return merge(dst, src, append(opts, WithOverride)...)
}
// WithTransformers adds transformers to merge, allowing to customize the merging of some types.
func WithTransformers(transformers Transformers) func(*Config) {
return func(config *Config) {
config.Transformers = transformers
}
}
// WithOverride will make merge override non-empty dst attributes with non-empty src attributes values.
func WithOverride(config *Config) {
config.Overwrite = true
}
// WithOverwriteWithEmptyValue will make merge override non empty dst attributes with empty src attributes values.
func WithOverwriteWithEmptyValue(config *Config) {
config.Overwrite = true
config.overwriteWithEmptyValue = true
}
// WithOverrideEmptySlice will make merge override empty dst slice with empty src slice.
func WithOverrideEmptySlice(config *Config) {
config.overwriteSliceWithEmptyValue = true
}
// WithoutDereference prevents dereferencing pointers when evaluating whether they are empty
// (i.e. a non-nil pointer is never considered empty).
func WithoutDereference(config *Config) {
config.ShouldNotDereference = true
}
// WithAppendSlice will make merge append slices instead of overwriting it.
func WithAppendSlice(config *Config) {
config.AppendSlice = true
}
// WithTypeCheck will make merge check types while overwriting it (must be used with WithOverride).
func WithTypeCheck(config *Config) {
config.TypeCheck = true
}
// WithSliceDeepCopy will merge slice element one by one with Overwrite flag.
func WithSliceDeepCopy(config *Config) {
config.sliceDeepCopy = true
config.Overwrite = true
}
func merge(dst, src interface{}, opts ...func(*Config)) error {
if dst != nil && reflect.ValueOf(dst).Kind() != reflect.Ptr {
return ErrNonPointerArgument
}
var (
vDst, vSrc reflect.Value
err error
)
config := &Config{}
for _, opt := range opts {
opt(config)
}
if vDst, vSrc, err = resolveValues(dst, src); err != nil {
return err
}
if vDst.Type() != vSrc.Type() {
return ErrDifferentArgumentsTypes
}
return deepMerge(vDst, vSrc, make(map[uintptr]*visit), 0, config)
}
// IsReflectNil is the reflect value provided nil
func isReflectNil(v reflect.Value) bool {
k := v.Kind()
switch k {
case reflect.Interface, reflect.Slice, reflect.Chan, reflect.Func, reflect.Map, reflect.Ptr:
// Both interface and slice are nil if first word is 0.
// Both are always bigger than a word; assume flagIndir.
return v.IsNil()
default:
return false
}
}
vendor/github.com/imdario/mergo/mergo.go
Generated Vendored
-81
View File
@@ -1,81 +0,0 @@
// Copyright 2013 Dario Castañé. All rights reserved.
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Based on src/pkg/reflect/deepequal.go from official
// golang's stdlib.
package mergo
import (
"errors"
"reflect"
)
// Errors reported by Mergo when it finds invalid arguments.
var (
ErrNilArguments = errors.New("src and dst must not be nil")
ErrDifferentArgumentsTypes = errors.New("src and dst must be of same type")
ErrNotSupported = errors.New("only structs, maps, and slices are supported")
ErrExpectedMapAsDestination = errors.New("dst was expected to be a map")
ErrExpectedStructAsDestination = errors.New("dst was expected to be a struct")
ErrNonPointerArgument = errors.New("dst must be a pointer")
)
// During deepMerge, must keep track of checks that are
// in progress. The comparison algorithm assumes that all
// checks in progress are true when it reencounters them.
// Visited are stored in a map indexed by 17 * a1 + a2;
type visit struct {
typ reflect.Type
next *visit
ptr uintptr
}
// From src/pkg/encoding/json/encode.go.
func isEmptyValue(v reflect.Value, shouldDereference bool) bool {
switch v.Kind() {
case reflect.Array, reflect.Map, reflect.Slice, reflect.String:
return v.Len() == 0
case reflect.Bool:
return !v.Bool()
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
return v.Int() == 0
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
return v.Uint() == 0
case reflect.Float32, reflect.Float64:
return v.Float() == 0
case reflect.Interface, reflect.Ptr:
if v.IsNil() {
return true
}
if shouldDereference {
return isEmptyValue(v.Elem(), shouldDereference)
}
return false
case reflect.Func:
return v.IsNil()
case reflect.Invalid:
return true
}
return false
}
func resolveValues(dst, src interface{}) (vDst, vSrc reflect.Value, err error) {
if dst == nil || src == nil {
err = ErrNilArguments
return
}
vDst = reflect.ValueOf(dst).Elem()
if vDst.Kind() != reflect.Struct && vDst.Kind() != reflect.Map && vDst.Kind() != reflect.Slice {
err = ErrNotSupported
return
}
vSrc = reflect.ValueOf(src)
// We check if vSrc is a pointer to dereference it.
if vSrc.Kind() == reflect.Ptr {
vSrc = vSrc.Elem()
}
return
}
vendor/github.com/in-toto/in-toto-golang/in_toto/attestations.go
Generated Vendored
+99
View File
@@ -0,0 +1,99 @@
package in_toto
import (
"github.com/in-toto/in-toto-golang/in_toto/slsa_provenance/common"
slsa01 "github.com/in-toto/in-toto-golang/in_toto/slsa_provenance/v0.1"
slsa02 "github.com/in-toto/in-toto-golang/in_toto/slsa_provenance/v0.2"
slsa1 "github.com/in-toto/in-toto-golang/in_toto/slsa_provenance/v1"
)
const (
// StatementInTotoV01 is the statement type for the generalized link format
// containing statements. This is constant for all predicate types.
StatementInTotoV01 = "https://in-toto.io/Statement/v0.1"
// PredicateSPDX represents a SBOM using the SPDX standard.
// The SPDX mandates 'spdxVersion' field, so predicate type can omit
// version.
PredicateSPDX = "https://spdx.dev/Document"
// PredicateCycloneDX represents a CycloneDX SBOM
PredicateCycloneDX = "https://cyclonedx.org/bom"
// PredicateLinkV1 represents an in-toto 0.9 link.
PredicateLinkV1 = "https://in-toto.io/Link/v1"
)
// Subject describes the set of software artifacts the statement applies to.
type Subject struct {
Name string `json:"name"`
Digest common.DigestSet `json:"digest"`
}
// StatementHeader defines the common fields for all statements
type StatementHeader struct {
Type string `json:"_type"`
PredicateType string `json:"predicateType"`
Subject []Subject `json:"subject"`
}
/*
Statement binds the attestation to a particular subject and identifies the
of the predicate. This struct represents a generic statement.
*/
type Statement struct {
StatementHeader
// Predicate contains type speficic metadata.
Predicate interface{} `json:"predicate"`
}
// ProvenanceStatementSLSA01 is the definition for an entire provenance statement with SLSA 0.1 predicate.
type ProvenanceStatementSLSA01 struct {
StatementHeader
Predicate slsa01.ProvenancePredicate `json:"predicate"`
}
// ProvenanceStatementSLSA02 is the definition for an entire provenance statement with SLSA 0.2 predicate.
type ProvenanceStatementSLSA02 struct {
StatementHeader
Predicate slsa02.ProvenancePredicate `json:"predicate"`
}
// ProvenanceStatementSLSA1 is the definition for an entire provenance statement with SLSA 1.0 predicate.
type ProvenanceStatementSLSA1 struct {
StatementHeader
Predicate slsa1.ProvenancePredicate `json:"predicate"`
}
// ProvenanceStatement is the definition for an entire provenance statement with SLSA 0.2 predicate.
// Deprecated: Only version-specific provenance structs will be maintained (ProvenanceStatementSLSA01, ProvenanceStatementSLSA02).
type ProvenanceStatement struct {
StatementHeader
Predicate slsa02.ProvenancePredicate `json:"predicate"`
}
// LinkStatement is the definition for an entire link statement.
type LinkStatement struct {
StatementHeader
Predicate Link `json:"predicate"`
}
/*
SPDXStatement is the definition for an entire SPDX statement.
This is currently not implemented. Some tooling exists here:
https://github.com/spdx/tools-golang, but this software is still in
early state.
This struct is the same as the generic Statement struct but is added for
completeness
*/
type SPDXStatement struct {
StatementHeader
Predicate interface{} `json:"predicate"`
}
/*
CycloneDXStatement defines a cyclonedx sbom in the predicate. It is not
currently serialized just as its SPDX counterpart. It is an empty
interface, like the generic Statement.
*/
type CycloneDXStatement struct {
StatementHeader
Predicate interface{} `json:"predicate"`
}
vendor/github.com/in-toto/in-toto-golang/in_toto/envelope.go
Generated Vendored
+166
View File
@@ -0,0 +1,166 @@
package in_toto
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"os"
"github.com/secure-systems-lab/go-securesystemslib/cjson"
"github.com/secure-systems-lab/go-securesystemslib/dsse"
"github.com/secure-systems-lab/go-securesystemslib/signerverifier"
)
// PayloadType is the payload type used for links and layouts.
const PayloadType = "application/vnd.in-toto+json"
// ErrInvalidPayloadType indicates that the envelope used an unknown payload type
var ErrInvalidPayloadType = errors.New("unknown payload type")
type Envelope struct {
envelope *dsse.Envelope
payload any
}
func loadEnvelope(env *dsse.Envelope) (*Envelope, error) {
e := &Envelope{envelope: env}
contentBytes, err := env.DecodeB64Payload()
if err != nil {
return nil, err
}
payload, err := loadPayload(contentBytes)
if err != nil {
return nil, err
}
e.payload = payload
return e, nil
}
func (e *Envelope) SetPayload(payload any) error {
encodedBytes, err := cjson.EncodeCanonical(payload)
if err != nil {
return err
}
e.payload = payload
e.envelope = &dsse.Envelope{
Payload: base64.StdEncoding.EncodeToString(encodedBytes),
PayloadType: PayloadType,
}
return nil
}
func (e *Envelope) GetPayload() any {
return e.payload
}
func (e *Envelope) VerifySignature(key Key) error {
verifier, err := getSignerVerifierFromKey(key)
if err != nil {
return err
}
ev, err := dsse.NewEnvelopeVerifier(verifier)
if err != nil {
return err
}
_, err = ev.Verify(context.Background(), e.envelope)
return err
}
func (e *Envelope) Sign(key Key) error {
signer, err := getSignerVerifierFromKey(key)
if err != nil {
return err
}
es, err := dsse.NewEnvelopeSigner(signer)
if err != nil {
return err
}
payload, err := e.envelope.DecodeB64Payload()
if err != nil {
return err
}
env, err := es.SignPayload(context.Background(), e.envelope.PayloadType, payload)
if err != nil {
return err
}
e.envelope = env
return nil
}
func (e *Envelope) Sigs() []Signature {
sigs := []Signature{}
for _, s := range e.envelope.Signatures {
sigs = append(sigs, Signature{
KeyID: s.KeyID,
Sig: s.Sig,
})
}
return sigs
}
func (e *Envelope) GetSignatureForKeyID(keyID string) (Signature, error) {
for _, s := range e.Sigs() {
if s.KeyID == keyID {
return s, nil
}
}
return Signature{}, fmt.Errorf("no signature found for key '%s'", keyID)
}
func (e *Envelope) Dump(path string) error {
jsonBytes, err := json.MarshalIndent(e.envelope, "", " ")
if err != nil {
return err
}
// Write JSON bytes to the passed path with permissions (-rw-r--r--)
err = os.WriteFile(path, jsonBytes, 0644)
if err != nil {
return err
}
return nil
}
func getSignerVerifierFromKey(key Key) (dsse.SignerVerifier, error) {
sslibKey := getSSLibKeyFromKey(key)
switch sslibKey.KeyType {
case signerverifier.RSAKeyType:
return signerverifier.NewRSAPSSSignerVerifierFromSSLibKey(&sslibKey)
case signerverifier.ED25519KeyType:
return signerverifier.NewED25519SignerVerifierFromSSLibKey(&sslibKey)
case signerverifier.ECDSAKeyType:
return signerverifier.NewECDSASignerVerifierFromSSLibKey(&sslibKey)
}
return nil, ErrUnsupportedKeyType
}
func getSSLibKeyFromKey(key Key) signerverifier.SSLibKey {
return signerverifier.SSLibKey{
KeyType: key.KeyType,
KeyIDHashAlgorithms: key.KeyIDHashAlgorithms,
KeyID: key.KeyID,
Scheme: key.Scheme,
KeyVal: signerverifier.KeyVal{
Public: key.KeyVal.Public,
Private: key.KeyVal.Private,
Certificate: key.KeyVal.Certificate,
},
}
}
vendor/github.com/in-toto/in-toto-golang/in_toto/keylib.go
Generated Vendored
+14 -16
View File
@@ -13,7 +13,6 @@ import (
"errors"
"fmt"
"io"
"io/ioutil"
"os"
"strings"
@@ -325,7 +324,7 @@ func (k *Key) LoadKeyReader(r io.Reader, scheme string, KeyIDHashAlgorithms []st
return ErrNoPEMBlock
}
// Read key bytes
pemBytes, err := ioutil.ReadAll(r)
pemBytes, err := io.ReadAll(r)
if err != nil {
return err
}
@@ -344,7 +343,7 @@ func (k *Key) LoadKeyReaderDefaults(r io.Reader) error {
return ErrNoPEMBlock
}
// Read key bytes
pemBytes, err := ioutil.ReadAll(r)
pemBytes, err := io.ReadAll(r)
if err != nil {
return err
}
@@ -366,7 +365,7 @@ func (k *Key) LoadKeyReaderDefaults(r io.Reader) error {
func getDefaultKeyScheme(key interface{}) (scheme string, keyIDHashAlgorithms []string, err error) {
keyIDHashAlgorithms = []string{"sha256", "sha512"}
switch key.(type) {
switch k := key.(type) {
case *rsa.PublicKey, *rsa.PrivateKey:
scheme = rsassapsssha256Scheme
case ed25519.PrivateKey, ed25519.PublicKey:
@@ -374,7 +373,7 @@ func getDefaultKeyScheme(key interface{}) (scheme string, keyIDHashAlgorithms []
case *ecdsa.PrivateKey, *ecdsa.PublicKey:
scheme = ecdsaSha2nistp256
case *x509.Certificate:
return getDefaultKeyScheme(key.(*x509.Certificate).PublicKey)
return getDefaultKeyScheme(k.PublicKey)
default:
err = ErrUnsupportedKeyType
}
@@ -382,11 +381,10 @@ func getDefaultKeyScheme(key interface{}) (scheme string, keyIDHashAlgorithms []
return scheme, keyIDHashAlgorithms, err
}
func (k *Key) loadKey(key interface{}, pemData *pem.Block, scheme string, keyIDHashAlgorithms []string) error {
switch key.(type) {
func (k *Key) loadKey(keyObj interface{}, pemData *pem.Block, scheme string, keyIDHashAlgorithms []string) error {
switch key := keyObj.(type) {
case *rsa.PublicKey:
pubKeyBytes, err := x509.MarshalPKIXPublicKey(key.(*rsa.PublicKey))
pubKeyBytes, err := x509.MarshalPKIXPublicKey(key)
if err != nil {
return err
}
@@ -396,7 +394,7 @@ func (k *Key) loadKey(key interface{}, pemData *pem.Block, scheme string, keyIDH
case *rsa.PrivateKey:
// Note: RSA Public Keys will get stored as X.509 SubjectPublicKeyInfo (RFC5280)
// This behavior is consistent to the securesystemslib
pubKeyBytes, err := x509.MarshalPKIXPublicKey(key.(*rsa.PrivateKey).Public())
pubKeyBytes, err := x509.MarshalPKIXPublicKey(key.Public())
if err != nil {
return err
}
@@ -404,16 +402,16 @@ func (k *Key) loadKey(key interface{}, pemData *pem.Block, scheme string, keyIDH
return err
}
case ed25519.PublicKey:
if err := k.setKeyComponents(key.(ed25519.PublicKey), []byte{}, ed25519KeyType, scheme, keyIDHashAlgorithms); err != nil {
if err := k.setKeyComponents(key, []byte{}, ed25519KeyType, scheme, keyIDHashAlgorithms); err != nil {
return err
}
case ed25519.PrivateKey:
pubKeyBytes := key.(ed25519.PrivateKey).Public()
if err := k.setKeyComponents(pubKeyBytes.(ed25519.PublicKey), key.(ed25519.PrivateKey), ed25519KeyType, scheme, keyIDHashAlgorithms); err != nil {
pubKeyBytes := key.Public()
if err := k.setKeyComponents(pubKeyBytes.(ed25519.PublicKey), key, ed25519KeyType, scheme, keyIDHashAlgorithms); err != nil {
return err
}
case *ecdsa.PrivateKey:
pubKeyBytes, err := x509.MarshalPKIXPublicKey(key.(*ecdsa.PrivateKey).Public())
pubKeyBytes, err := x509.MarshalPKIXPublicKey(key.Public())
if err != nil {
return err
}
@@ -421,7 +419,7 @@ func (k *Key) loadKey(key interface{}, pemData *pem.Block, scheme string, keyIDH
return err
}
case *ecdsa.PublicKey:
pubKeyBytes, err := x509.MarshalPKIXPublicKey(key.(*ecdsa.PublicKey))
pubKeyBytes, err := x509.MarshalPKIXPublicKey(key)
if err != nil {
return err
}
@@ -429,7 +427,7 @@ func (k *Key) loadKey(key interface{}, pemData *pem.Block, scheme string, keyIDH
return err
}
case *x509.Certificate:
err := k.loadKey(key.(*x509.Certificate).PublicKey, pemData, scheme, keyIDHashAlgorithms)
err := k.loadKey(key.PublicKey, pemData, scheme, keyIDHashAlgorithms)
if err != nil {
return err
}
vendor/github.com/in-toto/in-toto-golang/in_toto/model.go
Generated Vendored
+79 -185
View File
@@ -7,7 +7,6 @@ import (
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"os"
"reflect"
"regexp"
@@ -15,10 +14,6 @@ import (
"strings"
"time"
"github.com/in-toto/in-toto-golang/in_toto/slsa_provenance/common"
slsa01 "github.com/in-toto/in-toto-golang/in_toto/slsa_provenance/v0.1"
slsa02 "github.com/in-toto/in-toto-golang/in_toto/slsa_provenance/v0.2"
"github.com/secure-systems-lab/go-securesystemslib/cjson"
"github.com/secure-systems-lab/go-securesystemslib/dsse"
)
@@ -30,7 +25,7 @@ and private keys in PEM format stored as strings. For public keys the Private
field may be an empty string.
*/
type KeyVal struct {
Private string `json:"private"`
Private string `json:"private,omitempty"`
Public string `json:"public"`
Certificate string `json:"certificate,omitempty"`
}
@@ -48,9 +43,6 @@ type Key struct {
Scheme string `json:"scheme"`
}
// PayloadType is the payload type used for links and layouts.
const PayloadType = "application/vnd.in-toto+json"
// ErrEmptyKeyField will be thrown if a field in our Key struct is empty.
var ErrEmptyKeyField = errors.New("empty field in key")
@@ -73,23 +65,6 @@ var ErrNoPublicKey = errors.New("the given key is not a public key")
// for example: curve size = "521" and scheme = "ecdsa-sha2-nistp224"
var ErrCurveSizeSchemeMismatch = errors.New("the scheme does not match the curve size")
const (
// StatementInTotoV01 is the statement type for the generalized link format
// containing statements. This is constant for all predicate types.
StatementInTotoV01 = "https://in-toto.io/Statement/v0.1"
// PredicateSPDX represents a SBOM using the SPDX standard.
// The SPDX mandates 'spdxVersion' field, so predicate type can omit
// version.
PredicateSPDX = "https://spdx.dev/Document"
// PredicateCycloneDX represents a CycloneDX SBOM
PredicateCycloneDX = "https://cyclonedx.org/bom"
// PredicateLinkV1 represents an in-toto 0.9 link.
PredicateLinkV1 = "https://in-toto.io/Link/v1"
)
// ErrInvalidPayloadType indicates that the envelope used an unkown payload type
var ErrInvalidPayloadType = errors.New("unknown payload type")
/*
matchEcdsaScheme checks if the scheme suffix, matches the ecdsa key
curve size. We do not need a full regex match here, because
@@ -702,6 +677,67 @@ func validateLayout(layout Layout) error {
return nil
}
type Metadata interface {
Sign(Key) error
VerifySignature(Key) error
GetPayload() any
Sigs() []Signature
GetSignatureForKeyID(string) (Signature, error)
Dump(string) error
}
func LoadMetadata(path string) (Metadata, error) {
jsonBytes, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var rawData map[string]*json.RawMessage
if err := json.Unmarshal(jsonBytes, &rawData); err != nil {
return nil, err
}
if _, ok := rawData["payloadType"]; ok {
dsseEnv := &dsse.Envelope{}
if rawData["payload"] == nil || rawData["signatures"] == nil {
return nil, fmt.Errorf("in-toto metadata envelope requires 'payload' and 'signatures' parts")
}
if err := json.Unmarshal(jsonBytes, dsseEnv); err != nil {
return nil, err
}
if dsseEnv.PayloadType != PayloadType {
return nil, ErrInvalidPayloadType
}
return loadEnvelope(dsseEnv)
}
mb := &Metablock{}
// Error out on missing `signed` or `signatures` field or if
// one of them has a `null` value, which would lead to a nil pointer
// dereference in Unmarshal below.
if rawData["signed"] == nil || rawData["signatures"] == nil {
return nil, fmt.Errorf("in-toto metadata requires 'signed' and 'signatures' parts")
}
// Fully unmarshal signatures part
if err := json.Unmarshal(*rawData["signatures"], &mb.Signatures); err != nil {
return nil, err
}
payload, err := loadPayload(*rawData["signed"])
if err != nil {
return nil, err
}
mb.Signed = payload
return mb, nil
}
/*
Metablock is a generic container for signable in-toto objects such as Layout
or Link. It has two fields, one that contains the signable object and one that
@@ -767,17 +803,13 @@ func checkRequiredJSONFields(obj map[string]interface{},
Load parses JSON formatted metadata at the passed path into the Metablock
object on which it was called. It returns an error if it cannot parse
a valid JSON formatted Metablock that contains a Link or Layout.
Deprecated: Use LoadMetadata for a signature wrapper agnostic way to load an
envelope.
*/
func (mb *Metablock) Load(path string) error {
// Open file and close before returning
jsonFile, err := os.Open(path)
if err != nil {
return err
}
defer jsonFile.Close()
// Read entire file
jsonBytes, err := ioutil.ReadAll(jsonFile)
jsonBytes, err := os.ReadFile(path)
if err != nil {
return err
}
@@ -803,54 +835,14 @@ func (mb *Metablock) Load(path string) error {
return err
}
// Temporarily copy signed to opaque map to inspect the `_type` of signed
// and create link or layout accordingly
var signed map[string]interface{}
if err := json.Unmarshal(*rawMb["signed"], &signed); err != nil {
payload, err := loadPayload(*rawMb["signed"])
if err != nil {
return err
}
if signed["_type"] == "link" {
var link Link
if err := checkRequiredJSONFields(signed, reflect.TypeOf(link)); err != nil {
return err
}
mb.Signed = payload
data, err := rawMb["signed"].MarshalJSON()
if err != nil {
return err
}
decoder := json.NewDecoder(strings.NewReader(string(data)))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&link); err != nil {
return err
}
mb.Signed = link
} else if signed["_type"] == "layout" {
var layout Layout
if err := checkRequiredJSONFields(signed, reflect.TypeOf(layout)); err != nil {
return err
}
data, err := rawMb["signed"].MarshalJSON()
if err != nil {
return err
}
decoder := json.NewDecoder(strings.NewReader(string(data)))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&layout); err != nil {
return err
}
mb.Signed = layout
} else {
return fmt.Errorf("the '_type' field of the 'signed' part of in-toto" +
" metadata must be one of 'link' or 'layout'")
}
return jsonFile.Close()
return nil
}
/*
@@ -866,7 +858,7 @@ func (mb *Metablock) Dump(path string) error {
}
// Write JSON bytes to the passed path with permissions (-rw-r--r--)
err = ioutil.WriteFile(path, jsonBytes, 0644)
err = os.WriteFile(path, jsonBytes, 0644)
if err != nil {
return err
}
@@ -883,6 +875,14 @@ func (mb *Metablock) GetSignableRepresentation() ([]byte, error) {
return cjson.EncodeCanonical(mb.Signed)
}
func (mb *Metablock) GetPayload() any {
return mb.Signed
}
func (mb *Metablock) Sigs() []Signature {
return mb.Signatures
}
/*
VerifySignature verifies the first signature, corresponding to the passed Key,
that it finds in the Signatures field of the Metablock on which it was called.
@@ -965,109 +965,3 @@ func (mb *Metablock) Sign(key Key) error {
mb.Signatures = append(mb.Signatures, newSignature)
return nil
}
// Subject describes the set of software artifacts the statement applies to.
type Subject struct {
Name string `json:"name"`
Digest common.DigestSet `json:"digest"`
}
// StatementHeader defines the common fields for all statements
type StatementHeader struct {
Type string `json:"_type"`
PredicateType string `json:"predicateType"`
Subject []Subject `json:"subject"`
}
/*
Statement binds the attestation to a particular subject and identifies the
of the predicate. This struct represents a generic statement.
*/
type Statement struct {
StatementHeader
// Predicate contains type speficic metadata.
Predicate interface{} `json:"predicate"`
}
// ProvenanceStatementSLSA01 is the definition for an entire provenance statement with SLSA 0.1 predicate.
type ProvenanceStatementSLSA01 struct {
StatementHeader
Predicate slsa01.ProvenancePredicate `json:"predicate"`
}
// ProvenanceStatementSLSA02 is the definition for an entire provenance statement with SLSA 0.2 predicate.
type ProvenanceStatementSLSA02 struct {
StatementHeader
Predicate slsa02.ProvenancePredicate `json:"predicate"`
}
// ProvenanceStatement is the definition for an entire provenance statement with SLSA 0.2 predicate.
// Deprecated: Only version-specific provenance structs will be maintained (ProvenanceStatementSLSA01, ProvenanceStatementSLSA02).
type ProvenanceStatement struct {
StatementHeader
Predicate slsa02.ProvenancePredicate `json:"predicate"`
}
// LinkStatement is the definition for an entire link statement.
type LinkStatement struct {
StatementHeader
Predicate Link `json:"predicate"`
}
/*
SPDXStatement is the definition for an entire SPDX statement.
This is currently not implemented. Some tooling exists here:
https://github.com/spdx/tools-golang, but this software is still in
early state.
This struct is the same as the generic Statement struct but is added for
completeness
*/
type SPDXStatement struct {
StatementHeader
Predicate interface{} `json:"predicate"`
}
/*
CycloneDXStatement defines a cyclonedx sbom in the predicate. It is not
currently serialized just as its SPDX counterpart. It is an empty
interface, like the generic Statement.
*/
type CycloneDXStatement struct {
StatementHeader
Predicate interface{} `json:"predicate"`
}
/*
DSSESigner provides signature generation and validation based on the SSL
Signing Spec: https://github.com/secure-systems-lab/signing-spec
as describe by: https://github.com/MarkLodato/ITE/tree/media-type/ITE/5
It wraps the generic SSL envelope signer and enforces the correct payload
type both during signature generation and validation.
*/
type DSSESigner struct {
signer *dsse.EnvelopeSigner
}
func NewDSSESigner(p ...dsse.SignVerifier) (*DSSESigner, error) {
es, err := dsse.NewEnvelopeSigner(p...)
if err != nil {
return nil, err
}
return &DSSESigner{
signer: es,
}, nil
}
func (s *DSSESigner) SignPayload(body []byte) (*dsse.Envelope, error) {
return s.signer.SignPayload(PayloadType, body)
}
func (s *DSSESigner) Verify(e *dsse.Envelope) error {
if e.PayloadType != PayloadType {
return ErrInvalidPayloadType
}
_, err := s.signer.Verify(e)
return err
}
vendor/github.com/in-toto/in-toto-golang/in_toto/runlib.go
Generated Vendored
+96 -43
View File
@@ -4,7 +4,7 @@ import (
"bytes"
"errors"
"fmt"
"io/ioutil"
"io"
"os"
"os/exec"
"path/filepath"
@@ -44,7 +44,7 @@ normalized to Unix-style line separators (LF) before hashing file contents.
func RecordArtifact(path string, hashAlgorithms []string, lineNormalization bool) (map[string]interface{}, error) {
supportedHashMappings := getHashMapping()
// Read file from passed path
contents, err := ioutil.ReadFile(path)
contents, err := os.ReadFile(path)
hashedContentsMap := make(map[string]interface{})
if err != nil {
return nil, err
@@ -92,10 +92,10 @@ the following format:
If recording an artifact fails the first return value is nil and the second
return value is the error.
*/
func RecordArtifacts(paths []string, hashAlgorithms []string, gitignorePatterns []string, lStripPaths []string, lineNormalization bool) (evalArtifacts map[string]interface{}, err error) {
func RecordArtifacts(paths []string, hashAlgorithms []string, gitignorePatterns []string, lStripPaths []string, lineNormalization bool, followSymlinkDirs bool) (evalArtifacts map[string]interface{}, err error) {
// Make sure to initialize a fresh hashset for every RecordArtifacts call
visitedSymlinks = NewSet()
evalArtifacts, err = recordArtifacts(paths, hashAlgorithms, gitignorePatterns, lStripPaths, lineNormalization)
evalArtifacts, err = recordArtifacts(paths, hashAlgorithms, gitignorePatterns, lStripPaths, lineNormalization, followSymlinkDirs)
// pass result and error through
return evalArtifacts, err
}
@@ -118,7 +118,7 @@ the following format:
If recording an artifact fails the first return value is nil and the second
return value is the error.
*/
func recordArtifacts(paths []string, hashAlgorithms []string, gitignorePatterns []string, lStripPaths []string, lineNormalization bool) (map[string]interface{}, error) {
func recordArtifacts(paths []string, hashAlgorithms []string, gitignorePatterns []string, lStripPaths []string, lineNormalization bool, followSymlinkDirs bool) (map[string]interface{}, error) {
artifacts := make(map[string]interface{})
for _, path := range paths {
err := filepath.Walk(path,
@@ -160,18 +160,35 @@ func recordArtifacts(paths []string, hashAlgorithms []string, gitignorePatterns
if err != nil {
return err
}
info, err := os.Stat(evalSym)
if err != nil {
return err
}
targetIsDir := false
if info.IsDir() {
if !followSymlinkDirs {
// We don't follow symlinked directories
return nil
}
targetIsDir = true
}
// add symlink to visitedSymlinks set
// this way, we know which link we have visited already
// if we visit a symlink twice, we have detected a symlink cycle
visitedSymlinks.Add(path)
// We recursively call RecordArtifacts() to follow
// We recursively call recordArtifacts() to follow
// the new path.
evalArtifacts, evalErr := recordArtifacts([]string{evalSym}, hashAlgorithms, gitignorePatterns, lStripPaths, lineNormalization)
evalArtifacts, evalErr := recordArtifacts([]string{evalSym}, hashAlgorithms, gitignorePatterns, lStripPaths, lineNormalization, followSymlinkDirs)
if evalErr != nil {
return evalErr
}
for key, value := range evalArtifacts {
artifacts[key] = value
if targetIsDir {
symlinkPath := filepath.Join(path, strings.TrimPrefix(key, evalSym))
artifacts[symlinkPath] = value
} else {
artifacts[path] = value
}
}
return nil
}
@@ -189,8 +206,7 @@ func recordArtifacts(paths []string, hashAlgorithms []string, gitignorePatterns
}
}
// Check if path is unique
_, existingPath := artifacts[path]
if existingPath {
if _, exists := artifacts[path]; exists {
return fmt.Errorf("left stripping has resulted in non unique dictionary key: %s", path)
}
artifacts[path] = artifact
@@ -273,8 +289,8 @@ func RunCommand(cmdArgs []string, runDir string) (map[string]interface{}, error)
}
// TODO: duplicate stdout, stderr
stdout, _ := ioutil.ReadAll(stdoutPipe)
stderr, _ := ioutil.ReadAll(stderrPipe)
stdout, _ := io.ReadAll(stdoutPipe)
stderr, _ := io.ReadAll(stderrPipe)
retVal := waitErrToExitCode(cmd.Wait())
@@ -293,14 +309,10 @@ and materials at the passed materialPaths. The returned link is wrapped in a
Metablock object. If command execution or artifact recording fails the first
return value is an empty Metablock and the second return value is the error.
*/
func InTotoRun(name string, runDir string, materialPaths []string, productPaths []string,
cmdArgs []string, key Key, hashAlgorithms []string, gitignorePatterns []string,
lStripPaths []string, lineNormalization bool) (Metablock, error) {
var linkMb Metablock
materials, err := RecordArtifacts(materialPaths, hashAlgorithms, gitignorePatterns, lStripPaths, lineNormalization)
func InTotoRun(name string, runDir string, materialPaths []string, productPaths []string, cmdArgs []string, key Key, hashAlgorithms []string, gitignorePatterns []string, lStripPaths []string, lineNormalization bool, followSymlinkDirs bool, useDSSE bool) (Metadata, error) {
materials, err := RecordArtifacts(materialPaths, hashAlgorithms, gitignorePatterns, lStripPaths, lineNormalization, followSymlinkDirs)
if err != nil {
return linkMb, err
return nil, err
}
// make sure that we only run RunCommand if cmdArgs is not nil or empty
@@ -308,16 +320,16 @@ func InTotoRun(name string, runDir string, materialPaths []string, productPaths
if len(cmdArgs) != 0 {
byProducts, err = RunCommand(cmdArgs, runDir)
if err != nil {
return linkMb, err
return nil, err
}
}
products, err := RecordArtifacts(productPaths, hashAlgorithms, gitignorePatterns, lStripPaths, lineNormalization)
products, err := RecordArtifacts(productPaths, hashAlgorithms, gitignorePatterns, lStripPaths, lineNormalization, followSymlinkDirs)
if err != nil {
return linkMb, err
return nil, err
}
linkMb.Signed = Link{
link := Link{
Type: "link",
Name: name,
Materials: materials,
@@ -327,14 +339,25 @@ func InTotoRun(name string, runDir string, materialPaths []string, productPaths
Environment: map[string]interface{}{},
}
linkMb.Signatures = []Signature{}
// We use a new feature from Go1.13 here, to check the key struct.
// IsZero() will return True, if the key hasn't been initialized
if useDSSE {
env := &Envelope{}
if err := env.SetPayload(link); err != nil {
return nil, err
}
// with other values than the default ones.
if !reflect.ValueOf(key).IsZero() {
if err := env.Sign(key); err != nil {
return nil, err
}
}
return env, nil
}
linkMb := &Metablock{Signed: link, Signatures: []Signature{}}
if !reflect.ValueOf(key).IsZero() {
if err := linkMb.Sign(key); err != nil {
return linkMb, err
return nil, err
}
}
@@ -347,14 +370,13 @@ in order to provide evidence for supply chain steps that cannot be carries out
by a single command. InTotoRecordStart collects the hashes of the materials
before any commands are run, signs the unfinished link, and returns the link.
*/
func InTotoRecordStart(name string, materialPaths []string, key Key, hashAlgorithms, gitignorePatterns []string, lStripPaths []string, lineNormalization bool) (Metablock, error) {
var linkMb Metablock
materials, err := RecordArtifacts(materialPaths, hashAlgorithms, gitignorePatterns, lStripPaths, lineNormalization)
func InTotoRecordStart(name string, materialPaths []string, key Key, hashAlgorithms, gitignorePatterns []string, lStripPaths []string, lineNormalization bool, followSymlinkDirs bool, useDSSE bool) (Metadata, error) {
materials, err := RecordArtifacts(materialPaths, hashAlgorithms, gitignorePatterns, lStripPaths, lineNormalization, followSymlinkDirs)
if err != nil {
return linkMb, err
return nil, err
}
linkMb.Signed = Link{
link := Link{
Type: "link",
Name: name,
Materials: materials,
@@ -364,9 +386,26 @@ func InTotoRecordStart(name string, materialPaths []string, key Key, hashAlgorit
Environment: map[string]interface{}{},
}
if useDSSE {
env := &Envelope{}
if err := env.SetPayload(link); err != nil {
return nil, err
}
if !reflect.ValueOf(key).IsZero() {
if err := env.Sign(key); err != nil {
return nil, err
}
}
return env, nil
}
linkMb := &Metablock{Signed: link, Signatures: []Signature{}}
linkMb.Signatures = []Signature{}
if !reflect.ValueOf(key).IsZero() {
if err := linkMb.Sign(key); err != nil {
return linkMb, err
return nil, err
}
}
@@ -380,25 +419,39 @@ created by InTotoRecordStart and records the hashes of any products creted by
commands run between InTotoRecordStart and InTotoRecordStop. The resultant
finished link metablock is then signed by the provided key and returned.
*/
func InTotoRecordStop(prelimLinkMb Metablock, productPaths []string, key Key, hashAlgorithms, gitignorePatterns []string, lStripPaths []string, lineNormalization bool) (Metablock, error) {
var linkMb Metablock
if err := prelimLinkMb.VerifySignature(key); err != nil {
return linkMb, err
func InTotoRecordStop(prelimLinkEnv Metadata, productPaths []string, key Key, hashAlgorithms, gitignorePatterns []string, lStripPaths []string, lineNormalization bool, followSymlinkDirs bool, useDSSE bool) (Metadata, error) {
if err := prelimLinkEnv.VerifySignature(key); err != nil {
return nil, err
}
link, ok := prelimLinkMb.Signed.(Link)
link, ok := prelimLinkEnv.GetPayload().(Link)
if !ok {
return linkMb, errors.New("invalid metadata block")
return nil, errors.New("invalid metadata block")
}
products, err := RecordArtifacts(productPaths, hashAlgorithms, gitignorePatterns, lStripPaths, lineNormalization)
products, err := RecordArtifacts(productPaths, hashAlgorithms, gitignorePatterns, lStripPaths, lineNormalization, followSymlinkDirs)
if err != nil {
return linkMb, err
return nil, err
}
link.Products = products
linkMb.Signed = link
if useDSSE {
env := &Envelope{}
if err := env.SetPayload(link); err != nil {
return nil, err
}
if !reflect.ValueOf(key).IsZero() {
if err := env.Sign(key); err != nil {
return nil, err
}
}
return env, nil
}
linkMb := &Metablock{Signed: link, Signatures: []Signature{}}
if !reflect.ValueOf(key).IsZero() {
if err := linkMb.Sign(key); err != nil {
return linkMb, err
vendor/github.com/in-toto/in-toto-golang/in_toto/slsa_provenance/v0.2/provenance.go
Generated Vendored
+7
View File
@@ -11,6 +11,13 @@ const (
PredicateSLSAProvenance = "https://slsa.dev/provenance/v0.2"
)
// These are type aliases to common to avoid backwards incompatible changes.
type (
DigestSet = common.DigestSet
ProvenanceBuilder = common.ProvenanceBuilder
ProvenanceMaterial = common.ProvenanceMaterial
)
// ProvenancePredicate is the provenance predicate definition.
type ProvenancePredicate struct {
// Builder identifies the entity that executed the invocation, which is trusted to have
vendor/github.com/in-toto/in-toto-golang/in_toto/slsa_provenance/v1/provenance.go
Generated Vendored
+151
View File
@@ -0,0 +1,151 @@
package v1
import (
"time"
"github.com/in-toto/in-toto-golang/in_toto/slsa_provenance/common"
)
const (
// PredicateSLSAProvenance represents a build provenance for an artifact.
PredicateSLSAProvenance = "https://slsa.dev/provenance/v1"
)
// ProvenancePredicate is the provenance predicate definition.
type ProvenancePredicate struct {
// The BuildDefinition describes all of the inputs to the build. The
// accuracy and completeness are implied by runDetails.builder.id.
//
// It SHOULD contain all the information necessary and sufficient to
// initialize the build and begin execution.
BuildDefinition ProvenanceBuildDefinition `json:"buildDefinition"`
// Details specific to this particular execution of the build.
RunDetails ProvenanceRunDetails `json:"runDetails"`
}
// ProvenanceBuildDefinition describes the inputs to the build.
type ProvenanceBuildDefinition struct {
// Identifies the template for how to perform the build and interpret the
// parameters and dependencies.
// The URI SHOULD resolve to a human-readable specification that includes:
// overall description of the build type; schema for externalParameters and
// systemParameters; unambiguous instructions for how to initiate the build
// given this BuildDefinition, and a complete example.
BuildType string `json:"buildType"`
// The parameters that are under external control, such as those set by a
// user or tenant of the build system. They MUST be complete at SLSA Build
// L3, meaning that that there is no additional mechanism for an external
// party to influence the build. (At lower SLSA Build levels, the
// completeness MAY be best effort.)
// The build system SHOULD be designed to minimize the size and complexity
// of externalParameters, in order to reduce fragility and ease
// verification. Consumers SHOULD have an expectation of what “good” looks
// like; the more information that they need to check, the harder that task
// becomes.
ExternalParameters interface{} `json:"externalParameters"`
// The parameters that are under the control of the entity represented by
// builder.id. The primary intention of this field is for debugging,
// incident response, and vulnerability management. The values here MAY be
// necessary for reproducing the build. There is no need to verify these
// parameters because the build system is already trusted, and in many cases
// it is not practical to do so.
InternalParameters interface{} `json:"internalParameters,omitempty"`
// Unordered collection of artifacts needed at build time. Completeness is
// best effort, at least through SLSA Build L3. For example, if the build
// script fetches and executes “example.com/foo.sh”, which in turn fetches
// “example.com/bar.tar.gz”, then both “foo.sh” and “bar.tar.gz” SHOULD be
// listed here.
ResolvedDependencies []ResourceDescriptor `json:"resolvedDependencies,omitempty"`
}
// ProvenanceRunDetails includes details specific to a particular execution of a
// build.
type ProvenanceRunDetails struct {
// Identifies the entity that executed the invocation, which is trusted to
// have correctly performed the operation and populated this provenance.
//
// This field is REQUIRED for SLSA Build 1 unless id is implicit from the
// attestation envelope.
Builder Builder `json:"builder"`
// Metadata about this particular execution of the build.
BuildMetadata BuildMetadata `json:"metadata,omitempty"`
// Additional artifacts generated during the build that are not considered
// the “output” of the build but that might be needed during debugging or
// incident response. For example, this might reference logs generated
// during the build and/or a digest of the fully evaluated build
// configuration.
//
// In most cases, this SHOULD NOT contain all intermediate files generated
// during the build. Instead, this SHOULD only contain files that are
// likely to be useful later and that cannot be easily reproduced.
Byproducts []ResourceDescriptor `json:"byproducts,omitempty"`
}
// ResourceDescriptor describes a particular software artifact or resource
// (mutable or immutable).
// See https://github.com/in-toto/attestation/blob/main/spec/v1.0/resource_descriptor.md
type ResourceDescriptor struct {
// A URI used to identify the resource or artifact globally. This field is
// REQUIRED unless either digest or content is set.
URI string `json:"uri,omitempty"`
// A set of cryptographic digests of the contents of the resource or
// artifact. This field is REQUIRED unless either uri or content is set.
Digest common.DigestSet `json:"digest,omitempty"`
// TMachine-readable identifier for distinguishing between descriptors.
Name string `json:"name,omitempty"`
// The location of the described resource or artifact, if different from the
// uri.
DownloadLocation string `json:"downloadLocation,omitempty"`
// The MIME Type (i.e., media type) of the described resource or artifact.
MediaType string `json:"mediaType,omitempty"`
// The contents of the resource or artifact. This field is REQUIRED unless
// either uri or digest is set.
Content []byte `json:"content,omitempty"`
// This field MAY be used to provide additional information or metadata
// about the resource or artifact that may be useful to the consumer when
// evaluating the attestation against a policy.
Annotations map[string]interface{} `json:"annotations,omitempty"`
}
// Builder represents the transitive closure of all the entities that are, by
// necessity, trusted to faithfully run the build and record the provenance.
type Builder struct {
// URI indicating the transitive closure of the trusted builder.
ID string `json:"id"`
// Version numbers of components of the builder.
Version map[string]string `json:"version,omitempty"`
// Dependencies used by the orchestrator that are not run within the
// workload and that do not affect the build, but might affect the
// provenance generation or security guarantees.
BuilderDependencies []ResourceDescriptor `json:"builderDependencies,omitempty"`
}
type BuildMetadata struct {
// Identifies this particular build invocation, which can be useful for
// finding associated logs or other ad-hoc analysis. The exact meaning and
// format is defined by builder.id; by default it is treated as opaque and
// case-sensitive. The value SHOULD be globally unique.
InvocationID string `json:"invocationID,omitempty"`
// The timestamp of when the build started.
StartedOn *time.Time `json:"startedOn,omitempty"`
// The timestamp of when the build completed.
FinishedOn *time.Time `json:"finishedOn,omitempty"`
}
vendor/github.com/in-toto/in-toto-golang/in_toto/util.go
Generated Vendored
+43
View File
@@ -1,9 +1,15 @@
package in_toto
import (
"encoding/json"
"errors"
"fmt"
"reflect"
"strings"
)
var ErrUnknownMetadataType = errors.New("unknown metadata type encountered: not link or layout")
/*
Set represents a data structure for set operations. See `NewSet` for how to
create a Set, and available Set receivers for useful set operations.
@@ -145,3 +151,40 @@ func (s Set) IsSubSet(subset Set) bool {
}
return true
}
func loadPayload(payloadBytes []byte) (any, error) {
var payload map[string]any
if err := json.Unmarshal(payloadBytes, &payload); err != nil {
return nil, fmt.Errorf("error decoding payload: %w", err)
}
if payload["_type"] == "link" {
var link Link
if err := checkRequiredJSONFields(payload, reflect.TypeOf(link)); err != nil {
return nil, fmt.Errorf("error decoding payload: %w", err)
}
decoder := json.NewDecoder(strings.NewReader(string(payloadBytes)))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&link); err != nil {
return nil, fmt.Errorf("error decoding payload: %w", err)
}
return link, nil
} else if payload["_type"] == "layout" {
var layout Layout
if err := checkRequiredJSONFields(payload, reflect.TypeOf(layout)); err != nil {
return nil, fmt.Errorf("error decoding payload: %w", err)
}
decoder := json.NewDecoder(strings.NewReader(string(payloadBytes)))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&layout); err != nil {
return nil, fmt.Errorf("error decoding payload: %w", err)
}
return layout, nil
}
return nil, ErrUnknownMetadataType
}
vendor/github.com/in-toto/in-toto-golang/in_toto/verifylib.go
Generated Vendored
+134 -117
View File
@@ -12,7 +12,6 @@ import (
"io"
"os"
"path"
osPath "path"
"path/filepath"
"reflect"
"regexp"
@@ -23,6 +22,8 @@ import (
// ErrInspectionRunDirIsSymlink gets thrown if the runDir is a symlink
var ErrInspectionRunDirIsSymlink = errors.New("runDir is a symlink. This is a security risk")
var ErrNotLayout = errors.New("verification workflow passed a non-layout")
/*
RunInspections iteratively executes the command in the Run field of all
inspections of the passed layout, creating unsigned link metadata that records
@@ -41,8 +42,8 @@ If executing the inspection command fails, or if the executed command has a
non-zero exit code, the first return value is an empty Metablock map and the
second return value is the error.
*/
func RunInspections(layout Layout, runDir string, lineNormalization bool) (map[string]Metablock, error) {
inspectionMetadata := make(map[string]Metablock)
func RunInspections(layout Layout, runDir string, lineNormalization bool, useDSSE bool) (map[string]Metadata, error) {
inspectionMetadata := make(map[string]Metadata)
for _, inspection := range layout.Inspect {
@@ -51,14 +52,14 @@ func RunInspections(layout Layout, runDir string, lineNormalization bool) (map[s
paths = []string{runDir}
}
linkMb, err := InTotoRun(inspection.Name, runDir, paths, paths,
inspection.Run, Key{}, []string{"sha256"}, nil, nil, lineNormalization)
linkEnv, err := InTotoRun(inspection.Name, runDir, paths, paths,
inspection.Run, Key{}, []string{"sha256"}, nil, nil, lineNormalization, false, useDSSE)
if err != nil {
return nil, err
}
retVal := linkMb.Signed.(Link).ByProducts["return-value"]
retVal := linkEnv.GetPayload().(Link).ByProducts["return-value"]
if retVal != float64(0) {
return nil, fmt.Errorf("inspection command '%s' of inspection '%s'"+
" returned a non-zero value: %d", inspection.Run, inspection.Name,
@@ -67,11 +68,11 @@ func RunInspections(layout Layout, runDir string, lineNormalization bool) (map[s
// Dump inspection link to cwd using the short link name format
linkName := fmt.Sprintf(LinkNameFormatShort, inspection.Name)
if err := linkMb.Dump(linkName); err != nil {
if err := linkEnv.Dump(linkName); err != nil {
fmt.Printf("JSON serialization or writing failed: %s", err)
}
inspectionMetadata[inspection.Name] = linkMb
inspectionMetadata[inspection.Name] = linkEnv
}
return inspectionMetadata, nil
}
@@ -80,10 +81,10 @@ func RunInspections(layout Layout, runDir string, lineNormalization bool) (map[s
// type MATCH. See VerifyArtifacts for more details.
func verifyMatchRule(ruleData map[string]string,
srcArtifacts map[string]interface{}, srcArtifactQueue Set,
itemsMetadata map[string]Metablock) Set {
itemsMetadata map[string]Metadata) Set {
consumed := NewSet()
// Get destination link metadata
dstLinkMb, exists := itemsMetadata[ruleData["dstName"]]
dstLinkEnv, exists := itemsMetadata[ruleData["dstName"]]
if !exists {
// Destination link does not exist, rule can't consume any
// artifacts
@@ -94,9 +95,9 @@ func verifyMatchRule(ruleData map[string]string,
var dstArtifacts map[string]interface{}
switch ruleData["dstType"] {
case "materials":
dstArtifacts = dstLinkMb.Signed.(Link).Materials
dstArtifacts = dstLinkEnv.GetPayload().(Link).Materials
case "products":
dstArtifacts = dstLinkMb.Signed.(Link).Products
dstArtifacts = dstLinkEnv.GetPayload().(Link).Products
}
// cleanup paths in pattern and artifact maps
@@ -140,7 +141,7 @@ func verifyMatchRule(ruleData map[string]string,
// Construct corresponding destination artifact path, i.e.
// an optional destination prefix plus the source base path
dstPath := path.Clean(osPath.Join(ruleData["dstPrefix"], srcBasePath))
dstPath := path.Clean(path.Join(ruleData["dstPrefix"], srcBasePath))
// Try to find the corresponding destination artifact
dstArtifact, exists := dstArtifacts[dstPath]
@@ -180,7 +181,7 @@ DISALLOW rule to fail overall verification, if artifacts are left in the queue
that should have been consumed by preceding rules.
*/
func VerifyArtifacts(items []interface{},
itemsMetadata map[string]Metablock) error {
itemsMetadata map[string]Metadata) error {
// Verify artifact rules for each item in the layout
for _, itemI := range items {
// The layout item (interface) must be a Link or an Inspection we are only
@@ -207,7 +208,7 @@ func VerifyArtifacts(items []interface{},
}
// Use the item's name to extract the corresponding link
srcLinkMb, exists := itemsMetadata[itemName]
srcLinkEnv, exists := itemsMetadata[itemName]
if !exists {
return fmt.Errorf("VerifyArtifacts could not find metadata"+
" for item '%s', got: '%s'", itemName, itemsMetadata)
@@ -215,8 +216,8 @@ func VerifyArtifacts(items []interface{},
// Create shortcuts to materials and products (including hashes) reported
// by the item's link, required to verify "match" rules
materials := srcLinkMb.Signed.(Link).Materials
products := srcLinkMb.Signed.(Link).Products
materials := srcLinkEnv.GetPayload().(Link).Materials
products := srcLinkEnv.GetPayload().(Link).Products
// All other rules only require the material or product paths (without
// hashes). We extract them from the corresponding maps and store them as
@@ -364,9 +365,9 @@ Products, the first return value is an empty Metablock map and the second
return value is the error.
*/
func ReduceStepsMetadata(layout Layout,
stepsMetadata map[string]map[string]Metablock) (map[string]Metablock,
stepsMetadata map[string]map[string]Metadata) (map[string]Metadata,
error) {
stepsMetadataReduced := make(map[string]Metablock)
stepsMetadataReduced := make(map[string]Metadata)
for _, step := range layout.Steps {
linksPerStep, ok := stepsMetadata[step.Name]
@@ -379,16 +380,16 @@ func ReduceStepsMetadata(layout Layout,
// Get the first link (could be any link) for the current step, which will
// serve as reference link for below comparisons
var referenceKeyID string
var referenceLinkMb Metablock
for keyID, linkMb := range linksPerStep {
referenceLinkMb = linkMb
var referenceLinkEnv Metadata
for keyID, linkEnv := range linksPerStep {
referenceLinkEnv = linkEnv
referenceKeyID = keyID
break
}
// Only one link, nothing to reduce, take the reference link
if len(linksPerStep) == 1 {
stepsMetadataReduced[step.Name] = referenceLinkMb
stepsMetadataReduced[step.Name] = referenceLinkEnv
// Multiple links, reduce but first check
} else {
@@ -396,11 +397,11 @@ func ReduceStepsMetadata(layout Layout,
// TODO: What should we do if there are more links, than the
// threshold requires, but not all of them are equal? Right now we would
// also error.
for keyID, linkMb := range linksPerStep {
if !reflect.DeepEqual(linkMb.Signed.(Link).Materials,
referenceLinkMb.Signed.(Link).Materials) ||
!reflect.DeepEqual(linkMb.Signed.(Link).Products,
referenceLinkMb.Signed.(Link).Products) {
for keyID, linkEnv := range linksPerStep {
if !reflect.DeepEqual(linkEnv.GetPayload().(Link).Materials,
referenceLinkEnv.GetPayload().(Link).Materials) ||
!reflect.DeepEqual(linkEnv.GetPayload().(Link).Products,
referenceLinkEnv.GetPayload().(Link).Products) {
return nil, fmt.Errorf("link '%s' and '%s' have different"+
" artifacts",
fmt.Sprintf(LinkNameFormat, step.Name, referenceKeyID),
@@ -408,7 +409,7 @@ func ReduceStepsMetadata(layout Layout,
}
}
// We haven't errored out, so we can reduce (i.e take the reference link)
stepsMetadataReduced[step.Name] = referenceLinkMb
stepsMetadataReduced[step.Name] = referenceLinkEnv
}
}
return stepsMetadataReduced, nil
@@ -421,7 +422,7 @@ command, as per the layout. Soft verification means that, in case a command
does not align, a warning is issued.
*/
func VerifyStepCommandAlignment(layout Layout,
stepsMetadata map[string]map[string]Metablock) {
stepsMetadata map[string]map[string]Metadata) {
for _, step := range layout.Steps {
linksPerStep, ok := stepsMetadata[step.Name]
// We should never get here, layout verification must fail earlier
@@ -430,9 +431,9 @@ func VerifyStepCommandAlignment(layout Layout,
"', no link metadata found.")
}
for signerKeyID, linkMb := range linksPerStep {
for signerKeyID, linkEnv := range linksPerStep {
expectedCommandS := strings.Join(step.ExpectedCommand, " ")
executedCommandS := strings.Join(linkMb.Signed.(Link).Command, " ")
executedCommandS := strings.Join(linkEnv.GetPayload().(Link).Command, " ")
if expectedCommandS != executedCommandS {
linkName := fmt.Sprintf(LinkNameFormat, step.Name, signerKeyID)
@@ -502,11 +503,11 @@ return value is an empty map of Metablock maps and the second return value is
the error.
*/
func VerifyLinkSignatureThesholds(layout Layout,
stepsMetadata map[string]map[string]Metablock, rootCertPool, intermediateCertPool *x509.CertPool) (
map[string]map[string]Metablock, error) {
stepsMetadata map[string]map[string]Metadata, rootCertPool, intermediateCertPool *x509.CertPool) (
map[string]map[string]Metadata, error) {
// This will stores links with valid signature from an authorized functionary
// for all steps
stepsMetadataVerified := make(map[string]map[string]Metablock)
stepsMetadataVerified := make(map[string]map[string]Metadata)
// Try to find enough (>= threshold) links each with a valid signature from
// distinct authorized functionaries for each step
@@ -515,7 +516,7 @@ func VerifyLinkSignatureThesholds(layout Layout,
// This will store links with valid signature from an authorized
// functionary for the given step
linksPerStepVerified := make(map[string]Metablock)
linksPerStepVerified := make(map[string]Metadata)
// Check if there are any links at all for the given step
linksPerStep, ok := stepsMetadata[step.Name]
@@ -528,12 +529,12 @@ func VerifyLinkSignatureThesholds(layout Layout,
// verification passes. Only good links are stored, to verify thresholds
// below.
isAuthorizedSignature := false
for signerKeyID, linkMb := range linksPerStep {
for signerKeyID, linkEnv := range linksPerStep {
for _, authorizedKeyID := range step.PubKeys {
if signerKeyID == authorizedKeyID {
if verifierKey, ok := layout.Keys[authorizedKeyID]; ok {
if err := linkMb.VerifySignature(verifierKey); err == nil {
linksPerStepVerified[signerKeyID] = linkMb
if err := linkEnv.VerifySignature(verifierKey); err == nil {
linksPerStepVerified[signerKeyID] = linkEnv
isAuthorizedSignature = true
break
}
@@ -544,7 +545,7 @@ func VerifyLinkSignatureThesholds(layout Layout,
// If the signer's key wasn't in our step's pubkeys array, check the cert pool to
// see if the key is known to us.
if !isAuthorizedSignature {
sig, err := linkMb.GetSignatureForKeyID(signerKeyID)
sig, err := linkEnv.GetSignatureForKeyID(signerKeyID)
if err != nil {
stepErr = err
continue
@@ -563,13 +564,13 @@ func VerifyLinkSignatureThesholds(layout Layout,
continue
}
err = linkMb.VerifySignature(cert)
err = linkEnv.VerifySignature(cert)
if err != nil {
stepErr = err
continue
}
linksPerStepVerified[signerKeyID] = linkMb
linksPerStepVerified[signerKeyID] = linkEnv
}
}
@@ -614,30 +615,30 @@ ignored. Only a preliminary threshold check is performed, that is, if there
aren't at least Threshold links for any given step, the first return value
is an empty map of Metablock maps and the second return value is the error.
*/
func LoadLinksForLayout(layout Layout, linkDir string) (map[string]map[string]Metablock, error) {
stepsMetadata := make(map[string]map[string]Metablock)
func LoadLinksForLayout(layout Layout, linkDir string) (map[string]map[string]Metadata, error) {
stepsMetadata := make(map[string]map[string]Metadata)
for _, step := range layout.Steps {
linksPerStep := make(map[string]Metablock)
linksPerStep := make(map[string]Metadata)
// Since we can verify against certificates belonging to a CA, we need to
// load any possible links
linkFiles, err := filepath.Glob(osPath.Join(linkDir, fmt.Sprintf(LinkGlobFormat, step.Name)))
linkFiles, err := filepath.Glob(path.Join(linkDir, fmt.Sprintf(LinkGlobFormat, step.Name)))
if err != nil {
return nil, err
}
for _, linkPath := range linkFiles {
var linkMb Metablock
if err := linkMb.Load(linkPath); err != nil {
linkEnv, err := LoadMetadata(linkPath)
if err != nil {
continue
}
// To get the full key from the metadata's signatures, we have to check
// for one with the same short id...
signerShortKeyID := strings.TrimSuffix(strings.TrimPrefix(filepath.Base(linkPath), step.Name+"."), ".link")
for _, sig := range linkMb.Signatures {
for _, sig := range linkEnv.Sigs() {
if strings.HasPrefix(sig.KeyID, signerShortKeyID) {
linksPerStep[sig.KeyID] = linkMb
linksPerStep[sig.KeyID] = linkEnv
break
}
}
@@ -677,14 +678,14 @@ Signatures and keys are associated by key id. If the key map is empty, or the
Metablock's Signature field does not have a signature for one or more of the
passed keys, or a matching signature is invalid, an error is returned.
*/
func VerifyLayoutSignatures(layoutMb Metablock,
func VerifyLayoutSignatures(layoutEnv Metadata,
layoutKeys map[string]Key) error {
if len(layoutKeys) < 1 {
return fmt.Errorf("layout verification requires at least one key")
}
for _, key := range layoutKeys {
if err := layoutMb.VerifySignature(key); err != nil {
if err := layoutEnv.VerifySignature(key); err != nil {
return err
}
}
@@ -700,29 +701,35 @@ NOTE: The assumption is that the steps mentioned in the layout are to be
performed sequentially. So, the first step mentioned in the layout denotes what
comes into the supply chain and the last step denotes what goes out.
*/
func GetSummaryLink(layout Layout, stepsMetadataReduced map[string]Metablock,
stepName string) (Metablock, error) {
func GetSummaryLink(layout Layout, stepsMetadataReduced map[string]Metadata,
stepName string, useDSSE bool) (Metadata, error) {
var summaryLink Link
var result Metablock
if len(layout.Steps) > 0 {
firstStepLink := stepsMetadataReduced[layout.Steps[0].Name]
lastStepLink := stepsMetadataReduced[layout.Steps[len(layout.Steps)-1].Name]
summaryLink.Materials = firstStepLink.Signed.(Link).Materials
summaryLink.Materials = firstStepLink.GetPayload().(Link).Materials
summaryLink.Name = stepName
summaryLink.Type = firstStepLink.Signed.(Link).Type
summaryLink.Type = firstStepLink.GetPayload().(Link).Type
summaryLink.Products = lastStepLink.Signed.(Link).Products
summaryLink.ByProducts = lastStepLink.Signed.(Link).ByProducts
summaryLink.Products = lastStepLink.GetPayload().(Link).Products
summaryLink.ByProducts = lastStepLink.GetPayload().(Link).ByProducts
// Using the last command of the sublayout as the command
// of the summary link can be misleading. Is it necessary to
// include all the commands executed as part of sublayout?
summaryLink.Command = lastStepLink.Signed.(Link).Command
summaryLink.Command = lastStepLink.GetPayload().(Link).Command
}
result.Signed = summaryLink
if useDSSE {
env := &Envelope{}
if err := env.SetPayload(summaryLink); err != nil {
return nil, err
}
return result, nil
return env, nil
}
return &Metablock{Signed: summaryLink}, nil
}
/*
@@ -731,11 +738,11 @@ so, recursively resolves it and replaces it with a summary link summarizing the
steps carried out in the sublayout.
*/
func VerifySublayouts(layout Layout,
stepsMetadataVerified map[string]map[string]Metablock,
superLayoutLinkPath string, intermediatePems [][]byte, lineNormalization bool) (map[string]map[string]Metablock, error) {
stepsMetadataVerified map[string]map[string]Metadata,
superLayoutLinkPath string, intermediatePems [][]byte, lineNormalization bool) (map[string]map[string]Metadata, error) {
for stepName, linkData := range stepsMetadataVerified {
for keyID, metadata := range linkData {
if _, ok := metadata.Signed.(Layout); ok {
if _, ok := metadata.GetPayload().(Layout); ok {
layoutKeys := make(map[string]Key)
layoutKeys[keyID] = layout.Keys[keyID]
@@ -861,55 +868,60 @@ Metablock object.
NOTE: Artifact rules of type "create", "modify"
and "delete" are currently not supported.
*/
func InTotoVerify(layoutMb Metablock, layoutKeys map[string]Key,
func InTotoVerify(layoutEnv Metadata, layoutKeys map[string]Key,
linkDir string, stepName string, parameterDictionary map[string]string, intermediatePems [][]byte, lineNormalization bool) (
Metablock, error) {
var summaryLink Metablock
var err error
Metadata, error) {
// Verify root signatures
if err := VerifyLayoutSignatures(layoutMb, layoutKeys); err != nil {
return summaryLink, err
if err := VerifyLayoutSignatures(layoutEnv, layoutKeys); err != nil {
return nil, err
}
// Extract the layout from its Metablock container (for further processing)
layout := layoutMb.Signed.(Layout)
useDSSE := false
if _, ok := layoutEnv.(*Envelope); ok {
useDSSE = true
}
// Extract the layout from its Metadata container (for further processing)
layout, ok := layoutEnv.GetPayload().(Layout)
if !ok {
return nil, ErrNotLayout
}
// Verify layout expiration
if err := VerifyLayoutExpiration(layout); err != nil {
return summaryLink, err
return nil, err
}
// Substitute parameters in layout
layout, err = SubstituteParameters(layout, parameterDictionary)
layout, err := SubstituteParameters(layout, parameterDictionary)
if err != nil {
return summaryLink, err
return nil, err
}
rootCertPool, intermediateCertPool, err := LoadLayoutCertificates(layout, intermediatePems)
if err != nil {
return summaryLink, err
return nil, err
}
// Load links for layout
stepsMetadata, err := LoadLinksForLayout(layout, linkDir)
if err != nil {
return summaryLink, err
return nil, err
}
// Verify link signatures
stepsMetadataVerified, err := VerifyLinkSignatureThesholds(layout,
stepsMetadata, rootCertPool, intermediateCertPool)
if err != nil {
return summaryLink, err
return nil, err
}
// Verify and resolve sublayouts
stepsSublayoutVerified, err := VerifySublayouts(layout,
stepsMetadataVerified, linkDir, intermediatePems, lineNormalization)
if err != nil {
return summaryLink, err
return nil, err
}
// Verify command alignment (WARNING only)
@@ -922,18 +934,18 @@ func InTotoVerify(layoutMb Metablock, layoutKeys map[string]Key,
stepsMetadataReduced, err := ReduceStepsMetadata(layout,
stepsSublayoutVerified)
if err != nil {
return summaryLink, err
return nil, err
}
// Verify artifact rules
if err = VerifyArtifacts(layout.stepsAsInterfaceSlice(),
stepsMetadataReduced); err != nil {
return summaryLink, err
return nil, err
}
inspectionMetadata, err := RunInspections(layout, "", lineNormalization)
inspectionMetadata, err := RunInspections(layout, "", lineNormalization, useDSSE)
if err != nil {
return summaryLink, err
return nil, err
}
// Add steps metadata to inspection metadata, because inspection artifact
@@ -944,51 +956,48 @@ func InTotoVerify(layoutMb Metablock, layoutKeys map[string]Key,
if err = VerifyArtifacts(layout.inspectAsInterfaceSlice(),
inspectionMetadata); err != nil {
return summaryLink, err
return nil, err
}
summaryLink, err = GetSummaryLink(layout, stepsMetadataReduced, stepName)
summaryLink, err := GetSummaryLink(layout, stepsMetadataReduced, stepName, useDSSE)
if err != nil {
return summaryLink, err
return nil, err
}
return summaryLink, nil
}
/*
InTotoVerifyWithDirectory provides the same functionality as IntotoVerify, but
InTotoVerifyWithDirectory provides the same functionality as InTotoVerify, but
adds the possibility to select a local directory from where the inspections are run.
*/
func InTotoVerifyWithDirectory(layoutMb Metablock, layoutKeys map[string]Key,
func InTotoVerifyWithDirectory(layoutEnv Metadata, layoutKeys map[string]Key,
linkDir string, runDir string, stepName string, parameterDictionary map[string]string, intermediatePems [][]byte, lineNormalization bool) (
Metablock, error) {
var summaryLink Metablock
var err error
Metadata, error) {
// runDir sanity checks
// check if path exists
info, err := os.Stat(runDir)
if err != nil {
return Metablock{}, err
return nil, err
}
// check if runDir is a symlink
if info.Mode()&os.ModeSymlink == os.ModeSymlink {
return Metablock{}, ErrInspectionRunDirIsSymlink
return nil, ErrInspectionRunDirIsSymlink
}
// check if runDir is writable and a directory
err = isWritable(runDir)
if err != nil {
return Metablock{}, err
return nil, err
}
// check if runDir is empty (we do not want to overwrite files)
// We abuse File.Readdirnames for this action.
f, err := os.Open(runDir)
if err != nil {
return Metablock{}, err
return nil, err
}
defer f.Close()
// We use Readdirnames(1) for performance reasons, one child node
@@ -996,55 +1005,63 @@ func InTotoVerifyWithDirectory(layoutMb Metablock, layoutKeys map[string]Key,
_, err = f.Readdirnames(1)
// if io.EOF gets returned as error the directory is empty
if err == io.EOF {
return Metablock{}, err
return nil, err
}
err = f.Close()
if err != nil {
return Metablock{}, err
return nil, err
}
// Verify root signatures
if err := VerifyLayoutSignatures(layoutMb, layoutKeys); err != nil {
return summaryLink, err
if err := VerifyLayoutSignatures(layoutEnv, layoutKeys); err != nil {
return nil, err
}
// Extract the layout from its Metablock container (for further processing)
layout := layoutMb.Signed.(Layout)
useDSSE := false
if _, ok := layoutEnv.(*Envelope); ok {
useDSSE = true
}
// Extract the layout from its Metadata container (for further processing)
layout, ok := layoutEnv.GetPayload().(Layout)
if !ok {
return nil, ErrNotLayout
}
// Verify layout expiration
if err := VerifyLayoutExpiration(layout); err != nil {
return summaryLink, err
return nil, err
}
// Substitute parameters in layout
layout, err = SubstituteParameters(layout, parameterDictionary)
if err != nil {
return summaryLink, err
return nil, err
}
rootCertPool, intermediateCertPool, err := LoadLayoutCertificates(layout, intermediatePems)
if err != nil {
return summaryLink, err
return nil, err
}
// Load links for layout
stepsMetadata, err := LoadLinksForLayout(layout, linkDir)
if err != nil {
return summaryLink, err
return nil, err
}
// Verify link signatures
stepsMetadataVerified, err := VerifyLinkSignatureThesholds(layout,
stepsMetadata, rootCertPool, intermediateCertPool)
if err != nil {
return summaryLink, err
return nil, err
}
// Verify and resolve sublayouts
stepsSublayoutVerified, err := VerifySublayouts(layout,
stepsMetadataVerified, linkDir, intermediatePems, lineNormalization)
if err != nil {
return summaryLink, err
return nil, err
}
// Verify command alignment (WARNING only)
@@ -1057,18 +1074,18 @@ func InTotoVerifyWithDirectory(layoutMb Metablock, layoutKeys map[string]Key,
stepsMetadataReduced, err := ReduceStepsMetadata(layout,
stepsSublayoutVerified)
if err != nil {
return summaryLink, err
return nil, err
}
// Verify artifact rules
if err = VerifyArtifacts(layout.stepsAsInterfaceSlice(),
stepsMetadataReduced); err != nil {
return summaryLink, err
return nil, err
}
inspectionMetadata, err := RunInspections(layout, runDir, lineNormalization)
inspectionMetadata, err := RunInspections(layout, runDir, lineNormalization, useDSSE)
if err != nil {
return summaryLink, err
return nil, err
}
// Add steps metadata to inspection metadata, because inspection artifact
@@ -1079,12 +1096,12 @@ func InTotoVerifyWithDirectory(layoutMb Metablock, layoutKeys map[string]Key,
if err = VerifyArtifacts(layout.inspectAsInterfaceSlice(),
inspectionMetadata); err != nil {
return summaryLink, err
return nil, err
}
summaryLink, err = GetSummaryLink(layout, stepsMetadataReduced, stepName)
summaryLink, err := GetSummaryLink(layout, stepsMetadataReduced, stepName, useDSSE)
if err != nil {
return summaryLink, err
return nil, err
}
return summaryLink, nil
vendor/github.com/moby/buildkit/api/services/control/control.pb.go
Generated Vendored
+372 -660
View File
File diff suppressed because it is too large Load Diff
vendor/github.com/moby/buildkit/api/types/worker.pb.go
Generated Vendored
+65 -110
View File
@@ -1,6 +1,6 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.35.2
// protoc-gen-go v1.36.6
// protoc v3.11.4
// source: github.com/moby/buildkit/api/types/worker.proto
@@ -12,6 +12,7 @@ import (
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
unsafe "unsafe"
)
const (
@@ -22,16 +23,15 @@ const (
)
type WorkerRecord struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
ID string `protobuf:"bytes,1,opt,name=ID,proto3" json:"ID,omitempty"`
Labels map[string]string `protobuf:"bytes,2,rep,name=Labels,proto3" json:"Labels,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
Platforms []*pb.Platform `protobuf:"bytes,3,rep,name=platforms,proto3" json:"platforms,omitempty"`
GCPolicy []*GCPolicy `protobuf:"bytes,4,rep,name=GCPolicy,proto3" json:"GCPolicy,omitempty"`
BuildkitVersion *BuildkitVersion `protobuf:"bytes,5,opt,name=BuildkitVersion,proto3" json:"BuildkitVersion,omitempty"`
CDIDevices []*CDIDevice `protobuf:"bytes,6,rep,name=CDIDevices,proto3" json:"CDIDevices,omitempty"`
state protoimpl.MessageState `protogen:"open.v1"`
ID string `protobuf:"bytes,1,opt,name=ID,proto3" json:"ID,omitempty"`
Labels map[string]string `protobuf:"bytes,2,rep,name=Labels,proto3" json:"Labels,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
Platforms []*pb.Platform `protobuf:"bytes,3,rep,name=platforms,proto3" json:"platforms,omitempty"`
GCPolicy []*GCPolicy `protobuf:"bytes,4,rep,name=GCPolicy,proto3" json:"GCPolicy,omitempty"`
BuildkitVersion *BuildkitVersion `protobuf:"bytes,5,opt,name=BuildkitVersion,proto3" json:"BuildkitVersion,omitempty"`
CDIDevices []*CDIDevice `protobuf:"bytes,6,rep,name=CDIDevices,proto3" json:"CDIDevices,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *WorkerRecord) Reset() {
@@ -107,17 +107,16 @@ func (x *WorkerRecord) GetCDIDevices() []*CDIDevice {
}
type GCPolicy struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
All bool `protobuf:"varint,1,opt,name=all,proto3" json:"all,omitempty"`
KeepDuration int64 `protobuf:"varint,2,opt,name=keepDuration,proto3" json:"keepDuration,omitempty"`
Filters []string `protobuf:"bytes,4,rep,name=filters,proto3" json:"filters,omitempty"`
state protoimpl.MessageState `protogen:"open.v1"`
All bool `protobuf:"varint,1,opt,name=all,proto3" json:"all,omitempty"`
KeepDuration int64 `protobuf:"varint,2,opt,name=keepDuration,proto3" json:"keepDuration,omitempty"`
Filters []string `protobuf:"bytes,4,rep,name=filters,proto3" json:"filters,omitempty"`
// reservedSpace was renamed from freeBytes
ReservedSpace int64 `protobuf:"varint,3,opt,name=reservedSpace,proto3" json:"reservedSpace,omitempty"`
MaxUsedSpace int64 `protobuf:"varint,5,opt,name=maxUsedSpace,proto3" json:"maxUsedSpace,omitempty"`
MinFreeSpace int64 `protobuf:"varint,6,opt,name=minFreeSpace,proto3" json:"minFreeSpace,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *GCPolicy) Reset() {
@@ -193,13 +192,12 @@ func (x *GCPolicy) GetMinFreeSpace() int64 {
}
type BuildkitVersion struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
state protoimpl.MessageState `protogen:"open.v1"`
Package string `protobuf:"bytes,1,opt,name=package,proto3" json:"package,omitempty"`
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"`
Revision string `protobuf:"bytes,3,opt,name=revision,proto3" json:"revision,omitempty"`
unknownFields protoimpl.UnknownFields
Package string `protobuf:"bytes,1,opt,name=package,proto3" json:"package,omitempty"`
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"`
Revision string `protobuf:"bytes,3,opt,name=revision,proto3" json:"revision,omitempty"`
sizeCache protoimpl.SizeCache
}
func (x *BuildkitVersion) Reset() {
@@ -254,14 +252,13 @@ func (x *BuildkitVersion) GetRevision() string {
}
type CDIDevice struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
state protoimpl.MessageState `protogen:"open.v1"`
Name string `protobuf:"bytes,1,opt,name=Name,proto3" json:"Name,omitempty"`
AutoAllow bool `protobuf:"varint,2,opt,name=AutoAllow,proto3" json:"AutoAllow,omitempty"`
Annotations map[string]string `protobuf:"bytes,3,rep,name=Annotations,proto3" json:"Annotations,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
OnDemand bool `protobuf:"varint,4,opt,name=OnDemand,proto3" json:"OnDemand,omitempty"`
unknownFields protoimpl.UnknownFields
Name string `protobuf:"bytes,1,opt,name=Name,proto3" json:"Name,omitempty"`
AutoAllow bool `protobuf:"varint,2,opt,name=AutoAllow,proto3" json:"AutoAllow,omitempty"`
Annotations map[string]string `protobuf:"bytes,3,rep,name=Annotations,proto3" json:"Annotations,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
OnDemand bool `protobuf:"varint,4,opt,name=OnDemand,proto3" json:"OnDemand,omitempty"`
sizeCache protoimpl.SizeCache
}
func (x *CDIDevice) Reset() {
@@ -324,90 +321,49 @@ func (x *CDIDevice) GetOnDemand() bool {
var File_github_com_moby_buildkit_api_types_worker_proto protoreflect.FileDescriptor
var file_github_com_moby_buildkit_api_types_worker_proto_rawDesc = []byte{
0x0a, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x6d, 0x6f, 0x62,
0x79, 0x2f, 0x62, 0x75, 0x69, 0x6c, 0x64, 0x6b, 0x69, 0x74, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x74,
0x79, 0x70, 0x65, 0x73, 0x2f, 0x77, 0x6f, 0x72, 0x6b, 0x65, 0x72, 0x2e, 0x70, 0x72, 0x6f, 0x74,
0x6f, 0x12, 0x16, 0x6d, 0x6f, 0x62, 0x79, 0x2e, 0x62, 0x75, 0x69, 0x6c, 0x64, 0x6b, 0x69, 0x74,
0x2e, 0x76, 0x31, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x1a, 0x2c, 0x67, 0x69, 0x74, 0x68, 0x75,
0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x6d, 0x6f, 0x62, 0x79, 0x2f, 0x62, 0x75, 0x69, 0x6c, 0x64,
0x6b, 0x69, 0x74, 0x2f, 0x73, 0x6f, 0x6c, 0x76, 0x65, 0x72, 0x2f, 0x70, 0x62, 0x2f, 0x6f, 0x70,
0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0xa3, 0x03, 0x0a, 0x0c, 0x57, 0x6f, 0x72, 0x6b,
0x65, 0x72, 0x52, 0x65, 0x63, 0x6f, 0x72, 0x64, 0x12, 0x0e, 0x0a, 0x02, 0x49, 0x44, 0x18, 0x01,
0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x49, 0x44, 0x12, 0x48, 0x0a, 0x06, 0x4c, 0x61, 0x62, 0x65,
0x6c, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x30, 0x2e, 0x6d, 0x6f, 0x62, 0x79, 0x2e,
0x62, 0x75, 0x69, 0x6c, 0x64, 0x6b, 0x69, 0x74, 0x2e, 0x76, 0x31, 0x2e, 0x74, 0x79, 0x70, 0x65,
0x73, 0x2e, 0x57, 0x6f, 0x72, 0x6b, 0x65, 0x72, 0x52, 0x65, 0x63, 0x6f, 0x72, 0x64, 0x2e, 0x4c,
0x61, 0x62, 0x65, 0x6c, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x06, 0x4c, 0x61, 0x62, 0x65,
0x6c, 0x73, 0x12, 0x2a, 0x0a, 0x09, 0x70, 0x6c, 0x61, 0x74, 0x66, 0x6f, 0x72, 0x6d, 0x73, 0x18,
0x03, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x0c, 0x2e, 0x70, 0x62, 0x2e, 0x50, 0x6c, 0x61, 0x74, 0x66,
0x6f, 0x72, 0x6d, 0x52, 0x09, 0x70, 0x6c, 0x61, 0x74, 0x66, 0x6f, 0x72, 0x6d, 0x73, 0x12, 0x3c,
0x0a, 0x08, 0x47, 0x43, 0x50, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b,
0x32, 0x20, 0x2e, 0x6d, 0x6f, 0x62, 0x79, 0x2e, 0x62, 0x75, 0x69, 0x6c, 0x64, 0x6b, 0x69, 0x74,
0x2e, 0x76, 0x31, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x47, 0x43, 0x50, 0x6f, 0x6c, 0x69,
0x63, 0x79, 0x52, 0x08, 0x47, 0x43, 0x50, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x12, 0x51, 0x0a, 0x0f,
0x42, 0x75, 0x69, 0x6c, 0x64, 0x6b, 0x69, 0x74, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18,
0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x6d, 0x6f, 0x62, 0x79, 0x2e, 0x62, 0x75, 0x69,
0x6c, 0x64, 0x6b, 0x69, 0x74, 0x2e, 0x76, 0x31, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x42,
0x75, 0x69, 0x6c, 0x64, 0x6b, 0x69, 0x74, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x52, 0x0f,
0x42, 0x75, 0x69, 0x6c, 0x64, 0x6b, 0x69, 0x74, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x12,
0x41, 0x0a, 0x0a, 0x43, 0x44, 0x49, 0x44, 0x65, 0x76, 0x69, 0x63, 0x65, 0x73, 0x18, 0x06, 0x20,
0x03, 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x6d, 0x6f, 0x62, 0x79, 0x2e, 0x62, 0x75, 0x69, 0x6c, 0x64,
0x6b, 0x69, 0x74, 0x2e, 0x76, 0x31, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x43, 0x44, 0x49,
0x44, 0x65, 0x76, 0x69, 0x63, 0x65, 0x52, 0x0a, 0x43, 0x44, 0x49, 0x44, 0x65, 0x76, 0x69, 0x63,
0x65, 0x73, 0x1a, 0x39, 0x0a, 0x0b, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x45, 0x6e, 0x74, 0x72,
0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03,
0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01,
0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, 0xc8, 0x01,
0x0a, 0x08, 0x47, 0x43, 0x50, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x61, 0x6c,
0x6c, 0x18, 0x01, 0x20, 0x01, 0x28, 0x08, 0x52, 0x03, 0x61, 0x6c, 0x6c, 0x12, 0x22, 0x0a, 0x0c,
0x6b, 0x65, 0x65, 0x70, 0x44, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01,
0x28, 0x03, 0x52, 0x0c, 0x6b, 0x65, 0x65, 0x70, 0x44, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e,
0x12, 0x18, 0x0a, 0x07, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28,
0x09, 0x52, 0x07, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x73, 0x12, 0x24, 0x0a, 0x0d, 0x72, 0x65,
0x73, 0x65, 0x72, 0x76, 0x65, 0x64, 0x53, 0x70, 0x61, 0x63, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28,
0x03, 0x52, 0x0d, 0x72, 0x65, 0x73, 0x65, 0x72, 0x76, 0x65, 0x64, 0x53, 0x70, 0x61, 0x63, 0x65,
0x12, 0x22, 0x0a, 0x0c, 0x6d, 0x61, 0x78, 0x55, 0x73, 0x65, 0x64, 0x53, 0x70, 0x61, 0x63, 0x65,
0x18, 0x05, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0c, 0x6d, 0x61, 0x78, 0x55, 0x73, 0x65, 0x64, 0x53,
0x70, 0x61, 0x63, 0x65, 0x12, 0x22, 0x0a, 0x0c, 0x6d, 0x69, 0x6e, 0x46, 0x72, 0x65, 0x65, 0x53,
0x70, 0x61, 0x63, 0x65, 0x18, 0x06, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0c, 0x6d, 0x69, 0x6e, 0x46,
0x72, 0x65, 0x65, 0x53, 0x70, 0x61, 0x63, 0x65, 0x22, 0x61, 0x0a, 0x0f, 0x42, 0x75, 0x69, 0x6c,
0x64, 0x6b, 0x69, 0x74, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x12, 0x18, 0x0a, 0x07, 0x70,
0x61, 0x63, 0x6b, 0x61, 0x67, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x70, 0x61,
0x63, 0x6b, 0x61, 0x67, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e,
0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x12,
0x1a, 0x0a, 0x08, 0x72, 0x65, 0x76, 0x69, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x03, 0x20, 0x01, 0x28,
0x09, 0x52, 0x08, 0x72, 0x65, 0x76, 0x69, 0x73, 0x69, 0x6f, 0x6e, 0x22, 0xef, 0x01, 0x0a, 0x09,
0x43, 0x44, 0x49, 0x44, 0x65, 0x76, 0x69, 0x63, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x4e, 0x61, 0x6d,
0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x4e, 0x61, 0x6d, 0x65, 0x12, 0x1c, 0x0a,
0x09, 0x41, 0x75, 0x74, 0x6f, 0x41, 0x6c, 0x6c, 0x6f, 0x77, 0x18, 0x02, 0x20, 0x01, 0x28, 0x08,
0x52, 0x09, 0x41, 0x75, 0x74, 0x6f, 0x41, 0x6c, 0x6c, 0x6f, 0x77, 0x12, 0x54, 0x0a, 0x0b, 0x41,
0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x0b,
0x32, 0x32, 0x2e, 0x6d, 0x6f, 0x62, 0x79, 0x2e, 0x62, 0x75, 0x69, 0x6c, 0x64, 0x6b, 0x69, 0x74,
0x2e, 0x76, 0x31, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x43, 0x44, 0x49, 0x44, 0x65, 0x76,
0x69, 0x63, 0x65, 0x2e, 0x41, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x45,
0x6e, 0x74, 0x72, 0x79, 0x52, 0x0b, 0x41, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e,
0x73, 0x12, 0x1a, 0x0a, 0x08, 0x4f, 0x6e, 0x44, 0x65, 0x6d, 0x61, 0x6e, 0x64, 0x18, 0x04, 0x20,
0x01, 0x28, 0x08, 0x52, 0x08, 0x4f, 0x6e, 0x44, 0x65, 0x6d, 0x61, 0x6e, 0x64, 0x1a, 0x3e, 0x0a,
0x10, 0x41, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x45, 0x6e, 0x74, 0x72,
0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03,
0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01,
0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x42, 0x3b, 0x5a,
0x39, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x6d, 0x6f, 0x62, 0x79,
0x2f, 0x62, 0x75, 0x69, 0x6c, 0x64, 0x6b, 0x69, 0x74, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x74, 0x79,
0x70, 0x65, 0x73, 0x3b, 0x6d, 0x6f, 0x62, 0x79, 0x5f, 0x62, 0x75, 0x69, 0x6c, 0x64, 0x6b, 0x69,
0x74, 0x5f, 0x76, 0x31, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74,
0x6f, 0x33,
}
const file_github_com_moby_buildkit_api_types_worker_proto_rawDesc = "" +
"\n" +
"/github.com/moby/buildkit/api/types/worker.proto\x12\x16moby.buildkit.v1.types\x1a,github.com/moby/buildkit/solver/pb/ops.proto\"\xa3\x03\n" +
"\fWorkerRecord\x12\x0e\n" +
"\x02ID\x18\x01 \x01(\tR\x02ID\x12H\n" +
"\x06Labels\x18\x02 \x03(\v20.moby.buildkit.v1.types.WorkerRecord.LabelsEntryR\x06Labels\x12*\n" +
"\tplatforms\x18\x03 \x03(\v2\f.pb.PlatformR\tplatforms\x12<\n" +
"\bGCPolicy\x18\x04 \x03(\v2 .moby.buildkit.v1.types.GCPolicyR\bGCPolicy\x12Q\n" +
"\x0fBuildkitVersion\x18\x05 \x01(\v2'.moby.buildkit.v1.types.BuildkitVersionR\x0fBuildkitVersion\x12A\n" +
"\n" +
"CDIDevices\x18\x06 \x03(\v2!.moby.buildkit.v1.types.CDIDeviceR\n" +
"CDIDevices\x1a9\n" +
"\vLabelsEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xc8\x01\n" +
"\bGCPolicy\x12\x10\n" +
"\x03all\x18\x01 \x01(\bR\x03all\x12\"\n" +
"\fkeepDuration\x18\x02 \x01(\x03R\fkeepDuration\x12\x18\n" +
"\afilters\x18\x04 \x03(\tR\afilters\x12$\n" +
"\rreservedSpace\x18\x03 \x01(\x03R\rreservedSpace\x12\"\n" +
"\fmaxUsedSpace\x18\x05 \x01(\x03R\fmaxUsedSpace\x12\"\n" +
"\fminFreeSpace\x18\x06 \x01(\x03R\fminFreeSpace\"a\n" +
"\x0fBuildkitVersion\x12\x18\n" +
"\apackage\x18\x01 \x01(\tR\apackage\x12\x18\n" +
"\aversion\x18\x02 \x01(\tR\aversion\x12\x1a\n" +
"\brevision\x18\x03 \x01(\tR\brevision\"\xef\x01\n" +
"\tCDIDevice\x12\x12\n" +
"\x04Name\x18\x01 \x01(\tR\x04Name\x12\x1c\n" +
"\tAutoAllow\x18\x02 \x01(\bR\tAutoAllow\x12T\n" +
"\vAnnotations\x18\x03 \x03(\v22.moby.buildkit.v1.types.CDIDevice.AnnotationsEntryR\vAnnotations\x12\x1a\n" +
"\bOnDemand\x18\x04 \x01(\bR\bOnDemand\x1a>\n" +
"\x10AnnotationsEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01B;Z9github.com/moby/buildkit/api/types;moby_buildkit_v1_typesb\x06proto3"
var (
file_github_com_moby_buildkit_api_types_worker_proto_rawDescOnce sync.Once
file_github_com_moby_buildkit_api_types_worker_proto_rawDescData = file_github_com_moby_buildkit_api_types_worker_proto_rawDesc
file_github_com_moby_buildkit_api_types_worker_proto_rawDescData []byte
)
func file_github_com_moby_buildkit_api_types_worker_proto_rawDescGZIP() []byte {
file_github_com_moby_buildkit_api_types_worker_proto_rawDescOnce.Do(func() {
file_github_com_moby_buildkit_api_types_worker_proto_rawDescData = protoimpl.X.CompressGZIP(file_github_com_moby_buildkit_api_types_worker_proto_rawDescData)
file_github_com_moby_buildkit_api_types_worker_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_github_com_moby_buildkit_api_types_worker_proto_rawDesc), len(file_github_com_moby_buildkit_api_types_worker_proto_rawDesc)))
})
return file_github_com_moby_buildkit_api_types_worker_proto_rawDescData
}
@@ -445,7 +401,7 @@ func file_github_com_moby_buildkit_api_types_worker_proto_init() {
out := protoimpl.TypeBuilder{
File: protoimpl.DescBuilder{
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: file_github_com_moby_buildkit_api_types_worker_proto_rawDesc,
RawDescriptor: unsafe.Slice(unsafe.StringData(file_github_com_moby_buildkit_api_types_worker_proto_rawDesc), len(file_github_com_moby_buildkit_api_types_worker_proto_rawDesc)),
NumEnums: 0,
NumMessages: 6,
NumExtensions: 0,
@@ -456,7 +412,6 @@ func file_github_com_moby_buildkit_api_types_worker_proto_init() {
MessageInfos: file_github_com_moby_buildkit_api_types_worker_proto_msgTypes,
}.Build()
File_github_com_moby_buildkit_api_types_worker_proto = out.File
file_github_com_moby_buildkit_api_types_worker_proto_rawDesc = nil
file_github_com_moby_buildkit_api_types_worker_proto_goTypes = nil
file_github_com_moby_buildkit_api_types_worker_proto_depIdxs = nil
}

Some files were not shown because too many files have changed in this diff Show More