# Confmap

<!-- status autogenerated section -->
| Status        |           |
| ------------- |-----------|
| Stability     | [stable]: logs, metrics, traces   |
| Issues        | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector?query=is%3Aissue%20is%3Aopen%20label%3Apkg%2Fconfmap%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector/issues?q=is%3Aopen+is%3Aissue+label%3Apkg%2Fconfmap) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector?query=is%3Aissue%20is%3Aclosed%20label%3Apkg%2Fconfmap%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector/issues?q=is%3Aclosed+is%3Aissue+label%3Apkg%2Fconfmap) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner)    | [@mx-psi](https://www.github.com/mx-psi), [@evan-bradley](https://www.github.com/evan-bradley) |

[stable]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stable
<!-- end autogenerated section -->

# High Level Design

## Conf

The [Conf](confmap.go) represents the raw configuration for a service (e.g. OpenTelemetry Collector).

## Provider

The [Provider](provider.go) provides configuration, and allows to watch/monitor for changes. Any `Provider`
has a `<scheme>` associated with it, and will provide configs for `configURI` that follow the "<scheme>:<opaque_data>" format.
This format is compatible with the URI definition (see [RFC 3986](https://datatracker.ietf.org/doc/html/rfc3986)).
The `<scheme>` MUST be always included in the `configURI`. The scheme for any `Provider` MUST be at least 2
characters long to avoid conflicting with a driver-letter identifier as specified in
[file URI syntax](https://datatracker.ietf.org/doc/html/rfc8089#section-2).

## Converter

The [Converter](converter.go) allows implementing conversion logic for the provided configuration. One of the most
common use-case is to migrate/transform the configuration after a backwards incompatible change.

## Resolver

The `Resolver` handles the use of multiple [Providers](#provider) and [Converters](#converter)
simplifying configuration parsing, monitoring for updates, and the overall life-cycle of the used config providers.
The `Resolver` provides two main functionalities: [Configuration Resolving](#configuration-resolving) and
[Watching for Updates](#watching-for-updates).

### Configuration Resolving

The `Resolver` receives as input a set of `Providers`, a list of `Converters`, and a list of configuration identifier
`configURI` that will be used to generate the resulting, or effective, configuration in the form of a `Conf`,
that can be used by code that is oblivious to the usage of `Providers` and `Converters`.

`Providers` are used to provide an entire configuration when the `configURI` is given directly to the `Resolver`,
or an individual value (partial configuration) when the `configURI` is embedded into the `Conf` as a values using
the syntax `${configURI}`.

**Limitation:**
- When embedding a `${configURI}` the uri cannot contain dollar sign ("$") character unless it embeds another uri.
- The number of URIs is limited to 100.

```terminal
              Resolver                   Provider
   Resolve       │                          │
────────────────►│                          │
                 │                          │
              ┌─ │        Retrieve          │
              │  ├─────────────────────────►│
              │  │          Conf            │
              │  │◄─────────────────────────┤
  foreach     │  │                          │
  configURI   │  ├───┐                      │
              │  │   │Merge                 │
              │  │◄──┘                      │
              └─ │                          │
              ┌─ │        Retrieve          │
              │  ├─────────────────────────►│
              │  │    Partial Conf Value    │
              │  │◄─────────────────────────┤
  foreach     │  │                          │
  embedded    │  │                          │
  configURI   │  ├───┐                      │
              │  │   │Replace               │
              │  │◄──┘                      │
              └─ │                          │
                 │            Converter     │
              ┌─ │     Convert    │         │
              │  ├───────────────►│         │
    foreach   │  │                │         │
   Converter  │  │◄───────────────┤         │
              └─ │                          │
                 │                          │
◄────────────────┤                          │
```

The `Resolve` method proceeds in the following steps:

1. Start with an empty "result" of `Conf` type.
2. For each config URI retrieves individual configurations, and merges it into the "result".
3. For each embedded config URI retrieves individual value, and replaces it into the "result".
4. For each "Converter", call "Convert" for the "result".
5. Return the "result", aka effective, configuration.

#### (Experimental) Append merging strategy for lists

You can opt-in to experimentally combine slices instead of discarding the existing ones by enabling the `confmap.enableMergeAppendOption` feature flag. Lists are appended in the order in which they appear in their configuration sources.
This will **not** become the default in the future, we are still deciding how this should be configured and want your feedback on [this issue](https://github.com/open-telemetry/opentelemetry-collector/issues/8754).

##### Example
Consider the following configs,

```yaml
# main.yaml
receivers:
  otlp/in:
processors:
  attributes/example:
    actions:
      - key: key
        value: "value"
        action: upsert

exporters:
  otlp/out:
extensions:
  file_storage:

service:
  pipelines:
    traces:
      receivers: [ otlp/in ]
      processors: [ attributes/example ]
      exporters: [ otlp/out ]
  extensions: [ file_storage ]
```


```yaml
# extra_extension.yaml
processors:
  batch:
extensions:
  healthcheckv2:

service:
  extensions: [ healthcheckv2 ]
  pipelines:
    traces:
      processors: [ batch ]
```

If you run the Collector with following command,
```
otelcol --config=main.yaml --config=extra_extension.yaml --feature-gates=confmap.enableMergeAppendOption
```
then the final configuration after config resolution will look like following:

```yaml
# main.yaml
receivers:
  otlp/in:
processors:
  attributes/example:
    actions:
      - key: key
        value: "value"
        action: upsert
  batch:
exporters:
  otlp/out:
extensions:
  file_storage:
  healthcheckv2:

service:
  pipelines:
    traces:
      receivers: [ otlp/in ]
      processors: [ attributes/example, batch ]
      exporters: [ otlp/out ]
  extensions: [ file_storage, healthcheckv2 ]
```

Notice that the `service::extensions` list is a combination of both configurations. By default, the value of the last configuration source passed, `extra_extension`, would be used, so the extensions list would be: `service::extensions: [healthcheckv2]`.

> [!NOTE]
> By enabling this feature gate, all the lists in the given configuration will be merged. 

### Watching for Updates
After the configuration was processed, the `Resolver` can be used as a single point to watch for updates in the
configuration retrieved via the `Provider` used to retrieve the “initial” configuration and to generate the “effective” one.

```terminal
         Resolver              Provider
            │                     │
   Watch    │                     │
───────────►│                     │
            │                     │
            .                     .
            .                     .
            .                     .
            │      onChange       │
            │◄────────────────────┤
◄───────────┤                     │

```

The `Resolver` does that by passing an `onChange` func to each `Provider.Retrieve` call and capturing all watch events.

Calling the `onChange` func from a provider triggers the collector to re-resolve new configuration:

```terminal
         Resolver              Provider
            │                     │
   Watch    │                     │
───────────►│                     │
            │                     │
            .                     .
            .                     .
            .                     .
            │      onChange       │
            │◄────────────────────┤
◄───────────┤                     │
            |                     |
  Resolve   │                     │
───────────►│                     │
            │                     │
            │      Retrieve       │
            ├────────────────────►│
            │        Conf         │
            │◄────────────────────┤
◄───────────┤                     │
```

An example of a `Provider` with an `onChange` func that periodically gets notified can be found in provider_test.go as UpdatingProvider

## Troubleshooting

### Null Maps

Due to how our underlying merge library, [koanf](https://github.com/knadh/koanf), behaves, configuration resolution
will treat configuration such as

```yaml
processors:
```

as null, which is a valid value. As a result if you have configuration `A`:

```yaml
receivers:
    nop:

processors:
    nop:

exporters:
    nop:

extensions:
    nop:

service:
    extensions: [nop]
    pipelines:
        traces:
            receivers: [nop]
            processors: [nop]
            exporters: [nop]
```

and configuration `B`:

```yaml
processors:
```

and do `./otelcorecol --config A.yaml --config B.yaml`

The result will be an error:

```
Error: invalid configuration: service::pipelines::traces: references processor "nop" which is not configured
2024/06/10 14:37:14 collector server run finished with error: invalid configuration: service::pipelines::traces: references processor "nop" which is not configured
```

This happens because configuration `B` sets `processors` to null, removing the `nop` processor defined in configuration `A`,
so the `nop` processor referenced in configuration `A`'s pipeline no longer exists.

This situation can be remedied 2 ways:
1. Use `{}` when you want to represent an empty map, such as `processors: {}` instead of `processors:`.
2. Omit configuration like `processors:` from your configuration.
