Skip to content

[FLINK-39365][connector-http] Support custom HTTP headers via ConfigOption #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
featzhang wants to merge 3 commits into
base: main
Choose a base branch
from
Open

[FLINK-39365][connector-http] Support custom HTTP headers via ConfigOption #32

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
71802de
[FLINK-HTTP-3] Support custom HTTP headers via ConfigOption
featzhang Mar 12, 2026
d9412eb
[FLINK-HTTP-3] Document custom HTTP headers feature in connector docs
featzhang Mar 12, 2026
cc10e85
Update connector key from http-sink to http-async-sink
featzhang Mar 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/content.zh/docs/connectors/datastream/http.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ These options are specified on the builder using the setProperty method.

| Option | Required | Description/Value |
|---------------------------------------------------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| connector | required | Specify what connector to use. For HTTP Sink it should be set to _'http-sink'_. |
| connector | required | Specify what connector to use. For HTTP Sink it should be set to _'http-async-sink'_. |
Comment thread
featzhang marked this conversation as resolved.
| format | required | Specify what format to use. |
| url | required | The base URL that should be used for HTTP requests. For example _http://localhost:8080/client_. |
| insert-method | optional | Specify which HTTP method to use in the request. The value should be set either to `POST` or `PUT`. |
Expand Down
45 changes: 37 additions & 8 deletions docs/content.zh/docs/connectors/table/http.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ The HTTP source connector supports [Lookup Joins](https://nightlies.apache.org/f
* [Working with HTTP sink tables](#working-with-http-sink-tables)
* [HTTP Sink](#http-sink)
* [Sink Connector Options](#sink-connector-options)
* [Sink Http headers](#sink-http-headers)
* [Sink table HTTP status codes](#sink-table-http-status-codes)
* [Request submission](#request-submission)
* [Batch submission mode](#batch-submission-mode)
Expand Down Expand Up @@ -206,6 +207,7 @@ Note the options with the prefix _http_ are the HTTP connector specific options,
| http.request.query-param-fields | optional | Used for the `http-generic-json-url` query creator. The names of the fields that will be mapped to query parameters. The parameters are separated by semicolons, such as `param1;param2`. |
| http.request.body-template | optional | Used for the `http-generic-json-url` query creator. A JSON template string for constructing the request body for PUT and POST operations. Use `{{fieldName}}` placeholders to reference top-level columns from the lookup table. Supports creating complex nested JSON structures with both placeholders and literal values. See the [Body Template](#body-template) section for details and examples. |
| http.request.url-map | optional | Used for the `http-generic-json-url` query creator. The map of insert names to column names used as url segments. Parses a string as a map of strings. For example if there are table columns called `customerId` and `orderId`, then specifying value `customerId:cid,orderID:oid` and a url of https://myendpoint/customers/{cid}/orders/{oid} will mean that the url used for the lookup query will dynamically pickup the values for `customerId`, `orderId` and use them in the url e.g. https://myendpoint/customers/cid1/orders/oid1. The expected format of the map is: `key1:value1,key2:value2`. |
| http.source.lookup.header.* | optional | Custom HTTP headers to be added to every lookup request. Each header is specified as an individual property using the prefix `http.source.lookup.header.`, followed by the header name. For example: `'http.source.lookup.header.Content-Type' = 'application/json'`. Multiple headers can be defined by adding multiple properties with this prefix. |

### Query Creators

Expand Down Expand Up @@ -346,11 +348,14 @@ To create a custom format user has to implement Flink's `SerializationSchema` an
`resources/META-INF.services/org.apache.flink.table.factories.Factory` file. This is common Flink mechanism for providing custom implementations for various factories.

### Http headers
It is possible to set HTTP headers that will be added to HTTP request sent by lookup source connector.
Headers are defined via property key `http.source.lookup.header.HEADER_NAME = header value` for example:
`http.source.lookup.header.X-Content-Type-Options = nosniff`.
It is possible to set custom HTTP headers that will be added to every HTTP request sent by the lookup source connector.
Headers are specified as individual properties using the prefix `http.source.lookup.header.`, followed by the header name.
For example: `'http.source.lookup.header.Content-Type' = 'application/json'`.

Headers can be set using http lookup source table DDL. In example below, HTTP request done for `http-lookup` table will contain three headers:
> **Note:** As of [FLINK-HTTP-3], custom HTTP headers are formally supported as a `ConfigOption` (`http.source.lookup.header.*`),
> enabling proper validation and discoverability of header configuration.

Headers can be set using the HTTP lookup source table DDL. In the example below, every HTTP request for the `http-lookup` table will contain three headers:
- `Origin`
- `X-Content-Type-Options`
- `Content-Type`
Expand Down Expand Up @@ -431,7 +436,7 @@ CREATE TABLE http (
id bigint,
some_field string
) WITH (
'connector' = 'http-sink',
'connector' = 'http-async-sink',
'url' = 'http://example.com/myendpoint',
'format' = 'json'
)
Expand All @@ -451,7 +456,7 @@ another format name.

| Option | Required | Description/Value |
|-------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| connector | required | Specify what connector to use. For HTTP Sink it should be set to _'http-sink'_. |
| connector | required | Specify what connector to use. For HTTP Sink it should be set to _'http-async-sink'_. |
| format | required | Specify what format to use. |
| url | required | The base URL that should be used for HTTP requests. For example _http://localhost:8080/client_. |
| insert-method | optional | Specify which HTTP method to use in the request. The value should be set either to `POST` or `PUT`. |
Expand All @@ -472,6 +477,30 @@ another format name.
| http.sink.writer.thread-pool.size | optional | Sets the size of pool thread for HTTP Sink request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 1 thread will be used. |
| http.sink.writer.request.mode | optional | Sets the Http Sink request submission mode. Two modes are available: `single` and `batch`. Defaults to `batch` if not specified. |
| http.sink.request.batch.size | optional | Applicable only for `http.sink.writer.request.mode = batch`. Sets number of individual events/requests that will be submitted as one HTTP request by HTTP sink. The default value is 500 which is same as HTTP Sink `maxBatchSize` |
| http.sink.header.* | optional | Custom HTTP headers to be added to every sink request. Each header is specified as an individual property using the prefix `http.sink.header.`, followed by the header name. For example: `'http.sink.header.Content-Type' = 'application/json'`. Multiple headers can be defined by adding multiple properties with this prefix. |

### Sink Http headers
It is possible to set custom HTTP headers that will be added to every HTTP request sent by the HTTP sink connector.
Headers are specified as individual properties using the prefix `http.sink.header.`, followed by the header name.

> **Note:** As of [FLINK-HTTP-3], custom HTTP headers are formally supported as a `ConfigOption` (`http.sink.header.*`),
> enabling proper validation and discoverability of header configuration.

Headers can be set using the HTTP sink table DDL:

```roomsql
CREATE TABLE http_sink (
id bigint,
some_field string
) WITH (
'connector' = 'http-async-sink',
'url' = 'http://example.com/myendpoint',
'format' = 'json',
'http.sink.header.Content-Type' = 'application/json',
'http.sink.header.Authorization' = 'Bearer my-token',

@davidradl davidradl Apr 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fyi there is OIDC support for lookup joins - which uses bearer tokens under the covers. This is another candidate to be implemented for sinks.

'http.sink.header.X-Custom-Header' = 'my-value'
)
```

### Sink table HTTP status codes
You can configure a list of HTTP status codes that should be treated as errors for HTTP sink table.
Expand Down Expand Up @@ -541,7 +570,7 @@ CREATE TABLE http (
id bigint,
some_field string
) WITH (
'connector' = 'http-sink',
'connector' = 'http-async-sink',
'url' = 'http://example.com/myendpoint',
'format' = 'json',
'http.sink.request.batch.size' = '50'
Expand All @@ -559,7 +588,7 @@ CREATE TABLE http (
id bigint,
some_field string
) WITH (
'connector' = 'http-sink',
'connector' = 'http-async-sink',
'url' = 'http://example.com/myendpoint',
'format' = 'json',
'http.sink.writer.request.mode' = 'single'
Expand Down
2 changes: 1 addition & 1 deletion docs/content/docs/connectors/datastream/http.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ These options are specified on the builder using the setProperty method.

| Option | Required | Description/Value |
|---------------------------------------------------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| connector | required | Specify what connector to use. For HTTP Sink it should be set to _'http-sink'_. |
| connector | required | Specify what connector to use. For HTTP Sink it should be set to _'http-async-sink'_. |
| format | required | Specify what format to use. |
| url | required | The base URL that should be used for HTTP requests. For example _http://localhost:8080/client_. |
| insert-method | optional | Specify which HTTP method to use in the request. The value should be set either to `POST` or `PUT`. |
Expand Down
Loading
Loading