# Data streams API
You can send data to monq using HTTP requests. Collector module provides users with HTTP POST listening point – http(s)://<instance-name>/api/public/cl/v1/stream-data
# Request format
POST /api/public/cl/v1/stream-data
# Request parameters
Parameter | Type | Required | Description |
---|---|---|---|
streamKey | string | yes | The key of the data stream to which you want to send data. The key can be obtained on the "Integrations" page. |
streamKey
parameter required, if the key is not specified as an HTTP header parameter.
# Request header parameters
Parameter | Type | Required | Description |
---|---|---|---|
x-smon-stream-key | string | yes | The key of the data stream to which you want to send data. The key can be obtained on the "Integrations" page. |
Content-Type | string | yes | It is required to specify the type of data being sent. The following types are allowed for processing: application/json , application/x-ndjson , text/xml , text/plain . |
x-smon-stream-key
parameter required, if the key is not specified as an HTTP header parameter.
# Request body
The body of the request is a string with the data that you want to send for processing.
You can specify a structured data type application/json
, application/x-ndjson
, text/xml
. In this case, when receiving data a validation will be carried out. If the data format does not match the type, the event will be discarded and an error will be returned HTTP StatusCode 400 BadRequest
.
For messages of type application/json
, application/x-ndjson
it is possible to set the field “@timestamp”: “2019-07-25T09:16:39.284622281Z“
, in which date of the event will be set. If such a field is specified, then the date of the event in monq will use the value of the field, unless another instruction is specified in the data preprocessor. If the field is missing and is not defined in the preprocessor, then the value of the field will be the date and time of the event.
For messages of type text/xml
,text/plain
, you need to implement your own handlers (parsers) in the configuration template of the corresponding stream, for subsequent display and processing in the monq system.
For example, to write data incoming in the text/plain
format to ClickHouse, you need to add a handler in which to convert the incoming data to a JSON object.
labels_add('postRoutings',{'cl.stream-ready-event.new'})
local var_json = {value = source}
next_step(json.encode(var_json))
where the variable source
contains data in the format text/plain
.
# CURL query example
curl --location --request POST 'https://sm.monq.cloud/api/public/cl/v1/stream-data?streamKey=e3b81397-b71a-4f95-9327-eea4c82e17cf' \
--header 'Content-Type: application/json' \
--data-raw '{
"@timestamp": "2020-04-29T12:12:56.000000+00:00",
"stream": "stdout",
"docker": {
"container_id": "e988087df502e32cdcb16233ca5974398497061a8d6fed1586ec9336cfd63c2b"
},
"kubernetes": {
"container_name": "nginx-ingress-controller",
"namespace_name": "kube-system",
"pod_name": "nginx-ingress-controller-c95qd",
"container_image": "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0",
"pod_id": "79ff1d4c-89ea-11ea-b298-fabbba8ed4fb",
"host": "d3-d-node-02",
"labels": {
"app": "nginx-ingress",
"controller-revision-hash": "3781051395",
"pod-template-generation": "16"
},
"master_url": "https://10.16.0.1:443/api",
"namespace_id": "c7f77c99-d6d7-11e8-bd7f-7ee4f896140a",
"namespace_labels": { "certmanager_k8s_io/disable-validation": "true" }
},
"remote": "10.18.0.81",
"realip": "10.18.0.81",
"user": "-",
"method": "GET",
"path": "/api/v1/nodes/d3-d-node-02/proxy/metrics",
"protocol": "HTTP/1.1",
"code": 200,
"size": 8172,
"agent": "Prometheus/2.14.0",
"request_length": 344,
"request_time": 0.064,
"upstream_response_length": 8172,
"upstream_response_time": 0.065,
"upstream_status": 200,
"req_id": "b82905a8aadaca97b7b1662e6d1536f5\n"
}
'
# Response codes
Code | Description |
---|---|
200 | The request was successfully completed. |
400 | The data did not pass validation. |
422 | The data stream with a key {streamKey} not declared in the system. |
422 | The data stream with a key {streamKey} not active. |
422 | The data stream with a key {streamKey} pending deletion. |
500 | An unexpected error occurred while processing a request on the server side. |