Data streams API

You can send data to monq using HTTP requests. Collector module provides users with HTTP POST listening point – http(s)://<instance-name>/api/public/cl/v1/stream-data

Request format

POST /api/public/cl/v1/stream-data

Request parameters

Parameter	Type	Required	Description
streamKey	string	yes	The key of the data stream to which you want to send data. The key can be obtained on the "Integrations" page.

streamKey parameter required, if the key is not specified as an HTTP header parameter.

Request header parameters

Parameter	Type	Required	Description
x-smon-stream-key	string	yes	The key of the data stream to which you want to send data. The key can be obtained on the "Integrations" page.
Content-Type	string	yes	It is required to specify the type of data being sent. The following types are allowed for processing: application/json, application/x-ndjson, text/xml, text/plain.

x-smon-stream-key parameter required, if the key is not specified as an HTTP header parameter.

Request body

The body of the request is a string with the data that you want to send for processing.

You can specify a structured data type application/json, application/x-ndjson, text/xml. In this case, when receiving data a validation will be carried out. If the data format does not match the type, the event will be discarded and an error will be returned HTTP StatusCode 400 BadRequest.

For messages of type application/json, application/x-ndjson it is possible to set the field “@timestamp”: “2019-07-25T09:16:39.284622281Z“, in which date of the event will be set. If such a field is specified, then the date of the event in monq will use the value of the field, unless another instruction is specified in the data preprocessor. If the field is missing and is not defined in the preprocessor, then the value of the field will be the date and time of the event.

For messages of type text/xml,text/plain, you need to implement your own handlers (parsers) in the configuration template of the corresponding stream, for subsequent display and processing in the monq system. For example, to write data incoming in the text/plain format to ClickHouse, you need to add a handler in which to convert the incoming data to a JSON object.

labels_add('postRoutings',{'cl.stream-ready-event.new'})
local var_json = {value = source}
next_step(json.encode(var_json))

where the variable source contains data in the format text/plain.

CURL query example

curl --location --request POST 'https://sm.monq.cloud/api/public/cl/v1/stream-data?streamKey=e3b81397-b71a-4f95-9327-eea4c82e17cf' \
--header 'Content-Type: application/json' \
--data-raw '{
  "@timestamp": "2020-04-29T12:12:56.000000+00:00",
  "stream": "stdout",
  "docker": {
    "container_id": "e988087df502e32cdcb16233ca5974398497061a8d6fed1586ec9336cfd63c2b"
  },
  "kubernetes": {
    "container_name": "nginx-ingress-controller",
    "namespace_name": "kube-system",
    "pod_name": "nginx-ingress-controller-c95qd",
    "container_image": "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0",
    "pod_id": "79ff1d4c-89ea-11ea-b298-fabbba8ed4fb",
    "host": "d3-d-node-02",
    "labels": {
      "app": "nginx-ingress",
      "controller-revision-hash": "3781051395",
      "pod-template-generation": "16"
    },
    "master_url": "https://10.16.0.1:443/api",
    "namespace_id": "c7f77c99-d6d7-11e8-bd7f-7ee4f896140a",
    "namespace_labels": { "certmanager_k8s_io/disable-validation": "true" }
  },
  "remote": "10.18.0.81",
  "realip": "10.18.0.81",
  "user": "-",
  "method": "GET",
  "path": "/api/v1/nodes/d3-d-node-02/proxy/metrics",
  "protocol": "HTTP/1.1",
  "code": 200,
  "size": 8172,
  "agent": "Prometheus/2.14.0",
  "request_length": 344,
  "request_time": 0.064,
  "upstream_response_length": 8172,
  "upstream_response_time": 0.065,
  "upstream_status": 200,
  "req_id": "b82905a8aadaca97b7b1662e6d1536f5\n"
}
'

Response codes

Code	Description
200	The request was successfully completed.
400	The data did not pass validation.
422	The data stream with a key {streamKey} not declared in the system.
422	The data stream with a key {streamKey} not active.
422	The data stream with a key {streamKey} pending deletion.
500	An unexpected error occurred while processing a request on the server side.