Publishing

Simplest way to start publishing to Hermes is sending POST request on topic endpoint (currently you can publish only one message per request):

curl -X POST -H "Content-Type: application/json" http://hermes-frontend/topics/group.topic -d '{"message": "Hello world!"}'

Creating group

As data model describes, topics are gathered into topic groups. If you don't have a group yet, you need to create one by sending POST with application/json content type at

/groups

Request body must contain groupName field, which is the name of the group.

Sample request:

{
    "groupName": "my.group"
}

Creating topic

Use Hermes Management REST API to create topic by sending POST request with application/json content type on topics resource:

/topics

Request body must contain at least:

• name: fully qualified name of topic including group name, separated with a dot (see: naming convention)
• description: topic description
• contentType: format of data sent to Kafka, either JSON or AVRO
• retentionTime: time to keep data in Kafka in days
• owner: who's the owner of this topic – id is an identifier meaningful in a given source implementation. Default implementation is Plaintext, where id is simply a label.

Minimal request:

{
    "name": "my.group.my-topic",
    "description": "This is my topic",
    "contentType": "JSON",
    "retentionTime": {
        "duration": 1
    },
    "owner": {
        "source": "Plaintext",
        "id": "My Team"
    }
}

Other options:

Option	Description	Options	Default value
ack	acknowledgement level	ALL, LEADER	LEADER
trackingEnabled	track incoming messages?	-	false

Request that specifies all available options:

{
  "name": "my.group.my-topic",
  "description": "This is my topic",
  "ack": "LEADER",
  "retentionTime": {
    "duration": 1
  },
  "trackingEnabled": false,
  "contentType": "JSON",
  "owner": {
    "source": "Plaintext",
    "id": "My Team"
  }
}

Message format

Each topic has a defined content type that describes the format of data sent to Kafka. At this point Hermes supports messages sent in JSON and AVRO.

JSON

When topic has content type set to JSON it will accept messages in JSON format and they will be stored as JSON in Kafka.

This mode is dedicated for simple use-cases and offers no validation for published messages.

Avro

Avro is the recommended message format for topics in Hermes. It has many advantages over plain JSON, e.g. has built-in message validation (against defined schema) and lowers the volume of data sent to Kafka.

Read detailed documentation for publishing messages in Avro format here.

Response format

Message Id

Response will contain special header: Hermes-Message-Id. This is event UUID generated by Hermes, which can be used to track how event flew through the system.

Response codes

Success status:

• 201 Created - event received and acknowledged by Kafka

Failure statuses:

• 400 Bad Message - message did not pass validation (see docs about each data format for more)
• 404 Not Found - topic does not exist
• 408 Request Timeout - message was not sent to Hermes within timeout (took too much time on the network)
• 500 Internal Server Error - something went terribly bad
• 503 Service Unavailable - node is in shutdown mode

Acknowledgment level

Each topic can define level of acknowledgement (ACK):

• leader ACK - only one Kafka node (leader) needs to acknowledge reception of message
• all ACK - at least min.insync.replicas nodes must acknowledge reception of message

ACK configuration has the following consequences:

• with ACK leader message writes are replicated asynchronously, thus the acknowledgment latency will be low. However, message write may be lost when there is a topic leadership change - e.g. due to rebalance or broker restart.
• with ACK all messages writes are synchronously replicated to replicas. Write acknowledgement latency will be much higher than with leader ACK, it will also have higher variance due to tail latency. However, messages will be persisted as long as the whole replica set does not go down simultaneously.

Publishers are advised to select topic ACK level based on their latency and durability requirements.

Remote DC fallback

Hermes supports a remote datacenter fallback mechanism for multi datacenter deployments.

Fallback is configured on per topic basis, using a fallbackToRemoteDatacenterEnabled property:

PUT /topics/my.group.my-topic

{
  "fallbackToRemoteDatacenterEnabled": true
}

When using this setting for a topic, Hermes will try to send a message to a local datacenter Kafka first and will fall back to remote datacenter Kafka if the local send fails.

Hermes also provides a speculative fallback mechanism which will send messages to remote Kafka if the local Kafka is not responding in a timely manner. Speculative send is performed after frontend.kafka.fail-fast-producer.speculativeSendDelay elapses.

When using remote DC fallback, Hermes attempts to send a message to Kafka for the duration of frontend.handlers.maxPublishRequestDuration property. If after maxPublishRequestDuration Hermes has not received an acknowledgment from Kafka, it will respond with 500 status code to the client.

Table below summarizes remote fallback configuration options:

Option	Scope	Default value
fallbackToRemoteDatacenterEnabled	topic	false
frontend.kafka.fail-fast-producer.speculativeSendDelay	global	250ms
frontend.handlers.maxPublishRequestDuration	global	500ms

Partition assignment

Partition-Key header can be used by publishers to specify Kafka key which will be used for partition assignment for a message. This will ensure that all messages with given Partition-Key will be sent to the same Kafka partition.

Partition assignment is controlled by the Kafka producer's built-in partitioner. The default partitioner behavior can be overridden by setting frontend.kafka.producer.partitionerClass to a fully qualified class name (e.g. org.apache.kafka.clients.producer.RoundRobinPartitioner). See frontend tuning for details.