Fixed Header Format

Note: All data values will be in little-endian order, that is, lower order bytes preceed higher order bytes. A 16 bit word would be presented on the wire as LSB, then MSB.
The only exception to this the length field of a UTF-encoded string, which is defined by the standard to be network-ordered, i.e. big-endian.
 

A fixed length header will always be present for each message. The fixed length header will consist of 2 bytes containing the message Type, Flags, and one byte of the Remaining Length field.

The format of the fixed length header is as follows:
 
 

bit	7	6	5	4	3	2	1	0
byte 1	Message Type				DUP flag	QoS level		RETAIN
byte 2	Remaining Length

Message Type Field

The Message Type field is represented as a 4 bit unsigned value. The enumerations of the Type field defined at this version of the protocol are as follows:
 
 

Note that message type 0 is not a valid message code.

Flags Fields

The interpretation of these flags is as follows:
 

DUP: Duplicate delivery

The DUP bit will be set any time the client or broker tries to deliver a PUBLISH message that has already been sent. This only applies to messages of QoS > 0 which require acknowledgement (see below). By definition, when the DUP bit is set, the Variable Header will include a Message Identifier.
 

QoS: Quality of Service

The QoS bits are used to indicate the level of assurance of delivery of a PUBLISH message. There are four possible values which can be represented by the two QoS bits. The QoS levels are defined as follows:
 

RETAIN: RETAIN Flag

The RETAIN flag is an indication to the broker that this message should be held, if possible, in the broker, and should be sent to any new subscriber to this Topic as an initial message. This allows a complete "current state" of a number of Topics to be quickly established by a new client upon connecting to the broker. This is particularly useful if publishers are only sending messages on a "Report By Exception" basis: it may be a very long time before a new subscriber receives any data on a particular Topic. The data is  known as the "Retained", or "Last Known Good" (LKG) value.

After a SUBSCRIBE to one or more Topics, a subscriber will receive a SUBACK, and then one message for each of the newly subscribed Topics for which there is currently a Retained value. The Retained value is published from the broker to the subscriber with  the RETAIN flag set, and with the same QoS with which it was originally published, and so will be subject to the usual QoS delivery assurances. The RETAIN flag is set in the message to the subscribers to distinguish it from "live" data, so that it can be handled appropriately by the subscriber.

Note that there is no guarantee that a previous Retained PUBLISH to the broker will still be held by the broker, and so the subscriber might not receive an initial Retained PUBLISH on a Topic.
 
 

Remaining Length Field

The Remaining Length field represents the number of bytes remaining withing the current message, including data in the Variable Header portion of the message, and the user-defined Payload.
The field is encoded using a variable-length scheme which allows the use of just a single byte for message lengths up to 127 bytes, but which also allows larger messages to be carried when required. The encoding scheme is as follows:

Seven bits of each byte are used to encode the Remaining Length data, and the eighth bit in each byte is used to indicate whether or not there are any following bytes in the representation. Each byte thus encodes 128 values and a "continuation bit".

For example, the number 64 decimal is encoded as by a single byte, decimal value 64, hex 0x40.
The number 321 decimal (=128x2 + 65) is encoded as two bytes, least significant first:
First byte: 2+128 = 130 (note the top bit being set to indicate there is at least one following byte)
Second byte: 65

This version of the protocol limits the number of bytes in the representation to a maximum of four (4). This permits a single message of up to 268,435,455 (256MB) to be sent. It is felt that this will be more than adequate for the foreseeable future.
The representation of this number on the wire would be 0xFF, 0xFF, 0xFF, 0x7F.

Using this encoding scheme, the Remaining Length values which can be represented by increasing numbers of bytes is as follows:
 
 

digits	from	to
1	0 (0x00)	127 (0x7F)
2	128 (0x80, 0x01)	16,383 (0xFF, 0x7F)
3	16,384 (0x80, 0x80, 0x01)	2,097,151 (0xFF, 0xFF, 0x7F)
4	2,097,152 (0x80, 0x80, 0x80, 0x01)	268,435,455 (0xFF, 0xFF, 0xFF, 0x7F)

The algorithm for encoding a decimal number into this format is quite straightforward, and looks like this:

X is the number to convert to variable length encoding scheme
 

do

digit = X MOD 128
x = X DIV 128

// if there are more digits to encode, set the top bit of this digit
if ( X > 0 )

digit = digit OR 0x80

endif

'output' digit

while ( X > 0 )

Note: MOD is the modulo operator ('%' in C), DIV is integer division ('/' in C), and OR is bit-wise or ('|' in C).

The algorithm for decoding the Remaining Length field as it comes off the wire is similarly straightforward:
 

multiplier = 1
value = 0

do

digit = 'next digit from stream'

value += (digit AND 127) * multiplier;
multiplier *= 128;

while ((digit AND 128) != 0);

When this algorithm terminates, 'value' contains the Remaining Length in bytes.

Note that the Remaining Length encoding is not part of the Variable Header portion of the message, and so the number of bytes taken to encode the Remaining Length does not itself contribute to the value of the Remaining Length. The "extension bytes" of the variable length encoding should therefore be thought of as part of the Fixed Header, rather than part of the Variable Header.
 

Discussion

ASC on the SECurity flag

It has been decided (with Chris Sharp), that since Authentication belongs at the session level, and Encryption is an application level thing which does not involve the broker in any way, the SEC flag has no place in the protocol. So we cheerfully give its place over to the RETAIN flag.

 

ASC on messages types

We are getting close to the end of our 15 available message type codes. If we need to extend the verb set further in the future, we could coalesce the publish acknowledgements into one message type, and have an additional byte in the variable header to indicate the acknowledgement type. That will free up 3 type codes.

 

AN on message type 0

During our testing, we have occasionally attempted to decode an empty (zero) first byte of a message. We had a place holder in Protocol version 1 for PING in message type 0. This would erroneously lead to the interpretation of the invalid fixed-header byte as a PING packet. To avoid this risk in the future, we will mark message type 0 as reserved but unused.

 

AN on DUP bit

We should retain this field. Since field devices should be able to deliver data to the Argo broker reliably, in a potential network error condition any queued publishes that have not been acknowledged should be re-delivered by setting the DUP bit in the header. Note that guaranteed delivery in either direction should observe a  Time To Live in the sense that publish messages from the broker to field devices need to expire when used in conjunction with command based messaging. For example, command based publishes to open or close a valve should happen within near real time. If the message delivery fails, system operation should NOT guarantee the command be delivered at a later time.

ASC comment

The Time To Live/Expire field has subsequently been dropped, as it is an application level attribute, and so should appear in the payload.

 

AN on SEC bit

I believe that at this point in time we should retain this field within the header but will leave any definition as to the security specification to a later date. To date, all known systems of this type are already running within secure networks. Additionally, the overhead of encryption to a specification targeted for low level embedded systems may preclude some systems.

 

 ASC on economy of space

What is the breakpoint on the tradeoff between using a smaller type field (combined with the version number), which would require more bit twiddling (i.e. CPU) on the client to get data in and out of it, versus the ease and speed of using a whole byte, despite it being wasteful of space?

AN comment

Bit twiddling is perfectly alright.

 

ASC on COMP (Compression) flags

Could we move this flag into the Connect packet? Then there might be the issue of some topics requiring compression and others not requiring it. Also, what happens if compression "fails" (i.e. makes the packet bigger) for a certain packet - then we would need to unset the bit. So, I think the answer is NO... leave it in each message.

Comment

This is an application level attribute, and will thus be moved out of the Fixed Header and is left up to the application to put appropriate flags into the Payload.

 

TNH on Session policies

We could consider the option of "inheritance" of policies from the Connection/Session level, which would be passed on by default, but which could be selectively over-ridden where necessary. An example would be the COMP flag, which could be set "on" for a particular session, but turned off when a piece of binary data was to be sent.

Comment

This should be considered as a possible future enhancement to the protocol, but will not be included at   this level of the specification.

BACK to index