wiki:Notes/Messaging

Context Navigation

Messaging

Messaging

A big subject. Even Wikipedia seems confused about it.

Part of is the multi-level reference implied by the word 'message'.

The unitary message, as in
| message 1 | message 2 | message 3 |

The content of the unitary message, varies according to message format.
| <msg-header>message-1</msg-header><message-body>This is the message number 1<message-body> | <msg-header>message-2</msg-header><message-body> ...

The message body, such as
"This is the message number 1" or "E.T. Call Home"

The intention of the message,
"Buy Crunchy Flakes" or "Republican policies cause economic catastrophe".

A Message as Messages

In wiki:/PyWacket, note the reference to "keeping track of message threads".

In other words, message or messages is/are more than one. Message includes a sequence of messages. Can be templated ?

Wikipedia on 'Message'

http://en.wikipedia.org/wiki/Message

http://en.wikipedia.org/wiki/Message_queue

https://en.wikipedia.org/wiki/Message_queue

There are often numerous options as to the exact semantics of message passing, including:

Durability - messages may be kept in memory, written to disk, or even committed to a DBMS if the need for reliability indicates a more resource-intensive solution.

Security policies - which applications should have access to these messages?

Message purging policies - queues or messages may have a "time to live"

Message filtering - some systems support filtering data so that a subscriber may only see messages matching some pre-specified criteria of interest

Delivery policies - do we need to guarantee that a message is delivered at least once, or no more than once?

Routing policies - in a system with many queue servers, what servers should receive a message or a queue's messages?

Batching policies - should messages be delivered immediately? Or should the system wait a bit and try to deliver many messages at once?

Queuing criteria - when should a message be considered "enqueued"? When one queue has it? Or when it has been forwarded to at least one remote queue? Or to all queues?

Receipt notification - A publisher may need to know when some or all subscribers have received a message.

http://en.wikipedia.org/wiki/Message_broker

Interesting who's missing from the list of message brokers ...

http://en.wikipedia.org/wiki/Message-oriented_middleware

http://en.wikipedia.org/wiki/Enterprise_messaging_system

An enterprise messaging system (EMS) is a set of published enterprise-wide standards that allows organizations to send semantically precise messages between computer systems. EMS systems promote loosely coupled architectures that allow changes in the formats of messages to have minimum impact on message subscribers. EMS systems are facilitated by the use of structured messages (such as using XML or JSON), and appropriate protocols, such as DDS, MSMQ, AMQP or SOAP with web services.

EMS usually takes into account the following considerations:

Security: Messages must be encrypted if they travel over public interfaces. Messages must be authenticated or digitally signed if the receiver is to have confidence that the messages have not been tampered with in transit.

Routing: Messages need to be routed efficiently from the sender to the receiver. Intermediate nodes may need to route the messages if the body of the message is encrypted.

Metadata: The body of the document contains information that must be unambiguously interpreted. Metadata registries should be used to create precise definitions for each data element.

Subscription: Systems should be able to subscribe to all messages that match a specific pattern. Messages with a specific content may be routed differently. For example some messages may have different priority or security policies.

Policy: Enterprise messaging systems should provide some consideration for a centralized policy of messages such as what classes or roles of users can access different fields of any message.

EMS has reference to http://en.wikipedia.org/wiki/Service-oriented_architecture.

https://en.wikipedia.org/wiki/Staged_event-driven_architecture

The staged event-driven architecture (SEDA) refers to an approach to software architecture that decomposes a complex, event-driven application into a set of stages connected by queues.

It avoids the high overhead associated with thread-based concurrency models[clarify] (i.e. locking, unlocking, and polling for locks), and decouples event and thread scheduling from application logic. By performing admission control on each event queue, the service can be well-conditioned to load, preventing resources from being overcommitted when demand exceeds service capacity.

Instant Messaging

http://en.wikipedia.org/wiki/Instant_messaging

http://en.wikipedia.org/wiki/Category:Instant_messaging_server_software

http://en.wikipedia.org/wiki/Category:Instant_messaging_protocols

XMPP Messaging

http://xmpp.org/

http://en.wikipedia.org/wiki/XMPP

http://en.wikipedia.org/wiki/Comparison_of_XMPP_server_software

See wiki:/Notes/JavaOpenfire

Also, incredibly, see wiki:/Notes/NodeJS#NodeXMPP

http://chatme.im/ - seems quick and easy

See Gtalk or Chatme.im XMPP plugin for Firefox

See PythonXMPP, PythonServers#XMPP

Search wiki for 'xmpp'

AMQP Messaging

http://www.amqp.org/

http://queues.io/

Job queues, message queues and other queues. Almost all of them in one place.

http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol

The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware. The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security.

AMQP mandates the behavior of the messaging provider and client to the extent that implementations from different vendors are interoperable, in the same way as SMTP, HTTP, FTP, etc. have created interoperable systems.

Previous standardizations of middleware have happened at the API level (e.g. JMS) and were focused on standardizing programmer interaction with different middleware implementations, rather than on providing interoperability between multiple implementations. Unlike JMS, which defines an API and a set of behaviors that a messaging implementation must provide, AMQP is a wire-level protocol.

A wire-level protocol is a description of the format of the data that is sent across the network as a stream of octets. Consequently any tool that can create and interpret messages that conform to this data format can interoperate with any other compliant tool irrespective of implementation language.

A semi-classical examination of three AMQP platforms - http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/

See PythonAMQP, PythonServers#AMQP

Search wiki for 'amqp'

Rabbit MQ

See Erlang#RabbitMQ

Cloud AMQP

https://addons.heroku.com/

https://addons.heroku.com/?q=cloud

https://devcenter.heroku.com/articles/cloudamqp

https://addons.heroku.com/cloudamqp#lemur

CloudAMQP from $0/mo RabbitMQ as a Service

Platform - Message queueing

Queue job messages for your workers. CloudAMQP gives you in-order, no-duplicates guarantees and high availability as the messages can be replicated between multiple RabbitMQ nodes.

Application decoupling

Instead of building a massive application many teams find it beneficial to decouple different concerns in your application and only communicate between them asynchronously with messages. That way different parts of your application can evolve independently, be written in different languages and/or maintained by complete separated teams.

Database - Offload your data store

Instead of polling your data store, publish a message when new data is inserted. Interested parties will be notified immediately and your data store will be ready to answer qualified queries instead.

Conversation - Realtime applications

CloudAMQP is an excellent backend for realtime applications. Notifications and message streaming is handled very effectively by RabbitMQ, you’ll be able to push thousands of messages per second.

High Availability

All our servers are clustered and all queues are by default mirrored over all nodes. The load-balancer will automatically detect and temporarily remove unhealthy nodes. We provide RabbitMQ clusters you can rely on.

Red Hat Messaging

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_MRG/3/html/Messaging_Programming_Reference/index.html

Basically is Qpid, plus some value added. Part of Red Hat Cloud ?

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_MRG/1.1/html/Messaging_Tutorial/chap-Messaging_Tutorial-Using_RHM_with_Python.html

Good doc ...

Search wiki for 'amqp'

MQTT

http://mqtt.org/

MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol. It was designed as an extremely lightweight publish/subscribe messaging transport. It is useful for connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium.

http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html

MQTT is a Client Server publish/subscribe messaging transport protocol. It is light weight, open, simple, and designed so as to be easy to implement. These characteristics make it ideal for use in many situations, including constrained environments such as for communication in Machine to Machine (M2M) and Internet of Things (IoT) contexts where a small code footprint is required and/or network bandwidth is at a premium.

The protocol runs over TCP/IP, or over other network protocols that provide ordered, lossless, bi-directional connections. Its features include:

Use of the publish/subscribe message pattern which provides one-to-many message distribution and decoupling of applications.

A messaging transport that is agnostic to the content of the payload.

Three qualities of service for message delivery:

"At most once", where messages are delivered according to the best efforts of the operating environment. Message loss can occur. This level could be used, for example, with ambient sensor data where it does not matter if an individual reading is lost as the next one will be published soon after.

"At least once", where messages are assured to arrive but duplicates can occur.

"Exactly once", where message are assured to arrive exactly once. This level could be used, for example, with billing systems where duplicate or lost messages could lead to incorrect charges being applied.

A small transport overhead and protocol exchanges minimized to reduce network traffic.

A mechanism to notify interested parties when an abnormal disconnection occurs.

http://en.wikipedia.org/wiki/MQTT

MQTT (formerly Message Queue Telemetry Transport) is a publish-subscribe based "light weight" messaging protocol for use on top of the TCP/IP protocol. It is designed for connections with remote locations where a "small code footprint" is required and/or network bandwidth is limited. The Publish-Subscribe messaging pattern requires a message broker ...

Historically, the 'MQ' in 'MQTT' came from IBM's MQ message queuing product line. However, queuing per se is not required to be supported as a standard feature in all situations.

https://github.com/mqtt/mqtt.github.io/wiki/servers

https://github.com/mqtt/mqtt.github.io/wiki/List-of-links

... Completely unsorted! Totally unscientific !

Sounds familiar.

List includes Mosquitto, RabbitMQ, Apache Apollo and Mosca

https://github.com/mqtt/mqtt.github.io/wiki/server-support

Mosca MQTT Broker

http://www.mosca.io/

MQTT broker as a NodeJS module

https://github.com/mcollina/mosca

https://github.com/mcollina/mosca/wiki

Mosca is a MQTT broker. That means that mosca implements the mqtt protocol, so that you can run your own mqtt server on node.js.

http://mcollina.github.io/mosca/docs/

A bit sparse ...

Mosquitto MQTT Broker

Mosquitto seems to be the leading broker so far. Provides its own Python client

http://mosquitto.org/

Mosquitto is an open source (BSD licensed) message broker that implements the MQ Telemetry Transport protocol versions 3.1 and 3.1.1.

MQTT provides a lightweight method of carrying out messaging using a publish/subscribe model. This makes it suitable for "machine to machine" messaging such as with low power sensors or mobile devices such as phones, embedded computers or microcontrollers like the Arduino.

API - http://mosquitto.org/api/files/mosquitto-h.html

Wiki - https://github.com/mqtt/mqtt.github.io/wiki

Run well on LinuxServers#RaspberryPi

Search wiki for 'mqtt'

Brokerless Messaging

Search Google for 'brokerless messaging' - https://www.google.com/search?q=brokerless+messaging

Zero MQ

Fast, runs everywhere, good documentation ...

http://zeromq.org/

http://zguide.zeromq.org/

ØMQ (also known as ZeroMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework.

It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply.

It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks.

https://github.com/zeromq/libzmq

The 0MQ lightweight messaging kernel is a library which extends the standard socket interfaces with features traditionally provided by specialised messaging middleware products. 0MQ sockets provide an abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more.

Seems to assume Linux. Mentions CygWin and !MingW32...

http://zguide.zeromq.org/page%3aall#Service-Oriented-Reliable-Queuing-Majordomo-Pattern

About Windwos - http://zeromq.org/distro:microsoft-windows

http://zeromq.org/docs:welcome-from-amqp

If you've been using AMQP/0.8 or AMQP/0.9.1, here is a short guide to help you understand ØMQ ...

An AMQP network consists of a broker and applications connected in a star. Brokers can be duplicated for redundancy, and sometimes connected in federations. AMQP adopts the HTTP view of the Internet as a set of large hubs serving many users.

A ØMQ network consists of applications and brokers that interconnect in a distributed fashion. ØMQ adopts the TCP/IP view of the Internet as a dumb elastic fabric serving a massive number of users.

ØMQ does not come with a predefined broker. It is a library of broker-like functionality that you include into every application and use to build custom brokers (aka. 'devices') where you need them.

http://www.fedmsg.com/en/latest/overview/

http://www.aosabook.org/en/zeromq.html

Broker versus Brokerless - http://zeromq.org/whitepapers:brokerless

A Fairly Import Article about Distributed ( Peer to Peer ) Computing

http://zguide.zeromq.org/

Support for many different languages

Python -> http://zguide.zeromq.org/py:all

http://zguide.zeromq.org/py:chapter8

We'll cover:

Requirements for distributed computing

The pros and cons of WiFi for proximity networking

Discovery using UDP and TCP

A message-based API

Creating a new open source project

Peer-to-peer connectivity (the Harmony pattern)

Tracking peer presence and disappearance

Group messaging without central coordination

Large-scale testing and simulation

Dealing with high-water marks and blocked peers

Distributed logging and monitoring

Whether we're connecting a roomful of mobile devices over WiFi? or a cluster of virtual boxes over simulated Ethernet, we will hit the same kinds of problems. These are:

Discovery: how do we learn about other nodes on the network? Do we use a discovery service, centralized mediation, or some kind of broadcast beacon?

Presence: how do we track when other nodes come and go? Do we use some kind of central registration service, or heartbeating or beacons?

Connectivity: how do we actually connect one node to another? Do we use local networking, wide-area networking, or do we use a central message broker to do the forwarding?

Point-to-point messaging: how do we send a message from one node to another? Do we send this to the node's network address, or do we use some indirect addressing via a centralized message broker?

Group messaging: how do we send a message from one node to a group of others? Do we work via a centralized message broker, or do we use a pub-sub model like ZeroMQ?

Testing and simulation: how do we simulate large numbers of nodes so we can test performance properly? Do we have to buy two dozen Android tablets, or can we use pure software simulation?

Distributed Logging: how do we track what this cloud of nodes is doing so we can detect performance problems and failures? Do we create a main logging service, or do we allow every device to log the world around it?

Content distribution: how do we send content from one node to another? Do we use server-centric protocols like FTP or HTTP, or do we use decentralized protocols like FileMQ?

Official Python bindings - http://zeromq.org/bindings:python

See PythonClients#ZeroMQ, PythonServers#ZeroMQ