10 Principles for AMQP

Written: 03 May 2009 12:06 (amqp community sermon)

Ten ways that AMQP can be made simpler, more backwards compatible, more interesting, and overall more enjoyable and successful for all who work on it and use it.

Introduction

The more you know about something, the harder it is to build it.

We went live with the first production deployment of AMQP (on OpenAMQ) at the end of 2006, handling about half a billion messages a day. It was a massive project: design a new protocol, build an industrial-strength implementation, and migrate a major application onto this new infrastructure. Two and a half years sounds a lot but in that time we made three complete redesigns of AMQP, and OpenAMQ, before we had designs that were simple enough to work reliably, and still do what was needed. Functional simplicity is the hardest aspect of design.

It's been hard to translate that success into a final AMQP specification. Part of the problem was failure to agree on what "final" meant. Part of the problem is that AMQP addresses a large problem that cannot be solved quickly. Part of the problem is that the people working on AMQP - including myself - were still building the tools and experience needed.

AMQP will not, in my opinion, be solved in one or two steps, nor in one or two years. The work that has been done already over four years is very important. It has already created a healthy market of competing, interoperable, messaging products. The AMQP/0.9.1 specification, which was published last year, raises the bar: it is lean, precise, coherent, and tested.

But AMQP/0.9.1 is not the final destination. As a contract of interoperability and stability, it's excellent, almost perfect. But it has a limited view of the problem, and it is still more complex than it should be.

The thing about infrastructure is that once it is in place, it is horribly expensive to fix. The complexity in AMQP/0.9.1, if not solved, will be multiplied many times when AMQP expands to cover the fully-reliable messaging, low latency, multicast, and other features we expect to see in AMQP/1.0.

Complexity is easy, simplicity is hard. My day job is to design simplicity by identifying and removing complexity. In this article I'll do that to AMQP, and identify ten areas where there is unnecessary, and in the long term dangerous, complexity, and in each case I'll provide recommendations for solving that complexity.

My ten principles for AMQP are:

  1. Make small pieces: AMQP should be a fabric of protocols, not one large protocol.
  2. A layered architecture: these protocols should form a clearly layered architecture.
  3. Delegate the design work: each protocol is a job for a small, competent team.
  4. Leverage the community: open the design process to anyone who wants to participate.
  5. Leverage competition: allow competing teams to make competing designs, at any level.
  6. Leverage the market: use adoption, over time, to identify which designs are best.
  7. Use natural syntax: the framing at each level should fit the needs of that level.
  8. Use natural semantics: the interactions between peers at each level should be simple.
  9. Push blame to the edges: brokers should never compensate for poorly-written applications.
  10. Deconstruct the broker: the architecture should support zero, one, or more brokers.

Since theory without practice is useless, the RestMS project acts as a demonstrator of most of these principles.

Make small pieces

AMQP is on paper a single protocol1. This makes it easier to market. It makes it much harder to improve, and the 0-10 specification shows this: it's 197 pages, compared to about 40 for AMQP/0.9.1. I've recommended since 2006 that we refactor AMQP into smaller pieces, each solvable by a small team of 1-3 people. For example, separate protocols for version negotiation, for control commands, for message transfer, for transactions. These pieces need freedom to evolve independently, so that experimentation can be cleanly (contractually) separated from standardization.

A layered architecture

AMQP has no clearly layered architecture. The refactoring of AMQP into small protocols must be based around a solid, coherent, layered architecture. This should be the core of what AMQP 'is', the fundamental agreement around which detailed designs can be built. The architecture should specify what each layer does, in terms of its interactions or interfaces with higher and lower layers. The goal of the architecture is to compartementalize innovation, to guarantee space for contributions, and to ensure the overall coherence of AMQP. The layered architecture is what defines the major version of the protocol.

Delegate the design work

AMQP is specified by a tiny, exclusve team. When a complex problem is broken into pieces, shaped by a clear architecture, it becomes possible for teams to work independently and asynchronously, on different pieces. This is the only way to solve large problems: break them down and let people specialize in different areas. The AMQP process is today a technical process. It should become an administrative one, defining frameworks for collaboration, and approving specifications when they emerge and have been proved.

Leverage the community

The AMQP process excludes expert user contributors. Given that messaging users are often highly competent engineers who at the least can act as "competent clients" for designers, this is a problem. There are only two justifications for excluding participation. One: it's legally complex. Two: it's technically unsafe. Both these can be solved, by defining suitable legal frameworks and by breaking AMQP into small pieces that anyone can experiment at implementing.

Leverage competition

AMQP aims to get one authoritative design. This assumes that there are no commercial rivalries, but there clearly are. Rather than suppress these, we can use them constructively by allowing more than one answer to any given problem. If one team proposes a way of delivering messages, and another team thinks it can make a better way, it should be allowed to. Perhaps the two teams can agree, but if they don't we should allow two answers to sit on the table at once. This enables competition, so better designs can emerge. It also forces layers to be more orthogonal, and validates the overall architecture.

Leverage the market

AMQP is shielded from market opinion. If users prefer a particular API, for example, that does not feed back into the AMQP process. But it should: market opinion, especially with open source products, is a very valuable indicator of quality. When it comes to protocol designs, quality valuation by the market can be measured by the number of independent open source implementations, and the number of users of such implementations. Rather than try to make quality decisions up-front, it is simpler to use a green-brown model where experimentation allows multiple "green" specifications that are pruned over time to leave a small number, or one, brown specification.

Use the natural syntax

AMQP uses binary framing for control commands ("Queue.Declare") as well as message transfer ("Basic.Deliver"). This is not ideal. Each piece should use the natural syntax. Command protocols should use text, not binary framing. This is how every successful IETF command protocol works. Text framing makes backwards compatibility trivial. There are no performance issues with command protocols, so parsing is not an issue. Text framing is easy to understand, easy to implement. Message transfer protocols should use binary framing, and can be significantly simpler than AMQP's current framing: smaller envelopes, no verbs, no channels, no dynamic addressing.

Use natural semantics

AMQP currently mixes asynchronous and synchronous conversations in one protocol. It makes the combined protocol complex, and it makes error handling tricky. When we split control commands and message transfer into two separate protocols we can make each simple. Control commands work best with synchronous, pessimistic dialogues: each request gets a success/failure response. Message transfer works best with asynchronous, optimistic dialogues: messages are sent with no confirmation, and reliability is layered on top, as higher level protocols based on acknowledgements and retransmissions, transactions, and so on. Messages can be batched, commands should not.

Push blame to the edges

AMQP currently pushes problems upstream. Application private queues are held on the broker. Slow consumers cause these queues to back-up, and probably the biggest issue for production use is that servers run out of memory, and crash. Applying flow-control to publishers is the wrong solution. Messages should be pushed without pity to the edges, and if these are too slow, that should count as a defect in the edge application, and handled locally: drop old or new messages, or raise a fatal error. Problems at the edge should not be allowed to move upstream, period.

Deconstruct the broker

Currently, AMQP considers a broker to be a mainframe: big, important, central. This is certainly one use case but it is a very bad design for high data volumes: one big central bottleneck. It is wiser to treat the broker as a host for arbitrary queues and routers, and to accept that in many cases, such applications optimally reside at the edges, not a central point. For example, holding private queues centrally makes brokers fragile, while pushing private queues to the edges makes brokers more robust (by pushing blame to the edges). Similarly, routing can be done at the publisher edge. Only the Wolfpack pattern (one-to-one-of-many) absolutely remands a central broker.

Conclusions

The AMQP protocol and process are complex in ways that make it difficult to build a Version 1.0 and which in the long run may affect the success of the AMQP project itself. I've outlined ten principles that will in my view solve the complexity. Most of these proposals are years old and have been made many times to the AMQP working group.

Comments: 3, Rating: 0


AMQP, Sea, Sun, and Desert

Written: 08 Apr 2009 17:35 (amqp photos sandiego)

A brief report from the AMQP Conference at San Diego (AMQP'09) with photos of the event and the nearby Anza-Borrego desert state park.

We came to the fourth annual AMQP face-to-face, and the first one with a public conference. The organization by Matthew Arrot, Rita Bauer and the rest of his team was impeccable. You'll see from the first photo set, the USCD location was breath taking and we were spoilt with wonderful lunches, views, weather.

AMQP'09 was a unique event in many ways but mostly because it brought together such a diverse group of people, both an expanding AMQP work group, but also the core of a community of expert users. One of the weaknesses in the AMQP vision up to now has been that the "end user" has been represented only by large global businesses. At AMQP'09 the public seemed mainly to represent small teams, and their needs form an important part of the mix. iMatix is, of course a small team, but ironically our clients are all huge firms. Nonetheless, I often find the response of someone who has precious little time and money to spend on learning and using a technology more enlightening that that of someone with budgets.

The main reason for the public session on Day 1 was to present AMQP/1.0. Rafi and Rob did an excellent job presenting their work. Everyone who came to the event left with their own conclusions. Gone are the exchanges, just as everyone was starting to understand these. At one time on Day 2, the closed group was discussing "XML exchanges" for an hour or so, before I raised my hand to remind: "gentlemen, in 1.0 exchanges are gone." Our mental models take time to come to life, and they die hard.

In place of algorithmic exchanges that feed messages into a population of queues, we will have queues of messages that are scanned by a population of algorithmic links. While the design seems to work for mainstream scenarios, it is not yet clear exactly what this fixes from the 0.9.1 design. We lack a clear explanation of what is broken in 0.9.1, what the possible solutions are, and why the design chosen for 1.0 is the best solution. The argumentation, as one would say.

The argumentation is going to be needed, as more and more people invest in 0.9.1, and will continue to do so over the next year or so until AMQP/1.0 is mature enough to be supported by working products.

Some teams have exploited the exchange-binding-queue model elegantly: FastMQ's ZeroMQ does brokerless messaging by moving the exchange and binding to one peer, and the queue to another. It's unclear how this can work using the new queue-link model.

The difference between the old and the new models is, as far as I can see that the old model is easier to visualize as a flow of messages through exchanges into queues. The new model feels more like pointer manipulation. Maybe more efficient, but also more abstract. Given that it took several years to explain the old AMQP model (it is always totally obvious after six months of reflection), the extra effort needed to understand the new model may prove to be a problem.

People who are used to AMQP/0.9.1 should try to recall how they felt when first reading the exchange-binding-queue explanations. It was obvious, then it was not, and then after much re-reading, it became obvious again. Perhaps this was just alien technology. Or maybe these models are really hard work to understand, even (especially) when they are simple.

My in-progress judgement on AMQP/1.0 is that it looks good, but it is unclear why the changes were needed, and why an evolutionary approach was not possible. I'd have liked to have seen experimentation with the new model, using the existing AMQP/0.9.1 transport, for example. After all, it is trivial to define new classes in AMQP to work with a new set of broker-side objects. Or, I'd like to see the new proposed transport layer running with the existing AMQP/0.9.1 model. Bringing out a completely new AMQP model running on a completely new AMQP transport layer forces all implementers and users - brokers, clients and applications - to throw away their old code and move to new code, in one step. This seems designed to fail.

Having said this, if a new AMQP model is needed, or if a new AMQP transport is needed, we should make them. AMQP is still young enough to absorb this kind of change and if there were serious flaws with the exchange-binding-queue model, it's exactly the early adopters who will understand the need for redesign.

However: such change must be managed and its downstream impact must be minimized. It should be introduced in phases so that the new designs can be matured and validated by real use and competing implementations.

On the afternoon of Day 2 the group split into separate tracks, one to discuss technology and the other to discuss process, and legal aspects. Patents. It's a subject that causes distress because at their core patents (due to their cost and the risk they bring) are a game for large firms only. It is one of the few areas where large and small firms play in the same field, and whenever patents enter the picture the field tilts inexorably away from the smaller teams.

And like I said, it's the smaller teams that are key IMO to AMQP's success.

I'd like to report that the working group came to a firm decision to keep patents far away from AMQP but unfortunately this is not what happened. There were no decisions as such. There was no real criticism of Red Hat's act of patenting that famous XML exchange. Someone joked that one benefit of dropping exchanges in 1.0 was that this patent would become irrelevant.

However, standards committees are slow machines and the AMQP working group moves faster than most. We did make excellent progress on the idea that AMQP was more than one document, and consisted of a sort of ecosystem of specifications. We already have this, informally. The ecosystem is inevitable because AMQP is not a 100% solution. It needs supporting specs in all directions. Either vendors make these privately, with a chance of patenting, or they make them publicly, with a unilateral patent license such as the wiki.amqp.org site uses.

One can never stop a firm from patenting some extension to AMQP. But one can encourage firms to publish their specs in a form that precludes this. Not publishing the spec for functionality then becomes a clear statement of intent. Patents are an issue. but transparency around exactly what is safe goes a long way to solving that issue.

For a firm like iMatix, whether or not we invest in AMQP/1.0 depends very much on whether the AMQP process expands to include such an ecology of open, unilateral specifications. A big jump to new designs is one thing. A jump into a space that may have patent ambushes, quite another. If we're vocally sensitive to this issue it's because we've been hit by it before. In 2005, a meagre patent claim by Belgian firm AllisBlue caused us to exit the mobile applications market and abandon about Euro 250,000 and three years of investment into a mobile apps platform called SMS@. Any patenting activity around AMQP, no matter what the stated intentions are, is a threat to implementer and users of AMQP, and will keep smaller companies away from the table.

I'll continue to blog on this issue, and the progress (or lack of it) that the AMQP working group makes, over the next months and years.

{"module":"wiki\/image\/FlickrGalleryModule","params":{"p":"4","userName":"hintjens","tags":"AMQP09","sort":"date-taken-asc"}}

On Day 2… well, on Friday I decided to skip class and head for the desert. The 9-hour jet lag and committee discussions on software patents began to turn my brain to mush. The state desert park of Anza-Borrego, the largest park in California, is just two hours' drive north-west from San Diego. A stretch of highway, and then a long windy trail over mountains and valleys, until one comes to this massive area surrounded by mountains.

When one drives in the desert, one uses an all-wheel drive and GPS. Halfway into Anza-Borrego my GPS died. My heart froze. GPS, while giving a godlike wisdom of where to turn, also makes one stupid. A paper map would make a good backup. In any case I had a gallon of water (89c at Walmart leaving San Diego) and stout shoes. Then I saw a small 'reset' hole on the GPS. Found tiny twig, inserted, pressed, and yay! GPS rebooted and I was safe again.

The hot and cold deserts, along with impenetrable rain forests, are those rare parts of the dry surface of the planet that have not been moulded by human activity. Some roads cross Anza-Borrego but mostly it's moulded only by nature: rising crusts, erosion from water and wind. The plain is hot and dry. The plants are all thorns and bristles. Many were in flower. The beauty of the desert flower is accentuated by its rarity in the dry landscape.

As one walks the hills, one is wafted by the scents of flowers and plants. One particularly sweet smell, a cross between sage and lavender, I traced to a small pine shrub which just had particularly fragrant leaves.

The air rustles with the noises of plants in the wind and the buzz of invisible insects. The ground is sandy where it's flat, rocky on the slopes. There are burrows of mice, or snakes. I saw neither. There are palms where the locals pump water up from the water bed. In places it comes up by itself, and springs dot the park. In the hills you see the work of flash floods, which carve out gullies in the rock. The desert comes alive at night but as the sun set I decided to find a place with a real bed.

I filled up and overnighted in a motel in Borrego Springs, a town of retirees with legs muscled from climbing the desert paths. Like the sheep the area is named after. I did not see any Borrego sheep, they hid well.

The empty wilderness contrasted with the utterly refined beaches of La Jolla (which, Rainey explained, is pronounced "la Hoya"). Neither felt anything like home, which is presently a calm street in the old canal zone of Brussels.

But one thing I realized: AMQP, like any city, and unlike the desert plains and mountains of Anza-Borrego, thrive or fail according to the diversity and energy of the people that inhabit it. It's people and openness, not technology, that make standards a success. And bringing together such a diverse and energetic group of people, the AMQP 2009 event smelled of success.

{"module":"wiki\/image\/FlickrGalleryModule","params":{"p":"4","userName":"hintjens","tags":"Anza-Borrego","sort":"date-taken-asc"}}

Comments: 0, Rating: 0


Red Hat Claims AMQP

Written: 26 Mar 2009 13:46 (amqp lecture patent)

Brian Che is Product Manager at Red Hat for their MRG messaging product, which includes an AMQP server.

Brian welcomed Microsoft to the AMQP working group:

Just as Red Hat has been adding native AMQP support into the Linux platform and ecosystem at Fedora and through Red Hat Enterprise MRG, Microsoft is bringing AMQP support to Windows and its ecosystem. Between Linux and Windows, AMQP will become a standard messaging facility on the vast majority of operating systems and server platforms.

And he listed the Working group members, omitting the other implementers (Rabbit Technologies and iMatix):

The AMQP working group already has a well-esteemed set of members, ranging from software vendors like Red Hat to hardware vendors like Cisco to end-users like JP Morgan Chase, Goldman Sachs, Credit Suisse, and Deutsche Börse (see the full list of participants at http://amqp.org).

Ironically, given the later exposure of a stealth patent very close to AMQP by Red Hat, he wrote:

So, there is no threat of Microsoft holding the AMQP standard hostage via patent threats.

So much for Red Hat's public statements. But what do they say behind doors, to their clients, to journalists, and to partners? Well, Brian has been helpful in pointing us to a position paper he wrote for an Indiana University workshop on Cyberinfrastructure Software Sustainability on March 26 & 27th 2009.

In that paper, which I've copied here in case it disappears from that site, Brian explains Red Hat's true position:

Red Hat teamed up with one of its customers, JP Morgan Chase (JPMC), to create an open protocol standard around messaging, AMQP.

This is strange news. iMatix was working closely with JPMC in 2004-6 when we (iMatix and JPMC) designed AMQP, and we never saw anyone from Red Hat until they reviewed the specifications and helped create the AMQP workgroup. And to the best of my knowledge JPMC was not a Red Hat customer except for Linux licenses.

Maybe Brian just meant to type "iMatix" and wrote "Red Hat" out of habit instead. This happens. I do it all the time. But sadly there is more…

JPMC, like many other banks, had developed its own messaging software to meet its high-end messaging requirements. However, JPMC had also written down the specification of its work, and this proved to be a good starting point for creating an open messaging protocol standard. Red Hat and JPMC created a legal contract to form the AMQP working group, which would develop this new standard as AMQP in an open and IP-unencumbered manner. Then, they started bringing in many additional companies to collaborate in this working group.

I will now state this for the record. iMatix was hired by JPMC in 2004 specifically to (a) develop AMQP, (b) develop an open source implementation, and (c) migrate JPMC's largest investment bank application onto this new stack. It was a tough triple project done on brutal deadlines with a deep emotional cost. To demonstrate how much iMatix invested in this project: my wife was pregnant with our second baby, which died in utero at eight months1. While she was in hospital in Brussels, I was at JPMC in London with my team, where we had spent most of the previous months, making things work.

We took this work utterly seriously and in 2006, we were ready with a very solid specification that is essentially unchanged - many fixes, and unused parts removed - from the AMQP/0.9.1 specification that is used by most AMQP users today. Red Hat received this specification and their major contribution to it, infamous as an example of incompetent protocol 'design' pushed through by bluster and force, was the AMQP/0.9-work-in-progress specification. Read it, and weep, if you have the courage.

It is quite trivial to check that iMatix registered the AMQP IANA port (5672), and the amqp.org, amqp.com, and amqp.net domain names.

Brian's text continues to celebrate the "Linux to Windows interoperability" that seems to echo through Red Hat's approach to software patents over the years.

But back to Red Hat's mythical invention of AMQP. Maybe Brian Che is just ignorant. Let's roll with that for a second. All this specification and accuracy stuff is complex, after all. But in 2007 - yes, eighteen months ago - I pointed the AMQP work group to a Red Hat news article which said,

Indeed, Red Hat's APQM[sic] began its life as proprietary messaging software at financial services giant JP Morgan Chase, said Red Hat Chief Technology Officer Brian Stevens in an interview here during Oracle OpenWorld.

Different Brian, same set of lies. The AMQP work group asked Red Hat for an explanation for this article. "The journalist got it wrong", we were told. We were invited to speak to a Red Hat VP if we wanted clarification.

But I think Brian Che has given us all the clarification we need.

Comments: 9, Rating: 1


Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License