created: 1241352409|%e %B %Y, %H:%M
TAGS: amqp community sermon
Ten ways that AMQP can be made simpler, more backwards compatible, more interesting, and overall more enjoyable and successful for all who work on it and use it.
Introduction
The more you know about something, the harder it is to build it.
We went live with the first production deployment of AMQP (on OpenAMQ) at the end of 2006, handling about half a billion messages a day. It was a massive project: design a new protocol, build an industrial-strength implementation, and migrate a major application onto this new infrastructure. Two and a half years sounds a lot but in that time we made three complete redesigns of AMQP, and OpenAMQ, before we had designs that were simple enough to work reliably, and still do what was needed. Functional simplicity is the hardest aspect of design.
It's been hard to translate that success into a final AMQP specification. Part of the problem was failure to agree on what "final" meant. Part of the problem is that AMQP addresses a large problem that cannot be solved quickly. Part of the problem is that the people working on AMQP - including myself - were still building the tools and experience needed.
AMQP will not, in my opinion, be solved in one or two steps, nor in one or two years. The work that has been done already over four years is very important. It has already created a healthy market of competing, interoperable, messaging products. The AMQP/0.9.1 specification, which was published last year, raises the bar: it is lean, precise, coherent, and tested.
But AMQP/0.9.1 is not the final destination. As a contract of interoperability and stability, it's excellent, almost perfect. But it has a limited view of the problem, and it is still more complex than it should be.
The thing about infrastructure is that once it is in place, it is horribly expensive to fix. The complexity in AMQP/0.9.1, if not solved, will be multiplied many times when AMQP expands to cover the fully-reliable messaging, low latency, multicast, and other features we expect to see in AMQP/1.0.
Complexity is easy, simplicity is hard. My day job is to design simplicity by identifying and removing complexity. In this article I'll do that to AMQP, and identify ten areas where there is unnecessary, and in the long term dangerous, complexity, and in each case I'll provide recommendations for solving that complexity.
My ten principles for AMQP are:
- Make small pieces: AMQP should be a fabric of protocols, not one large protocol.
- A layered architecture: these protocols should form a clearly layered architecture.
- Delegate the design work: each protocol is a job for a small, competent team.
- Leverage the community: open the design process to anyone who wants to participate.
- Leverage competition: allow competing teams to make competing designs, at any level.
- Leverage the market: use adoption, over time, to identify which designs are best.
- Use natural syntax: the framing at each level should fit the needs of that level.
- Use natural semantics: the interactions between peers at each level should be simple.
- Push blame to the edges: brokers should never compensate for poorly-written applications.
- Deconstruct the broker: the architecture should support zero, one, or more brokers.
Since theory without practice is useless, the RestMS project acts as a demonstrator of most of these principles.
Make small pieces
AMQP is on paper a single protocol1. This makes it easier to market. It makes it much harder to improve, and the 0-10 specification shows this: it's 197 pages, compared to about 40 for AMQP/0.9.1. I've recommended since 2006 that we refactor AMQP into smaller pieces, each solvable by a small team of 1-3 people. For example, separate protocols for version negotiation, for control commands, for message transfer, for transactions. These pieces need freedom to evolve independently, so that experimentation can be cleanly (contractually) separated from standardization.
A layered architecture
AMQP has no clearly layered architecture. The refactoring of AMQP into small protocols must be based around a solid, coherent, layered architecture. This should be the core of what AMQP 'is', the fundamental agreement around which detailed designs can be built. The architecture should specify what each layer does, in terms of its interactions or interfaces with higher and lower layers. The goal of the architecture is to compartementalize innovation, to guarantee space for contributions, and to ensure the overall coherence of AMQP. The layered architecture is what defines the major version of the protocol.
Delegate the design work
AMQP is specified by a tiny, exclusve team. When a complex problem is broken into pieces, shaped by a clear architecture, it becomes possible for teams to work independently and asynchronously, on different pieces. This is the only way to solve large problems: break them down and let people specialize in different areas. The AMQP process is today a technical process. It should become an administrative one, defining frameworks for collaboration, and approving specifications when they emerge and have been proved.
Leverage the community
The AMQP process excludes expert user contributors. Given that messaging users are often highly competent engineers who at the least can act as "competent clients" for designers, this is a problem. There are only two justifications for excluding participation. One: it's legally complex. Two: it's technically unsafe. Both these can be solved, by defining suitable legal frameworks and by breaking AMQP into small pieces that anyone can experiment at implementing.
Leverage competition
AMQP aims to get one authoritative design. This assumes that there are no commercial rivalries, but there clearly are. Rather than suppress these, we can use them constructively by allowing more than one answer to any given problem. If one team proposes a way of delivering messages, and another team thinks it can make a better way, it should be allowed to. Perhaps the two teams can agree, but if they don't we should allow two answers to sit on the table at once. This enables competition, so better designs can emerge. It also forces layers to be more orthogonal, and validates the overall architecture.
Leverage the market
AMQP is shielded from market opinion. If users prefer a particular API, for example, that does not feed back into the AMQP process. But it should: market opinion, especially with open source products, is a very valuable indicator of quality. When it comes to protocol designs, quality valuation by the market can be measured by the number of independent open source implementations, and the number of users of such implementations. Rather than try to make quality decisions up-front, it is simpler to use a green-brown model where experimentation allows multiple "green" specifications that are pruned over time to leave a small number, or one, brown specification.
Use the natural syntax
AMQP uses binary framing for control commands ("Queue.Declare") as well as message transfer ("Basic.Deliver"). This is not ideal. Each piece should use the natural syntax. Command protocols should use text, not binary framing. This is how every successful IETF command protocol works. Text framing makes backwards compatibility trivial. There are no performance issues with command protocols, so parsing is not an issue. Text framing is easy to understand, easy to implement. Message transfer protocols should use binary framing, and can be significantly simpler than AMQP's current framing: smaller envelopes, no verbs, no channels, no dynamic addressing.
Use natural semantics
AMQP currently mixes asynchronous and synchronous conversations in one protocol. It makes the combined protocol complex, and it makes error handling tricky. When we split control commands and message transfer into two separate protocols we can make each simple. Control commands work best with synchronous, pessimistic dialogues: each request gets a success/failure response. Message transfer works best with asynchronous, optimistic dialogues: messages are sent with no confirmation, and reliability is layered on top, as higher level protocols based on acknowledgements and retransmissions, transactions, and so on. Messages can be batched, commands should not.
Push blame to the edges
AMQP currently pushes problems upstream. Application private queues are held on the broker. Slow consumers cause these queues to back-up, and probably the biggest issue for production use is that servers run out of memory, and crash. Applying flow-control to publishers is the wrong solution. Messages should be pushed without pity to the edges, and if these are too slow, that should count as a defect in the edge application, and handled locally: drop old or new messages, or raise a fatal error. Problems at the edge should not be allowed to move upstream, period.
Deconstruct the broker
Currently, AMQP considers a broker to be a mainframe: big, important, central. This is certainly one use case but it is a very bad design for high data volumes: one big central bottleneck. It is wiser to treat the broker as a host for arbitrary queues and routers, and to accept that in many cases, such applications optimally reside at the edges, not a central point. For example, holding private queues centrally makes brokers fragile, while pushing private queues to the edges makes brokers more robust (by pushing blame to the edges). Similarly, routing can be done at the publisher edge. Only the Wolfpack pattern (one-to-one-of-many) absolutely remands a central broker.
Conclusions
The AMQP protocol and process are complex in ways that make it difficult to build a Version 1.0 and which in the long run may affect the success of the AMQP project itself. I've outlined ten principles that will in my view solve the complexity. Most of these proposals are years old and have been made many times to the AMQP working group.
Rate this post:
Pieter,
I think this is a great list of items.
However, the best architecture/protocol/software is hard to use if the management and control of it is not an integral part. (Note: integral does NOT mean "has to be there at all times"; it means there are hooks allowing its introduction at any point and granularity.)
I am not sure if this really warrants a separate bullet, but, I saw no particular place in your list to put it, so made another bullet!
Cheers, John
John Apps +1 719 534 3910
I agree. Ages ago we proposed a simple XML-based management API that was developed by iMatix and JPMorganChase for management and control in 2005. The other vendors did not accept it, so we have no standard for management. It'd be fine if everyone documented and promoted their APIs so others could interoperate but that does not happen.
Since vendor rivalry seems inevitable, the architecture needs to embrace that: allow multiple management API options, if people insist on inventing their own, but all published fairly. When one gains dominance, clients can then push other vendors to conform, and we get a standard.
This is the collaborative/competitive model we built for http://wiki.amqp.org but it remains to be seen whether this can work, or whether iMatix is an aberration in systematically documenting their interfaces and extensions so that others can use them.
Portfolio
I think AMQP WG, and V1 already address Pieter's concerns.
AMQP is my vision; a drive to create a useful, credible, dominant standard for "business messaging" (thanks to Alexis Richardson for inventing that term, which sounds so much better than "MOM").
To solve the problem of what AMQP should be, the big users of this stuff (including me) got together and defined what success should look like.
Want to know what "enterprise" customers need from a full MOM offering? Check it out:
http://jira.amqp.org/confluence/display/AMQP/AMQP+1.0+Business+Requirements+-+Revised
AMQP 1.0 is the first revision of the protocol to tick all the boxes; and it learned from the work that went before.
We have a diverse group of companies working together both users and suppliers of technology, and strong competitors of all types sitting around the table. Where we've got to, and the fact that this group of companies has met every week for nearly 3 years, speaks volumes.
The 1.0 PR is published as one document to reduce the opportunity for confusion and fragmentation in a new spec, but it is layered with each layer in its own independent chapter. We are working hard to clear up the layering down at the lowest level, but there are people from at least 3 organisations working on that.
That's why we have SIG's in AMQP Working Group.
But working with others is slower and harder than working alone - it's always so bothersome and time-consuming to argue points of view. In business terms, time is money, and this process is just too expensive for some people to stick at. Which is unfortunate. But its not for want of openness or a call to action.
I know that the combination of ideas argued honestly is in the end better. 1.0 proves that.
But whose community? There is an AMQP community, albeit a bit small. There is the Rabbit Community, the Apache Community, the OpenAMQ community. But most people like the pleasure and freedom of building their own codes than the tough exercise of working with (potential) competitors. There's also wiki.amqp.org which has a smaller community than www.amqp.org itself. But we agreed to help fix that up and clarify it.
I think we've got that covered in AMQP Working Group.
One day in San Diego when we had in one room iMatix, Red Hat, Microsoft, 29 West, Tervela and Solace talking technology was quite something. Oh, and JPM and DBS, we know our tech too.
The messaging market cannot evolve without a standard to rally around, that the users care about.
Let's see who shifts the most brokers…. let the market decide.
Good design demands simplicity, and that simplicity is notoriously hard to find. AMQP 1.0 is conceptually much simpler than previous versions, given that Exchanges have been factored into their essence, and disappeared.
This depends on how you think of the network.
AMQP introduces the concept of trusted intermediaries into the network, and its very useful.
The whole idea is to factor that difficult stuff out of applications. It can't disappear completely, but applications get a whole lot simpler in this model. Some of the largest applications in the world are written to this model.
The trust contract is that it won't accept something unless it can (publisher flow control) and it will safe keep it until the client comes to collect.
AMQP is about client->broker communication. AMQP says nothing about how the brokers network presence is represented. For example, Red Hat's MRG has a clustered multi-node broker - a bit like Oracle RAC. Every node is the same broker. Rabbit leverages Erlang clustering.
I hope that one day, like multicast, the entire network will be AMQP aware.
All of todays most successful, scalable networks are intermediated. That's what a modern Ethernet switch/router is.
AMQP is a higher value intermediation, and a different addressing scheme which is more locationally and temporally robust for the convenience of end users.
Pieter has made a significant contribution to the evolution of AMQP up 'til v0-91.
And so have Robert Godfrey, Rafi Schloming, Gordon Sim, and Tony Garnock-Jones from a technical viewpoint, and Carl Trieloff and Alexis Richardson from a market awareness viewpoint.
And many others.
And of course then there me's. I just started with this idea to standardise MOM in 2003….
John
Post preview:
Close preview