Implementing Scalable Publish-Subscribe in a Managed Runtime Environment

The reliable multicast, publish-subscribe, and group communication paradigms are highly effective in support of replication and event notification, and could serve as the enabling technologies for new types of applications that are both interactive and decentralized. To fully realize this vision, we need a high-performance, scalable, and reliable multicast engine, as an integral part of the runtime environment. Since the majority of development today is done in managed, strongly-typed environments such as Java or.NET, integration with such environments is of particular importance. What factors limit performance and scalability of a reliable multicast engine in a managed environment? What support from the runtime could improve performance, avoid instabilities, or make such systems easier to build? We focused on answering these questions by analyzing the performance of QuickSilver Scalable Multicast (QSM), a new multicast protocol and a system we’ve built entirely in.NET. We found that memory-related overheads and scheduling-related phenomena dominate the behavior of our system, and that most problems can be alleviated by techniques such as restructuring the protocol stack to limit caching and buffering, and introducing a custom, priority-based scheduling policy.