The ability of a caching solution to rapidly adapt to changes in web content and overall traffic mix and evolve over time is critical. The list of top domains is in constant flux – there have been several instances of new web sites, unknown until recently, generating a significant amount of traffic within a short duration. Current transparent caching solutions require complete software upgrades to respond effectively to changes in content providers’ caching rules. In addition, most solutions use the Uniform Resource Locator (URL) as the unique identifier of a cached object. As a result, they have difficulty in delivering an effective cache hit ratio for domains that use dynamic URLs – they have to store a separate copy derived from each URL. For example, portals like YouTube have specifically developed techniques that prevent their videos from being effectively distributed by caches. Opening the same video several times results in completely different URLs and URL parameters. Specifically, YouTube uses a redirection model for content distribution, which means the same video objects often originate from a different server with a different URL.
MediaWarp has developed a plugin architecture that can rapidly respond to market needs with plugin modules that can be released between major release cycles. Moreover, they are customer extensible and modules can be stacked on top of one another to extend functionality. This allows operators to respond rapidly to ongoing changes in the Internet. Plugins can be downloaded while the system is in production which avoids system downtime. For example, MediaWarp has developed a specific YouTube plugin that converts all dynamic URLs to a standard URL using the metadata in the URL. As shown in the example, the plugin converts a rather long URL to a much shorter and static URL using the unique video ID metadata. This enables the plugin to recognize that the different URLs actually refer to the same video and thereby to locate the unique video in the cache.
If YouTube changes the metadata in the future, MediaWarp could very easily modify the plugin to adapt to the change. Using simple interfaces, plugins can be used to modify incoming requests as well as proxied responses. A very rich and flexible cache policy framework can also be developed using plugins.
In summary, a plugin architecture is required to maintain the effectiveness of a caching solution by responding quickly to changes in content providers’ caching rules and the emergence of new ‘hot traffic’ domains.