Fred van den Driessche

f is for feeds

Continuing the Atlas A-Z series, this article is brought to you by the letter F. F is for Feeds.

feeds in atlas

Just as Atlas ingests data from a multitude of different sources it’s also an award-winning supplier of feeds to industry partners. The feeds provided can all be found in the atlas-feeds repository. They take a number of different forms but all of them are based on transformations of the core Atlas data model. The transformations produced are then uploaded to remote systems as required.

Feeds demonstrate one of Atlas’ central strengths: due to its normalized data model a feed for one data set can often be used for further data sets with few changes.

schedule transforms

Transferring data through broadcast schedules is one of the most important and common methods since remote systems predominantly still consider TV and radio to be linear. Since Atlas maintains schedule indexes for all of the broadcasts it ingests, it’s fairly straightforward to transform that data into the required format.

Atlas currently handles two forms of schedule-based feeds:

source-set feeds

Another feed form that Atlas supplies is a transformation on a full source data set. These feeds tend to be quite large so they’re often filtered to exclude unavailable content, for example.

There are two main source-set feeds at present in Atlas:

  • Xbox feeds upload the available Channel 4 dataset so that it can be indexed and linked to for viewing on an Xbox.

  • Sitemaps feeds provide references in the Sitemaps format for indexing by search engines. Currently this is the only feed that doesn’t actively push its results to a remote location.

delta feeds

Delta feeds provide via an initial bootstrap of the entire relevant data set and then a continuing series of delta updates containing just the relevant changes to that data set. The main requirement these feeds make on the data is ubiquitous and accurate change times so the deltas are sufficient to keep the resultant remote dataset synchronized.

There are two feeds of this form:

  • Interlinking feeds provide deltas for full content hierarchies including both broadcasts and on-demand locations.

  • TV-Anytime feeds upload delta files in TV-Anytime XML for indexing.

feeds in the future

Feeds are a powerful idea: once data is translated into Atlas’ core model it becomes simple to transmit it to remote systems in industry-standard formats, allowing Atlas to act as a hub. In the future, with the release of deer and its more complete filtering capabilities, it may be possible to decentralize the feeds. Rather than communicating with the database directly each feed generator would make requests for the required data to the HTTP API. Each feed generator can then be run as a separate system performing the transformations and uploads independently.

That’s about all you need to know about feeds in Atlas. If you have any comments or questions please get in touch below or on the Atlas discussion group.

blog comments powered by Disqus