The current version of Atlas has served us very well over the last year, and continues to do so. In that same time we’ve been building various prototypes on top of it, such as TellyTopic, which have highlighted functionality we’d like to add into Atlas in the next major version. Key improvements we’d like to make include:
- Flexible aggregations: you don’t want to have to keep coming back to Atlas to get enough data for a single page. We’ve already made a start on this with the brand_summary and series_summary annotations, but we’d like to do a lot more
- Realtime updates: when there’s a last-minute schedule change, it needs to get reflected quickly
- Handling more data: we’re constantly adding new data sources and extending the scope of existing ones
All of this whilst continuing to be fast and cope with large volumes of traffic.
- We are going to switch to using Cassandra as our database. It will scale with our larger datasets and let us better control what is stored in memory
- Some of our processes are full scan batch jobs right now. As our datasets grow this doesn’t work, so we’re going to be introducing ActiveMQ for change stream updates
- To support more flexible queries we’ll be using ElasticSearch
when’s this happening?
These changes will be done in phases, which we’ve already started on: some new data sources are now stored in Cassandra instead of Mongo. We’ll be posting regular updates on the blog over the coming months, going into more detail on each of the changes.
what does it mean to you?
Along with giving you more data, such as music from musicbrainz, we’ll also be revising the API. It’ll be more consistent and resource-based, much like the new topic and content_group endpoints. These endpoints will give you more useful information that meets common use cases through aggregations, so you’ll not need to call many endpoints.
We will, of course, be keeping you updated with progress on the mailing list, so sign up to watch things progress.
Sergio will be telling you all about Cassandra early next week and in my next post I’ll be talking about how we’re going to be using ActiveMQ in a bit more detail, so stay tuned!