Sunday, April 28, 2013

Netflix new API: post-REST, Functional, Reactive

At Netflix scale (that is at peaks 1/3 of internet traffic in USA),
even REST style of communication is not efficient and good enough.
So they have internally started implementing more optimized style of communication
that is more efficient on network traffic,
and more efficient on server side processing.

  • The first problem with REST is a need to send multiple requests
    (and wait for responses) that is very dependent on specifics of
    more than 800 types of supported devices.

    The solution are custom, more high-level messages,
    that are implemented for and by client team and running on severs
    to prepare complete data and return in one optimized message.

  • Second issue is how to make this server-side processing efficient,
    without creating too much additional load on servers.
    For this they have implemented "Reactive Functional Programming",
    that is first introduced by Microsoft Research's Eric Meyer as RX Framework
    and added to .NET and other tools.

    Netflix has adopted same style, and implemented for Java platform
    but not Java language: instead they are using Grails in JavaVM.

    article: The Netflix API Optimization Story
    Before (REST):


    After (Rx Client Adapters):


    presentation: Netflix API Revolution @ InfoQ

    presentation: Functional Reactive Programming in the Netflix API@ InfoQ

    blog post: @ Neflix blog

    Interview: Ben Christensen on Resilience at Netflix with Hystrix, Reactive Programming for the JVM with RxJava @ InfoQ


    All we need now is a clever catchy name for the new style protocols,
    and a simplified portable open source implementation, i.e. for Node.js :)
    If that can be a part of some PhD thesis (like REST was), even better :)

    Would "web sockets" and "comet-style" connections be effective here?

  • No comments: