Sunday, February 15, 2015

Apache Mesos, Data Center Operating System (DCOS)

A16z Podcast: The Datacenter Needs an Operating System | Andreessen Horowitz
"Mesosphere’s Benjamin Hindman, the co-creator of Apache Mesos (which came out ofthe U.C. Berkeley AMPLab), joins Steven Sinofsky, a16z board partner (who formerly oversaw the Windows division at Microsoft) for a discussion about all-things-OS."

"Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively."

"What is Mesos? A distributed systems kernel
Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments. Project Features:
  • Scalability to 10,000s of nodes
  • Fault-tolerant replicated master and slaves using ZooKeeper
  • Support for Docker containers
  • Native isolation between tasks with Linux Containers
  • Multi-resource scheduling (memory, CPU, disk, and ports)
  • Java, Python and C++ APIs for developing new parallel applications
  • Web UI for viewing cluster state"

This is raising level of abstraction in data center to "task", that could include multiple processes running on virtual or physical servers, or preferably in containers.  Already used at scale in Twitter, Netflix, PayPal, Airbnb, and others:

Mesos @ InfoQ

No comments: