“‘Startups Don’t Need to Spend Big to Get Big Results,’ Says CEO Robert Nishihara as Ray Summit Highlights AI Innovations”

The relentless complexity of AI infrastructure is a barrier for many companies eager to harness the power of artificial intelligence. From scaling AI platforms to managing distributed computing, developers often find themselves stuck in technical challenges, rather than focusing on innovation. This is the very problem Anyscale, a cutting-edge enterprise tech company, set out to solve when it was founded in 2019.

In the words of Robert Nishihara, Anyscale’s co-founder and CEO: “We started Ray and Anyscale because of our own experience struggling with the scaling challenges around machine learning. It’s striking to see how far the project has come, with industry leaders like Uber, OpenAI, Shopify, and ByteDance building their next-generation ML platforms using Ray.”

Born from Scaling Challenges at Berkeley

The idea behind Anyscale began long before the company itself. It traces its origins to the University of California, Berkeley’s RISELab, where Nishihara and his co-founders, Ion Stoica, and Philipp Moritz, were grappling with the difficulties of scaling AI workloads. They identified a gap in the available tools, particularly in distributed computing for machine learning (ML), which led them to create Ray, an open-source project that would eventually serve as the bedrock for Anyscale.

Ray, introduced in 2017, was a game-changing distributed compute framework designed to handle machine learning workloads at scale, simplifying the task for developers who lacked large infrastructure teams. Ion Stoica, Anyscale’s co-founder and executive chairman, highlighted the project’s importance: “Scaling applications on clusters is a daunting challenge for machine learning engineers who don’t have teams of infrastructure experts behind them. The time wasted on low-level engineering would be better spent on designing algorithms.”

Anyscale’s Mission

With Ray gaining momentum in the open-source community, the founders realized the need for a broader platform that could simplify AI scaling for enterprises. Thus, Anyscale was born. The company’s mission is clear: democratize AI by making it easier for developers to build, manage, and scale AI applications without getting bogged down in the complexities of infrastructure.

Over the past few years, Anyscale’s platform has gained traction across the tech world. Nishihara noted, “As more and more organizations integrate AI into day-to-day operations, we’ve seen Anyscale usage soar 44x over the past year, and Ray is now orchestrating more than 1 million clusters per month.”

Funding and Growth

Anyscale’s rapid rise is a testament to its innovation and the growing demand for scalable AI solutions. The company has attracted substantial funding to fuel its expansion, securing $259.6 million over four funding rounds. Each round marked a significant milestone in the company’s journey:

• Series A (December 2019): Anyscale raised $20.6 million, led by Andreessen Horowitz (a16z), with participation from NEA, Intel Capital, and others.
• Series B (October 2020): A $40 million round, again led by NEA, with Andreessen Horowitz and Intel Capital backing the company.
• Series C (December 2021): This round brought Anyscale’s valuation to $1 billion, solidifying its position as a unicorn in the AI space.
• Series C-II (August 2022): Anyscale raised $99 million, continuing its growth trajectory.

Ben Horowitz, co-founder of Andreessen Horowitz, underscored Anyscale’s potential: “Ray is one of the fastest-growing open-source projects we’ve ever tracked, and it’s being used in production at some of the largest and most sophisticated companies.”

Keerti Melkote‘s recent appointment as CEO of Anyscale represents a pivotal moment for the AI infrastructure firm. With a wealth of experience in scaling startups—most notably founding and steering Aruba Networks to more than $5 billion in revenue—Melkote is poised to provide crucial leadership to Anyscale during a transformative period in the AI sector. The demand for robust AI infrastructure is surging, and Anyscale’s Ray platform has experienced significant adoption, leading to a remarkable quadrupling of the company’s revenue in 2024 and a sixfold increase in Ray usage year over year. Under Melkote’s guidance, the company aims to harness this momentum while tackling the challenges of scaling AI workloads across diverse industries. He emphasizes the pressing market need by stating, “Many organizations lack the resources to build their own proprietary platforms, and Anyscale’s managed services help to fill this gap.”

Scaling AI Workloads Across Industries

Ray’s versatility and power have made it the go-to framework for companies across industries, from Uber and Shopify to OpenAI and ByteDance. Whether it’s Instacart optimizing delivery routing or McKinsey leveraging AI for competitive machine design, Ray is enabling enterprises to scale their AI applications with unprecedented ease.

Key use cases include:

• Instacart: Real-time optimization of delivery routes.
• Riot Games and J.P. Morgan: Scaling reinforcement learning using Ray’s RLlib library.
• Amazon and Ant Group: Processing petabytes of data per day, with Ray deployed across hundreds of thousands of cores.
• McKinsey: Using Ray in high-stakes competitions like the America’s Cup.

These successes underscore Ray’s ability to handle a diverse range of AI workloads, from classical machine learning to generative AI. The framework’s impact is magnified by its role in orchestrating over 1 million clusters per month.

Anyscale’s impact is evident in its work with Canva, a design platform serving over 170 million monthly active users. Canva, managing over 100 machine learning models in production, faced significant challenges in scaling its AI operations efficiently. Anyscale’s collaboration with Canva provided a modern AI platform that dramatically improved Canva’s capabilities, allowing for distributed training that processed millions of images with remarkable efficiency.

The results of this partnership were profound: Canva’s innovation velocity increased by up to 12x, with many machine learning models now training 4-6x faster. For example, an image classification model that previously took 90 minutes to complete a single epoch now finishes in just 6 minutes. Additionally, Anyscale’s solution maximized hardware efficiency by fully utilizing Canva’s GPUs at peak load, eliminating data ingestion bottlenecks.

Canva is doing amazing things in AI!

For more on their AI journey, check out their case study: https://t.co/vZBoFVr1Ov https://t.co/AuUu8WPSex

— Anyscale (@anyscalecompute) October 1, 2024

Uber, another major adopter of Anyscale Ray, has utilized the platform to significantly enhance its AI capabilities. By leveraging Ray, Uber can now run over 20,000 model training jobs each month on its clusters and data centers. Ray has enabled Uber to efficiently use GPUs and maintain scalable, heterogeneous computation clusters for diverse AI workloads.

Uber uses Ray to process a wide array of data types, including tabular data, unstructured Parquet files, and multimodal data for computer vision and video applications. Moreover, by elastically sharing resources through Kubernetes clusters, Uber has optimized both time and costs. The company is moving toward a hybrid cloud setup, and Ray is playing a pivotal role in evolving Uber’s machine learning infrastructure across a multi-cloud, multi-region, and multi-provider environment.

During her fireside chat at #raysummit, Smitha Shyam with @Uber shares how Ray has helped transform their internal processes, and addressed pain points that were previously hard to solve. pic.twitter.com/IGPP8uttgz

— Anyscale (@anyscalecompute) August 24, 2022

Zhitao Li, Director of Engineering for AI and Model Infrastructure at Uber, emphasized the platform’s importance: “With Ray, we are able to run many clusters. I believe we are doing more than 20,000 model training jobs every month on our clusters and in our data centers, and this would not be possible without Ray.”

The Ray Summit 2024: Innovating for the Future

Anyscale continues to push the boundaries of AI development, as demonstrated at the Ray Summit 2024. Held from September 30 to October 2, the summit brought together top industry leaders from companies like Meta, OpenAI, Runway, and a16z to discuss the future of AI infrastructure.

Key takeaways from the summit include:

• The AI landscape faces significant challenges, including increased complexity of AI infrastructure, high costs, inefficiencies in managing AI workloads, and fragmented tech stacks causing operational inefficiencies. Anyscale aims to simplify these barriers with its evolving solutions.
• The importance and volume of unstructured data—text, audio, images, and video—have risen significantly. Ray is tackling this by offering performance improvements and cost savings for data ingestion and batch pre-processing with Ray Data, positioning itself as a critical tool for enterprises handling such data.
• Anyscale now allows developers to create hybrid clusters spanning both cloud and on-prem environments, enabling hybrid infrastructures and simplifying setup for large-scale AI deployment.
• One of the biggest operational challenges—manually determining cost-effective instance types—is being addressed by Anyscale’s newly released feature. This product automatically selects the best instance types based on resource requirements specified in code, and implements region-aware scaling, allowing developers to focus on application development instead of infrastructure management.
•  Organizations across nearly all sectors are using Ray today to enhance their research, product features, or operational efficiency with AI: healthcare, retail, transportation & logistics, education & research, software, finance, media & entertainment, manufacturing, and more.

The event was marked by several key announcements:

• RayTurbo: An optimized version of Ray designed for peak performance, delivering up to 4.5X faster data processing and significantly lowering costs.
• Ray Data: A powerful library for processing unstructured data at scale, supporting data formats like Hudi, Iceberg, and Delta Lake.
• Anyscale Operator for Kubernetes: A breakthrough integration enabling users to deploy Ray on Kubernetes, bringing Anyscale’s managed services into modern software development environments.

Nishihara emphasized Anyscale’s long-term vision at the summit: “There’s going to be a massive infrastructure build-out for AI, with both a hardware and a software piece. Nvidia has nailed the hardware, but there’s still much to be done on the software side. That’s where we come in.”

RayTurbo

One of the most exciting developments to emerge from the Ray Summit is RayTurbo, a game-changing upgrade to the Ray platform. Available exclusively on the Anyscale platform, RayTurbo offers substantial performance improvements, including 5X faster node launching, 60% higher query-per-second (QPS) serving, and 90% cost savings through spot instance support.

By leveraging RayTurbo, companies can process massive datasets at unprecedented speeds, optimizing AI workloads from distributed training to model serving.

The Future of AI with Anyscale

As AI becomes an integral part of every industry, Anyscale is positioning itself at the heart of the infrastructure revolution. With its cutting-edge platform, Ray, and now RayTurbo, the company is setting new standards in AI development.

In the words of Nishihara: “Our goal is to enable every developer and every team to succeed with AI without needing to worry about building and managing infrastructure. We want to remove distributed systems expertise from the critical path of realizing the business potential of AI.”

Solving AI Infrastructure Complexity, One Innovation at a Time

Anyscale’s journey is a testament to the power of vision, innovation, and perseverance. What began as a project in a Berkeley lab has grown into a leading force in AI development, empowering developers and enterprises alike. With Ray orchestrating millions of clusters and new advancements like RayTurbo leading the charge, Anyscale continues to redefine what’s possible in AI.

Their focus on democratizing access to scalable infrastructure isn’t just a technological advancement—it’s a fundamental shift in how businesses of all sizes can leverage AI. This forward-thinking approach has put Anyscale on a trajectory that could see them defining the next era of AI infrastructure. With partnerships deepening and adoption growing, the company is not only solving present challenges but also opening doors to AI’s limitless potential.

The question now is: As AI becomes an indispensable part of modern business, how many companies will build their future on the foundation that Anyscale is laying today? Robert Nishihara. “In other words, nascent startups don’t have to spend big bucks to get the same AI technology that powers big businesses.”