In a move that will excite curious developers, Vespa is now open source. Image: ImYanis/Shutterstock
Oath, the company born of the Yahoo-AOL merger, releases Vespa data processing engine’s source code.
Vespa, the tool used to power search on the Yahoo network of sites, was yesterday released on GitHub by Oath for any curious developers or companies to take a look.
Although Yahoo web search has mostly been powered by Bing in recent years, Vespa technology is used within the network of Oath sites itself, such as Flickr, Yahoo.com, Yahoo News, Yahoo Sports, Yahoo Finance and Yahoo Gemini.
Now that Vespa is open source, people can now build apps that “compute responses to user requests, over large datasets, at real time and at internet scale”.
Vespa provides efficiency and speed
Applications traditionally have to compute over large datasets at serving time, in search and recommendation, to deliver a list of search results or some articles a reader may like based on the page they are currently on.
Vespa helps to make serving a speedy process, while offering the ability to perform these requests over large datasets more efficiently with algorithms, data management and memory management, all in a single, neat package.
Currently, Vespa is used to manage billions of requests across many Oath sites, processing and serving content and ads almost 90,000 times per second.
Avoiding pesky data bottlenecks
According to a statement from Jon Bratseth, distinguished architect at Vespa: “Vespa distributes data and computation over many machines without any single master as a bottleneck. Where conventional applications work by pulling data into a stateless tier for processing, Vespa instead pushes computations to the data.”
Vespa can be run on premises or in the cloud, and developers can now use it to compute over massive datasets in real time rather than cumbersome management of clusters and data, with it apparently only taking 10 minutes to get an app up and running by following documentation provided.
For small-scale internet companies just starting out, or developers whose interests are piqued by the processing power of Vespa, this open-source software could be a valuable resource.
In a move that will excite curious developers, Vespa is now open source. Image: ImYanis/Shutterstock
