OpenSearch, the AWS-sponsored Elasticsearch fork, reaches 1.0 milestone

The AWS-sponsored OpenSearch project has released version 1.0 of its Elasticsearch fork, an evolution of its former project Open Distro for Elasticsearch.

"OpenSearch is a community-driven, open source search and analytics suite derived from Apache 2.0 licensed Elasticsearch 7.10.2 & Kibana 7.10.2. It consists of a search engine daemon (OpenSearch), a visualization and user interface (OpenSearch Dashboards), and advanced features from Open Distro for Elasticsearch like security, alerting, anomaly detection and more," said AWS in a post.

Tweaks since the beta version include the introduction of ARM64 architecture support on Linux, and several new features.

"Now that OpenSearch has added test automation, built infrastructure, made updates to ensure the code is suitable for production use, and thoroughly tested, the project will be able to deliver on a more regular release cadence," AWS added.

A roadmap of what is to come is here.

How did we get here?

OpenSearch has a confusing and colourful history. AWS launched its Elasticsearch service in October 2015, with Werner Vogels, AWS CTO, declaring on Twitter that it was "a great partnership between elastic and AWS" (tweet since deleted); however, according to Elastic co-founder and CEO Shay Banon, there was no collaboration and Elastic protested at the use of its Elasticsearch trademark.

Elasticsearch was originally mostly licensed under Apache 2.0 but Elastic also offered X-Pack or "Elastic Stack Features", first as closed source then under its own less open Elastic licence.

In March 2019, AWS created its own distribution, calling it the Open Distro for Elasticsearch, based on those parts of the product that are Apache 2.0 licensed. AWS VP and software architect Adrian Cockcroft complained that "since June 2018, we have witnessed significant intermingling of proprietary code into the code base" of Elasticsearch and that this was "making it very difficult for developers who want to only work on open source to contribute and participate."

However, he added: "Our intention is not to fork Elasticsearch, and we will be making contributions back to the Apache 2.0-licensed Elasticsearch upstream project."

Elastic remained unhappy with the use of the Elasticsearch name and in September 2019 commenced a lawsuit against AWS, stating: "Due to Amazon's misleading use of the ELASTICSEARCH mark, consumers of search and analytics software are, at least, likely to be confused as to whether Elastic sponsors or approves AESS [Amazon Elasticsearch Service] and Open Distro."

Banon also claimed last year that "unfortunately, our copyrights and trademarks have been abused and misused" and at the start of 2021 changed the product to be dual-licensed under its own Elastic License 2.0 or the SSPL (Server Side Public License) devised by MongoDB.

This had the effect of making it difficult for cloud providers to run the code as a service since they would have to publish all their proprietary code for managing the service. Elastic's move was unpopular with the open-source community, especially those who had contributed code believing it would always be free to use.

AWS responded by introducing OpenSearch in April, based on Open Distro but with a new name and this time explicitly a fork "derived from Elasticsearch 7.10.2." The company also introduced OpenSearch Dashboards, a fork of Kibana 7.10.2, a visualisation tool for Elasticsearch data. AWS said that "we plan to rename our existing Amazon Elasticsearch Service to Amazon OpenSearch Service," though at the time of writing it remains under the old name.

• Open-source companies gather to gripe: Cloud giants sell our code as a service – and we get the square root of nothing
• Pentagon scraps $10bn JEDI winner-takes-all cloud contract
• New mystery AWS product 'Infinidash' goes viral — despite being entirely fictional
• Open-source developers under corporate pressure to adopt less-permissive licenses, Percona CEO says

Towards the end of last month, lawyers for Elasticsearch and AWS filed a court document suggesting that the litigation may soon be settled. It notes the forks and the plans to rename the Amazon Elasticsearch Service.

The document informs us: "The parties remain actively engaged in substantive settlement discussions that seek to resolve this dispute in its entirety, the parties have exchanged multiple iterations of a potential settlement term sheet, and have significantly narrowed the areas remaining in settlement discussions."

The document requests an extension of the court deadlines on the basis of "the possibility of a near-term resolution" of the Elasticsearch claims. It appears therefore that this long-running matter is nearing conclusion.

Elastic will continue its proprietary product development while AWS offers its open-source fork. Note that customers can, for instance, run the official Elastic Cloud on AWS via a marketplace offering with an Amazon account, or on AWS via Elastic's cloud.elastic.co with no Amazon account necessary.

The post by AWS introducing OpenSearch 1.0 says that much effort has gone into "continual removal of proprietary code and marks." The site also notes that "OpenSearch is a registered trademark of Amazon Web Services."

"This whole saga has left a pretty sour taste," observed a user on Hacker News who maintains another open-source search project, Typesense, noting that open source is great for attracting contributors but that it "becomes an albatross around your neck" when it comes to generating revenue.

AWS sells hosted infrastructure and has a strong business with both free and proprietary software. The key question now is how successful the company is in building a community around OpenSearch.

"It's early days but it looks healthy so far," said one contributor.

Another "long-term Elasticsearch user" was less positive, saying: "I've been digging through the commit logs to see if this project is attracting new developers. There's a little bit but most of it is trivial stuff. So, not a lot yet."

They added: "They forked the code but not the developer community. Not yet at least." ®