Magento2 Elasticsearch error: cluster_block_exception [FORBIDDEN/12/index read-only / allow delete (api)], flood stage disk watermark exceeded

Posted by

Elasticsearch error: cluster_block_exception

尝试像往常一样将文档发布到 Elasticsearch 时,我收到了这个错误:

cluster_block_exception [FORBIDDEN/12/index read-only / allow delete (api)];

我还在 Elasticsearch 日志中看到了这条消息:

flood stage disk watermark [95%] exceeded ... all indices on this node will marked read-only

默认情况下,Elasticsearch 的决定基于可用磁盘空间的百分比,因此在大磁盘上,即使您有许多 GB 的可用空间,也会发生这种情况。

认情况下,flood stage watermark (洪水阶段)的水印为 95%,因此在 1TB 驱动器上,您至少需要 50GB 的可用空间,否则 Elasticsearch 会将自身置于只读模式。

关洪水阶段水印的文档,请参阅:https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html

正确的解决方案取决于上下文 – 例如生产环境与开发环境。

解决方案 1:释放磁盘空间

释放足够的磁盘空间以便超过 5% 的磁盘可用将解决此问题。但是,一旦有足够的磁盘可用,Elasticsearch 不会自动退出只读模式,您必须执行以下操作来解锁索引:

$ curl -XPUT -H "Content-Type: application/json" https://[YOUR_ELASTICSEARCH_ENDPOINT]:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

解决方案2:更改洪水阶段水印设置

将“cluster.routing.allocation.disk.watermark.flood_stage”设置更改为其他内容。它可以设置为较低的百分比或绝对值。以下是如何从文档更改设置的示例:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "100gb",
    "cluster.routing.allocation.disk.watermark.high": "50gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "10gb",
    "cluster.info.update.interval": "1m"
  }
}

同样,在这样做之后,您必须使用上面的 curl 命令来解锁索引,但之后它们不应再次进入只读模式。

默认情况下,当您的可用磁盘空间不足 5% 时,安装的 Elasticsearch 会进入只读模式。如果您看到与此类似的错误:

Elasticsearch::Transport::Transport::Errors::Forbidden: [403] {"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}
Elasticsearch::Transport::Transport::Errors::Forbidden: [403] {"error":{"root_cause":[{"type":"cluster_block_exception","re​​ason":"blocked by: [FORBIDDEN/12/索引只读/允许删除 (api)];"}],"type":"cluster_block_exception","re​​ason":"被阻止:[FORBIDDEN/12/index 只读/允许删除 (api)];" },"状态":403}

或者在 /usr/local/var/log/elasticsearch.log 中,您可以看到类似于以下内容的日志:

在 [nCxquc7PTxKvs6hLkfonvg][nCxquc7][/usr/local/var/lib/elasticsearch/nodes/0] 上超出了洪水阶段磁盘水印 [95%] 免费:15.3gb[4.1%],此节点上的所有索引都将标记为已读-只要

然后,您可以通过运行以下命令来修复它:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

本文参考