Fix / Repair Index Stuck at Initializing or Unassigned

Fix / Repair Index Stuck at Initializing or Unassigned

STEP 1 - FIND OUT WHICH SHARD IS BROKEN

(use Elastic head) 

STEP 2 - GO TO THE ES NODE and EXECUTE THIS

cd /usr/share/elasticsearch/lib

java -cp lucene-core*.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /data_ssd/es/bh/nodes/0/indices/bh_clk/12/index/ -fix

STEP 3 - May need to execute the above command multiple time until all warning/error are gone.

repeat the step above (fixing elasticsearch index using lucene) until it is no longer complaining of any error.

STEP 4 - Repeat this step to ALL your ElasticSearch nodes

Since your index shard can exists in many of your ElasticSearch node (because elasticsearch automatically relocate your shard for balancing and data distribution), you need to perform this fix to all elasticsearch node(s) that may contain the bad shard.

Don't worry... if the bad shard does not exists in the node you are trying to repair, no harm will be done. It will just tell you it does not exists.

STEP 5 - Check HEALTH to make sure your elasticsearch cluster is now GREEN

curl -XGET "http://10.0.20.$i:9200/_cluster/health?pretty=true"