2020-06-25T20:26:15Z

I am using Elasticsearch 6.8.10 with Spring Boot 2.2.7 and Spring Data Elasticsearch.

I have streams of stats and trends data being stored in Kafka topics. These topics are read using Spring Kafka and stored into MongoDB and Elasticsearch for analysis and reporting. The problem I am having is that when the queues are being processed and the data is written to Elasticsearch, Elasticsearch CPU consumption is continuously around 250%. This leads to sporadic timeout errors across the application. I understand that indexing is an intensive operation but I am trying to understand what I can do to reduce the CPU usage.

Data:

Approx stats queue items (1.2M)
Stats document size (220 bytes)

The VM config details are:

4 CPU, 16GB Memory, 20GB disk (SSD)
Running on a VM in Google Cloud Platform.
VM only used for Elasticsearch

The Docker Elasticsearch config details:

I am using single node (for the moment)

version: '2.4'
services:

  elasticsearch:
    container_name: elasticsearch
    image: 'docker.elastic.co/elasticsearch/elasticsearch:6.8.10'
    ports:
      - '9200:9200'
      - '9300:9300'
    mem_limit: 16GB
    environment:
      - discovery.type=single-node
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms8g -Xmx8g"      
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - 'esdata1:/usr/share/elasticsearch/data'
    restart: always

volumes:
  esdata1:
    driver: local

Example of Spring Stat document:

Shards = 1, replicas = 0

@Document(indexName = "stats_test", type = "stat", shards = 1, replicas = 0)
public class EsStat {

    @Id
    @Field(type = FieldType.Keyword)
    private String id;

    @Field(type = FieldType.Keyword)
    private String entityOrRelationshipId;

    @Field(type = FieldType.Keyword)
    private String articleId;

    @Field(type = FieldType.Keyword)
    private String status;

    @Field(type = FieldType.Keyword)
    private String type;

    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd'T'HH:mm:ss.SSSZ")
    @Field(type = FieldType.Date, format = DateFormat.custom, pattern = "yyyy-MM-dd'T'HH:mm:ss.SSSZ")
    private ZonedDateTime date;

    @JsonProperty("type")
    @Field(type = FieldType.Keyword)
    private String dataSource;

    // getter and setters 
}

Stats Spring repository:

Indexing is done via Spring Data Elasticsearch repository:

public interface StatElasticsearchRepository extends ElasticsearchRepository<EsStat, String> {
}

Stats mapping:

{
  "stats": {
    "mappings": {
      "stat": {
        "properties": {
          "_class": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "articleId": {
            "type": "keyword"
          },
          "dataSource": {
            "type": "keyword"
          },
          "date": {
            "type": "date",
            "format": "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
          },
          "entityOrRelationshipId": {
            "type": "keyword"
          },
          "id": {
            "type": "keyword"
          },
          "status": {
            "type": "keyword"
          },
          "type": {
            "type": "keyword"
          }
        }
      }
    }
  }
}

What can I do to identify why the CPU usage is so high and what can I do to reduce it?

Any advice or recommendations would be much appreciated. I'm happy to add more configuration/output if required.

2025-12-07T19:11:57Z

I am trying to implement multi-tenancy in a Spring Boot application using Spring Data Elasticsearch.

My goal is:

Automatically inject a tenantId filter into all Elasticsearch queries
This must apply to:

Repository-derived queries (findBy…)
Custom @Query JSON queries
CRUD operations like findById, save, etc.
Low-level operations triggered internally (because findById() does not call search(); it uses different ES ops under the hood)
Derived queries → findByPersonTypeName
Custom @Query → findByIdentityNo
findById / getById (which do not use a search query)
Low-level ES operations invoked by Spring Data Elasticsearch internally

Basically, I want something like Hibernate Filters but for ElasticsearchRepository.

I want a global multi-tenant filter that automatically injects:

{ "term": { "tenantId": <currentTenant> } }

Dependencies

Spring Boot brings:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
    <version>3.3.10</version>
</dependency>

<dependency>
    <groupId>org.springframework.data</groupId>
    <artifactId>spring-data-elasticsearch</artifactId>
    <version>5.3.10</version>
</dependency>

<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-client</artifactId>
    <version>8.13.4</version>
</dependency>

Document model

@Document(indexName = PersonDocument.PERSON_INDEX)
public class PersonDocument extends TenantDocument {
    public static final String PERSON_INDEX = "person_index";

    @Id
    @Field(type = FieldType.Long)
    private Long personNo;
}

public class TenantDocument {

    @Field(type = FieldType.Long)
    protected Long tenantId;

    public Long getTenantId() { return tenantId; }
    public void setTenantId(Long tenantId) { this.tenantId = tenantId; }
}

Repository

@Repository
public interface PersonDocumentRepository
        extends ElasticsearchRepository<PersonDocument, Long> {

    List<PersonDocument> findPersonDocumentByPersonTypeName(String name);

    @Query("{\"bool\":{\"must\":[{\"term\":{\"identityNo\":\"?0\"}}]}}")
    PersonDocument findByIdentityNo(String identityNo);
}

2024-11-17T15:17:00Z

I am upgrading the elasticsearch version from 8.14.3 to 8.15.3 and there are some issues with this query-

return new RangeQuery.Builder() .field(fieldName) .gte(JsonData.of(dateToday)) .build() ._toQuery;

Here, .field seems to be deprecated. Is there any other way to rewrite the query in the new version?

2025-10-05T13:52:06Z

Using SpringBoot data elasticsearch

Need to print the Elasticsearch JSON representation of the query

Creating a org.springframework.data.elasticsearch.core.query.CriteriaQuery that is passed to org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate::search()

Can print the criteriaQuery.getCriteria().getCriteriaChain() and get some good information but need to get the JSON query that is created (I believe) by the REST client

Looking at various page e.g., it seems to be a tricky affair https://github.com/spring-projects/spring-data-elasticsearch/issues/1808

Tried log level like: logging.level.org.springframework.data.elasticsearch.core=TRACE

But also no joy

What is the simplest/easiest way to get the JSON ?

Any help is much appreciated

2025-09-11T07:38:32Z

I have a SpringBoot application in which I query an Elasticsearch cluster using an Elasticsearch repository. This is working fine at first. I have the following method in my repository, among others:

List<FooDocument> findByEvent_Guid(String guid);

This method also returns the desired results. However, there are problems: as soon as the result set is larger than 10,000, an "UncategorizedElasticsearchException (all shards failed)" is thrown.

This seems to be a known issue/feature because it can no longer paginate properly with such large data sets. I found a way to circumvent this by using a PageRequest with a pageSize max of 10,000, so that the error no longer occurs, but after 10,000 hits, no more results appear on subsequent pages.

How can I access the additional results?

2025-09-08T19:49:19Z

I faced a problem connected with Elasticsearch while implementing full text search. When I try to find documents in index I face this problem

2025-08-27T09:15:59.544+03:00 ERROR 19622 --- [Muse] [nio-8080-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.data.elasticsearch.UncategorizedElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed] with root cause

co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed

This is the dependency tree:

andrejburlacko@MacBook-Air-Andrej team28 % mvn dependency:tree | grep elasticsearch
[INFO] +- org.springframework.boot:spring-boot-starter-data-elasticsearch:jar:3.5.4:compile
[INFO] |  \- org.springframework.data:spring-data-elasticsearch:jar:5.5.2:compile
[INFO] |     +- co.elastic.clients:elasticsearch-java:jar:8.18.1:compile
[INFO] |     +- org.elasticsearch.client:elasticsearch-rest-client:jar:8.18.1:compile

This is how I run elastic in docker:

docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  -e "ES_JAVA_OPTS=-Xms512m -Xmx512m" \
  elasticsearch:8.18.1

This how my indexes look like:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size dataset.size

yellow open questions _nkW3NzETNanGDBXuhiQyQ 1 1 50 0 23.7kb 23.7kb 23.7kb

yellow open tags BdaRQbRWTzSGQIWxod96VQ 1 1 0 0 249b 249b 249b

This is how mapping structure looks like for question:

{
  "questions" : {
    "mappings" : {
      "properties" : {
        "_class" : {
          "type" : "keyword",
          "index" : false,
          "doc_values" : false
        },
        "id" : {
          "type" : "keyword"
        },
        "questionText" : {
          "type" : "text"
        },
        "score" : {
          "type" : "integer"
        },
        "title" : {
          "type" : "text"
        },
        "viewsCount" : {
          "type" : "integer"
        }
      }
    }
  }
}

When I try search by hands through elastic api, I get my questions successfully

GET http://localhost:9200/questions/_search?q=title:производительность&pretty

{
    "took": 59,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 2,
            "relation": "eq"
        },
        "max_score": 2.920971,
        "hits": [
            {
                "_index": "questions",
                "_id": "b23e4dcc-cc13-4ba3-a354-9aee75238375",
                "_score": 2.920971,
                "_source": {
                    "_class": "ru.t1.usernamedrew.muse.entity.elastic.SearchQuestion",
                    "id": "b23e4dcc-cc13-4ba3-a354-9aee75238375",
                    "title": "SQL индексы производительность",
                    "questionText": "Как индексы влияют на производительность SQL запросов?",
                    "viewsCount": 126,
                    "score": 12
                }
            },
            {
                "_index": "questions",
                "_id": "d54852a3-b6bd-4e99-8376-faa19cfe9a45",
                "_score": 2.5565581,
                "_source": {
                    "_class": "ru.t1.usernamedrew.muse.entity.elastic.SearchQuestion",
                    "id": "d54852a3-b6bd-4e99-8376-faa19cfe9a45",
                    "title": "Java Stream API производительность",
                    "questionText": "Влияет ли использование Stream API на производительность Java приложений?",
                    "viewsCount": 95,
                    "score": 9
                }
            }
        ]
    }
}

This is how my search service looks like:

@Service
@RequiredArgsConstructor
public class SearchServiceImpl implements SearchService {
    private final SearchQuestionRepository searchQuestionRepository;
    private final SearchTagRepository searchTagRepository;
    private final QuestionService questionService;
    private final TagService tagService;
    private final ElasticsearchOperations elasticsearchOperations;

    // Поиск вопросов по заголовку
    @Override
    public Slice<QuestionPreviewDTO> searchQuestions(String query, Pageable pageable) {
        List<UUID> questionIds = searchQuestionRepository.findByTitleContainingIgnoreCase(query, pageable) // <- it falls then findByTitleIgnoreCase try’s to make a query
                .getContent().stream()
                .map(SearchQuestion::getId)
                .toList();

        return questionService.getQuestionsByIds(questionIds, pageable);
    }

    // Поиск тегов по заголовку
    @Override
    public Slice<TagDTO> searchTags(String query, Pageable pageable) {
        List<UUID> tagsIds = searchTagRepository.findByTitleContaining(query, pageable)
                .getContent().stream()
                .map(SearchTag::getId)
                .toList();

        return tagService.findAllByIds(tagsIds, pageable);
    }

    // Индексация вопроса
    @Override
    public void indexQuestion(QuestionPreviewDTO question) {
        searchQuestionRepository.save(convertToSearchQuestion(question));
    }

    @Override
    public void indexTags(Tag tags) {
        // TODO
    }

    @Override
    @Transactional
    public void reindexAllQuestions() {
        searchQuestionRepository.deleteAll();

        int batchSize = 100;
        int offset = 0;

        while (true) {
            Pageable pageable = PageRequest.of(offset, batchSize);
            Slice<QuestionPreviewDTO> questionSlice = questionService.getQuestions(pageable);

            if (questionSlice.getContent().isEmpty()) {
                break;
            }

            List<SearchQuestion> searchQuestions = questionSlice.getContent().stream()
                    .map(this::convertToSearchQuestion)
                    .toList();

            searchQuestionRepository.saveAll(searchQuestions);

            offset++;
        }
    }

    private SearchQuestion convertToSearchQuestion(QuestionPreviewDTO question) {
        return SearchQuestion.builder()
                .id(question.getId())
                .title(question.getTitle())
                .questionText(question.getQuestionText())
                .viewsCount(question.getViewsCount())
                .score(question.getScore())
                .build();
    }
}

My documents:

@Document(indexName = "questions")
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class SearchQuestion {
    @Id
    @Field(type = FieldType.Keyword)
    private UUID id;

    @Field(type = FieldType.Text)
    private String title;

    @Field(type = FieldType.Text)
    private String questionText;

    @Field(type = FieldType.Integer)
    private Integer viewsCount;

    @Field(type = FieldType.Integer)
    private Integer score;
}

@Document(indexName = "tags")
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class SearchTag {
    @Id
    @Field(type = FieldType.Keyword)
    private UUID id;

    @Field(type = FieldType.Text)
    private String title;

    @Field(type = FieldType.Text)
    private String description;
}

And elastic search repositories:

public interface SearchQuestionRepository extends ElasticsearchRepository<SearchQuestion, UUID> {
    Slice<SearchQuestion> findByTitleContainingIgnoreCase(String title, Pageable pageable);
}

public interface SearchTagRepository extends ElasticsearchRepository<SearchTag, UUID> {
    Slice<SearchTag> findByTitleContaining(String title, Pageable pageable);
}

Controller:

@RestController
@RequestMapping("/api/search")
@RequiredArgsConstructor
@Tag(name = "Контроллер поиска", description = "Поиск вопросов по названию и тегам")
public class SearchController {
    private final SearchService searchService;

    @GetMapping("/questions")
    @Operation(summary = "Поиск вопросов по заголовку с использованием ElasticSearch")
    public ResponseEntity<Slice<QuestionPreviewDTO>> searchQuestions(
            @RequestParam("q") String prefix,
            @PageableDefault(size = 20, sort = "createdAt") Pageable pageable) {
        return ResponseEntity.ok(searchService.searchQuestions(prefix, pageable)); // falls in search service, reindex works well
    }

    @GetMapping("/tags")
    @Operation(summary = "Поиск тега по названию с использованием ElasticSearch")
    public ResponseEntity<Slice<TagDTO>> searchTags(
            @RequestParam("q") String prefix,
            @PageableDefault(size = 20) Pageable pageable) {
        return ResponseEntity.ok(searchService.searchTags(prefix, pageable));
    }

    @GetMapping("/reindex")
    @Operation(summary = "Реиндексация вопросов в ElasticSearch")
    public void reindex() {
        searchService.reindexAllQuestions();
    }
}

Full text of exception:

co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed
    at co.elastic.clients.transport.ElasticsearchTransportBase.getApiResponse(ElasticsearchTransportBase.java:357) ~[elasticsearch-java-8.18.1.jar:na]
    at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:141) ~[elasticsearch-java-8.18.1.jar:na]
    at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:5374) ~[elasticsearch-java-8.18.1.jar:na]
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.lambda$doSearch$14(ElasticsearchTemplate.java:356) ~[spring-data-elasticsearch-5.5.2.jar:5.5.2]
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.execute(ElasticsearchTemplate.java:689) ~[spring-data-elasticsearch-5.5.2.jar:5.5.2]
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.doSearch(ElasticsearchTemplate.java:356) ~[spring-data-elasticsearch-5.5.2.jar:5.5.2]
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.search(ElasticsearchTemplate.java:349) ~[spring-data-elasticsearch-5.5.2.jar:5.5.2]
    at org.springframework.data.elasticsearch.core.SearchOperations.searchOne(SearchOperations.java:91) ~[spring-data-elasticsearch-5.5.2.jar:5.5.2]
    at org.springframework.data.elasticsearch.repository.query.AbstractElasticsearchRepositoryQuery.execute(AbstractElasticsearchRepositoryQuery.java:130) ~[spring-data-elasticsearch-5.5.2.jar:5.5.2]
    at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:170) ~[spring-data-commons-3.5.2.jar:3.5.2]
    at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:158) ~[spring-data-commons-3.5.2.jar:3.5.2]
    at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:170) ~[spring-data-commons-3.5.2.jar:3.5.2]
    at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:149) ~[spring-data-commons-3.5.2.jar:3.5.2]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.2.9.jar:6.2.9]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223) ~[spring-aop-6.2.9.jar:6.2.9]
    at jdk.proxy2/jdk.proxy2.$Proxy180.findByTitleContainingIgnoreCase(Unknown Source) ~[na:na]
    at ru.t1.usernamedrew.muse.service.impl.SearchServiceImpl.searchQuestions(SearchServiceImpl.java:40) ~[classes/:na] **<-- it falls when it try's to make a query in SearchQuestionRepository**
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:360) ~[spring-aop-6.2.9.jar:6.2.9]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:724) ~[spring-aop-6.2.9.jar:6.2.9]
    at ru.t1.usernamedrew.muse.service.impl.SearchServiceImpl$$SpringCGLIB$$0.searchQuestions(<generated>) ~[classes/:na]
    at ru.t1.usernamedrew.muse.controller.SearchController.searchQuestions(SearchController.java:30) ~[classes/:na]
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:258) ~[spring-web-6.2.9.jar:6.2.9]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:191) ~[spring-web-6.2.9.jar:6.2.9]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) ~[spring-webmvc-6.2.9.jar:6.2.9]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:991) ~[spring-webmvc-6.2.9.jar:6.2.9]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:896) ~[spring-webmvc-6.2.9.jar:6.2.9]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-6.2.9.jar:6.2.9]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) ~[spring-webmvc-6.2.9.jar:6.2.9]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) ~[spring-webmvc-6.2.9.jar:6.2.9]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-6.2.9.jar:6.2.9]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.2.9.jar:6.2.9]
    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564) ~[tomcat-embed-core-10.1.43.jar:6.0]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.2.9.jar:6.2.9]
    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) ~[tomcat-embed-core-10.1.43.jar:6.0]

I've tried different types of queries. Tried to use NativeQuery, repositories with Page and List return types. Tried to set number_of_replicas to 0. What can I do next to fix this problem, also I tried to configure elastic with java code. All my shards are green and yellow. May be I wrote something wrong in elastic's repositories but I can't identify what's the problem. Spring Data Elastic documentation says it should work with Page and Slice.

2023-03-16T06:21:17Z

I have an existing index "my-index" in Elastic. I'm trying to update this existing index with new fields without losing data.

I also tried to update the existing index with .putMapping function of RestHighLevelClient but I still got an error like this

Elasticsearch exception [type=mapper_parsing_exception, reason=Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:...

Here are the mappings of my existing index

{
  "mappings": {
    "_doc": {
      "properties": {
        "_class": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 420
            }
          }
        },
        "taskType": {
          "type": "integer"
        },
        "updateTime": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss"
        },
        "workType": {
          "type": "integer"
        }
      }
    }
  }
}

And my expected index is

{
  "mappings": {
    "_doc": {  
    "mappings": {
    "dynamic_templates": [
      {
        "strings": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword"
          }
        }
      },
      {
        "dates": {
          "match_mapping_type": "date",
          "mapping": {
            "type": "date",
            "format": "yyyy-MM-dd HH:mm:ss"
          }
        }
      }
    ],
      "properties": {
        "_class": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 420
            }
          }
        },
        "date": {
          "type": "long"
        },
        "favorite_color": {
          "type": "keyword"
        },
        "is_active": {
          "type": "boolean"
        },
        "number_float": {
          "type": "float"
        },
        "number_long": {
          "type": "long"
        },
        "taskType": {
          "type": "integer"
        },
        "updateTime": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss"
        },
        "updatedDate": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss"
        },
        "workType": {
          "type": "integer"
        }
      }
    }
  }
}

Can I update the existing index without losing any data using spring-data-elasticsearch repository??? Thanks for helping.

2024-04-15T14:28:30Z

I have a test project with the following configuration:

spring boot 2.7.18
spring-boot-starter-data-elasticsearch
opensearch 2.6.0

The connection to OpenSearch is configured like this:

@Configuration
class OpensearchConfig: ReactiveElasticsearchConfiguration() {
    val httpHeaders = HttpHeaders()
    httpHeaders.add("Content-type", "application/json")

    override fun clientConfiguration(): ClientConfiguration {
        val config = ClientConfiguration.builder()
            .connectedTo("localhost:9200")
            .usingSsl()
            .withDefaultHeaders(httpHeaders)
            .withBasicAuth("admin", "admin")
            .build()

        return config
    }
}

I'm trying to save a simple entity:

val isIndexExists = opensearch.indexOps(Person::class.java).exists().awaitSingle()

if (!isIndexExists) {
     opensearch.indexOps(Person::class.java).create().subscribe()
}

opensearch.save(Person(UUID.randomUUID().toString(), name))
            .doOnNext(System.out::println)
            .subscribe()

And I got the following error:

org.springframework.dao.DataAccessResourceFailureException: status: 200, [es/indices.exists] Missing [X-Elastic-Product] header. Please check that you are connecting to an Elasticsearch instance, and that any networking filters are preserving that header.; nested exception is java.lang.RuntimeException: status: 200, [es/indices.exists] Missing [X-Elastic-Product] header. Please check that you are connecting to an Elasticsearch instance, and that any networking filters are preserving that header.] with root cause

I didn't find an answer in the documentation whether spring-boot-starter-data-elasticsearch supports OpenSearch, so I had a question - maybe I'm configuring the connection to OpenSearch incorrectly? Or do I need to use opensearch-rest-client instead of spring-boot-starter-data-elasticsearch?

2021-10-04T11:10:10Z

I am new to reactive programming and I´m trying to switch to ReactiveElasticsearchClient.

Now the old code uses RestHighLevelClient and does a scroll search with a do-while loop like the following:

SearchRequest searchRequest = searchRequest(indexName)
        .source(searchSource()
                        .query(QueryBuilders.matchAllQuery())
                        .fetchSource(true)
                        .sort("id", SortOrder.ASC)
                        .size(1000))
        .scroll(TimeValue.timeValueSeconds(60L));

SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
SearchHits searchHits = searchResponse.getHits();
String scrollId = searchResponse.getScrollId();

do {
    /*
    * work with the search hits and
    * and do some logic here
    *
    * */

    SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId).scroll(TimeValue.timeValueSeconds(60));

    try {
        searchResponse = restHighLevelClient.scroll(scrollRequest, RequestOptions.DEFAULT);
        searchHits = searchResponse.getHits();
        scrollId = searchResponse.getScrollId();

    } catch (Exception e) {

        LOGGER.error(e.getMessage());
    }

} while (searchHits.getHits().length > 0);

ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
clearScrollRequest.addScrollId(scrollId);
restHighLevelClient.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);

How can I do the same in a reactive pipeline with the ReactiveElasticsearchClient? Some of the classes that the RestHighLevelClient uses are not available for the reactive client.

2021-05-17T18:36:40Z

I am trying to connect to an existing Elasticsearch instance. The instance is set up and working fine, which I can confirm by accessing it through the browser under https://localhost:9200. I am using Spring Data Elasticsearch version 4.1.2 and I am following the official instructions from https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch.clients.rest.

Unfortunately, when starting the application, the initial health checks of the elasticsearch instance fail, because the code is trying to use http instead of https:

o.s.w.r.f.client.ExchangeFunctions       : [377e90b0] HTTP HEAD http://localhost:9200/

Therefore I added the ".usingSsl()" to the ClientConfiguration, but it does not seem to have any effect and I just don't understand why. The "funny" thing is, if I implement the reactive client as described here (https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch.clients.reactive), https suddenly works and the applications starts perfectly fine. However, I want to use the regular high level client.

Can you help me change the configuration in a way so that it finally makes use of https instead of http?

@Configuration
class ElasticsearchClientConfig: AbstractElasticsearchConfiguration() {
    
    @Bean
    override fun elasticsearchClient(): RestHighLevelClient {
           val clientConfiguration = ClientConfiguration.builder()
                 .connectedTo("localhost:9200")
                 .usingSsl()
                 .build();

        return RestClients.create(clientConfiguration).rest();
    }
}

2025-05-12T11:35:32Z

I have a Spring Boot 3.3.x application that uses Spring Data Elasticsearch to query a single Elasticsearch index.

The query is built dynamically using the Criteria API. Now, I need to add a new condition using a match_phrase query.

However, I couldn't find any documentation or examples showing how to use match_phrase with the Criteria API.

One option would be to refactor the entire solution to use native queries, but that doesn't seem reasonable or trivial in our case.

Another workaround I found is to use NativeQueryBuilder, set the CriteriaQuery as the main query, and then add a manually constructed match_phrase query as a filter:

return new NativeQueryBuilder()
    .withQuery(criteriaQuery)
    .withFilter(nativeMatchPhraseQuery) 
    .withPageable(pageable)
    .build();

While this kind of works, it doesn't feel clean or intuitive.

Is there a way to use match_phrase within a Criteria query directly? Or a better pattern to combine Criteria and native queries more elegantly?

2025-04-23T02:22:36Z

This is a followup question to (Is there a way to have field level audit in elastic search?)

Step 1:

#push a sample doc
PUT my_index/_doc/1
{
  "created_at": "2025-02-24T13:00:00Z",
  "email": "john@example.com",
  "name": "dogan",
  "field_update_timestamps": {
    "email": "2025-02-24T13:00:00Z",
    "name": "2025-02-24T13:00:00Z"
  }
}

Step 2:

#update the doc with `_update` API call
POST my_index/_update/1
{
  "script": {
    "source": """
      if (!ctx._source.containsKey('field_update_timestamps')) {ctx._source.field_update_timestamps = [:]; }
      for (entry in params.entrySet()) {
        ctx._source[entry.key] = entry.value; ctx._source.field_update_timestamps[entry.key] = 
        new java.text.SimpleDateFormat('yyyy-MM-dd\'T\'HH:mm:ss\'Z\'').format(new java.util.Date());
      }
    """,
    "lang": "painless",
    "params": {
      "name": "musab"
    }
  }
}

So can we update the same record in the index concurrently? I mean if we start adding different domain like attendance in office or worked from home. {"wfh": "", "wfo": "" concurrently. Please note no 2 separate threads will put the same fields concurrently ie there will be no another thread which will be updating "wfh" or "wfo", however there can be another thread writing "email". Will there be any optimistic lock exception or data loss?

Thanks

Cc: @Musab Dugan

2024-10-04T08:52:35Z

I have a custom Kafka library that for business reasons I'm forced to use. My use case is to read each record from Kafka and save to a repository. The entry to my repository is a callback I write that implements something like

interface Listener<T> {
    Response apply(T val);
}

The library takes my listener, iterates over the ConsumerRecords<T> from Kafka, and away I go. I wanted to see if I could use a reactive repository in Spring Data to save everything. Something like

Response apply(T val) {

    myRepo.save(val)
       .subscribe()

...
    return response;

}

So far I've only tried calling subscribe on the returned value of the Repository.save, but I don't have a way to dispose of all of these Publishers

But I immediately run into questions:

I'm creating a Mono for every iteration within ConsumerRecords, but I'm never disposing of any of them. I can't actually tell when to dispose of each Mono so this sounds like it would just create a memory leak
I know for a fact that the library starts with the ConsumerRecords collection, so it seems like this should naturally be a Flux, but since my entry is the callback I can't figure out how to make this happen

What would be ideal is to have some way to declare a Flux an set it so that each payload passed in is added to the flux, then call MyRepository.saveAll(Flux vals) and return a flux I can dispose of later, but if I understand the Reactor guide, I can't just add something to an exising Flux.

Is this just a hopeless cause? If I alternatively just take the Mono instance that comes back from calling MyRepository.save(T), when do I call dispose on it?

2025-03-19T06:07:25Z

How should I implement script sort in spring-data-elasticsearch 5.3.x

NativeQueryBuilder nativeQueryBuilder= new NativeQueryBuilder();
nativeQueryBuilder.withQuery(query);
nativeQueryBuilder.withSort(Sort.Order.desc('key'));
return elasticsearchTemplate.search(nativeQueryBuilder.build(), xxx.class);

Now, I want to add a script-based sorting on top of this. I know how to write the script, but it seems that script-based sorting is not supported in the context of NativeQueryBuilder.

2021-04-10T02:08:12Z

Across the web, I can see that spring-data-elasticsearch has some configuration properties that you can define in your application.properties, such as:

spring.data.elasticsearch.repositories.enabled=true
spring.data.elasticsearch.cluster-nodes=localhost:9300
spring.data.elasticsearch.cluster-name=elasticsearch

elasticsearch.index.name=my_index
elasticsearch.user.type=user

However, in IntelliJ, I can see that, for example:

spring.data.elasticsearch.cluster-nodes=localhost:9300
spring.data.elasticsearch.cluster-name=elasticsearch

... are actually now deprecated. However, I can't seem to find anything in the spring-data-elasticsearch documentations that either lists what the available properies are, or, what the deprecated ones should be replaced with instead.

Any help would be welcome. Thanks in advance!

2021-06-23T14:14:44Z

I'm facing a weird issue. I have 2 laptops, one running under Windows where everything works fine and on the other side, a Mac Mini M1.

Assuming, I'm developing a basic Spring Boot application with Elasticsearch as Repository. In the application.yml, as usual I have the following properties :

spring:
  elasticsearch:
    rest:
      uris: https://dummy-cloud-provider.cloud.com:12833
      username: myusername
      password: mypassword

But everytime I want to start the application on my Mac, I have the following error :

2021-06-23 15:55:51.972 ERROR [reactor-http-nio-3] r.c.p.Operators - Operator called default onErrorDropped
reactor.core.Exceptions$ErrorCallbackNotImplemented: org.springframework.data.elasticsearch.client.NoReachableHostException: Host 'localhost:9200' not reachable. Cluster state is offline.
Caused by: org.springframework.data.elasticsearch.client.NoReachableHostException: Host 'localhost:9200' not reachable. Cluster state is offline.
    at org.springframework.data.elasticsearch.client.reactive.SingleNodeHostProvider.lambda$lookupActiveHost$3(SingleNodeHostProvider.java:101)
    at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:102)
    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:220)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2397)
    ...

I don't understand why it always skip my application.yml variables and try to reach localhost:9200.

I hope someone can help me :)

Best regards

2025-02-12T11:52:20Z

Custom @ReadingConverter isn't being triggered but when I was initially implementing this code it worked normally.

Configuration class:

@Configuration
public class ElasticsearchConfig {

  @Bean
  public MappingElasticsearchConverter mappingElasticsearchConverter() {
    return new MappingElasticsearchConverter(new SimpleElasticsearchMappingContext());
  }

  @Bean
  public ElasticsearchCustomConversions elasticsearchCustomConversions(
      MappingElasticsearchConverter mappingElasticsearchConverter, ObjectMapper objectMapper) {
    return new ElasticsearchCustomConversions(List.of(
        new LogEntryReadingConverter(mappingElasticsearchConverter, objectMapper)
    ));
  }
}

Here is the implementation of the converter:

@Component
@ReadingConverter
public class LogEntryReadingConverter implements Converter<Document, LogEntry> {

  private final MappingElasticsearchConverter elasticsearchConverter;
  private final ObjectMapper objectMapper;

  public LogEntryReadingConverter(MappingElasticsearchConverter elasticsearchConverter, ObjectMapper objectMapper) {
    this.elasticsearchConverter = elasticsearchConverter;
    this.objectMapper = objectMapper;
  }

  public LogEntry convert(Document source) {
    // Use Elasticsearch's internal conversion (respects @Field annotations)
    LogEntry logEntry = elasticsearchConverter.read(LogEntry.class, source);

    // some custom logic is here

    return logEntry;
  }

My document class:

@Data
@JsonIgnoreProperties(ignoreUnknown = true)
@Document(indexName = "logstash")
public class LogEntry implements ILogModel {

  @Id
  private String id;
  // more fields are here
}

My repository class:

@Repository
public interface HaProxyLogRepository extends ElasticsearchRepository<LogEntry, String> {

    Page<LogEntry> findByTimestampBetween(LocalDateTime from, LocalDateTime to, Pageable pageable);

}

I would expect the convert method to be called before execution of the repository method "findByTimestampBetween" on method call. It worked before, for some odd reason it never triggers now.

I am using Spring Boot 3.1.2. and Spring Data Elasticsearch 5.1.2.

2025-02-24T09:33:04Z

I have an index with say 10 fields. These fields may get modified from time to time. Is there a way to add any field level timestamp, which can store the timestamp when it was last updated and also can be retrieved by api?

Thanks in advance.

2016-09-17T16:39:21Z

I'm trying to use both Spring Data Cassandra & Spring Data ElasticSearch in an Spring Boot application. I already put Cassandra Repository & ElasticSearch Repository in different packages.

@EnableElasticsearchRepositories(basePackages = "com.my.app.es.repository")
public class ElasticsearchConfig {
   ...
}

@EnableCassandraRepositories(basePackages = { "com.my.app.cassandra.repository" })
public class CassandraConfig extends AbstractCassandraConfiguration {
   ...
}

When I tried to run, I got the following exception::

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'converter' defined in class path resource [com/my/app/config/CassandraConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.cassandra.convert.CassandraConverter]: Factory method 'converter' threw exception; nested exception is java.lang.IllegalStateException: @Bean method CassandraConfig.mappingContext called as a bean reference for type [org.springframework.data.cassandra.mapping.CassandraMappingContext] but overridden by non-compatible bean instance of type [org.springframework.data.elasticsearch.core.mapping.SimpleElasticsearchMappingContext]. Overriding bean of same name declared in: class path resource [org/springframework/boot/autoconfigure/data/elasticsearch/ElasticsearchDataAutoConfiguration.class]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:776) ~[spring-beans-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861) ~[spring-context-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541) ~[spring-context-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) ~[spring-boot-1.4.0.RELEASE.jar:1.4.0.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) [spring-boot-1.4.0.RELEASE.jar:1.4.0.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:369) [spring-boot-1.4.0.RELEASE.jar:1.4.0.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:313) [spring-boot-1.4.0.RELEASE.jar:1.4.0.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1185) [spring-boot-1.4.0.RELEASE.jar:1.4.0.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1174) [spring-boot-1.4.0.RELEASE.jar:1.4.0.RELEASE]

2022-07-12T20:17:51Z

I am having an issue trying to implement optimistic locking when executing a document upsert.

I followed all the guides, however I am getting the following error:

Validation Failed: 1: compare and write operations can not be used with upsert;

I can trace this error back to this validate method, however I cannot find an explanation as to why this cannot be done nor HOW to do an upsert without running into conflicts.

Here's where I am getting the error in my code:

private UpdateResponse doAction(UUID esId, List<Comment> comments, dSeqNoPrimaryTerm seqPrimNo, String indexName) {
        UpdateResponse response = null;
        Map<String, Object> params = new HashMap<>();
        params.put("comments", comments);
        Builder builder = UpdateQuery
                .builder(esId.toString())
                .withDocAsUpsert(true)
                .withDocument(Document.from(params))
                .withRefresh(Refresh.True);
        if (seqPrimNo != null) {
            builder.withIfSeqNo((int) seqPrimNo.getSequenceNumber());
            builder.withIfPrimaryTerm((int) seqPrimNo.getPrimaryTerm());
        }
        UpdateQuery query = builder.build();
        response = elasticsearchTemplate.update(query, IndexCoordinates.of(indexName));  // <-- ERRORS HERE
        return response;
    }

Does that mean that we cannot do optimistic locking when doing an upsert? What am I missing?

2023-05-16T06:03:47Z

I am new at stackoverflow and happy to learn from all

I am required to upgrade my Java spring boot program from:

Java 11 to Java 17
Spring Boot 2.4.2 to Spring Boot 3.0.5

After change the pom.xml:

java.version: 17
org.springframework.boot:spring-boot-starter-parent:3.0.5

My elasticsearch dependency (org.springframework.boot:spring-boot-starter-data-elasticsearch) also upgrade to 3.0.5. However, I found the following packages are missing:

org.apache.lucene.search.join.ScoreMode
org.elasticsearch.index.query.BoolQueryBuilder
org.elasticsearch.index.query.QueryBuilder
org.elasticsearch.index.query.QueryBuilders
org.elasticsearch.index.query.TermsQueryBuilder

So I have some questions about the new Elasticsearch client library:

Whether not support using the QueryBuilder again, so I require to change all of service code about using above package?
Whether not support reactive mode at 5.0.5 version? If both answer are YES, it is really painful for me to upgrade it.

The following is the base information of my ES: version: 7.10.1

Also, I have a stupid question want to confirm: according to the following information provided by Spring-Data-Elasticsearch: https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#preface.versions

If my ES version is 7.10.1, I only can upgrade my spring boot to 2.7.x? or I upgrade to 3.0.5 also can support the older version.

If there are any things missing, I will provide again. Sorry for my poor English and terms. Thank you for your answers.

I had tried change the pom with following pattern:

Case 1:

Upgrade Java from 11 to 17
Upgrade Java Spring Boot from 2.4.2 to 3.0.5 or 3.0.6
Let the org.springframework.boot:spring-boot-starter-data-elasticsearch version same as parent

Result: Some ES packages are missing and program cannot be run

Case 2:

Upgrade Java from 11 to 17
Keep Java Spring Boot 2.4.2
Upgrade org.springframework.boot:spring-boot-starter-data-elasticsearch to 3.0.5 or 3.0.6

Result: program runs without error

Case 3:

Upgrade Java from 11 to 17
Upgrade Java Spring Boot from 2.4.2 to 3.0.5 or 3.0.6
Keep org.springframework.boot:spring-boot-starter-data-elasticsearch:2.4.2

Result: some ES packages are missing and program cannot be run

Thus, It seems the missing package problems is caused by Upgrade Java Spring Boot from 2.4.2 to 3.0.x

2025-01-21T17:25:51Z

I'm using Elasticsearch filters and I would like to apply a must condition for selected regions (e.g., Occitanie and Île-de-France). Additionally, I want to ensure that profiles with the region "Toute la France" are also included in the search results, even if they are not part of the selected regions.

Currently, I am doing the following:

if (regions != null && !regions.isEmpty()) { 

BoolQuery.Builder finalQuery = new BoolQuery.Builder();
      
  for (String regionName : regions) {
            finalQuery.must(Query.of(q -> q.nested(n -> n
                    .path("regions")
                    .query(Query.of(inner -> inner.match(m -> m
                            .field("regions.regionName")
                            .query(regionName)
                    )))
            )));
        }

        finalQuery.should(Query.of(q -> q.nested(n -> n
                .path("regions")
                .query(Query.of(inner -> inner.match(m -> m
                        .field("regions.regionName")
                        .query("Toute la France")
                        .boost(4.0f)
                )))
        )));

        finalQuery.minimumShouldMatch("1");

        boolQuery.filter(Query.of(q -> q.bool(finalQuery.build())));
    }

However, this query only returns profiles that match the selected regions (must), but it does not include profiles with "Toute la France."

How can I modify the query to include profiles with "Toute la France" along with the selected regions?

Thanks in advance for your help!

2015-03-01T18:04:07Z

I have successfully created a query using ElasticSearch's _plugin/head interface. The query is meant to return the latest timestamp for a specific device at a specific location. The query looks as follows:

{  
   "query":{  
      "bool":{  
         "must":[  
            {  
               "term":{  
                  "deviceevent.location.id":"1"
               }
            },
            {  
               "term":{  
                  "deviceevent.deviceId":"AHE1LDD01"
               }
            }
         ]
      }
   },
   "from":0,
   "size":1,
   "sort":{  
      "timestamp":{  
         "order":"desc"
      }
   }
}

The above query works as intended. Now using Spring-Boot and Spring-Data-ElasticSearch, I defined my own ElasticSearchRepository which looks as follows:

package com.repository.elasticsearch;

import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;
import org.springframework.data.elasticsearch.annotations.Query;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;

import com.domain.DeviceEvent;

public interface DeviceEventRepository extends ElasticsearchRepository<DeviceEvent, String>
{
    @Query("{\"bool\":{\"must\":[{\"term\":{\"deviceevent.location.id\": \"?0\"}},{\"term\":{\"deviceevent.deviceId\": \"?1\"}}]}},\"from\": 0,\"size\": 1,\"sort\":{\"timestamp\":{\"order\":\"desc\"}}")
    DeviceEvent findLatestCheckInAtLocation(Long locationId, String deviceId);
}

The above code is breaking mainly because I would expect it to return one DeviceEvent, but it's actually returning a device events with count = 10 (The default Page size). It seems also that the results are not being ordered by the timestamp in a descending order. It's as if the size and order parts of the query are not being picked up.

What am I doing wrong here?

2025-01-08T09:24:08Z

Now, RestHighLevelClient is deprecated. So i want rewrite code with ElasticSearch Java Api Client in my spring project.

Most of the functions are being modified properly, but I have not found an appropriate way to modify this function.

Below is my existing code.

//    import org.elasticsearch.index.reindex.DeleteByQueryRequest;
    private String deleteLarge(String indexName, Long categoryNumber, Optional<Long> age) {
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery()
                .filter(QueryBuilders.matchQuery("category", categoryNumber));
        age.ifPresent(a -> boolQueryBuilder
                .filter(QueryBuilders.matchQuery("age", a)));

        DeleteByQueryRequest request = new DeleteByQueryRequest(indexName)
                .setQuery(boolQueryBuilder)
                .setSlices(10);

        try {
            TaskSubmissionResponse taskSubmissionResponse = restHighLevelClient.submitDeleteByQueryTask(request, RequestOptions.DEFAULT);
            logger.info("DeleteByQueryTask task id: {}", taskSubmissionResponse.getTask());
            return taskSubmissionResponse.getTask();
        } catch (Exception e) {
            logger.error(e.getMessage());
        }
        return null;
    }

The purpose of this code is to delete a very large volume of documents (ranging from tens of thousands to millions). Since it was difficult to estimate the execution time, I submitted a task to be performed internally by ES.

However, it seems that the ElasticSearch Java API Client does not have such a feature. In the official documentation, there is a Class like ElasticsearchTasksClient, which provides the ability to get info or cancel task, but not to submit new tasks.

Here are my questions:

Does ES no longer support tasks? Is it gradually being deprecated?
If not, is there another way to submit new tasks?
If it is indeed unavailable, is there another method to achieve similar functionality? Would using deleteByQuery in ElasticsearchAsyncClient be equivalent to submitting a task?

2025-01-08T15:19:39Z

The following code will throw an exception: [es/search] failed: [parsing_exception] unknown query [query].

    @Query(value = """
{
  "query": {
    "script_score": {
      "query": {
        "match_all": {}
      },
      "script": {
        "source": "cosineSimilarity(params.vector, doc['nameVector']) + cosineSimilarity(params.vector, doc['descriptionVector'])",
        "params": {
          "vector": ?0
        }
      }
    }
  },
  "sort": [
    { "_score": { "order": "desc" } }
  ]
}
""")

    Flux<SearchHit<ClubPO>> queryClubs(List<Double> vector);

The following code can run successfully.

    @Query("""
{
  "script_score": {
    "query": {
      "match_all": {}
    },
    "script": {
      "source": "cosineSimilarity(params.vector, doc['nameVector']) + cosineSimilarity(params.vector, doc['descriptionVector'])",
      "params": {
        "vector": ?0
      }
    }
  }
}
""")
    Flux<SearchHit<ClubPO>> queryClubs(List<Double> vector);

What should I do?

Are there any other annotations? For example, ones that support more raw query syntax?

# {"error":{"root_cause":[{"type":"parsing_exception","reason":"unknown query [query]","line":2,"col":12}],"type":"parsing_exception","reason":"unknown query [query]","line":2,"col":12,"caused_by":{"type":"named_object_not_found_exception","reason":"[2:12] unknown field [query]"}},"status":400}

2024-12-21T17:57:37Z

I want to use the following elastic search query in Java to search through all my documents in my index.

{
  "query": {
    "query_string": {
      "query": "my search input"
    }
  }
}

When I use CURL it works fine and i get results back, but when i try to use this query in Java, it complains and gives me the following exception:

co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [parsing_exception] unknown query [query]

This is what my code looks like in Java:

    @GetMapping("/search")
    public void searchResume(@RequestParam(name = "search") String query) {
        Query q = new StringQuery("""
                {
                  "query": {
                    "query_string": {
                      "query": "my search input"
                    }
                  }
                }
                """);
        System.out.println(operations.search(q, Resume.class));
    }

Does someone know why it works using CURL but does not work when using Java and spring-data-elasticsearch?

I tried googling a lot and even asked chatgpt for help but I can't seem to find out why it doesn't work. Thank you in advance.

2017-05-28T11:31:59Z

I'm new to the community so I apologise if I do something wrong.

I'm using spring data elasticsearch (2.0.4/2.4) And I would like to make a bulk insert and delete. But ElasticsearchTemplate only contains a method bulkInsert

@Override
public void bulkIndex(List<IndexQuery> queries) {
    BulkRequestBuilder bulkRequest = client.prepareBulk();
    for (IndexQuery query : queries) {
        bulkRequest.add(prepareIndex(query));
    }
    BulkResponse bulkResponse = bulkRequest.execute().actionGet();
    if (bulkResponse.hasFailures()) {
        Map<String, String> failedDocuments = new HashMap<String, String>();
        for (BulkItemResponse item : bulkResponse.getItems()) {
            if (item.isFailed())
                failedDocuments.put(item.getId(), item.getFailureMessage());
        }
        throw new ElasticsearchException(
                "Bulk indexing has failures. Use ElasticsearchException.getFailedDocuments() for detailed messages ["
                        + failedDocuments + "]", failedDocuments
        );
    }
}

So I have created a bulk method to handle both but I can't access the method prepareIndex which is private.

Are you aware of any solution to, in one bulk, index and delete documents or should I use reflection to change visibility of the prepareIndex method or is there any easy way to create an indexRequest from a model/POJO?

2024-10-30T22:06:14Z

I had an ELK server that I needed to shut down and replace with a new one. I had a Java Spring Boot project that was connected to the original ELK server. After moving to the new server, all the indices were reset, and I lost all previous data.

Now, I am encountering issues trying to connect my Spring Boot application to the new ELK server. I have verified that the ELK server is running and is accessible. However, when I start my Spring Boot application, I receive the following error:

2024-10-31T00:42:37.816+03:00  WARN 9020 --- [data-leak-service] [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'Controller' defined in file [Controller.class]: Unsatisfied dependency expressed through constructor parameter 0: Error creating bean with name 'Service' defined in file [Service.class]: Unsatisfied dependency expressed through constructor parameter 0: Error creating bean with name 'DataRepository' defined in DataRepository defined in @EnableElasticsearchRepositories declared on ElasticSearchConfiguration: Cannot resolve reference to bean 'elasticsearchTemplate' while setting bean property 'elasticsearchOperations' ...

I suspect that this issue may be related to the ELK configuration. Here is my current configuration:

@Configuration 
public class ElasticSearchConfiguration {

    @Value("${ElasticUrl}")
    private String elkUrl;

    @Value("${elasticsearch.username}")
    private String username;

    @Value("${elasticsearch.password}")
    private String password;

    @Bean
    public ElasticsearchClient elasticsearchClient() {
        // Set up credentials provider for basic authentication
        BasicCredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(
                AuthScope.ANY,
                new UsernamePasswordCredentials(username, password)
        );

        // Configure RestClient with authentication
        RestClient restClient = RestClient.builder(HttpHost.create(elkUrl))
                .setHttpClientConfigCallback(httpClientBuilder ->
                        httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider))
                .build();

        // Set up transport with Jackson JSON mapper
        ElasticsearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());

        // Return the authenticated Elasticsearch client
        return new ElasticsearchClient(transport);
    }
}

Additionally, I added the property for automatic index creation in my application.properties:

spring.elasticsearch.index.auto-create=true

I am using the following dependencies:

<dependency>
    <groupId>co.elastic.clients</groupId>
    <artifactId>elasticsearch-java</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>

I am using the Elasticsearch client version 8.15.1.

Previously, I had a different configuration that worked without authentication:

@Configuration
public class ElasticsearchConfig {

    @Bean
    public ElasticsearchClient elasticsearchClient() {
        // Create the low-level client (without authentication)
        RestClient restClient = RestClient.builder(HttpHost.create("http://elkip:9200")).build();

        // Create the transport with a Jackson JSON mapper
        ElasticsearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());

        // Create the Elasticsearch client
        return new ElasticsearchClient(transport);
    }
}

What could be causing this error, and how can I resolve it? Additionally, what steps do I need to take to enable automatic index creation?

2021-05-27T10:33:17Z

I read through https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#reference to begin with

My requirements

I want to use percolator. Is there any support for it in spring data elasticsearch? I don't see any in the above link although I understand that percolating is same as indexing (technically from using spring data elasticsearch's perspective). So I can use the indexing part of spring data elasticsearch but just checking if there are any that are specific to percolator.
I want to create an index dynamically. I do understand I can achieve that using SpEL template expression as mentioned in https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch.mapping.meta-model.annotations but my case is slightly different, I will get the index name via the RequestParam as part of the API call. So this means as of my knowledge I cannot use SpEL or try something like https://stackoverflow.com/a/33520421/4068218
I see I can use ElasticsearchOperations or ElasticsearchRepository to create Index. Because of #2 (i.e index name via request parameter) I think ElasticsearchOperations better suits but I see IndexOperations facilitating createMapping, createSettings but not both together. I see putMapping too but I dont see anything that says both mapping and settings. The reason I want both is I want to create something like below to begin with

  "settings" : {
                "index" : {
                  "number_of_shards" : 1,
                  "number_of_replicas" : 0
                }
              },
            "mappings": {
                "properties": {
                  "message": {
                    "type": "text"
                  },
                  "query": {
                    "type": "percolator"
                  }
                }
              }

Bottom line :- How do I create an index (name of the index will be dynamic via request param) with mappings, settings using ElasticsearchOperations?
Any lead/help is much appreciated

2021-11-09T10:59:41Z

Current Im working on a spring boot project to fetch the data from specific Elastic search index using Spring Data ElasticSearch (version 4.1.7). Im able to connect to elastic search and fetch all the data associated with a specific index. But there are some fields in the index which are returning null value. Where as the actual value exist in the index in elastic search .
Im having issues in retriving the specific fields due to incorrect field types. I have created a Document Class which

@Document(indexName = "netstat*")
@Setting(settingPath = "static/es-settings.json")
public class NetworkStats {
@Id
@Field(type = FieldType.Keyword)
private String id; 

@Field
private String timeStamp ;

@Field(type = FieldType.Text)
private String podName ; 

@Field(type = FieldType.Text)
private String iporHost ;

@Field(type = FieldType.Auto)
private String connectionCount ; 

@Field(type = FieldType.Text)
private String command ;    
}

Apart from the above I have getters & setters for the field as well defined.Im unabel to retrieve the values for fields connectionCount, iporHost, podName, timeStamp. Though the fields have values in the elastic index.
Sample data from ElasticSearch below

{
            "_index": "netstat-2021.11.09",
            "_type": "_doc",
            "_id": "0sJmA30BW7LXXVrK0nCg",
            "_score": 1.0,
            "_source": {
                "podname": "logindeployment-5479f7-4f2qr",
                "input": {
                    "type": "log"
                },
                "iporhost": "6200",
                "tags": [],
                "command": "OUTESTAB",
                "agent": {
                    "version": "7.15.1",
                    "id": "e8feeb4e-1f9e-41b2-bcba-38e507d176db",
                    "type": "filebeat",
                    "hostname": "BL1726L",
                    "name": "BL1726L",
                    "ephemeral_id": "c2859ece-3e95-4204-909d-af59322b9fa2"
                },
                "timestamp": "05-10-21T13:17:25.014719626",
                "ecs": {
                    "version": "1.11.0"
                },
                "connectionscount": "2",
                "@timestamp": "2021-11-09T06:31:29.104Z",
                "log": {
                    "offset": 1002,
                    "file": {
                        "path": "D:\\elastic_stack\\data\\log\\netStats.log"
                    }
                }
            }
        },

Require help on how to retrieve the data for the above mentioned fields.I guess there is some issue with field mapping. Any help on this is much appreciated.