Elasticsearch Nest Scroll API in C#

Feb 25, 2018

Example of using Elasticsearch’s scroll API with NEST in C#.

private void GetAllDocumentsInConversions(string clientId, string yesterdaysDate, 
string todaysDate, string scrollTimeout = "1m", int scrollSize = 200)
        {
            _conversions = new List<Conversion>();

            var elasticClient = new ElasticClient(new Uri("http://root:1ddqd1dkfaJKL2O522@prod-escluster.pixibo.com"));

            ISearchResponse<Conversion> initialResponse = elasticClient.Search<Conversion>
                (scr => scr.Index("tracking_prod")
                     .From(0)
                     .Take(scrollSize)
                     .Query(s =>
                        s.Term(d => d.Field(p => p.clientId).Value(clientId)) &&
                        s.DateRange(d => d.Field(p => p.timestamp).GreaterThanOrEquals(yesterdaysDate).LessThanOrEquals(todaysDate)))
                     .Scroll(scrollTimeout));

            if (!initialResponse.IsValid || string.IsNullOrEmpty(initialResponse.ScrollId))
                throw new Exception(initialResponse.ServerError.Error.Reason);

            if (initialResponse.Documents?.Any() == true)
                _conversions.AddRange(initialResponse.Documents.ToList());

            string scrollid = initialResponse.ScrollId;
            bool isScrollSetHasData = true;

            while (isScrollSetHasData)
            {
                Console.WriteLine(clientId + ": Scrolling Conversions: " + _conversions.Count());

                ISearchResponse<Conversion> loopingResponse = elasticClient.Scroll<Conversion>(scrollTimeout, scrollid);

                if (loopingResponse.IsValid)
                {
                    _conversions.AddRange(loopingResponse.Documents.ToList());
                    scrollid = loopingResponse.ScrollId;
                }
                isScrollSetHasData = loopingResponse.Documents.Any();
            }

            elasticClient.ClearScroll(new ClearScrollRequest(scrollid));
        }

Things to notice in this example of the Elasticsearch Nest Scroll API in C#…

• scrollTimeout only needs to be long enough to last between calls.  Each call to the scroll API with the scrollid resets the timer.
• After scrollTimeout expires Elasticsearch will delete the scroll dataset
• scrollSize (take) in this case controls how many documents are returned with each call to the Scroll API