Elasticsearch plugin - Query DSL not functioning as expected when referencing objects

dawid_tylman · January 21, 2021, 1:50pm

Hello!

I’ve recently started using InsightConnect and ran into a problem with the Elasticsearch plugin - The “Query” field seems to have an odd interaction when referencing object fields, e.g. pasting below query and hitting save:
{“query”:{“match_phrase”:{“related.ip”: “192.0.2.1”}}}
and opening the edit screen again will show us that the query’s been changed to this:
{“query”:{“match_phrase”:{“related.ip”:“192.0.2.1”,“related”:{“ip”:“192.0.2.1”}}}}
The query it creates is incorrect and produces errors in both Elastic and the InsightConnect’s Elastic plugin as well.

Getting rid of dots fixes the issue. Obviously it’s not a fix as we can no longer reference objects cleanly. Is there a solution to this? Am I doing something wrong?

Regards,
Dawid

joey_mcadams · January 21, 2021, 3:45pm

Give me a minute, I’m going to try to reproduce this.

dawid_tylman · January 21, 2021, 3:58pm

Cheers.

I sort of got around this issue by modifying the PATCH request and correcting it to what it should be but then I noticed another issue which seems to be related to the plugin running on the orchestrator itself.

I am getting below error message:

Failed validating ‘type’ in schema[‘properties’][‘hits’][‘properties’][‘total’]:
{‘description’: ‘Number of Total Hits’,
‘order’: 1,
‘title’: ‘Total’,
‘type’: ‘integer’}

On instance[‘hits’][‘total’]:
{‘relation’: ‘eq’, ‘value’: 1011}

As you can see, the schema expects the “total” field to be an integer but instead it is an object with nested values. This seems to be the case for all of the calls I make to Elastic (e.g. via cURL).

Schema for the plugin was last updated in November 2019:

github.com

rapid7/insightconnect-plugins/blob/master/elasticsearch/komand_elasticsearch/actions/search_documents/schema.py

# GENERATED BY KOMAND SDK - DO NOT EDIT
import komand
import json


class Component:
    DESCRIPTION = "Execute a search query and get back search hits that match the query"


class Input:
    _INDEX = "_index"
    _TYPE = "_type"
    QUERY = "query"
    ROUTING = "routing"
    

class Output:
    _SHARDS = "_shards"
    HITS = "hits"
    TIMED_OUT = "timed_out"

This file has been truncated. show original

BR

joey_mcadams · January 21, 2021, 3:59pm

Yup, that’s a bug on our end. I’ll put a ticket in and we’ll get on a fix.

Thanks for reporting this and sorry for the inconvenience. We’ll get a fix out as soon as we can.

joey_mcadams · January 21, 2021, 4:06pm

So this part where the paste was acting weird. That line has “smart quotes” in it.
“ vs "

That bombs out most string parsers (including JSON parsing). It’s a weird copy-paste bug in OSX we run into every now and again. There are instances where OSX just decides to convert quotes on the 2nd sunday of a month with a blue moon.

I’m not sure what we can do to improve the user experience around that, unfortunately.

dawid_tylman · January 21, 2021, 4:16pm

Hi Joey,

I am not using Mac. The whole string was also pasted into notepad prior to me copying and pasting it into the JSON Query field so I am not sure if that’s correct?

The very same query also works flawlessly in Kibana. Maybe it’s the Rapid7 forum that changed the quotation marks? Anyways, I have just went into the InsightConnect instance and typed in the whole string myself and I’m getting the same behaviour.

Could you test it with normal quotes instead of the smart quotes and see if you can reproduce it again?

BR

joey_mcadams · January 21, 2021, 4:55pm

Ok, got it. I’m putting in a ticket for this.

I think you’re right about the “.” being the problem. My suspicion is when we do the handlebar parsing it’s trying to replace based on the dot. Not sure yet though.

Thanks for sticking with me to get that bug right. We’ll get it fixed.

dawid_tylman · January 22, 2021, 8:36am

That’s much appreciated, thank you.

Is it possible you could also look into the other thing I mentioned? It looks like the schema.py file contains fields that’ve been incorrectly addressed so it throws an error when it tries to interpret an object field called “total” as integer.

BR

//EDIT

The “total” field is mentioned in the article below. Could it be that the schema.py was never updated to work with Elastic 7.X?
https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html