Added ExtractGrokPatterns to OTTL converters

[michalpristas]

Description:
Added converter to OTTL for parsing grok patterns

Link to tracking Issue: #32593

Testing:
added unit tests, e2e test

for manual test use this config

receivers:
  filelog:
    include: [ demo.log ]
    start_at: beginning

exporters:
  debug:
    verbosity: detailed
    sampling_initial: 10000
    sampling_thereafter: 10000

processors:
  transform:
    error_mode: ignore
    log_statements:
      - context: log
        statements: 
          - merge_maps(attributes, ExtractGrokPatterns(body, "%{WOOHOO}", true, ["WOOHOO=%{ELB_URI} otel"]), "insert")



service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [transform ]
      exporters:
        - debug

add this line to demo.log

http://user:[email protected]:80/path?query=string otel

Output should contain these attributes:

Attributes:
     -> log.file.name: Str(demo.log)
     -> url.username: Str(user)
     -> url.domain: Str(example.com)
     -> url.port: Int(80)
     -> url.path: Str(/path)
     -> url.query: Str(query=string)
     -> url.scheme: Str(http)

For default set of patterns check: http://user:[email protected]:80/path?query=string
This implementation uses a complete set defined in this directory:
https://github.com/elastic/go-grok/tree/main/patterns

%{ELB_URI} comes from AWS set and is equivalent to
((?P<url.scheme>[A-Za-z][A-Za-z0-9+\.-]+)://(?:(?P<url.username>([a-zA-Z0-9._-]+))(?::[^@]*)?@)?(?:((?P<url.domain>(?:((?:(((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|((?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))))|(\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b))))(?::(?P<url.port>\b[1-9][0-9]*\b))?))?(?:((?P<url.path>(/[A-Za-z0-9$.+!*'(){},~:;=@#%&_\-]+)+)(?:\?(?P<url.query>[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]<>]*))?))?)

Documentation:
updated ottl/readme

[TylerHelmuth]

I think there is a typo here:

custom pattern definition strings used `pattern` in a form of `PATTERN_NAME=PATTERN`

But I'm not sure because I don't fully understand what patternDefinitions does yet.

[TylerHelmuth]

Can you add some more in depth docs around what the optional parameters do and when a user would use them?

[TylerHelmuth]

I'm still pretty confused about what patternDefinitions is doing. Where did the value pass123 come from? I don't see it in the example string.

[michalpristas]

it's second part of the string, my bad there was x previously

[TylerHelmuth]

So what is

  - `PASSWORD=%{WORD}`
  - `USERINFO=%{GREEDYDATA}

providing in this scenario? It looks to me like %{USERNAME:user.name}:%{PASSWORD:user.password} is doing all the work

[michalpristas]

%{USERNAME:user.name}:%{PASSWORD:user.password} is a pattern so it is a format in which your logs occur in a file.

  - `PASSWORD=%{WORD}`
  - `USERINFO=%{GREEDYDATA}

is a pattern definition, it tells parser how %{USERNAME:user.name} translates to reges, in this case it is a WORD that comes from a default set and is \b\w+\b(definition)

s after compilation you end up with named capture <user.name> capturing regex \b\w+\b

[michalpristas]

I asked very nice people from our docs team to help me rephrase this so it is more clear