[receiver/filelog] Add option to insert offset as attribute

[francois07]

Component(s)

pkg/stanza, receiver/filelog

Is your feature request related to a problem? Please describe.

When sending logs to an intermediate service such as Kafka, there are times when unicity and order of messages are not 100% guaranteed.

Describe the solution you'd like

It would be useful to be able to insert the offset of each log record as an attribute, to be able to guarantee order and unicity of logs when sending them to an intermediate service before the backend.

A config parameter such as insert_offset seems appropriate

Describe alternatives you've considered

Using the transform processor, insert a timestamp and a UID to log records

Additional context

I'm not completely familiar with how stanza works, so maybe there is already an order and unicity guarantee ? If that's the case, please do tell me. Thank you !

[github-actions]

Pinging code owners:

• pkg/stanza: @djaglowski
• receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

You're correct there is not an order guarantee. In fact, the alternative suggestion of using transform processor won't work because of this.

I'm not opposed to this proposal in general but there are some mechanics that need to be worked out. Currently I believe we update the offset only once per file poll interval, but we may be able to track it log by log.

[francois07]

After investigating, it seems that #33530 does exactly what I'm trying to do. What do you think @djaglowski ?

[djaglowski]

@francois07 I think it should work for your use case, and is simpler than using the actual offset. We may someday want to implement your original suggestion but if you can work with the line number for now I think we can close this and reconsider later as necessary.