Lua Transform

The Vector lua transform accepts and outputs log and metric events, allowing you to transform events with a full embedded Lua engine.

Configuration

• simple
• inline
• module

vector.toml

[transforms.my_transform_id]
  # General
  type = "lua" # required
  inputs = ["my-source-or-transform-id"] # required
  version = "2" # required

  # Hooks
  hooks.process = """
  function (event, emit)
    event.log.field = "value" -- set value of a field
    event.log.another_field = nil -- remove field
    event.log.first, event.log.second = nil, event.log.first -- rename field

    -- Very important! Emit the processed event.
    emit(event)
  end
  """

5 items

simple inline modulecommonrequired

• tablecommonrequired

  hooks

  Configures hooks handlers.

  • stringoptional

    init

    A function which is called when the first event comes, before calling hooks.process

    • No default
    • View examples
  • stringcommonrequired

    process

    A function which is called for each incoming event. It can produce new events using emit function.

    • No default
    • View examples
  • stringoptional

    shutdown

    A function which is called when Vector is stopped. It can produce new events using emit function.

    • No default
    • View examples
• [string]optional

  search_dirs

  A list of directories to search when loading a Lua file via the require function. If not specified, the modules are looked up in the directories of Vector's configs. See Search Directories for more info.

  • No default
  • View examples
• stringoptional

  source

  The source which is evaluated when the transform is created.

  • No default
  • View examples
• [table]optional

  timers

  Configures timers which are executed periodically at given interval.

  • stringrequired

    handler

    Defines a handler function which is executed periodially at interval_seconds. It can produce new events using emit function.

    • No default
    • View examples
  • uint (seconds)required

    interval_seconds

    Defines the interval at which the timer handler would be executed.

    • No default
    • View examples
  • No default
  • View examples
• stringenumcommonrequired

  version

  Transform API version. Specifying this version ensures that Vector does not break backward compatibility.

  • No default
  • Enum, must be one of: "2"
  • View examples

Examples

Add, rename, & remove log fields

The following examples demonstrates how to add, rename, and remove a log event's fields:

vector.toml

# ...
hooks.process = """
function (event, emit)
  -- Add root level field
  event.log.field = "new value"

  -- Add nested field
  event.log.nested.field = "nested value"

  -- Rename field
  event.log.new_field = event.log.old_field
  event.log.old_field = nil

  -- Remove fields
  event.log.field = nil

  emit(event)
end
"""

How It Works

Add Fields & Tags

To add log fields and metric tags just set the desired key to the appropriate value:

-- Add root level field
event.log.field = "new value"

-- Add nested field
event.log.nested.field = "nested value"

-- Add tag
event.metric.tags.tag = "new value"

Defining Timestamps

To parse a timestamp with an optional milliseconds field, like 2020-04-07 06:26:02.643 or 2020-04-07 06:26:02:

timestamp_pattern = "(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)"

function parse_timestamp(str)
  local year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)
  local ms = 0
  if millis and millis ~= "" then
    ms = tonumber(millis)
  end
  return {
    year    = tonumber(year),
    month   = tonumber(month),
    day     = tonumber(day),
    hour    = tonumber(hour),
    min     = tonumber(min),
    sec     = tonumber(sec),
    nanosec = ms * 1000000
  }
end

parse_timestamp('2020-04-07 06:26:02.643')
parse_timestamp('2020-04-07 06:26:02')

Drop Events

To drop events, simply do not call the emitting function with it. For example:

function (event, emit)
  if not event["message"].match(str, "debug") then
    emit(event)
  end
end

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Iterate Over Fields & Tags

To iterate over all fields of an event use the pairs method. For example:

function (event, emit)
  -- Remove all fields where the value is "-"
  for f, v in pairs(event) do
    if v == "-" then
      event[f] = nil
    end
  end

  emit(event)
end

Learning Lua

In order to write non-trivial transforms in Lua, one has to have basic understanding of Lua. Because Lua is an easy to learn language, reading a few first chapters of the official book or consulting the manual would suffice.

Lua Version

Vector uses the rlua Rust crate which currently embeds Lua 5.3.

Remove Fields & Tags

You can remove log fields and metric tags by setting them to nil:

-- remove a log field
event.log.field = nil

-- remove a metric tag
event.metric.tags.tag = nil

Representation of Events

Events are represented as tables in Lua. Vector uses externally tagged representation to encode both log and metric events in a consistent fashion:

• Log events are represented as values of a key named log.
• Metric events are represented as values of a key named metric.

• Log
• Metric

For instance, a typical log event produced by the stdin source could have been created programmatically using the following code:

event = {
  log = {
    host = "localhost",
    message = "the message",
    timestamp = os.date("!*t")
  }
}

Data Types

The correspondence between Vector's data types and Lua data type is summarized by the following table:

Vector Type	Lua Type	Comment
String	string
Integer	integer
Float	number
Boolean	boolean
Timestamp	table	There is no dedicated timestamp type in Lua. Timestamps are represented as tables using the convention defined by os.date and os.time. The table representation of a timestamp contains the fields year, month, day, hour, min, sec, nanosec, yday, wday, and isdst. If such a table is passed from Lua to Vector, the fields yday, wday, and isdst can be omitted. In addition to the os.time representation, Vector supports sub-second resolution with a nanosec field in the table.
Null	empty string	In Lua setting the value of a table field to nil means deletion of this field. In addition, the length operator # does not work in the expected way with sequences containing nulls. Because of that Null values are encoded as empty strings.
Map	table
Array	sequence	Sequences are a special case of tables. Indexes start from 1, following the Lua convention.

Search Directories

Vector provides a search_dirs option that allows you to specify absolute paths that will be searched when using the Lua require function. If this option is not set, the directories of the configuration files will be used instead.