Simple Log Service:Output configuration of processing results

Dataset description

In the data transformation task, datasets defined by Simple Log Service Processing Language (SPL) rules are referenced and written to one or more specified targets. Named datasets are identified by their names, while unnamed datasets are indicated by the identifier __UNNAMED__. For more information, see SPL datasets.

Example

The results consist of two datasets: valid and __UNNAMED__.

-- Output unnamed dataset __UNNAMED__
*
| where mode is null;

-- Define named dataset src, do not output
.let src = * 
| where mode is not null;

-- Define dataset valid, do not output
.let valid = $src
| where mode = 'a'
| parse-regexp content, '(\S+)\s+(\S+)\s+(\S+)' as x, y, z
| project x, y, z;

-- Output named dataset valid
$valid; 

-- Output unnamed dataset __UNNAMED__
$src
| where mode = 'b'
| parse-csv content as u, v
| project u, v;

Dynamic output to destination project and logstore

The data transformation (new version) allows for dynamic specification of the destination project and logstore for outputting processing results using SPL rules. The following reserved fields are available:

• __tag__:__sls_etl_output_project__: indicates the destination project for the dynamic output.

• __tag__:__sls_etl_output_logstore__: indicates the destination logstore for the dynamic output.

Examples

• Example 1:

  Designate the target project for processing results as the value of the dst_project field:

  | extend "__tag__:__sls_etl_output_project__"=dst_project

• Example 2:

  Designate the destination logstore for processing results through string concatenation:

  | extend "__tag__:__sls_etl_output_logstore__"=concat('log-', cast(status as VARCHAR))

Specify hash key for output

The data transformation task (new version) supports dynamic specification of hash keys for routing processing results to a specific shard in the Simple Log Service logstore using SPL rules. For more information on hash key routing to shards, see PutLogs.

The reserved field:

__tag__:__sls_etl_output_hash_key__: indicates the hash key used when writing processing results to the destination logstore.

Example

Use the MD5 hash of the field as the hash key for writing to the destination logstore:

| extend "__tag__:__sls_etl_output_hash_key__"=to_hex(md5(to_utf8(field))