Quick Start Guide
I quick demonstration of creating a streaming pipe in IngeXt.
Walkthrough Demo those of you that just want to see things work, this guide is for you. The objective is to connect a data source to a data sink.
When you log into Ingext for the first time, you are greeted with an empty hone screen that has three buttons:
- Add Source
- Add Router
- Add Sink
Objective
In this example, you are going to create a HEC listener. You will then send a message to that listener, see it in the processing, and then send it to a drop sink.
What You Need
This is a basic Hello World program done IngeXt style.
- You will need to have a command prompt with the ability to send a
curl
command. - You will also need to have a IngeXt account.
Creating the Data Source
You are going to create an HEC Data Source.
Click the "Add Source" Button
This is the button on the far left. We wan to first have a way to get data into the pipes. In this example, you are going to use the HTTP Event Collector (HEC) service. This is a popular service for modern audit collection.
Clicking will bring up the pullout, and then select the HTTP Event Collector option.
Give it a name
Before you can save, you need to give the Data Source a name. The name is all one camel case work, meaning that you use capital letters to show the words inside the string.
In this case,
Connecting a Processor
For now we are just going to connect a "Pass through" processor. A Pass through processor simply exports what it receives. This is where the magic happens in streaming. It is common that we would put a parser here. But for now we are just going to do a simple echo, where we write to the sink what we are getting from the source.
Click the Add Router
We first need to create the router and assign a processor.
- Clicking the Add Router button, opens the "Adding a Router" pullout.
- We are going to select the Event_Passthrough for this example.
When you save this you should now see a router in the middle of the screen called Event_Passthrough.
View the Router
Processors are where the real power in Ingext is. Processors do parsing, metrics, transformation, and enrichment. Here we are just going to use the process to see what is going on.
Processors belong to Pipes. Routers can have multiple Pipes.
This is a the process pullout. There are three actions on this page.
- Expand processor code: This will do a popup of the code to make it more readable.
- Edit: This allows an edit, latest events, and code debugging page. We will use this later when we test the Hello World install.
- Remove From Pipe: This will remove the process all together.
We will get back to this later.
Connect a Data Sink
You are going to create a drop sink. A drop sink takes a message and discards it. It is not that exciting. Drop sinks are used to track message that are being discarded.
To create a drop sink
A data sink is where the data does. This would normally be a s3 bucket, a web hook, or a tool, like a SIEM. In this case, we are creating a Drop bucket. This just helps us track better that we are going to throw away the data and not keep it. In later sections, we will change this.
For now
- click the Add Sink button.
- Give the sink a name.
- Save
This creates a sink to discard the incoming events.
Testing the Sink
We are going to send a message to the HEC.
Sending an HEC event.
This is where the curl
command comes into play.
curl -H "Authorization: Ingext <TOKEN>" <HEC HTTP Address> -d '{"sourcetype": "curl prompt", "event": "Hello World"}'
You are to replace the
- with the toke the interface asked you to copy and save.**
- with the http address the interface provided.
Using a terminal, send this packet via curl
. You should receive an OK
response when this call is successful.
There is a delay in HEC as it buffers the data and updates the pipe. You may want to refresh the page to accelerate the page polling.
Seeing the event in the Data Source
Back to the Ingext interface, wait until the data source card becomes green. You can then click on the data source.
When you open the pullout, select the Event tab. This will present a count (number of records) and bytes (total size) histogram. It also, under the event logs, shows the last incoming message.
Seeing the event in the Processor
We can see more of the messages and eventual parse and transform these by looking into the parser we created.
The trickiest part of this is clicking on the Processor. To do this. Click on the parser in the configuration page and get the pullout.
You want to click on the processor button, highlighted above in red. This will open the Processor panel. Then you can click the edit button to go to the Processor coding page by clicking the edit button (View Processor hint).
This is the edit page for processor. It will allow you to transform and enrich the record before placing in the sink.
This is a powerful page. For now, just click on the Events button, and you will see a list of messages that have been processed by the processor. Clicking on an events clipboard will place that event into the input buffer of the processor for testing and coding purposes.
Demo
You can visit the demo in its own tab.
Updated about 1 month ago