Event Sink

We recently released a new provider channel that we call Event Sink. Its purpose is to provide a universal mechanism to forward events sent through Blotout to any HTTP endpoint that can receive a JSON payload via POST.

The reasoning behind this new channel is to enable our users to process raw events without an intermediary. If you ever wanted to build additional logic on top of Blotout's event stream, you can now do so by providing your own HTTP server that will consume these events in real time.

How events are sent to your endpoint

We use a Cloudflare Worker to receive events from the browser and other sources and in turn call specific endpoints configured on the tag hosted on the website. When you configure an Event Sink channel, you'll provide the HTTP endpoint URL along with any additional request headers you'd like us to append when sending an event.

Once you save and deploy the tag, the channel will be activated and every event generated on your tag will forward the event payload to your endpoint.

NOTE: Your endpoint should be able to handle the maximum throughput rate of events generated on your site. You can estimate the maximum throughput per minute your endpoint should handle by looking at the tag analytics for a typical day by setting the chart to Minute resolution and the window size to 1440 to get the last 24 hours worth of data:

The call will be made as a POST request to the URL you provide, and will send the event formatted as a JSON string, along with the Content-type: application/json; charset=utf-8 header and any others you have provided.

The top-level JSON object sent in the body will always receive three properties:

type JSONRequestBody = {
  action: 'tag' | 'user'
  payload: {
    /* payload dependent on type of action */
  }
  user: {
    /* user information */
  }
}

Requests that represent events will be tagged with action: 'tag' and actions that represent data collected from the user will be tagged with action: 'user'.

Example implementation

To illustrate how to build an endpoint that will be able to receive events, we'll create a Cloudflare Worker and use its publically accessible development endpoint to configure the Event Sink. The reason we would recommend using something like Cloudflare Workers is that these services automatically scale up the number of workers based on the incoming request rates, so you don't need to worry about scaling your own infrastructure to keep up with traffic spikes.

To create one, you need a free or paid account on Cloudflare. Once you log in and select your account, you'll be able to create a new worker by selecting the Hello world template:

After giving it a sensible name, you can deploy the worker and you're presented with the success screen showing the link to your newly created worker:

Click Edit code. This will take you to the code editor that will allow you to implement the endpoint.

We can now implement a simple handler that will read the JSON payload from the request object and log it to the console. Replace the entire contents of the file with this block of code:

export default {
  async fetch(request, env, ctx) {
    if (request.method == 'POST') {
      const requestPayload = await request.json()
      const headers = [...request.headers.entries()]
      console.log(requestPayload, { headers })
      return new Response(JSON.stringify({ success: true }), {
        headers: new Headers({
          'Content-Type': 'application/json; charset=utf-8',
        }),
      })
    }
    return new Response('Not found', { status: 404 })
  },
}

After deploying the worker, you can check that the endpoint works by firing a POST request to the URL with a valid JSON payload:

Now that we've verified that the endpoint can receive POST requests and log the payload, we can open realtime logs for the worker to observe the events coming through:

In another tab, we can open our EdgeTag site and add the Event Sink channel configured for our worker endpoint:

Saving and deploying this tag will start the event stream to our endpoint. Switching back to our Cloudflare tab, we should see the realtime logger displaying POST requests made to our worker, logging the payloads in the logs section:

Of course, this example just logs the payload and request headers to the console, but depending on your needs, you might want to write this payload to persistent storage, a database or perform filtering or transformation before forwarding it to another service.

You can also configure a secret header value in the provider form to send along with the request so that you can then compare against a value in the worker to authenticate the incoming request. This would prevent your endpoint from accepting any POST request from anything other than Blotout.

If you do decide to use this to build something interesting, we'd love to hear your thoughts!