Tutorials
Alchemy Login

How to Use Custom Webhooks for Web3 Data Ingestion

Learn How to Stream Web3 Data to Your Dapp with Custom Webhooks

Suggest Edits

Custom Webhooks are the latest additions to Alchemy’s Notify suite of web3 notification services.

Powered by Alchemy’s Supernode and GraphQL, they bring custom real-time notifications for any blockchain activity or contract-based event.

This article will cover an introduction to Custom Webhooks, the benefits of using them, how to set up an Express.js server, and three examples of how Custom Webhooks can be used to track on-chain data.

If you’re familiar with data streaming, you can skip to any of the following tutorials:

  1. How to Stream New Block Data with Custom Webhooks
  2. How to Implement a Custom Webhook for Created Contracts
  3. How to Track Block Transactions with Custom Webhooks

For more ways to use Custom Webhooks, explore our library of examples!

What is web3 data ingestion?

Web3 data ingestion is the process of collecting and importing data from various sources in the web3 ecosystem into a system for storage and analysis.

What is the problem with web3 data ingestion?

Web3 data ingestion has several major problem areas, including large and varied data, different data formats, and lack of scalability.

These problems have arisen because of the more than 1 million smart contracts deployed on the Ethereum mainnet alone.

The amount of information that has been recorded on the blockchain has increased exponentially during recent years.

Here are three major concerns in more detail:

1. Large and Varied Data

Firstly, the amount of data generated by the different blockchains are often large and varied, making it difficult to manage and process.

2. Different Data Formats

Secondly, the data may be in different formats, making it challenging to normalize and integrate.

3. Lack of Scalability

Finally, the process of web3 data ingestion needs to be efficient and scalable to keep up with the volume and speed of incoming data.

Presently, web3 data ingestion processing is not too scalable.

All these factors make data ingestion a complex and challenging problem for web3 applications.

Why is data ingestion an important product for web3 developers?

Data ingestion is important for web3 because decisions in web3 are time-sensitive, and the process of data handling needs to be scalable.

1. Web3 is Time Sensitive

In the context of web3, proper data ingestion is crucial for making time-sensitive decisions in web3 products.

Users need to stay informed about the latest price changes, token mints, the on-chain activity of a specific wallet, the latest NFT drop or track the on-chain activity of a whale’s wallet.

2. Web3 Products Require Scalability

Proper data ingestion can lead to efficient and scalable web3 products that can improve with time.

However, improper data handling could result in various problems, ranging from poor performance due to information overload and scalability to critical problems like wrong data filtering.

3. Complicated Handling of Blockchain Data

Usually, blockchain data can be tracked and recorded with event listeners that can be encoded into the decentralized application.

However, this may get complicated and time-consuming for the developer, especially if the dapp needs to listen to multiple events from multiple smart contracts or even multiple blockchains.

This is where webhooks come into play.

What are Custom Webhooks?

Alchemy’s Custom Webhooks are the latest addition to Alchemy’s web3 notification tool suite.

Alchemy’s query customization options make them superior to the existing notification solutions, which are based on a set of pre-defined triggers.

Why use Custom Webhooks?

Custom Webhooks provide a host of benefits for development. Here are some in more detail:

1. Precise Information Querying

Powered by Alchemy’s Supernode and GraphQL, Custom Webhooks allow for precise information querying so that the application receives only the information that is required by its business logic.

2. Cost-Effectiveness

Additionally, Custom Webhooks reduce data payload and computational resources, which makes them a more cost-effective method.

3. Ease of Integration

Lastly, compared to in-dapp event listeners, Custom Webhooks are way easier to integrate and provide the data in ready to digest way, which immensely contributes to the developer experience.

How to Setup an Express.js Server for Custom Webhooks

For this tutorial, a basic Express.js server will be created.

It will have a separate endpoint for each of the webhooks.

Prerequisites

The following list of tools is required:

  • Integrated Development Environment (IDE)
  • Package Manager (NPM)
  • Node.js
  • Express.js for server creation
  • Ngrok for exposing the localhost server
  • An Alchemy Account
  • An Alchemy Custom Webhook

1. Configure the Development Environment

Firstly, create a folder for the project.

Open the folder with VSCode, then open a new terminal inside and type the following command to initialize an npm project:

npm init -y

The -y flag prefills all the required information with its default values and speeds up the configuration process.

Then, install express by running:

npm i express

Ngrok will also be needed to expose the local Express server.

  1. Sign-up for a free Ngrok account.
  2. Install Ngrok using the Ngrok guide. On macOS, run brew install ngrok
  3. Connect the Ngrok account by running ngrok authtoken YOUR_AUTH_TOKEN

2. Configure the Server

The following code creates a basic Express server with separate endpoints for each of the custom hooks that will be created in the tutorial.

The information received by the webhooks is stored in three separate arrays in the memory of the server.

Create an index.js file in the root of the project’s directory and paste the following code inside it:

// Import express (after running 'npm install express').
const express = require("express");
const app = express();

// Configure server. 
const PORT = 8080;

// Decode JSON data.
app.use(express.json({ limit: "25mb" }));
let blocks = [];
let createdContracts = [];
let transactionLog = [];

// Create a route for the app.
app.get("/", (req, res) => {
    res.send(
        "Welchome to Alchemy Blockchain Information Fetcher. Access the designated endpoints to retrieve the desired information."
    );
});

// Create a POST block-info route. Use as a webhook URL.
app.post("/block-info", (req, res) => {
    const blockInfo = JSON.stringify(req.body.event.data.block);
    blocks.push(blockInfo);

    // Respond with status 200.
    res.send("Success.");
})

// Create a GET block-info route.
app.get("/block-info", (req, res) => {
    res.send(blocks);
});

// Create a POST created-contracts route. Use as a webhook URL.
app.post("/created-contracts", (req, res) => {
    const createdContractsInfo = req.body.event.data.block.logs;

    // Loop over the received object of block transactions 
    // save all transactions in which a contract has been created. 
    for (let transaction of createdContractsInfo) {
        if (transaction.transaction.createdContract) {
            createdContracts.push(transaction);
        }
    }

    // Respond with status 200. 
    res.send("Success.");
});

// Create a GET created-contracts route.
app.get("/created-contracts", (req, res) => {
    res.send(createdContracts);
});

// Create a POST transactions route. Use as a webhook URL.
app.post("/transactions", (req, res) => {
    const transactions = req.body.event.data.block;
    transactionLog.push(transactions);

    // Respond with status 200.
    res.send("Success.");
})

// Create a GET transactions route.
app.get("/transactions", (req, res) => {
    res.send(transactionLog);
});

// Make the server listen to requests.
app.listen(PORT, () => {
    console.log(`Server running at: http://localhost:${PORT}/`);
})

3. Start the Server

Next, type this in the terminal:

node index.js

The server should be started at a particular port as shown below:

Lastly, open another terminal and start ngrok with the following command:

ngrok http PORT_NUMBER_RETURNED_BY_EXPRESS

Now that the server and environment are configured and running, the next step is to connect Alchemy’s Custom Webhook with the server.

How to Stream New Block Data with Custom Webhooks

Using Alchemy’s Custom Webhooks, obtaining block information is simple.

The following steps will implement a simple custom webhook with Alchemy:

1. Sign Up with Alchemy

Firstly, head over to Alchemy, create a free account, and sign in.

2. Create Webhook using GraphQL

Next, navigate to the Notify dashboard.

At the GraphQL section, click + Create Webhook.

The page will redirect to the GraphQL query playground, where the webhook can be fine-tuned to fetch specific data from the blockchain.

📘

For the simplification of the tutorial, webhook verification is not implemented. Always verify the webhooks that interact with your decentralized application. Click here to learn how to do that.

The entry point for all EVM data is the block. This is why every query starts with it.

To get the general information for every upcoming block, paste this code into the query editor:

{
  block {
    hash,
    number,
    timestamp
  }
}

The code is a GraphQL query that asks for each produced block's hash, number, and timestamp.

3. Configure the Webhook URL

Next, configure the webhook URL.

This is the address through which the webhook will submit the queried information to the server.

In the VSCode terminal, used to initialize ngrok, find and copy the forwarding link.

Paste the forwarding link in the Webhook URL input field in the GraphQL section of the Notify dashboard.

Then, add /block-info at the end.

Now, the URL will point to the specific entry point in the server created for this webhook.

Click on the Create Webhook button in the lower-left corner.

The Chain and Networks field can be configured to fetch data from other blockchains like Arbitrum, Polygon, and Optimism, but for this tutorial, the Ethereum mainnet will be used.

The webhook is now set up and will notify the server with the queried information via a POST request whenever a new block is added to the blockchain.

4. Verify Retrieval of Data

Head over to the server to check the retrieved data.

In a new tab of an Internet browser, paste this URL:

http://localhost:YOUR_PORT_NUMBER/block-info

The server will return information about every upcoming block.

How to Implement a Custom Webhook for Created Contracts

Implementing a Custom Webhook is simple with Alchemy. In this guide, the webhook will fetch all the transactions included in the block.

If a contract has been created in one of them, the contract’s address is recorded and stored.

1. Create a Webhook for Created Contracts using GraphQL

Assuming you already have an Alchemy account, navigate to the Notify dashboard.

At the GraphQL section, click + Create Webhook.

The page will redirect to the GraphQL query playground, where the webhook can be fine-tuned to fetch specific data from the blockchain.

In the Custom Webhooks GraphQL playground, paste the following query:

{
  block {
    logs(filter: {addresses: [], topics: []}) { 
      transaction {
        createdContract {
          address
        }
      }
    }
  }
}

The query checks every transaction in the block, checks if it is associated with the creation of a smart contract, and returns its address.

If the transaction is not associated with creating a contract, the value will be null.

2. Configure the Webhook URL

In the webhook URL, copy the forwarding link, generated by ngrok and add /created-contracts at the end.

Click on the Create Webhook button in the lower-left corner.

3. Verify the Created Contract Data

Next, head over to the server to check the retrieved data.

In a new tab of an Internet browser, paste this URL:

http://localhost:YOUR_PORT_NUMBER/created-contracts

The contracts created should be shown as below:

If contracts are created in each upcoming block, their addresses will be recorded and returned by the server.

How to Track Block Transactions with Custom Webhooks

This example creates a webhook that fetches all the transactions included in a block.

More specifically, it includes the number of transactions as well as an array of transaction objects.

1. Create Webhook using GraphQL

Go back to the Notify dashboard, and in the GraphQL section, click + Create Webhook.

The page will redirect to the GraphQL query playground, where the webhook can be fine-tuned to fetch specific data from the blockchain.

In the Custom Webhooks GraphQL playground, paste the query below:

{
  block {
    transactionCount,
    transactions {
       hash,
        nonce,
        index,
        from {
          address
        },
        to {
					address
        },
        value,
        gasPrice,
        maxFeePerGas,
        maxPriorityFeePerGas,
        gas,
        status,
        gasUsed,
        cumulativeGasUsed,
        effectiveGasPrice,
        createdContract {
          address
        }
    }
  }
}

2. Configure the Webhook URL

In the webhook URL, copy the forwarding link, generated by ngrok and add /transactions at the end.

Click on the Create Webhook button in the lower-left corner.

3. Verify Retrieval of Data

Next, head over to the server to check the retrieved data.

In a new tab of an Internet browser, paste this URL:

http://localhost:YOUR_PORT_NUMBER/transactions

This will visualize the transactions in the following format:

Alchemy’s Custom Webhooks provide custom-tailored webhooks that supply only the information that a dapp would use. They are reliable, instant, and require only a few clicks to integrate.

With Custom Webhooks, end users can make well-informed and time-sensitive decisions about the latest NFT drop or track the on-chain activity of a whale’s wallet.

Read the FAQs for Custom Webhooks for more information, and sign up to start integrating Custom Webhooks into your dapp today, and start utilizing the power of real-time web3 data ingestion.

Updated over 1 year ago

ReadMe