To run the ln-history platform in real-time you need to have one (or multiple) Core Lightning nodes - with the gossip-publisher-zmq Core Lightning Plugin setup - running.

#Features of the plugin

The gossip-publisher-zmq plugin monitors the gossip_store file every Core Lightning node manages by itself to store the collected gossip messages. The plugin is able to find a valid offset in the gossip_store by itself and starts to publish collected gossip messages via a ZeroMQ PUB socket. Any service can connect to this pub socket which will be in this case the gossip-syncer.

Every gossip message will get parsed by utilizing the lnhistoryclient package.

#API of the plugin

Every message of the gossip-publisher-zmq pub socket will have the format of the PluginEventMetadata:

# PluginEvent refers to all events published by the gossip-publisher-zmq Core Lightning plugin
class PluginEventMetadata(TypedDict):
    type: int
    name: str
    timestamp: int
    sender_node_id: str
    length: int  # Length in bytes without starting 2-byte typ


# Base structure for all events
class BasePluginEvent(TypedDict):
    metadata: PluginEventMetadata
    raw_hex: str

#Setup the plugin

#Clone

Clone the plugin repository into your /plugins directory of your node using the following command:

#Setup virtual environment

Go inside the directory and install the necessary dependencies in your virtual environment

First create the virtual environment using the following command

This creates a new folder called .venv which will contain the dependencies for the gossip-publisher-zmq plugin

Next we activate this environment by executing the following command

Lastly we install the dependencies as defined in the requirements.txt using pip from our virtual environment.

#Update the shebang

To use the virtual environment we need to edit the first line of the main.py file to match the newly created virtual environment. Substitute the <your-user> with your user and update the first line with the following shebang.

#!/home/<your-user>/lightning/plugins/gossip-publisher-zmq/.venv/bin/python

#Update configuration

Add this line to your lightning.conf

plugin=lightning/plugins/gossip-publisher-zmq/main.py

#Environment variablesauthentication

The plugin can be configured with the following parameters

The cloned repository contains a .env.example which can be copied and renamed to .env In this file update the environment variable however you like.

🔐 Important: Never commit .env files containing credentials to version control.

#Notes about theSTART_AT_BYTE value

The plugin goes from the start to the end through the gossip_store file. If you only want to publish newly collected messages and not every collected message, please set this value to something close to the end. You could take a look at the file size of the gossip_store file in bytes, subtract 5000 from that and set the START_AT_BYTE to that value. The plugin will continue from that point until it finds a valid gossip message and starts to publish all following gossip messages.

#Restart

Just restart the Core Lightning node and your plugin should be running

#Status methods

The plugin provides two methods that can be called at any time to verify if the plugin is running correctly.

#Overall Status

You can check at any time the current status of the plugin by executing the following command.

 {
   "running": true,
   "zmq_endpoint": "tcp://127.0.0.1:5676",
   "sender_node_id": "<your-node-id>",
   "gossip_store_path": "/home/<your-user>/lightning/data/bitcoin/gossip_store",
   "offset_of_gossip_store": 53998754,
   "initialized": true,
   "last_10_processed_messages": [{"<message-1-data>"}, {"<message-2-data>"}, ]
}

#Last published messages

Additionally you can have a look at the last 100 processed messages using this command

[
    {
        "<message-1-data>"
    },
    {
        "<message-2-data>"
    }
]