• Andrea Amorosi (Hashnode | Twitter | GitHub)

• Pascal Vogel (Hashnode | Twitter)

Overview

Made popular together with large language models (LLMs), semantic search has proven to be a powerful new approach for providing relevant search results based on natural language queries.

Semantic search takes into account the meaning and intention behind search queries and does not rely on exact or near-exact matching of search terms to keywords.

For example, if I search for "warm-weather hat" on amazon.com, I get a lot of results for winter gear although what I'm actually looking for are hats to wear in the summer. In this case, a semantic search would be more successful in understanding my intention and can offer more relevant results.

With this Pinecone x Hashnode integration, a submission to the Hashnode APIs Hackathon in Category #1, you can easily set up semantic search for your Hashnode blog posts and query them using a serverless REST API endpoint.

To implement this, we rely on Pinecone, a managed vector database, the OpenAI API for embedding models, and AWS serverless services to host the solution in your own AWS account.

This project builds on HashBridge (GitHub) which makes it easy to set up integrations between Hashnode and AWS serverless functions with AWS Lambda.

In the rest of this blog post, we go into more detail on how you can set up and use the Pinecone x Hashnode integration, how it works, and the implementation decisions we made.

How to use Pinecone x Hashnode

If you want to get started quickly, watch this 8 minute demo to learn how to set up and use HashBridge with your Hashnode blog:

We provide detailed deployment instructions and the full source code in our GitHub repository. Your issues and contributions are very welcome!

How Pinecone x Hashnode works

The architecture of this Pinecone x Hashnode integration can roughly be split in two parts that work independently:

1. Content ingestion: whenever a blog post is published, updated, or deleted on our Hashnode blog, we need to update our vector database to reflect this change. In addition, we need to be able to do an initial ingestion of existing blog posts into the vector database.

2. Search: when a user submits a search query to our search API, we need to run a search in our vector database and return the most relevant results.

Sounds easy enough! In the following sections, we explain the architecture and underlying technical concepts of these two elements in detail. But first, let's do a quick primer on semantic search and vector embeddings.

Primer: semantic search and vector embeddings

Vector embeddings are the basis for semantic search. A vector embedding is a series of numbers (a vector) in a vector space that represents an object such as a word, sentence, or entire document. Each object is a point with a certain position in the vector space. By measuring the distance between points, we can assess and compare their similarity.

Similar sentences are positioned close-by in the vector space (Source:DeepAI).

In case of semantic search, we can use this to our advantage. We generate vector embeddings for our Hashnode blog posts which positions them in the vector space. When a user inputs a search query, we also generate a vector embedding that represents the query. Finally, we can compare the position of the search query with the positions of our blog content and find the content that is closest and therefore most similar to the query.

To generate vector embeddings, we can use an embedding model. This type of LLM takes a text input and returns a vector with a certain number of dimensions. Besides countless open-source models that you can host yourself, there are also managed models available from OpenAI or via Amazon Bedrock.

Ingesting Hashnode events and storing vector embeddings in Pinecone

Because this integration builds on HashBridge (GitHub), we already have a way to receive webhook events from Hashnode on an EventBridge event bus in our AWS account which triggers a serverless AWS Lambda function. HashBridge uses the Hashnode GraphQL API to enrich webhook events with additional data and metadata such as blog post content in markdown, author, URL, and more.

A Lambda function, written in TypeScript, contains our logic for generating embeddings and communicating with the Pinecone vector database.

Pinecone is a fully-managed vector database that enables AI use cases like semantic search, retrieval-augmented generation (RAG), and more. The database is integrated with popular frameworks like LangChain and LlamaIndex and offers clients for Python, Node.js, and Java as well as a REST API.

Pinecone offers both pod-based and serverless pricing options, the latter in preview. We tested this integration with the pod-based free tier but it should work similar in case of the serverless option.

One challenge when generating vector embeddings is the context size supported by embedding models. OpenAI’s ext-embedding-3-small embedding model, which we are using here, supports up to 8191 input tokens, which translates to roughly 32,000 characters in case of English-language text.

This means that for blog posts with more than 32,000 characters, we need to split the content into multiple chunks. We also need to consider that we are embedding the markdown content instead of text content to not lose URLs or code formatting metadata. Depending on the use case, different chunking strategies can be applied to improve retrieval accuracy. This Pinecone blog post goes into more detail.

In our case, we decide to break the markdown blog content into chunks of 500 characters with an overlap of 50 characters before generating vector embeddings for each chunk. LangChain’s Markdown splitter helps with keeping text with common context together during chunking based on the Markdown document structure.

To store these chunks which all make up one blog post, we follow Pinecone’s recommendation of upserting each chunk separately as a distinct record. We use ID prefixes to connect each chunk to its parent document. The resulting structure looks something like this:

{"id": "blogid1#chunk1", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, ...]},
{"id": "blogid1#chunk2", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, ...]},
{"id": "blogid1#chunk3", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, ...]},
{"id": "blogid1#chunk4", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, ...]}

In addition to embedding new posts, you might already have a whole bunch of posts in your Hashnode blog that you want to make searchable. In this case, you can use our initial embedding script which will perform the query, chunk, embed, and store steps necessary to get all of your blog posts into Pinecone.

To update or delete a blog post from Pinecone based on a Hashnode event, we need to be a bit more careful.

Using the search API

Our search API consists of a Lambda function with its function URL and IAM authentication enabled. The function URL is protected by a CloudFront distribution where a Lambda@Edge function signs each request using Sigv4. This setup is very similar to what we used to build HashBridge, so head over to the HashBridge announcement blog post and take a look at the Verifying webhook events with Lambda@Edge section for more details.

We process search queries in the Search API handler function. This function performs the following tasks:

1. Get vector embeddings for the search query text from OpenAI.

2. Run a query operation on Pinecone to identify the chunks most similar to query embeddings.

3. Extract the post ID from the three similar Pinecone query result chunks and use the Hashnode GraphQL API to get the post brief.

Let's look at an example where we ask a natural language question to the Hashnode Engineering Blog.

After deploying the integration, we can make a simple HTTP GET request to the search API endpoint and include our query as a URL parameter. In this case, we are using httpie to make the request from the command line for the query "peer review infra changes":

http https://dkzum3j6x4pzr.cloudfront.net/search text=="peer review infra changes"

The search API would return a response that contains the three most similar results, including their author, brief, similarity score, and more:

{
    "matches": [
        {
            "post": {
                "author": {
                    "name": "Shad Mirza",
                    "username": "iamshadmirza"
                },
                "brief": "Hey, everyone! If you've been using AWS, chances are you've come across CDK and building cloud apps with it. As seamless as it is to deploy apps using CDK, it is equally important to monitor changes in infrastructure code and prevent issues.\nThis gui...",
                "id": "647ed1c80fce655a6be9ac07",
                "title": "Simple Steps to Include CDK Diffs in GitHub PRs"
            },
            "similarity_score": 0.4194794
        },
        // 2 more matches
    ]
}

And, sure enough, when we check the result with the highest score, we find a relevant discussion covering the topic addressed in our query in the post content despite the exact terms not being present in the title or brief of the post (Simple Steps to Include CDK Diffs in GitHub PRs).

Conclusion

Our Pinecone x Hashnode integration makes it easy to add semantic search to your Hashnode blog. In particular if you are using Hashnode's headless mode, you may want to offer a unified semantic search experience across your existing website content and your Hashnode blog content. You can ingest all of these content types into Pinecone and our solution provides a simple but accurate search enpoint.

Let us know what you think in the comments or open a GitHub issue with any questions and feedback.

• Andrea Amorosi (Hashnode | Twitter | GitHub)

• Pascal Vogel (Hashnode | Twitter)

Overview

When we saw the Hashnode APIs Hackathon announcement, we thought long and hard about what integration could make an impactful addition to the Hashnode ecosystem for the hackathon Category #1. With this submission, we decided to build a re-usable integration that anyone can use to extend their Hashnode blog with Amazon Web Services (AWS).

HashBridge is an integration that allows you to connect your Hashnode blog to serverless AWS Lambda functions and other resources based on events in your Hashnode blog.

Each time you publish, update or delete a blog post, you trigger a serverless function that receives blog content and metadata such as URL, author, and more as an input. Inside your function, you can then write code in the programming language of your choice that communicates with AWS services or any external API.

For example, you could use a text-to-speech service to generate an audio version of your blog post, use AI to generate a cover image, or cross-post to social media. All of this without setting up and managing your own infrastructure and at close to zero cost for running HashBridge in your own AWS account.

In the rest of this blog post, we go into more detail on how you can set up and use HashBridge, how we built it, and the implementation decisions we made.

How to use HashBridge

Use the following quick instructions to deploy HashBridge to your AWS account. For more detailed deployment instructions and the full source code, head over to the GitHub repository. Your issues and contributions are very welcome.

As a prerequisite, you need an AWS account and Node.js installed. Next, follow these four steps to set up HashBridge:

1. Go to your Hashnode blog dashboard, choose Webhooks and then Add New Webhook. Do not input an URL just yet but select all of the post_* events and copy the secret value. Keep this window open and we'll come back later with the URL.

2. Create a new secret in AWS Secrets Manager in the us-east-1 region. Choose Other type of secret and input the Hashnode webhook secret value as Plaintext and save it. Call your secret hashnode/webhook-secret. Note that while this secret needs to be in us-east-1, you can deploy the CDK stack in the next step to any AWS region.

3. Clone the HashBridge GitHub repository, run npm ci and deploy with npm run cdk deploy -- --all.

4. Copy the CloudFront URL from the CDK outputs and paste it into the URL field from step 1. Choose Create and you’re done!

That's it, you will now receive events from your Hashnode blog in your AWS account. Edit the sample consumer Lambda function to react to these events.

How HashBridge works

HashBridge uses AWS serverless services to set up a lightweight and easy-to-use integration between Hashnode, Amazon EventBridge and AWS Lambda:

In the following sections, we explain each element of this architecture in more detail.

Hashnode webhooks and GraphQL API

On the Hashnode side, two features make HashBridge possible: webhooks and the GraphQL API.

Hashnode webhooks send an HTTP request to an API endpoint of your choice whenever you publish, update or delete a blog post or static page on your Hashnode blog. They are a great way to build event-driven integrations between your blog and third-party tools. You can then use these events to trigger static site rebuilds, post to social media, or anything else you can think of.

Hashnode’s GraphQL API lets you CRUD all aspects of your Hashnode blog, such as posts and their metadata, comments, or static pages. With this API, you can use Hashnode as a headless CMS to build your own blog frontend (e.g. using the starter kit), publish a monthly newsletter, and more.

HashBridge makes use of webhooks to receive events from Hashnode and then the GraphQL API to enrich these events with data and metadata such as a blog post’s content in markdown. You can customize the query and enrich the event to suit your needs.

Receiving webhook events with Lambda function URLs

When you activate webhooks, Hashnode will start sending events as requests to an HTTP endpoint. As we don’t want to operate a full-blown virtual machine for this, we can use serverless functions with AWS Lambda to receive and process the webhook requests, ensuring we only pay for what we need.

The easiest way to receive requests in Lambda are Lambda function URLs. A function URL is a dedicated HTTP(S) endpoint for a Lambda function and basically turns a function into a simple API endpoint.

Hashnode webhook requests contain the post ID of the blog post related to the event but they do not contain the full blog content. For example:

{
  "metadata": {
    "uuid": "66c9430c-1152-4243-9fa1-17478f56f2bf"
  },
  "data": {
    "publication": {
      "id": "65a965a9f60adbf4aeebde2f"
    },
    "post": {
      "id": "65b7f792400f6e60e52d0bf2"
    },
    "eventType": "post_published"
  }
}

To get the blog content, our Lambda function uses the Hashnode GraphQL API to query the corresponding post by id to get its content:

query {
  post(id: "65b7f792400f6e60e52d0bf2") {
    content {
      markdown
    }
  }
}

When we started to work on the hackathon, this query was not yet possible. But sure enough, Hashnode created this new endpoint within just a couple of hours after hearing our feedback. That’s the new standard for customer obsession!

Finally, we publish the webhook event enriched with the post content to an EventBridge event bus from where we can then flexibly consume events inside and outside of AWS.

This allows us to receive the event once and then use it many times in multiple places. By consuming the event from the event bus, we are also decoupling the webhook verification and enrichment logic from the consuming services.

Verifying webhook events with Lambda@Edge

Many Lambda-based webhook solutions stop at this point and just leave the Lambda function URL exposed to the internet. However, as you can see in the architecture diagram, we decided to go a step further to improve the flexibility and security of our implementation.

The main reason we consider this necessary is that Lambda function URLs have limited options when it comes to authentication and security today: they only support IAM authentication, don’t allow us to restrict request headers and HTTP methods, or to set up a WAF, caching, or DDoS protection. We also can’t set up a custom domain, so if we want to replace the function later we have to update the Hashnode webhook URL as well.

To improve on these points, we set up an Amazon CloudFront distribution and define the function URL as its origin. CloudFront is a CDN which means that we directly get some benefits out of the box: we make use of the AWS global network of edge locations, get DDoS protection and a stable URL endpoint (and could even set up a custom domain).

Even better, we can now use Lambda@Edge, a serverless edge computing service, to inspect and manipulate each request that arrives at our distribution before it reaches our Lambda function URL. This way, we can verify webhook requests from Hashnode close to where they originate and before they reach our Lambda function URL.

Verifying webhook requests with Lambda@Edge

In the Lambda@Edge function, called Auth handler in our case, we want to verify that the incoming events are actually coming from Hashnode. To make this possible, we can use the x-hashnode-signature header which Hashnode sends with each webhook request. It consists of a timestamp and the actual signature:

{
  "user-agent": "HashnodeWebhooks/1.0 (https://hashnode.com/)",
  "content-type": "application/json",
  "content-length": "187",
  "accept-encoding": "gzip, deflate, br",
  "x-hashnode-signature": "t=1706555289174,v1=e2971dae21c3bd9bb1ca16c842c84d3032cb0b8fd4e1e4d6dda7ff959e1243e8"
}

If we are in possession of the webhook secret, which is displayed in the Hashnode dashboard when creating the webhook, we can cryptographically verify the validity of an event’s signature and ensure it’s coming from Hashnode. Thanks to the timestamp, we are also protected from replay attacks where an attacker reuses a signature. Take a look at Hashnode’s webhook documentation to see details of how this works.

We securely store the Hashnode webhook secret in AWS Secrets Manager and access it in our Lambda@Edge function.

Besides validating that requests are actually coming from Hashnode, the Lambda@Edge function also enables us to turn on IAM authentication on the Lambda function URL so that it only accepts requests from callers with appropriate IAM permissions.

We achieve this by configuring our Lambda@Edge with the appropriate permissions and then having it sign the verified requests that we want to forward to the function URL with SigV4.

Ensuring idempotency

To make the integration more resilient against data inconsistencies, we also set up idempotency controls. Let’s imagine that you receive two identical events for the same blog post being published from the Hashnode webhook due to an error. Depending on what you built on top of Hashnode, this could mean two emails going out to customers instead of one, or two processes being launched in downstream systems.

This is where the concept of idempotency comes into play. An operation is idempotent if it can be applied multiple times without changing the result beyond the initial execution. In our case, we want to make sure that even if the same Hashnode event arrives multiple times, for example for a new blog post being published, only one event is published on the EventBridge event bus.

As our API handler Lambda function is written in TypeScript, we can simply use the idempotency utility included in the Powertools for AWS Lambda (TypeScript) to prevent the function from executing more than once on the same event payload. We use a serverless Amazon DynamoDB table as a lightweight and performant idempotency persistence layer.

Building your first serverless function based on HashBridge

To write your first serverless function that makes use of HashBridge, you can use the example function included in the HashBridge GitHub repository.

We set up this minimal example function with the Node.js runtime and TypeScript but you can adapt the CDK template to use any other Lambda runtime as well:

import type { EventBridgeEvent } from "aws-lambda";
import { Logger } from "@aws-lambda-powertools/logger";
import type { EventType, PostEvent } from "./types.js";

const logger = new Logger({ logLevel: "DEBUG" });

export const handler = (event: EventBridgeEvent<EventType, PostEvent>) => {
    const eventType = event["detail-type"];
    logger.debug("Received event", { eventType, event });
};

This function is subscribed to the EventBridge event bus and is triggered each time a webhook event is received from Hashnode.

Conclusion

HashBridge makes it easy to extend your Hashnode blog with event-driven serverless functions. We had a blast building HashBridge over the last two weeks and we're excited to see what you are going to build on top of it.

Reach out in the comments or via GitHub issue with any questions and feedback.

A big thank you to the Hashnode team for this Hackathon challenge and for answering our questions throughout the process.