How to Build an AI Autonomous Bot(AutoBot πŸ˜„) - Crawl & Scrape the Web -> Generate Twitter Threads



onΒ Β· 3 min read

Have you ever wondered how people use bots to 10x their workflow and productivity. We'll go over the step by step process to creating a bot for boosting your twitter productivity leveraging all the newest technologies using AI.

This blog post is inspired by the brilliant work done by Jason Zhou. The content on this blog comes from this video, we love the idea and want to provide a version available in blog format for our readers.

High Level Overview of the App

  1. Serper request to get list of relevant articles
  2. LLM to choose the best articles, and return urls
  3. Scrape the content of each article and create summaries
  4. Turn summarized data into a twitter thread

Detailed Steps

This guide assumes you have a python environment set up already. Visit our environment setup guide if you have any issues with setup.

Serper is a lightweight service that queries google and returns the search results.

Create an account on Serper and obtain an API Key

Serper homepage a screenshot with some title text and buttons for sign in and sign upA screenshot of the serpapi api key page with a button to reset the key

Serper Homepage

Menu -> API Key Page to get the API Key

Create an account on Browserless and obtain an API Key, we will use this to programmatically crawl webpages

browserless api key page home page of the website with a chart on it

From here we can create a new project directory and also initiate our



At this point, we have created the project and added all dependencies needed.

Now we'll add our api keys for services we need to access. We will be using 3 services to create this autoBot (Serper, Browserless, and ChatGPT). It wasn't listed above, but make sure to create an OpenAI account in order to obtain an API Key.

Screenshot of twitter-autobot workspace with vscode ide

From here, we'll break it down by going over the code for each step in the high level overview.

1. Serper Request - Obtain list of relevant articles

Add the necessary imports for this section


Write logic to perform Serper query:


2. Use ChatGPT to Help Us Choose the Best Articles

We'll use the LLM to select the most relevant articles from the list and return us their url's

First import a few more libraries to our



Next, we'll define the prompt to instruct our AI agent what to do:


3. Scrape the content of each article and create summaries

Import the rest of the needed libraries. BeautifulSoup is a great library for extracting text from html.


We'll be using browserless to scrape the website data. Browserless is a nifty library for scraping web data. ChatGPT to make summaries of each article so that we can provide as much information as possible.


4. Turn summarized data into a twitter thread

Now, we're pretty much following the pattern we've used in many of the functions above. We'll define a prompt and feed in the structure of the data.

The prompt can be optimized however you see fit. For now, we have the basics, but you can probably come up with some interesting modifications after some trial and error.


Let's Put our AutoBot to Work

Let's run the code and see what we get



πŸŽ‰ Our researcher created a thread, what's cool is it even has emojis and hashtags.

Some observations and potential improvements:

  • ChatGPT tends to be verbose, so adding modifiers to the prompt such as concise, summarize, simplify can help the results be less wordy.

About @workingforlatte

Hey there, I'm @workingforlatte, an engineer and solopreneur on a mission to make AI accessible and affordable while indulging in my love for lattes. One day, I hope to treat myself to a luxurious mcol latte, and who knows, maybe even upgrade to an hcol latte (a lofty dream, I know)!

With a deep passion for technology and AI, I bring my engineering expertise to the forefront as I explore the ever-evolving AI landscape. I'm thrilled to be part of this AI blog, sharing my unique insights and unraveling the mysteries of artificial intelligence.

Join me on this exciting journey as we navigate the intricate world of AI together. From tackling complex technical challenges to discovering the endless possibilities of AI, I'm here to inspire and empower fellow tech enthusiasts like you.

Together, let's embark on this adventure of uncovering the wonders of AI and embracing the incredible ways it will shape our future. Get ready to dive into the AI realm alongside me, and let's make AI knowledge accessible and enjoyable for everyone. Cheers to a future filled with AI breakthroughs and, of course, plenty of delicious lattes!

Copyright Β© 2024