Get Your Own FireCrawl Alternative without the limits in 30 min!
Are you tired of hitting usage limits on web scraping services? Perhaps you’ve been using Firecrawl or similar services and found yourself constantly bumping against request caps or facing high costs for additional capacity. This guide will show you how to set up SpiderForce4ai a powerful Firecrawl alternative – on a reliable Hetzner Cloud server for just $4.59 per month with unlimited requests.
By the end of this tutorial, you’ll have your own dedicated web-to-markdown conversion API that you can use for content extraction, research, or data collection without any usage restrictions.
What You’ll Get from This Setup
- A fully functional web crawler service with unlimited requests
- Your own dedicated server with a custom domain
- Automatic HTTPS encryption for secure communications
- A complete development environment ready for further customisation
- No monthly subscription fees beyond the basic server cost
Step 1: Create a Hetzner Cloud Account
- Visit the Hetzner Cloud website.
- Click on the “Sign Up” button.
- Enter your email address and create a secure password.
- Tick the checkbox to agree to Hetzner’s terms and conditions and data privacy policy.
- Click “Continue” to proceed with registration.
If you already have an account, simply log in using your credentials.
Step 2: Create a New Project
Once logged in:
- Click on “New project” button.
- Enter a project name – for example, “myapiservice” as shown in the screenshot.
- Click “Add project” to create it.
Your new project will appear in the dashboard, ready for you to create resources within it.
Step 3: Create a Cloud Server
Now it’s time to create the server that will host your web crawler service:
- Click on “Create Resource” and select “Servers” from the dropdown menu.
Choose a location for your server. For optimal performance, select a location closest to your target users. In our example, we’re selecting Helsinki.
Select an operating system image. Choose “Ubuntu 22.04” (important: don’t use Ubuntu 24 as it might have compatibility issues with our setup script).
For networking options, keep the defaults with both Public IPv4 and IPv6 enabled.
Name your server something descriptive like “myapiservices” to easily identify it later.
Review the price (should be $4.59/month for the CX22 plan) and click “Create & Buy now” to deploy your server.
After a minute or so, your server will be ready, and you’ll receive an email with login details.
Step 4: Connect to Your Server
With your server created, you’ll need to connect to it using SSH:
- If you’re on Windows, use PuTTY or another SSH client.
- If you’re on Mac or Linux, open Terminal and use the ssh command:
ssh root@YOUR_SERVER_IP
Replace YOUR_SERVER_IP with the IPv4 address from your server details.
When prompted about the unknown host key, type “yes” and press Enter to continue.
Step 5: Install the Development Environment
To streamline the setup process, we’ll use a pre-configured installation script that installs everything needed for our web crawler service:
cd $HOME && curl -sSL https://raw.githubusercontent.com/petertamai/TheBasicSetup/main/setup.sh -o setup.sh && chmod +x setup.sh && bash setup.sh
This script will install:
- Docker & Docker Compose
- Caddy Server (with automatic HTTPS)
- Node.js & npm
- PM2 process manager
- And other essential tools
During installation, you’ll be asked if you want to set up a sudo user. Choose “y” and follow the prompts to create a non-root user with administrative privileges.
Once the installation completes, you’ll see the success message and usage instructions.
Step 6: Deploy the Web Crawler Service
Now it’s time to deploy the SpiderForce4ai container, which provides the web crawling functionality:
docker run -d --restart unless-stopped -p 3004:3004 --name spiderforce2ai petertamai/spiderforce2ai:latest
This command:
- Pulls the SpiderForce4ai image from Docker Hub
- Sets it to restart automatically if the server reboots
- Maps port 3004 on your server to port 3004 in the container
- Names the container “spiderforce4ai” for easy reference
Verify the container is running with:
docker ps
Step 7: Set Up Your Domain
To access your crawler service through a custom domain:
- Go to your DNS provider (we’re using Cloudflare in this example).
- Add a new A record pointing a subdomain (like
web2mark.yourdomain.com) to your server’s IP address. - Disable proxying for this record (set to “DNS only”).
- Set a short TTL (Time To Live) like 1 minute to allow for quick updates.
Step 8: Configure Caddy Server
Finally, we’ll configure Caddy to provide HTTPS for your domain and proxy requests to the SpiderForce2ai service:
- Run the Caddy domain configuration tool:
sudo caddyAddDomain
- Enter your domain name (e.g.,
web2mark.yourdomain.com). - Enter the port number
3004where your SpiderForce2ai service is running.
Caddy will automatically:
- Obtain an SSL certificate from Let’s Encrypt
- Configure the necessary proxy settings
- Restart to apply the changes
That’s It! Your Unlimited Web Crawler Is Ready
You can now access your SpiderForce2ai web crawler at https://your-subdomain.yourdomain.com. The service provides a web-to-markdown conversion API that you can use for:
- Content extraction
- Web scraping
- Research data collection
- Article summarization
- And much more!
All with unlimited requests, no monthly subscription costs beyond the basic server charge, and full control over your data.
How to Use Your Web Crawler (The Simplest Way)
You can now access your SpiderForce2ai web crawler at https://your-subdomain.yourdomain.com. The service provides a web-to-markdown conversion API that you can use for:
- Content extraction
- Web scraping
- Research data collection
- Article summarization
- And much more!
All with unlimited requests, no monthly subscription costs beyond the basic server charge, and full control over your data.
https://your-subdomain.yourdomain.com/convert?url=https://website-to-scrape.com
Simply paste this URL into your browser, and you’ll receive the converted markdown of the target website. This same endpoint can be used in your applications, scripts, or tools that need web content extraction capabilities.
The API will return the complete markdown conversion of the webpage, ready for you to use however you need—no authentication required, no request limits, and no additional costs!
Final Thoughts: Freedom from Usage Limits
Setting up your own web crawler service might seem technically challenging at first, but as we’ve shown, it’s actually quite straightforward with the right tools. For just $4.59 per month, you’ve freed yourself from the arbitrary usage restrictions and high costs that commercial services impose.
The beauty of this setup isn’t just the cost savings – it’s the flexibility. You can extend this foundation to add custom features, integrate with other services, or scale up when needed. Your data remains under your control, and you’re not dependent on third-party service availability or pricing changes.
So why continue paying more for less? With your own Hetzner Cloud server running SpiderForce2ai, you’ve got unlimited web crawling capability at a fraction of what you’d pay elsewhere. Happy scraping!