Setting up a 4get Instance (Meta Search Engine)
I’d figure I’d just copy what I’ve written on the README.md for a proper tutorial on my website!
Installing with Caddy
Installing dependencies
1apt install caddy php8.2 php8.2-dom php8.2-imagick imagemagick php8.2-curl curl php8.2-apcu php8.2-fpm git
Cloning the repository
1cd /var/www
2git clone https://git.konakona.moe/diowo/4get
As you can see, this will download MY repository. The official repository is at https://git.lolcat.ca/lolcat/4get.
Set permission for the icons/ directory
1cd /var/www/4get/
2sudo chmod 777 -R icons/
Add an entry for 4get on your Caddyfile
/etc/caddy/Caddyfile
14get.konakona.moe {
2 root * /var/www/4get
3 file_server
4 encode gzip
5 php_fastcgi unix//var/run/php/php8.2-fpm.sock {
6 index index.php
7 }
8 try_files {path} {path}.php
9}
Caddy deals with SSL certificates automatically so you don’t have to mess with anything. Also if needed, a sample of my Caddyfile can be found here.
Restart Caddy
1systemctl restart caddy
Captcha
You will probably want to enable a captcha for this proxy if you want to host it publicly or else bots are going to rape it, literally.
The steps on enabling the captcha are a bit manual but here is more or less what you have to do:
- Acquire a bunch of images of objects or something, like idk: minecraft, fumos, konata, factorio.
- Then you will have to convert everything to
100x100pixels.png, and name them from1.pngtoN.png. - Then copy the folders over to
/var/www/4get/data/captcha/
- Now enable the captcha by editing
/var/www/4get/data/config.php(the number on the right being the number of images if you have 70 images then it’s 70 because I think the index starts at 1)
1const BOT_PROTECTION = 1;
2
3const CAPTCHA_DATASET = [
4 ["KSP", 84],
5 ["fumos", 121],
6 ["minecraft", 70],
7 ["factorio", 94],
8 ["konata", 143],
9 ["yui", 232]
10];
NB: When someone gets a captcha, the captcha will say you have to “Pick N images of <name of the directory>”.
To help you, here is a script that I wrote for FFmpeg:
https://git.konakona.moe/diowo/misc/src/branch/master/scripts/4get-captcha
1#!/bin/sh
2
3# Scale, crop, convert and rename files for the 4get captcha with FFmpeg.
4
5if [ ! -d "$1" ]; then
6 echo "\$1 doesn't exist!"
7 exit 1
8fi
9
10if [ ! -d "$2" ]; then
11 mkdir -p "$2"
12fi
13
14counter=1
15for file in "$1"/*; do
16 if [ -f "$file" ]; then
17 ffmpeg -i "$file" -vf "scale=100:100, crop=100:100" "$2/$counter.png"
18 counter=$((counter+1))
19 fi
20done
And a python script to scrape images off of 4get (only works with instances that have the captcha disabled for now!)
https://git.konakona.moe/diowo/misc/src/branch/master/scripts/4get-img.py
1#!/bin/python3
2import requests
3import mimetypes
4import uuid
5import sys
6
7
8url = "https://4get.konakona.moe/api/v1/images.php?s=" + '+'.join(sys.argv[1:])
9
10response = requests.get(url)
11
12if response.status_code == 200:
13 json = response.json()
14 for image in json["image"]:
15 img = image["source"][0]["url"]
16
17 try:
18 response = requests.get(img)
19 except:
20 print("Most likely timeout error.")
21
22 if response.status_code == 200:
23 extension = mimetypes.guess_extension(response.headers.get('content-type'))
24
25 if extension != None:
26 filename = str(uuid.uuid4()) + extension
27 with open(filename, 'wb') as file:
28 file.write(response.content)
29 print(f"File downloaded successfully as {filename}")
30 else:
31 print("Skipped because content-type could not be found on the headers.")
32 else:
33 print(f"Error: {response.status_code}")
34else:
35 print(f"Error: {response.status_code}")
Optional Tor support
If you want to server 4get on the Tor network, follow these steps.
Installing the tor daemon
Debian repositories for the tor daemon absolutely suck because they are old. You’ll have to use the tor project repository.
1apt install -y apt-transport-https gpg
2
3echo "deb [signed-by=/usr/share/keyrings/tor-archive-keyring.gpg] https://deb.torproject.org/torproject.org $(lsb_release -cs) main
4deb-src [signed-by=/usr/share/keyrings/tor-archive-keyring.gpg] https://deb.torproject.org/torproject.org $(lsb_release -cs) main" > /etc/apt/sources.list.d/tor.list
5
6curl -s https://deb.torproject.org/torproject.org/A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.asc | gpg --dearmor > /usr/share/keyrings/tor-archive-keyring.gpg
7
8apt update
9apt install tor deb.torproject.org-keyring
Make sure that $(lsb_release -cs) got replaced with the release name of Debian, in my case bookworm.
Now enable the tor daemon
1systemctl enable --now tor
Tor monitoring
You can monitor your tor node using an official tool called nyx.
1apt install nyx
Then to use it whenever you want
1nyx
Creating the Tor site
Add these lines to your torrc
/etc/tor/torrc
1HiddenServiceDir /var/lib/tor/4get/
2HiddenServicePort 80 127.0.0.1:1337
3HiddenServiceVersion 3
As you can see, I chose the port 1337, I think it’s important to choose something different like this specially if you have multiple sites running.
Now restart tor
1systemctl restart tor
Updating the Caddyfile
Caddy will now need to answer on the port 1337. To do this you will need to update the Caddyfile.
/etc/caddy/Caddyfile
14get.konakona.moe, :1337 {
2 header Onion-Location http://tvwj4wosozy5pyqbc3ivor4uyqvtnmvgtn3ktzvcpuujrovuuozgswid.onion{path}
3 root * /var/www/4get
4 file_server
5 php_fastcgi unix//var/run/php/php8.2-fpm.sock {
6 index index.php
7 }
8 try_files {path} {path}.php
9 encode gzip
10}
Add the lines header Onion-Location http://tvwj4wosozy5pyqbc3ivor4uyqvtnmvgtn3ktzvcpuujrovuuozgswid.onion{path} and :1337 on your Caddyfile. Your onion address will of course differ from mine, and to get it you will need to paste this command:
1cat /var/lib/tor/4get/hostname
Now change it to the one you got and restart caddy again.
1systemctl restart caddy
Banners
You can also make 400x100 banners to use on 4get! A way to add soul to this soulless web. The banners are stored in the folder /var/www/4get/banner/.
Here are mine for inspiration, yeah this section is just me flexing my cool banners:
4get-konakona.png
4get-seb.png
Created on: Friday, 1 March 2024
