Actually this article should have been written a long time ago, but I was too lazy to drag it till now. TG-bot is quite fun and can do many unexpected functions. Because the back end of the i-Book.in page is algolia, their maximum data storage for free users is only 1W. After several expansions, my current data volume has exceeded a lot, but I still haven't rectified it, because Flask is too difficult. And if the extra data is not indexed, I will find it very troublesome, so I set up a TG-bot, and the backend uses Elasticsearch to store the data, so that even millions of data can be easily handled. Telegram's API.

The following sections are some introductions to this bot.

# Instructions

This bot has only one command, which is /so. This command can search both the book title and author, and returns the closest result based on what you enter. It is recommended to enter the complete book title / author to get the most accurate result. Of course, if you search for books that aren't in the database, some keywords with similar results will be returned.

If you are interested, you can click {% btn https://telegram.me/ibookin, here,telegram fa-lg fa-fw %} to harass this bot in the TG group.

It is normal to find the book you want. After all, it is only a database with a small sample size. You must know that there are hundreds of thousands of new publications every year. The current total is even more None of them. I don't plan to make a profit based on this, I'm just interested in the data collation and extraction. Recently, I have been crawling information about some books, and I plan to do some sublimation of data, such as who has the most books in the database, which country has the most books, and what types of books have the most.

The whole operation step is simply: the bot first receives the information from the private chat and returns it to the back-end script, and the back-end retrieves the ES based on the content of the information, and then sends the information parsed back to the channel that received the message based on the returned data of the ES . I don't really want to write the specific details, because there is no database and I can't rewrite this bot, so I only plan to write some core content.

# Construct

Install docker

Because I'm not very proficient in ES, I use the docker version directly, without having to mess with java and messy configuration items, you can use pull directly.

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

Install & configure ES

After docker is installed, it is time to get ES:

docker pull elasticsearch:6.7.0

Create the mount directory and configuration file of the ES:

cd  /
mkdir-p mnt/elasticsearch
cd  mnt/elasticsearch
mkdir config
mkdir master
mkdir slave
chmod 777 master
chmod 777 slave

Put two configuration files in config:

cd config
touch master.yml
touch slave.yml

matser.yml：

cluster.name: elasticsearch-cluster
node.name: master
network.bind_host: 0.0.0.0
network.publish_host: `your ip`
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true 
node.data: true  
discovery.zen.ping.unicast.hosts: [" `your ip`:9300"," `your ip`:9301"]

slave.yml：

cluster.name: elasticsearch-cluster
node.name: slave
network.bind_host: 0.0.0.0
network.publish_host: `your ip`
http.port: 9202
transport.tcp.port: 9302
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: false
node.data: true  
discovery.zen.ping.unicast.hosts: ["`your ip`:9300","`your ip`:9301"]

Need to increase the number of JVM threads limit (otherwise an error will be reported):

nano /etc/sysctl.conf
#Add this
vm.max_map_count=262144 
#Execute this command after saving
sysctl -p

Initialize the container

master：

docker run -e ES_JAVA_OPTS="-Xms256m -Xmx256m" -d -p 9200:9200 -p 9300:9300 -v /mnt/elasticsearch/config/master.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /mnt/elasticsearch/master:/usr/share/elasticsearch/data --name es-master elasticsearch:6.7.0

slave：

 docker run -e ES_JAVA_OPTS="-Xms256m -Xmx256m" -d -p 9201:9201 -p 9301:9301 -v /mnt/elasticsearch/config/slave.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /mnt/elasticsearch/slave:/usr/share/elasticsearch/data --name es-slave elasticsearch:6.7.0

The rest is docked with the bot. I am updating this part of the system for a while because the bot has not been fully optimized and will be placed in Gtihub.

# postscript

Only when the amount of data is large can the data be played, so the book information I collected recently is not crawled based on my existing data, but crawled across the site when I see one, so if I have an e-book in the future File, you can directly output the data, and you do n’t need to search for data. The next step is to open source the information of the ipfs-free books on github (I remember that there are already existing ones, but I like to make wheels!).

Recently, I have just finished reading a book named 间客, and there is a sentence that appears many times: "The source of inequality in social classes is inequality in information." We have no way to equalize information to everyone, but if we have the ability to get more information, why not do it? I think everyone has this ability, just because of laziness, because of fear of trouble, because it feels difficult, because it feels time-consuming, so give up.Reading a book is a time-consuming and energy-intensive task. (Someone will definitely refute me. I said that after reading a few million words in a few days, I'm sorry, those are electronic drugs. There is no nutrition after reading.), Reading the book seriously can bring the mind to the characters in the book, you can experience the emotions and feelings that the author passes through the text, and you can learn a lot of new knowledge. This knowledge is not necessarily English, but it may be the practice of a certain dish, or it may be a very unpopular trick, but once you master it, it is your unique knowledge.

It may be necessary to use Telegram to increase the difficulty for some people to use this bot, but I think the threshold is necessary. If you do not find the book you want using this bot, please do not talk privately. Let me add it, I will not add it. I don't like to add them one by one, that's silly, unless I find another batch of data. But ... if there are hundreds of gigabytes of data lying in front of me, I'm still very shy.