I recently redesigned i-book.in and updated the automation script. I will not post the specific code. It is available on GitHub.

Github：Ebook-crawler

Because it does not involve core data, it is just the source code of the crawler, so I posted it on GitHub. Those who are interested in slowly crawling the data can git use it on their own server. If you are too lazy to use it, you can use i-book.in directly. Already.

The specific implementation method is quite simple:

1. Get the name of the book.

2. Go to Algolia to search by name, and confirm whether there is this book. Skip if there is, parse and download if not.

3. After downloading, upload the data to the ipfs network and parse the hash.

4. According to the parsed book data and ipfs hash data, combine them into json format characters that Algolia can accept.

5. Upload the json to Algolia, so that the database has this book, and it will not be downloaded next time.

# Anti-reptile

No matter what the site is, there will be crawlers, so after using a CF shield for the blog, I also used CF for i-book.in, and added anti-crawling measures to the operation of downloading e-books, which is the CF firewall rule. The internal settings use the suffix format of the e-book as keywords, and these keywords trigger human-computer verification. The specific settings are shown below.

# English original book

In addition to Enjing's study room, I have recently crawled the original English book. Although I may not understand it, I can also learn English if I look at it. The main thing is that crawling the data is fun.

This site is still crawling. The specific crawling code is waiting for me to debug, and it is also put on GitHub. I have to spit it out. Why is it strange for foreigners to name books, all kinds of punctuation? To replace it, otherwise it will fail to create a folder, causing subsequent downloads to fail.

After downloading for one afternoon, I downloaded a total of 3473 books, and there were 17 book errors. I was too lazy to debug these 17 books, as if it was still a special symbol problem.

# Algolia account merger

After the original English book is integrated into a database, the data in the database will exceed 1W, so i-book.in on the backend needs to use multiple Algolia accounts to search at the same time.

So the combined account has to be on the agenda, and I won't, so I plan to rewrite a python web service, built-in multiple Algolia accounts to expand the index, or directly go to sql, but my sql will not (the chicken is me That's right. -EOF-