blob: a4b7da7af8ddc05d108795c3fc21e48fdd63dd97 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
|
---
title: "Purging the 1MB Club"
layout: post
summary: "I finally got around to testing and purging 1MB Club member websites"
---
This project has been running for almost 3 years now, and I still enjoy adding more members to the club! The only issue is that I have never been great at ensuring the quality of the members remained consistent. This included removing dead links, sites that now forward to less-than-safe domains, etc.
That all changes today!
## The First Purge
I wrote up a very crude script that checks the status of each existing club member's URL and flags those that error-out. After running the check I found I could remove over 60 websites that were either dead or broken.
> **Note:** If you happen to notice your website has been incorrectly caught in the "purge-crossfire", then don't hesitate to shoot me an email letting me know! I'm only human after all.
## Checking URL Script
This `ruby` script is far from perfect but it works well for my own personal workflow. Feel free to steal this and tweak it for your own purposes as you see fit!
```
require 'httparty'
require 'nokogiri'
HTTParty::Basement.default_options.update(verify: false)
response = HTTParty.get('https://1mb.club')
document = Nokogiri::HTML(response.body)
website_urls = document.css("#container tr")
puts "Scanning website members URLs..."
website_urls.each do |single_site|
begin
url = single_site.css("a.site").first.attribute("href").value
response = HTTParty.get(url, timeout: 4)
puts "Checking: " + url
rescue Exception
puts "<!-------- ERROR: " + url
end
end
```
## More Posts to Come
I have been slacking in terms of web-related content on this blog. My plan is to improve that and I even have a few outlines ready to go. I'll be sure to keep you all posted when something new is published!
|
