I got Everything working!

I finally did it!

I had to spend the entire day yesterday and today to fix the server, but I did it!

Tomorrow I’ll try to catch up with the posts which should have gone up while I was doing the repairs.


The culprit was fail2ban

No clue why or how it happened, but every time I restored the backup (with fail2ban) I was unable to log back in after a reboot.

I basically got no response from the server over SSH.

Having no way of looking at the logs, I just assumed at first it had something to do with my SSH config.

I was wrong.


Why did it take me so long?

First things first, when you configure your Amazon S3 to move the files to Glacier after a while, well, Amazon obeys.

But when you need the files, you’ve got to unfreeze them.

If you don’t want to pay extra, that’s gonna take 3-5 hours.

This is a one-time time cost if you tell Amazon to keep the files unfrozen for long enough.


Then, reinitialising the server takes 10 minutes every time you mess up.

Installing the necessary packages to load the backup takes 5 minutes.

Downloading the entire backup usually takes around 10 minutes on the server.

Copying the files from the backup-folder to the live system takes another 5.

So that’s 30 minutes just to reset the server to a point where you can try to experiment with stuff.

That’s a HUGE iteration time, so progress is naturally quite slow.


You could skip all this if your host provides you with a KVM, which is a virtual physical access to the server.

But mine is so bad, I can’t even enter the login password before the connection resets :/


The oh so long search…


The first few hours were spent investigating the SSH config, making SSH gets started after a reboot…

After a while, I decided to only copy some files to see if I could pin down the culprit.

I immediately found out that the ‘etc/’ folder contained the issue (I looked at this one first because I initially suspected SSH).

After a few failed attempts, I changed strategy and tried to get the server running by only copying the folders I thought necessary.

After each modification, I had to reboot to check if it was the culprit.

I basically worked my way up, trying to figure out which folders were required by apache2 and MySQL by the errors they reported.

At some point, after a long search already, I opened the auth log and saw a few failed login attempts from China.

So I decided it would be time to copy fail2ban to resolve the issue.

I restarted and there it was! I couldn’t connect anymore!


It’s finally working!

So I went through all the process of loading the backup again, except that this time I excluded the fail2ban folder.

And everything worked.

Just like that.


The lessons learned

1: chown is dangerous! don’t be lazy and look up command syntax if you’re unsure and don’t just assume things!

2: I’ve done something like 10 full backup restores, so I should be good when I have to do the next one

3: If you can’t connect back to the server, make sure it isn’t fail2ban doing stuff

4: Use a KVM if available!

5: Unfreeze Glacier files beforehand!

6: I learned a lot of collateral knowledge while scavenging my way through this. I feel less like a toddler wandering around the Unix ecosystem and more like a small child understanding some of the basics required to make some things wörk.


Also, don’t break something which is working fine.

Unless you want to learn how it works.

Then yeah, go for it, break stuff, and fix the mess yourself!


Live long and prosper!

Ship Weapons

3 – Departure

You know how governments are, always trying to “protect” their citizens.

This was no exception, and army and police forces were sent to surround these threats and make sure to maintain order.

The aliens didn’t seem to be bothered by the commotion caused by their arrival.

A few people, which probably were as fascinated and attracted by the alien ship tried to get through the security lines, but were quickly stopped by the armed forces.

This was the first reaction from the aliens.

Huge, Giant structures appeared on the hull of the ship.

They looked similar to the weapons used by the humans, but were much bigger and probably thousands of times more effective.

The army reading the mood of the situation, stepped aside, giving free the path towards the aliens.

As the first few tried to run for it, even more guns appeared.

Only a select few were still in the mood to make the jump, and only moved forwards slowly, not knowing what to expect.

They were left alone, allowing them to board.

After a while, as none of them came back, it became clear to the masses: whoever joined the aliens wouldn’t come back.

People became more hesitant, and no one was finding the courage to follow in the footsteps of those that came before.

I had come prepared.

My backpack contained foods of different sorts, a bit of water, a breathing mask, my laptop containing an offline copy of Wikipedia and a bunch of offline video games, a change of clothes and a solar charging cell.

As soon as a path opened up in front of me, I went for it, passed the aliens and set foot inside the ship.

Google It

Last week, in the intro to online learning, I told you to use Google.

So that’s what we are going to do!

(you can use your favorite search engine, I just like the interface and the results are good enough for me)


Google it

So, first step when you don’t know something, type it into google.

Check out the first few websites, get a little more comfortable with the subject.

Now that you know a few theme specific keywords, you can do a second more specific search.

Once you’ve found the perfect website teaching you all the things you want to know, bookmark the website to not lose it, and do some learning.


Magical keywords

How-to, tutorial, DIY, online course…

Some keywords are just linked to the type of content you’re looking for.

You’ll figure out a few new ones along the way, discover topic-specific keywords which will greatly help with your search results.


If you haven’t found the perfect website yet, let google help you.

The Suggestions give you similar searches which might hold different results.

Maybe you’ll be lucky and find the nugget you were looking for.

I usually try out a few different phrasings and suggestions, open them in new tabs and then go through them one by one.


Image search

If you remember a visual of some concept but can’t find the name and a quick Google search didn’t help, you might want to switch to image search.

You’ll be able to glance over a lot more content in a short amount of time.

Once you’ve spotted your target, open the link to access the website hosting it.

Now that you have the domain-specific vocabulary, you can go back to a normal Google search and find all the websites.


Google power tools

There are a few more advanced options in Google to help you refine your search.

I’ll just link this post since it’s a little more advanced and there is no need for me to rewrite something this good.

I found this article by searching for “google search lifehack”.

Lifehack being the keyword here, since it’s all about optimisation and self improvement.

I’ll just quickly go over them:

“” : exact match

– : not

OR : or

* : wildcard


Huge Roadblocks

What I call a roadblock here is a similarly named and popular element unrelated to your search.

For example, if you search for “stones” (rocks, pebbles…), you’ll only get results for the Rolling Stones.

The way to cut through the humongous noise is to use a keyword which will only be used in your niche.

Here, I’ll search for “stones geology” and the Rolling Stones are gone!


The more you use google and analyse which search querries held the best results, the better you’ll speak the language of Google.

This is an evolving language, so you’ll have to keep learning!

Also, Google is learning from you to serve you better results the more you use it.

You might find it uncanny for some company to know so much about you, but I find it useful, so, I’ll deal with it once it bites me in the butt.


Live long and prosper!