Hurricane Charley and the life it changed.

Let me start by saying I get nostalgic about the old times. I miss them, and my friends. But I would never change a thing that happened. I am grateful for all the times we spent together, and will cherish those memories for as long as I can. We made a lot of memories – some good, some not so good – but those and the great memories will forever be in my heart!

Today marks 15 years since Hurricane Charley formed. In 4 days, it will be 15 years since it "wobbled" up the Peace River in Punta Gorda. It destroyed my home, my life, separated my group of friends. It destroyed my home town, and the quaintness it once had. It took so much from so many.

Out of destruction comes anew. The plot where our home was has long since been cleared, and I think there’s even a new house there now. My friends all started new lives, doing new things, and excelling as wonderful people. My home town has been re-built, losing it’s once beautful quaintness, but gaining a renewed aesthetic beauty and even a better economy. All of those other people have moved on, gotten stronger, doing better, and are for the most part living their lives as normal. It’s been an interesting journey for me, however.

It was about this time, 15 years ago (and maybe a month), that I met Jeff Bushey, and his little company SurityNet, and The PC Hospital. I had just applied for a job at a new restaurant opening soon in the strip mall, and from the sounds of it – I had gotten the job. I walked into the office looking to buy a PCI SCSI (80 pin at that) card for my PC, so I could retrieve some stuff off of my old Macintosh drive. (When did Macs stop using SCSI?) After talking to Jeff for a bit, telling him what I needed, and that I’m shopping around and can’t promise to buy anything. I explained the situation with work. By the time I left the office, I had gone through an interview, and was offered a position. Flabbergasted! I was exstatic! I’d been working for him for a very short time when the hurricane hit.

Enter August 13th, 2004. Charley is aiming for Tampa Bay. Seemingly everyone in Punta Gorda is having a hurricane party. My friends and I included. We’re baking a pizza, so I stay home when everyone else went down the road to our friends house to get them to come down. The usual thing – most of the group goes. They never came back. 20 minutes later, I get a call from my mom, telling me the hurricane changed directions, and is now heading for us. She told me she sent her husband to get me (and my friends if needed) By the time he drove the 10 minutes to where I was living, the winds had picked up, palm trees were bent over, leaves and debris flying through the air. Oh boy was I glad to be getting away from the water! I found out later, my friends had been told to stay at our other friend’s house, which was considerably better built. They all made it out OK!

It wasn’t until about 5 or 6 when Bear was able to take me back home. The roads were blocked with downed trees and power lines. I had to walk the last mile. Only, when I got home, there was no home. The roof was in the road, the windows were blown out, there was debris everywhere. My pizza! It was still in the oven. It had finished cooking! WOO! I have some food. Awesme! I’m now in the middle of a completely destroyed area, with no one else, and almost no supplies. I realized there was pretty much nothing I could do at the house, but there was plenty I could do to clear the roads. I made my way around, moving trees, and (DO NOT TRY THIS AT HOME!) power lines out of the way for emergency vehicles and residents. I made it back to the corner store. People were looting it. Kind of sad. One person wrote a note and put some money with it in a safe drop tube, and put it in the safe for the items he had taken. I needed water, and there was no running water – and if there was, I wouldn’t had trusted it. So, yes, I took a gallon water, a snickers and a can of mountain dew. I can’t justify my actions, they are what they are. I did go back about 3 months later, when the store opened, and offered to pay for what I had taken, and was told not to worry about it. The manager told me how many people had done the same, and honestly, I was put aback by that. Very cool. Insurance covered the losses, and it would had technically been illegal to now sell those items – or something.

I walked the 5 or so miles from there to town, moving branches and lines as I went. (Again, DO NOT MESS WITH DOWNED POWER LINES!!! I was dumb to do so, but did so as safely as I could) In all, I cleared probably 7 or 8 miles of roadway for traffic. When I got down to the old neighborhoods downtown, I spent a couple hours helping some friends of my friends clear some very large trees from the roads. From there, I walked down to the Highschool, where there’s some (at the time) newer apartment buildings. I knew Barb’s daughter lived in one, so I walked around looking for Barb’s car. Come to find out later, I had just missed her. It’s getting dark, and dark in that situation is dangerous. I had no choice but to find shelter or, preferably, to find a way back to Mom and Bear’s house. I went back to the previous neighborhood, and someone had a working cell phone. I tried calling, after the third time, I finally got through. Yay! I don’t have to sleep in a random place!

The next day, Bear and I drove up to SurityNet. I didn’t have my own transportation, and Bear wouldn’t be able to take me back and forth. I had to let Jeff know I couldn’t work any more, and would have to work with Bear, to be in exchange for now having to live there. Not a big deal, just sucks I can’t work at SurityNet any more. We go out and do some errands – checking on customer houses, getting some water and MREs and head home. Jeff calls. He’s got an offer for me. If I would be willing to pay rent, he had a spare room for me. He said he could take rent from my pay, and take me in to work with him, and bring me home too. Jackpot! I get to live in a nice, new home with airconditioning, get transportation to and from work, and get to keep my awesome PC repair job! This turned out to be one of the best things for me. Jeff and SurityNet introduced me to the wide world of IT at large.

Note: I’m going to skip a pretty regretful situation involving me moving to Kansas, being cheated on and having to move back to Florida. It wasn’t a plesant time at all.

It’s later 2005, and I’m ‘renting’ a room from Barb, helping her to take care of her dogs (about 20 or so) which were used for breeding. Every dog she had was loved and cared for greatly by her, myself and many of our friends. They were pets… that just happened to help pay for themselves. I stop by SurityNet one day, just to say hi to all the people, and Jeff offers me my position back. I took it. In hindsight, I probably could have made some better decisions – but I didn’t. Life was good, and I worked for Jeff for a few more years. We’re swamped with work. We need someone new.

Jeff gets some resumes in, and asks me to look over them. We settle on one, from Shawn, and Jeff calls him in for an interview. Shawn has amazing experience – from PC and printer repair to networks and firewalls. Top notch person to be working with us! Needless to say, Shawn gets hired. At this time, I’m back at Jeff’s. After a week or so, Shawn asks if we know of any RV parks near by, as he and his wife live in an RV. Jeff had sold his to help build up SurityNet, but still had his RV shed at the house. Jeff, being a kind and wonderful man, offered Shawn to park there. Wow, it’s like some kind of tech beta house now! Shawn, his wife and I became good friends. Some time later, they left, as he had an amazing opportunity for work. But Jeff nor I could blame him. We all missed them though.

The whole time this is going on, I’m chatting with friends in IRC. Good friends. I love me some IRC – so many interesting people, such good friendships can be built! We’re playing Neverwinter Nights, Guild Wars and some other games. In 2009, one of my friends suggest this new game from this Swedish indie game dev, called Minecraft. I’m broke, living in a travel trailer behind Mom’s, and no longer working for SurityNet. I tell him I can’t afford it, and my computer (a little laptop) probably wouldn’t run it well. It could barely run a 5 year old Neverwinter Nights. Turns out it was free, web based, and played much smoother than any other game (Oh my how I miss those times for Minecraft!) I hop on some servers, and am annoyed by players griefing my stuff. Every server I log into, just chaos and idiocy.

Then I found BuildSomethingFool. An amazing community run by a couple of potheads (at least I think they were) They had an amazing staff, and the players were very well behaved (or banned if they weren’t) I spent like 3 days building an Eiffel Tower build, with an underground area, gardens, etc. All without using hacks. No flying, nothing. The owners were amazed and offered me to be staff. I could now ban the little trouble makers! WOO! I also learned the game and could explain it very well to new players. One in particular was so confused, and so hapless – I couldn’t help but take pitty on her. I spent probably 20 hours with her, teaching her Minecraft, and embuing the knowledge I had gained. She was a quick learner – and soon became staff as well. By this time, Minecraft Beta was being released, and premium accounts were being sold. VueJohnson, her player name, wanted to thank me for everything I taught her, and purchased an account for me. I was so grateful. The only thing we could do at the time though was to change our skins.

I took a break from Minecraft for a while. Everyone was focusing on this new beta of Minecraft, and I wasn’t able to run it nearly as smooth as the old classic version. When I came back, I had a better computer, thanks to a wonderful friend who I haven’t spoken to in years, but always wished to be annonymous. I could play again! But it was kind of boring. I tried starting my own servers, and quickly ran into problems. Low and behold, where is support for these things, on IRC! I was right at home! I came for help, and stayed to provide the assistance I could to new people. I did this on and off for a few years. It was a great hobby for me. I ended up helping someone get their servers up and running, and really for the first time was doing something I truly enjoyed doing with Minecraft since beta came out. This is not where my life would have led me if I had not moved in with Jeff the first time.

Shawn and I get back in communication. I end up going to his place for a weekend to visit. Some time later, he’s going to move again, and I help him move.

I’m sitting doing some work on this other fella’s server, and keeping an eye on the IRC support channel I was in. This person, presumably a girl, asked some questions – I answered the best I could. After a few weeks, we had become pretty good friends. But I couldn’t tell how old she was – not that it mattered, our friendship was open, public and innocent, but I was just a bit confused. Some days it seemed like she was this mature adulting person who had life together, but then she sometimes seemed like a 12 year old – playful, creative and curious – You know, the good parts of 12 year olds. So I told her. She never told me her age, but confirmed she was much older than 12. Then one day, she shared a picture of her brand new swimming pool and spa. Oh! She’s either much older than 12, or REALLY has her life together for being 20 something. It’s a beautiful picture. I told her "One day, you wait and see, I’ll be swimming in that pool!" More to tease her, intentionally coming across as a bit creepy. We were at that level of friendship. Or so I thought. She didn’t reply for what seemed an eterity! I was crushed! I just ruined a good friendship over something silly.

Well, later that day, she did reply. She insinuated that me swimming in her new pool was not out of the question. Woah! I’m thinking we’d meet up for a lunch or something, I don’t know. We both live in the same state, within an hours drive. So, completely possible.

About this time, Shawn is moving out of state, to Pennsylvania. He asks me to help him move, again. So I do. I figure it’ll be a week or so. We get up there, and there’s a lot of work that needs to be done, so I offer to stay and help. I’m doing this for a good few months. Great times. We learn a lot about construction, remodeling, and even gardening! We’ve got one more trip down to Florida, for more stuff, and Shawn has some business to tend to. I let this woman I’ve been friends with know, and that I’d like to meet up with her one day while I’m down, to have lunch. She agrees! Amazing! She drove out to meet us, and had a fantastic time! But time’s up, and Shawn and I have to head back up north. She sends me along with a cell phone, so we can keep in contact. She’s going to London for Minecon. Yeah, things were a bit more serious than friends, I’m quite happy to say. We talked every evening. I had been fighting it for months, but after that weekend, I knew I was in love. She was too, apparently!

It’s August again, 2015. I’m done helping Shawn with what we can do. I’m planning to head home, when she tells me I should come visit her first. She’ll pick me up from the air port, and I can stay at her home for a while, and swim in that beautiful swimming pool she has! I never left. In fact, I married her in 2017. Something I never thought I would do in life is get married. It’s been an amazing 4 years. An amazing 4 years that I never would have had if Hurricane Charley hadn’t so wonderfully destroyed everything I knew those 15 years ago. It was a long road, but one so very worth it. It was a journey I had to take to be ready to be the person I am for myself, and for her. I can’t help but look back today and say "This was God’s plan all along, and I know he waited until I was ready to let her I meet!" To this day, I love my wife some very much, and would give the world for her.

There’s not a day where we’re separated due to work where I do not miss you with every ounce of my being. For so long, I felt a void in my life, in my being, in my soul, one which only you have ever been able to satisfy. I love you Cindy!

New Linux Install!

Today, I found a wonderful deal on a small VPS over at Ionos: 1 vCore (Xeon Gold 5120 @ 2.2GHz), 512MBs RAM and 10GBs SSD storage – all for $2 per month.
This might not sound like a whole heck of a lot of resources to you, and you’d be right. But for specific use cases, this is perfect.
(disclaimer: The above link is a referral which may provide financial gain for us, with referral rewards)

If you’re using another hosting company’s VPS, Dedicated server or VM, you might find a good bit of useful information here, especially the stuff past the Ionos setup and configuration stages. For initial hardening of an Ubuntu server, you might want to read this article here.

So, the first thing you’d need to do is to create an Ionos account (presuming you’ll be ordering a VPS from Ionos), and then order your VPS. Like most hosting companies, you can create an account with your first order. I actually really like Ionos account pages and provisioning and management interface. The one thing I do not like is having to use a customer id to login, but to each their own.

This is my first VPS with Ionos. Ionos was previously named 1&1, but has changed considerably since their merger and name change. We (My wife & business partner) have a dedicated server from Ionos, which we’ve had for about 8-10 months now. It’s a solid server with no issues. I’m expecting the same with this VPS.

Well, that’s partially true. I ran into a snafu with provisioning ipv6 on this VPS, and resorted to a fresh install. Both times, I had Ubuntu 18.04 installed, because it’s what I’m comfortable with. I really like apt/apt-get, and some tools made by the Ubuntu team, and feel they’re better suited running on Ubuntu itself.

The issue with the ipv6 provisioning was actually not an issue with provisioning, but a mis-understanding about Ionos’ hardware firewall, which sits outside of the VPS. I failed to realize that their firewall was what was blocking my attempts at ipv6 connectivity. Upon re-imaging of the VPS, I read the little pop-up, which stated something about firewalls – At that point, 2 hours of work were gone and I was face-desking pretty hard, because I knew that was my issue all along.

So, step 3 (1 & 2 are above) is to create a new firewall configuration, and, for the time being, allow all connections so that the firewall is not an issue for setup. I personally will be taking advantage of the hardware firewall, once I’ve got all my services provisioned and working. That way, if I run into any issues in setup, or afterward I can narrow down the cause. Some IS professionals would argue with me about not initially taking advantage of this firewall. They may be more correct. After creating the configuration, you’ll need to assign it as your active firewall rules for the VPS. The Linux server does not have to be restarted for this. (Note: I use a software based firewall within the Linux environment to restrict access to services, ports, etc. I personally use and have found UFW to be more than adequate to do the job in lieu of IPTables, another software firewall for Linux) The only other thing to mention here is that you must manually setup an IPv6 address through the management interface for the VPS, and to set up ipv6 firewall configs the same as for ipv4, if you plan to use IPv6 at all.

At this point, you should have an account and interface access for your hosting company, a VPS, and hardware firewall config(s).

Now, let’s get to it! Use your favorite SSH client to log into your fresh VPS. You’ll need the root password given to you from your hosting company, usually sent to you via email. Ionos, however, has the new-image generated password available on the management page. Pretty nifty! You should be able to connect with any SSH client over port 22/TCP. PuTTY, KiTTY (a fork of PuTTY), WinSSHTerm, SSH client built into Linux, as well as any other SSH standard-compliant client will work.

At this point, we’re through with Ionos, and everything here will be Ubuntu, if not GNU/Linux generally relevant.

CHANGE YOUR ROOT PASSWORD!!!

Once logged in as root, type {passwd} and then enter your new password.
(Again, for a better start to hardening your server for security, read the "Linux SSH login – a good starting point", linked above)
At this point, it is advised to create a new user account, with sudoers access, with a new password, and then log out of the VPS as root and log in with the new account. We’re going to ignore this for the time being as everything we’re going to do first requires root/sudo access, and in the event that someone manages to get into your system before you’re done, it’s not too troublesome to reimage the VPS.

UPDATE YOUR APT CACHE!!!

Before doing much else, you should run {apt update && apt upgrade}
This may (more than likely /will/) cause a kernel update, and will require a restart (shutdown -r now)

SETUP & INSTALL SOME BASIC STUFF!!!

Now, let’s get some administrative things out of the way. Namely, hostname and fqdn (fully qualified domain name), additional utilities, and some software & services.
UFW – Uncomplicated Firewall, easier to use firewall than IPTables. (IPTables has it’s place, but most don’t need that power) {apt install ufw}
fail2ban – Intrusion mitigation software to ban access after N unsuccessful authentication attempts. {apt install fail2ban}
Linux PAM – Pluggable Authentication Module, part of most modern distros. Ensure it’s installed.

  • additional reading and consideration for libpam_shield and pam_tally2 for additional levels.

htop – a better hardware resource monitoring tool, with CPU, RAM and cache graphs, process list, etc. {apt install htop}

GNU Screen – a virtual terminal service allowing easier management of full-time processes (tmux and fg/bg work too!) {apt install screen}

HAProxy – an HTTP(S) and TCP proxy, for routing connections (layers 4 & 7) to different ports and hosts. Not required, but useful {apt install haproxy}

HATop – a monitoring tool for HAProxy, requires reading documentation to use. {apt install hatop}

MariaDB – An enhanced fork of MySQL SQL database server – You’ll know if you require an SQL server. {apt install mariadb-server}

  • MariaDB setup will require you to have certain information available, and written down for later access. This can be done later in the overall setup process though.

Java – If you require a Java Virtual Machine (JVM), I highly suggest using Oracle’s JRE. This, however, requires adding an apt repository. Read more here to install Java 12 in ubuntu!

  • If you choose not to use Oracle’s JRE, you can use OpenJDK, with a simple {apt install openjdk-11-jre-headless}

Hiawatha – a security focused light weight (compared to Apache, anyways) web server. Requires source tarball to install latest version.

This should about do it for the additional software and utilities. At least as far as installation goes. Now, onto configuring hostname and fqdn!

With time, change comes. Change is good, needed and wanted. Sometimes it isn’t. Sometimes older technology works just as well, or even better in some cases. There’s various ways to set your new server’s hostname. We’re going to use the tried and true method.

There’s a couple places to set hostname and fqdn.
/etc/hosts and /etc/hostname are two files, where changes will be made.

In hosts, you’ll add your public IP (the same IP you used to connect to the server via SSH) and the fqdn you wish to associate with that IP.
{12.345.67.89 blog.bluntaboutit.com}
This assigns blog.bluntaboutit.com to the IP 12.345.67.89 (fake IP, do not use!)
{ff02:816:f00d:3475::1 blog.bluntaboutit.com}
This assigns the fqdn to the provided IPv6 address (also fake)
With this, blog.bluntaboutit.com will connect to either the v4 or v6 address.

In the hostname file, you’ll add your hostname, which will appear after the @ in bash, as well as identify the machine on the network and other spaces.
This is a single simple string.
{blog.bluntaboutit.com}

Make sure to use domains you actually own and can assign the IP addresses to in your DNS server. Otherwise you might find yourself in a heap of trouble, possibly even with your hosting provider.
Once you’ve edited your files, confirmed the data is correct, saved the files, confirmed the data is correct again, you can restart the VPS. This will solidify the settings and cause your server to use the new hostname and fqdn on start up. Another option, for temporarily setting the hostname is to use {hostname blog.bluntaboutit.com}

Now, I’m running this server as a POP server – point of presence. It’s a server dedicated to running a reverse proxy (HAProxy), where users will connect to and be forwarded to the real server. This is due to the real-time nature of the connections. Having this server will give more stable client-proxy connections to those in the region than doing a client-server connection directly. It adds a tiny bit of latency to the connection, but overall it’s more stable. The proxy-server connection is running through private infrastructure, and so is unencumbered by public traffic, and less hops. Ultimately, there is less latency for the client-proxy-server connection than for client-server connections for most users in the region of the world closer to this server.

With that, I won’t be using mariadb, screen, java, or hiawatha. However, I will still be using UFW, fail2ban, PAM, SSH keys (for login), htop, HAProxy and hatop. The afformentioned software is noted, mostly as these are things which I would normally use on a server, for various reasons and to varying degree. They may also be things which others may forget to install at a more appropriate time. And so, they’re listed as a reminder – just in case. Others may have other software which they consider to be basic stuff, and may want to add to the list of initial setup installables.

SETUP YOUR FIREWALL!!!

Now, there’s two firewalls you can use. I highly suggest using both your hosting provider’s hardware firewall, as well as UFW (Or IPTables, if you need the power it provides). UFW is super simple. But, there is a bit of a learning curve.

Setting up UFW:
Before you do ANYTHING with UFW (once you have it installed, that is) PLEASE do yourself a favor and add your ssh port.
{ufw allow 22/tcp}
This adds a rule to UFW to allow any connection (inside or outside the private network) to connect to the server to port 22 via TCP on IPv4 and IPv6 address (if IPv6 is enabled on your server)
UFW is still very powerful, but for admins looking only to open/block ports/IPs/IP ranges to/from their server, UFW is the easier, and honestly safer choice. IPTables configs can become very complex and can easily be mis-configured to a point of failure. UFW has sanity checks on the commands run against it, and will hint at why the command wasn’t accepted.
If you have, say, a service listening on TCP port 25565, and want everyone in the world to connect to it, but only to your IPv4 address, you would run
{ufw allow from any proto tcp to 12.345.67.89 port 25565}
This will allow any IP address capable of routing to the server’s IP of 12.345.67.89 to connect to TCP port 25565. Likewise, to allow any IP to connect to v4 or v6 addresses, from anywhere, the command can be simplified to the level of SSH’s rule:
{ufw allow 25565/tcp}

UFW also provides firewall access to allow/deny/route in-bound and out-bound traffic on several protocols.

Setting up hardware firewalls:
You’ll need to find the docs for your hosting provider or your own hardware firewall in order to configure and use. Being Ionos is still growing, I feel there is a chance that anything I write here about their hardware firewall setup may become outmoded and useless as time goes on. Their documentation is pretty clear however.

SETUP YOUR AUTHENTICATION PROTECTION!!!
We’ll be using fail2ban as one of several layers to our unauthorized access mitigation solution.
Being that fail2ban has a lot of really good write-ups already, I’m going to have you read A2 Hosting’s instructions. I could copy and paste their instructions, or just the commands they use, but since I use their docs often, I might as well toss them some love!

I will make some notes, however:
"enabled = false" – This setting, on or near line 117 of the default config as of fail2ban 0.10.2, should NOT be changed as indicated by A2 Hosting’s page. Doing so will enable EVERY jail, causing fail2ban to fail to start… and ban. In the individual sections for each jail (such as "[sshd]") add the line "enabled = true" to enable that jail.

"ignoreip" – If you have either a jump-box or a static IP, then you would add that IP to this list, and uncomment it. Otherwise, relying on this to save you from failed logins can bit you in the behind if your IP does change. Especially since now someone else is now potentially white-listed on your server to attempt to brute-force it over time. A jump-box is another server or VPS which only, or primary use is to SSH into, and then connect to other servers. This can be achieved either by logging into the jump-box and then starting a new SSH session from there to the target server, or by means of automatic redirection (i.e. a reverse proxy) If you do not specifically pay for a static IP, or specifically told you have one, usually with business accounts – then you more than likely do not have a static IP, even if your IP hasn’t changed in 6+ months. You can test this by removing all power from your cable/dsl modem for an hour, and comparing the IP address(es) from before it was powered off and after it’s powered back on. Disconnecting the ISP’s wire (telephone wire, coax cable or fibre cable) for at least an hour will usually also work. Exchanging the modem for a new one will too – if you have a Static IP, the new modem WILL have the new IP (unless your ISP sucks really bad)

"bantime" – The default and suggested is 10 minutes. If you’re not afraid of locking yourself out (Either because you’ve never failed log in more than N times, or are OK with accessing the remote console) OR you’re OK with waiting that length of time before logging in again, you can set this MUCH higher. Otherwise, leaving it at 10 minutes is probably OK. I set this much higher.

"findtime" – This is also default to 10 minutes. If this were set to 3 days, all accumulated login failures over 3 days will count towards N tries. If you fail login once a day on average, you can easily become banned if this is set too high. Usually script-kiddies will give up on a host if they’re banned quickly. Between this and the next setting, determines the solution for N tries per X time. Or, at default: 5 tries per 10 minutes, which results in a 10 minute ban. This is plenty for those script kiddies, but a dedicated hacker will just take a snack break and try again. I prefer 3 tries per 5 minutes, with a much longer ban. But I’m comfortable using the remote console.

"maxtry" – Again, 5 is the default, and I set mine to 3. This is simply the number of failed authentication attempts before the IP is banned.

Additionally, if you’re using UFW and want it to handle IP bans, change these keys to use ufw:
(ensure you have /etc/fail2ban/action.d/ufw.conf before relying on this!)
banaction = ufw
banaction_allports = ufw

As pointed out in the A2 Hosting write-up, there is a large selection of services fail2ban can monitor. Most of these settings are probably best left alone, unless you have a specific reason for changing them. Don’t forget to change enabled to = true!

Restart fail2ban service, and enjoy!
{service fail2ban restart} (and view status with {service fail2ban status} – ensure there’s no issues!)

Linux PAM has several config files, all which are optimally set by default. However if you wish to take a look, and make changes at risk of bricking your server, they’re in {/etc/pam.d/} These can be used to fine tune failure attempts. Be careful though, as you can easily negate fail2ban’s timings.

HARDEN YOUR SERVER MORE!!!
At this time, the server is minimally secured and ready to use. But it can still be brute-forced over time – just a much longer time. I highly suggest changing from password SSH authentication to using SSH key pairs. Having a password locked private key is also valuable to this, and should not be overlooked for convenience.
If however, you REALLY wish to continue to use passwords, will never be using SSH, or for what ever reason are not able to use SSH keys, you can bypass this section – but I strongly urge you to reconsider.

Passwords are great for keeping the kids off your desktop, out of your game, and away from specific files. But they can be cracked. And with newer CPUs, times are getting much shorter for cracking software. This applies to file locks, as well as account passwords. SSH Keys too can eventually be cracked, as can the password on an SSH private key – but we’re adding layers of security, which helps greatly to mitigate intrusions! Hardware firewall -> UFW -> fail2ban -> PAM -> account name -> SSH keys -> pk password. Lots of levels to get through.

Some additional methods to help mitigate intrusions include, but aren’t limited to disabling root login over SSH, restricting accounts from services and sudo, changing the SSH port from 22 to something else, requiring SSH access from (a) specific IP address(es) and you have yourself a pretty secure server. Denying all outbound connections, with exception to needed IP/ports can mitigate certain attacks as well as malicious software from "phoning home" Disabling and/or uninstalling any non-used services and software will limit attack vectors for exploits as well. There are also other software which can be installed, such as anti-malware software, spam filtering, and additional levels of authentication enforcement. It can get pretty crazy! Some systems need the protection.

Reminder: We’re still using the root account. At this point, it may be beneficial to create a new user account, with sudo permissions to use as your administrative account (using a not common word for the account name), and a user account without sudo power for running anything that will never need root privileges to operate. Most software does not require sudo/root privileges to run. Remote console can be used for true root access, if ever needed – such as if your administrative account becomes locked or corrupted. As the only software I am running requires root access, I will be creating a new user account with sudo power only, and locking root from ssh login.

Creating your first non-root administrative user account:
Let’s say your administrative account will be named stormbringer. (Let’s not use this name for accounts, ok?)
{adduser stormbringer}
This will prompt for additional information, starting with the new account’s password.
Next, the account must be added to the sudo group, which (should) give it sudo access:
{usermod -aG sudo stormbringer}
To test that the new user account functions, and that it has access to sudo:
{su stormbringer}
The bash prompt should now have replaced "root@" with "stormbringer@" – provided it does, do:
{sudo apt update}
This will prompt for stormbringer’s password. Enter it. This should update your apt cache. If it does, success! If not, go back up a few lines and try again.
To exit out of the stormbringer account back to root, simply do:
{exit}
The bash prompt will now read "root@"

SSH Keys are the key:
NOTE: Be sure to use your administrative user account (NOT root) when performing the below, unless you specifically need to allow the root user to have ssh key login authority.

We’re going to add SSH keys, and disable password login. For this, we need to generate a key-pair on our VPS. This does a couple of things – one, it lets you have a "master key" which can be put on your other servers for convenience, and be easily negated by generating a new key pair replacement in the event of a security breach or loss of key control. It also populates the file system with needed directories. We’re also going to use a separate key pair which belong to the admin. This allows changing the admin’s keys without affecting the other servers. It is good practice to replace key pairs which have been distributed often. Having a separate key for the admin account also allows the admin to retain access to all servers when the server-specific keys are replaced.

SSH keys will not prevent the need to use a password for privilege escalation once logged in with an account with sudo power. Keys can, however be used to disallow password authentication on SSH login.

Since we’re going to be denying SSH login to the root user account, go ahead and log in to the server with your administrative account. It’s best to use a new SSH session for this account, leaving the root session open for the time being. This is so that if there are any issues with connecting via the administrative account, the root account can quickly be accessed to assess the issue, and fix it. All instructions from now on will be done using the administrative account, and NOT the root user account.

To begin, we’ll generate a server specific key pair. Because this key pair will only ever be used for server-server communication, and getting to these keys is difficult, it can be seen as "mildly safe" to generate this pair without a password. In some instances, this can be more safe, as scripts written to rsync data across an SSH connection must store the private key’s password in plain text (unless you want to get really into it, and will encrypt the password, which is beyond most people)
{ssh-keygen}
Let the keys be generated to the default provided path. This will make life easier for you. However, security by obscurity is still a thing, and changing this could be seen as obscurity. Unless you have reason to password protect this key pair, simply leave the password request empty.
When the generation is complete, you will be given a nifty ascii art, followed by the bash prompt. Success!

To create an admin specific key pair, the same instructions can be followed above, either on the same machine (which will overwrite the current key pair), on another Linux host, or with some other tool, such as PuTTY’s key pair generator.

Success! You’ve got an admin specific key! (I’m going to assume you figured out how to do this, because I literally already told you)

To grant access to the administrative account via the admin key, the admin key pair PUBLIC key needs to be added to the server/account.
Create, if it does not exist: /home/stormbringer/.ssh/authorized_keys (using vi, nano, etc)
or with {cp /home/stormbringer/.ssh/id_rsa.pub /home/stormbringer/.ssh/authorized_keys}
Add your public keys to authorized_keys file, including the server and all admin public keys.
Add a comment (using "#") to identify the public key’s owner. This will allow the admin to quickly select and remove expired/compromised/orphaned/ keys and those of ex-admins/users. Do NOT delete or alter the id_rsa.pub file, or you will lose half of your server-specific key pair.
Add new keys, one per line, and only taking up one line. If word-wrap is enabled, the strings will appear on multiple lines, ensure they are in fact on a single line.
Repeat this for each admin public key which needs to be added for access to the account.

In your SSH client, you will need to associate your private key with the server/user profile. This is done differently depending on OS and client. On Windows, PuTTY’s Pageant program will run in the background, and require a password to unlock the private key, but will provide the key to many Windows based SSH clients, including PuTTY and WinSSHTerm v2.

Create a new, (if counting, third) SSH session. This time, using the administrative account (stormbringer, in my example here) – and if everything went right, the connection should quickly complete without requiring a password to be entered. (No cheating here, don’t add your password to an auto-fill script!) Once the administrative account is logged in using SSH keys, SUCCESS! Now we can move to disabling password authentication.

(At this point, you should be able to log in with your administrative account, using only SSH keys, and be able to use sudo to run programs, which will still require the administrative account’s password. If you cannot do these three things in this manner, you should review the instructions, or seek real-time/live assistance)

HARDEN THE SSH DAEMON!!!
Now that you can log in using SSH keys, let’s get rid of that gaping security hole known as "password authentication"! While doing this, there are some additional changes to the SSHD config that can be made. I’ll go over some of them here:
Edit this file with {sudo nano /etc/ssh/sshd_config} (replace nano with vi if you’re hardcore, or old school)
Being that the administrative account is being used, sudo is now required to alter system level config files.

Note: Towards the top of this file are some config options which can be changed, such as the port SSH listens on, and IPv4/6 addresses. Be sure to make appropriate changes in the hardware firewall and UFW/IPTables if these are changed!

These options are commented out, but are enable by default, allowing overrides with changes here. Changes require uncommenting the lines. I also uncomment lines which still apply default values and won’t be changed, but which I use, just as a visual aide when editing the file at a later time.
LoginGraceTime – default is 2 minutes before the session will timeout for no input. This can be changed to 30s when using SSH keys, unless very latent network connections are expected to be used. Leaving this at default is also fine.

  • PermitRootLogin – default is "prohibit-password" or "yes" – change this to "no" to completely disable root SSH login. Leaving this as "prohibit-password" will allow the use of SSH keys to login to the server via the root user. I will be setting this to "no"
    MaxAuthTries – default is 6, I prefer 4. fail2ban should kick in at 3, but just in case.
    MaxSessions – deault is 10. That’s a lot of sessions for a server with 99% no SSH usage at all. File servers accepting rsync over SSH may require more, however.
    * PubkeyAuthentication – default is yes, and commented out. This can be left alone, or uncommented for visual aide, or paranoia reasons.
    * PasswordAuthentication – default is yes. We’re changing this to "no" to prevent password attempts.
    * PermitEmptyPasswords – default is no. I uncomment anyways, even though the setting is nullified by the above setting.
    * ChallengeResponseAuthentication – default is yes. This can still allow brute-force password attacks. We’ll uncomment and set to "no"
    * UsePAM – default is yes. We’ll keep this uncommented and set to "yes" – This allows for less complex client setups.
    X11Forwarding – default is yes. This is a server, what’s a gui? Set this to "no"
    PrintMotd – default is no (at least on my ionos Ubuntu 18.04 image) – This can be changed to provide various info/data on login.
    Banner – default is commented out and set to "none" – I want a nice banner I can grin at on login. I’m setting to "/home/stormbringer/ssh-banner"

    • The ssh-banner file won’t exist, it needs to be created. This is where some nice ascii art, or a big "NO TRESPASSING" sign can be store for display.

The settings marked with * are ones we’re concerned with, regarding security and hardening the server. The rest are fluff and ancillary.
Now here’s something tricky. My config, at the very bottom, has "PermitRootLogin yes" and "PasswordAuthentication yes" – both uncommented. This would negate our previous settings. Ensure your file does not have duplicate entries. Review the file after restart as well, just in case something is messing with things.

Save your file. If you used sudo to run the editor, you should have no problems saving. If you cannot save, you didn’t sudo. Copy the contents, or save to an alternative location in the account’s directory. Then close the editor (if you failed to sudo, either sudo cp the saved file to the proper location, or re-open the proper file WITH sudo)

(If you want a banner, create the file you specified, with at least a word, so it will exist and not potentially cause issues with the config, or login)

Now, we need to test our config the best way we can – by putting it into use with the SSH service.
{sudo service ssh restart} and enter the account’s password. This may cause your connection to reset.
Create a new SSH session (counting still? Number 4) to ensure login is still possible. If not, you’ll need to fix your config. If your connection was reset, you’ll need to fix your config /using remote console/ – which can be a pain. If you were able to log in (and see the text from ssh-banner file if you created one) Success!

Now, go ahead and start your 5th SSH session, this time, using the root user. You may receive the text from ssh-banner file, but then be disconnected with a "no supported authentication methods available" message. If you, like I, do not want root to be able to log in with SSH – SUCCESS! Go ahead and close all but the original root user session and one of the administrative sessions.
At this time, it may be prudent to test {sudo apt update} and {su -} (from the administrative account)
Sudo will require the administrative account’s password. "su -" on the other hand should require the root user account password to access. Sudo should be enough for most things, however in rare cases, the actual root user account may need to be utilized to gain access to portions of the system or services.

Congratulations! You’ve made it to the end. Your reward? A wonderfully secure server. At least to what I consider to be a basic level of security!

BUT WAIT! THERE’S MORE!!!
Did we forget about the hardware firewall? Nope! (ok, well, maybe a little.)
By now you should know if you’re running IPv4 and/or IPv6, and what address(es) will be utilized for what purposes. You should also know at least some of the ports your services and software will be listening on. The hardware firewall configuration should mirror (at least mostly) the rules for allowed ports in UFW. There may be instances where UFW may have more open ports than the hardware firewall. This would be due to allowing monitoring services from your hosting provider, connections to/from other servers on the LAN/private network, or maybe other reasons, such as future use. The hardware firewall should never have any ports opened which are not explicitly in use on the server. Open ports are open attack vectors for exploits, Denial-of-service attacks, and other nefarious things. UFW can block a lot, but it uses server resources to do so. A flood of connections (DDoS) not mitigated by the hardware firewall can potentially overwhelm the Linux server, causing a crash, exploit, or full intrusion.

Here’s a starter kit of useful commands you can perform to inspect your server:
{df -h} (disk filesystem, human readable) will display the amount of space allocated, used and free on your drive, and where each portion is mounted. Generally "/" will be the most used, and largest partition. It’s also the partition that can be used up by extraneous software installs and file storage in the user’s /home/ directory.
{du -h} (du -sh) (disk usage, human readable (summary)) Can specify a directory to see how large that directory or it’s contents are.
{free -h} (available memory resource, again human readable) This shows some quality stats about your RAM can cache.
{lscpu} (list CPU information) This shows information about the CPU as reported to Linux via the hypervisor from the hardware. Modern VPSs generally have accurate info.
{lspci} (list PCI information) Not too horribly useful for VPSs, but can provided critical data on dedicated servers.
{jobs}/{fg}/{bg} – If you’ve ever ghosted a program, where it’s still open but can’t be accessed, try these commands.
{htop} – nice system monitoring tool, with colors! Can also

Some of these commands give good info without sudo. Some will give more info when run as root or via sudo.

This is a baseline of a good start to a fresh Linux install. Obviously, there’s many many many more things that can be done with a Linux server, in terms of use, and security both. But this should provide more than a firm footing for any new Linux server.

I, the author, and BluntAboutIT.com take NO responsibility for any loss of data, access, sanity or finances resulting in the failure (or successful) following of this guide. It is a GUIDE, not a set of axioms. Every admin should fully know, understand and carefully choose the routes they take with their servers, as well as with any and all configurations, software, etc. This guide is here for two reasons only: To help the education process for those who need a bit of help getting started, and for myself, so I have a "check list" of sorts when provisioning new servers. What works for me may not work for you, either technically or functionally. You’ve been warned. I, the author, and BluntAboutIT.com take NO responsibility for any loss of data, access, sanity or finances resulting in the failure (or successful) following of this guide. It is a GUIDE, not a set of axioms. Every admin should fully know, understand and carefully choose the routes they take with their servers, as well as with any and all configurations, software, etc. This guide is here for two reasons only: To help the education process for those who need a bit of help getting started, and for myself, so I have a "check list" of sorts when provisioning new servers. What works for me may not work for you, either technically or functionally. You’ve been warned.

Linux SSH login – a good starting point

The steps below were included in a later article I wrote, regarding new Linux server installations, here which includes much more information from that aspect. The information below is still valid, useful, educational information which should be read if intending to start the process of hardening a Linux server. I apologize for the sloppiness of this, but I see no reason to copy and paste the same information into this article when it flows very naturally in the new article. You will thank yourself for reading both articles, however!

My environment:
Ubuntu Server 18.04 hosted in a datacenter, with a public IP used for administration and public use.
Windows client computer with SSH terminal program. (I highly suggest WinSSHTerm v2 for higher level usage)

The goal:
To run a server with SSH key login only.
To use password authentication for privilege escalation only.
To prevent unauthorized access, login and escalation, through various methods.

The server software:
OpenSSH (sshd – ssh daemon service)
fail2ban (intrusion prevention service)
Linux PAM (Pluggable Authentication Module)
UFW (Uncomplicated FireWall, in lieu of IPTables)

Additional software used for demonstration puroses:
MariaDB (A fork of MySQL, with some enhancements)
HAProxy (A layer 4 and 7 routing service (HTTP(S) and TCP-only proxy)
Hiawatha (HTTP daemon akin to Apache, with simpler configs, and a security focus)

The method:
Using the above software services, the GNU/Linux installation will be secure from intrusion from unauthorized and unauthenticated users (and user-like software). This process will include allowing TCP port 22 incoming access, and denying incoming access to all other ports (opening 22, closing everything else), until such time as additional ports are needed for access into the machine. All outbound connections will be allowed in this tutorial. Once UFW is enabled, external clients may connect only to port 22/TCP. Being OpenSSH will run on port 22, SSH is the only thing that can connect to the server. The next step will be to allow SSH key logins, ensure this is working correctly, and then disable password authentication on SSH. After getting the server to a point where only SSH with a key pair can connect to the server, fail2ban and PAM will be utilized to help mitigate brute-force attacks for login and privilege escalation (i.e. sudo and su usage).

The end result:
We’ll be using 5 pieces of software (Linux, SSH, fail2ban, PAM and UFW) as a starting point to secure a Linux server installation. These instructions are based around Ubuntu 18.04 LTS, but may be applicable to other distros and version. 18.04 is a SystemD based system, and some differences will occur for older, non-SysD installations. Before a Linux server can be of any use, it must be accessible. This can be through a local console (keyboard & monitor), through a remote console (such as many hosting companies provide for direct access, or which can be set up using a serial cable and terminal (often using a laptop connected to the server) – or by remote terminal access, via SSH (at this time, I do not know any other ways to access the main console or terminals of a Linux host) For this tutorial, we will assume SSH access will be used, even if console access is also used.

Users (potentially just the server admin, you, the reader perhaps) will gain access to the server with an SSH compatible client. This client will connect to TCP port 22 (possibly changed, will go over later in this article) The server will go through various methods of authenticating the connect, the supplied account, and credentials (SSH key). Upon successful connection and authentication, the user will have access to the server. If the user is granted sudoers privileges, the user can then use ‘su’ and ‘sudo’ to gain escalated (root) privileges. If, however, authentication fails, the user’s connection will be terminated. Multiple failures will invoke the user being banned, and from connecting at all. The user information will be added to a fail2ban jail, with configurable ban time (or permanent – but this is dangerous as if the admin somehow fails to login properly multiple times, the admin will have to gain direct console access to resolve the issue)

There will be a minimalized guide at the bottom with the basic information needed as a refresher for admins who understand the software and steps needed, but lack the confidence in blindly following memory to achieve this basic setup.

A synopsis of the steps needed:
Step 1 – Install and setup the Linux server, and accounts. This is the only time the root user account will be used to gain access to the system.

Step 2 – Install and set up UFW. This will include choosing to use the default SSH port, or modifying it. Changing the SSH port is “security by obscurity” – which can help mitigate SSH probing attempts, causing a lack of interest in your server to any hackers. Using the default port is still highly suggested, and though probes may find the port to be open, it will be very difficult for hackers to gain access. (Warning: These methods do not encompass software exploits which may exist in the SSH daemon, Linux, or any other software being used – This covers conventional, brute-force and guessing-game type attacks only)

Step 3 – Secure the SSHd to use SSH keys, and to then disable password login. Passwords will still be used by the system for authenticated users to gain su/sudo access.

Step 4 – Use PAM to mitigate authenticated user brute-force and guessing-game password escalation attempts, and to assist with SSH key login.

Step 5 – Ensure fail2ban is set up to ban malicious connections, and mitigate attacks on connection and escalation.

Step 6 – To use some commonly used user-accessible software services to demonstrate how to allow access to these services, including non-authentication public access (HAproxy and Hiawatha) and secure, private access (MariaDB) These demonstrations will provide a basis for understanding how to grant access to nearly any hosted service in a secure manner.

These 6 steps will give a BASE LINE level of security, and should not be counted on for 100% of a system’s security. There are many additional methods which can be utilized to harden a server system. Some options are additional software, replacement software, hardware firewalls, VPNs (Software and hardware) The extensiveness of advanced security and hardening is beyond the scope of this guide, but should be understood and researched as needed.

BluntAboutIT.com and I, the author do NOT make any guarantee to any end, and CANNOT be held accountable for security failure for any system which is set up using this guide. Again, this is a base line guide, but the admin must ensure that ALL security needs are met, including ensuring that every security measure needed is in use and properly functioning and using the latest available software. With that said, this guide can provide a good starting point for admins to secure their servers.

(At this time, I am publishing this page, with it incomplete and lacking any actual instructions. The information here is enough for a smart person to do some research and be able to begin the process, if not complete it. This tutorial will be updated at a later time to include instructional steps for installing, setting up and utilizing the software mentioned thus far. There may also be additions to what is currently published)