Simon Mott

Moving from Wordpress to Hugo

Tue, 15 Sep 2026 17:35:12 +0000

Everyone’s probably sick of hearing about LLMs now, but I’ve been working with them a bit more recently and I decided it was probably time I used one to update my website.

Way back when, my website was originally some PHP I’d cobbled together. At the time it was… fine, I suppose. It had a basic admin section which allowed me to write posts, create custom pages etc. Referencing my recent OwnTracks post, this was the version of my site that had my current location for all to see - which, in hindsight, probably isn’t the smartest thing to publish on a public website… I was, and still am, terrible at creating good looking designs for things, so it was pretty basic, but it served my needs.

I had this version of my website from when I originally created it in 2009, through to sometime in late 2016.

Wordpress

After working in the Managed Services and Hosting industry for a couple of years, I’d decided I wanted to update my website with a shiny new look. Wordpress was the flavour of the month back then, so I figured I’d have a play with it and set that up. I managed to find a theme I mostly liked (I forget where I found it), but I had to apply some basic modifications to it in order to make it fit better on the page. Believe it or not, it was too narrow. I didn’t realize this at the time, but that decision would come back to bite me.

I used to see Wordpress websites getting compromised all the time at work, so I made the decision pretty early on to try and “harden” my install a little bit. This was done by keeping my installed plugins pretty light and anything under the wp-admin folder, including the main Wordpress login form, was restricted using Apache’s mod_auth_basic module, meaning anyone attempting to access it would be presented with a login prompt, which, importantly, was not handled by PHP. My thought process was that most sites seemed to get compromised through the admin plane, and if something tried automated methods, the auth requirement would foil it. Given this was handled entirely by Apache (and was a different set of credentials to the actual Wordpress authentication), it was less likely for anything to exploit any CVEs there, as they simply couldn’t access it. It was a little annoying with two login prompts whenever I wanted to access it myself, but as this was in Apache, I could whitelist my own IP addresses that were likely to access the admin portal. This seems to have worked pretty well for me overall - to my surprise, my website (as far as I know) never got compromised.

I am well-known amongst my friend groups to be pretty lax at applying timely updates. As I mentioned earlier, I’d modified the theme I used, so from that point onwards, I never updated the theme after the initial setup, as doing so would have undone my customizations. I didn’t really understand how theming worked in Wordpress, so I didn’t do my customizations in a way that meant I could easily update the theme. That pain point, plus being lazy with updates, meant I only really did it sporadically, which isn’t an ideal situation to be in.

In a similar manner, the VM the website ran on slowly aged, to the point where I found the thought of updating it annoying, and I simply couldn’t be bothered to migrate everything elsewhere wholesale.

Wordpress itself, or at least the combination of plugins and my theme, wasn’t exactly the most performant. If a crawler decided to request a lot of pages in quick succession, PHP would just churn CPU trying to generate the pages, consuming memory, and quite a few times it fell over due to running out of RAM and consuming all the swap. At this point I decided to play with putting some caching in front of the site. Sure I could have used nginx and been done with it, but I wanted to play around with Apache caching, which is the entire reason this blog post exists.

Overall, Wordpress has served me well for 10 years, give or take a month or two. Looking at archive.org, June 24th 2016 is the last entry for the old design, and August 31st 2016 is the first entry with Wordpress.

Hugo to the rescue

In general, I am a fan of dark mode for all the things. One thing that’s bugged me about my Wordpress site theme of late is how bright it is, with no easy way to change it. This, in combination with the self-imposed mountain-out-of-a-mole-hills of keeping Wordpress and the server updated, meant it was probably time to switch things up.

I’d been aware of Hugo for a while, but I never seriously looked at moving to it, as pulling all the bits I wanted out from Wordpress, reformatting the posts and generally reconfiguring things always seemed like a chore. Thankfully in this day and age we have LLMs that can do a lot of the heavy lifting for us.

I was chatting to Greboid on IRC recently, discussing websites in general, and he decided to start working on a Hugo conversion from Wordpress for me (I assume for fun?) and he came up with a design that is pretty much what this version of my website is based on - excluding a few tweaks I made after the fact.

I had a script that would crawl my website (originally to test the caching), which, as a nice side effect, produced a list of all the URLs reachable on my site. Armed with that, an export from Wordpress itself, and some vague hand-wavy comments and suggestions, an LLM agent dutifully created a Hugo file tree, converted all my posts, brought over Greg’s styling, handled old URL preservation and produced suitable Dockerfile and docker-compose files to build an image for my website.

A quick VM-with-docker template deployment later resulted in a shiny new server that houses my website (and a few other things) with pretty much minimal effort on my part.

A downside of using Hugo is that there are a lot more individual files I myself have to keep on top of instead of just letting Wordpress plugins/themes handle everything, but the main benefit to using Hugo over Wordpress is that it’s much simpler to chop and change things as I see fit. Of course, the main advantage here is that the website is a static site, so doesn’t rely on PHP or anything like that, and is blisteringly fast compared to the old site.

For nostalgia’s sake - here are some screenshots showing the 3 main styles from the original to now. I have come to realise my website design caters to my vanity as they all have a giant picture of me. The Hugo one is included here for if/when I change the design again in the future.

PHP - 2009 to 2016

Wordpress - 2016 to 2026

Hugo - 2026 to ??

OwnTracks and my Google Timeline

Sat, 22 Aug 2026 15:47:57 +0000

A while back, a friend showed me an application they had started using called OwnTracks, which periodically sends location updates from your phone to your own server. I used to collect similar location data back in the day via Google Latitude, so the idea of self-hosting my own tracking history intrigued me. Fast forward a few months, and I finally got around to setting it up.

Shortly after, another friend decided to set it up as well and mentioned wanting to import their historical Google Timeline exports into it. I hadn’t even considered that, but I loved the idea of giving it a shot since I had years of location data sitting around. A day or two later, I had my data fully ingested and viewable in OwnTracks, dating all the way back to mid-2010.

The first hurdle I ran into was that iOS Google Maps exports use a different format than the Android versions, requiring me to patch the import script to handle the iOS payload. Using an LLM coding assistant, I threw some sample data at it, and it quickly generated everything OwnTracks needed to ingest the files.

However, I quickly discovered that Google Maps on iOS logs far less granular data than it does on Android. Thankfully, I still had my old Android phone from when Google originally moved Timeline data on-device. That phone contained my complete, detailed history from my Android years up until my switch to iPhone, so I dutifully ingested that data too.

The next issue I ran into is that the built in map from the OwnTracks recorder, and even using the OwnTracks frontend was horribly laggy, even with just a years worth of data and I really wanted to see all my data on a map, all at once. Armed with my LLM, I thought It’d be pretty trivial to prompt something into existence that queries the OwnTracks recorder API and plot it all onto a map. I’d originally planned to create something I could use to entirely replace OwnTrack’s own frontend, so that’s where I started. It pretty quickly spiralled though, and it got way more complicated than I’d initially expected.

Most of this was striving to create something that could be suitable for people other than myself. After about a week or so of on-and-off prompting, I’d ended up with something that loads all my data, allows reasonably robust configuration of its appearance and probably too many tuning options. I am however pretty happy with the result even if its code is probably hot garbage.

It allows me to view all of my data, panning and scrolling (mostly) without lag, and lets me ultimately view everything all at once. What I find more interesting though, and mainly what I wanted to visualize are the heat maps - I’m not going to post my current or previous address in detail, but you can clearly see my home, my sister’s, the local shop I visited a lot, my mum’s and amusingly, the petrol station I used to frequent. This image shows at a high level, where I live, where I used to live, my old commute, and a giant hot spot near Birmingham for the many years I attended Insomnia Gaming Festival (rip) at the NEC.

Exploring the data further, clearly shows the kind of hot spots I wanted to see. When I visited Glasgow for work, there’s a hotspot for Glasgow Central, my old office and the hotel they put me up in when I visited, along with the venue of our annual Christmas party. I also took a look at my 2015 trip to Japan, and the Airbnb I stayed at is clearly visible, along with the two major transit hubs I used to get around.

Similarly for my trip to TwitchCon in Las Vegas back in 2023, but instead of a heat map, you can see route lines centered around my Airbnb, the strip, the airport and Fremont Street.

And finally - I attended an EPIC LUX LAN event recently too, and using OwnTrack’s data instead of Google Timeline, I was amused at the granularity. You can see the 4 main areas I spent my time in whilst at the event - The main event hall, my hotel room, the dining area, and in the lobby playing board games with friends!

One interesting observation after importing the Google data were strange little artifacts that are displayed on the map. Upon further investigation (mainly by Chris - chameth.com), the grid pattern visible turned out to be Google’s own spatial index showing through. Google indexes geography with a system called S2, which divides the Earth into a hierarchy of square-ish cells, each with a fixed, deterministic center point. Level 18 of that hierarchy produces cells about 30 metres across. When Google Timeline “fills in” your location history during stationary periods, it appears to snap those synthesized points to the centers of these cells - so the points don’t scatter like real GPS fixes, they land on a fixed ~30 m lattice that gets reused every time you’re in the same place. Plot them on a map and you see the index itself: repeating rows of dots, tilted slightly because the cell grid isn’t aligned with north.

A quick LLM task to parse my data and remove the matching S2 coordinates cleared it right up.

I look forward to see what data OwnTracks accumulates over the (hopefully many) years to come!

Using ffmpeg's vpp_qsv detail enhancement filter

Sun, 22 Dec 2024 05:04:27 +0000

As part of my last post, I was exploring video transcoding with an Intel Arc GPU and how different levels of ICQ affect video size and quality. During which, I noticed there’s a QSV compatible filter in ffmpeg called vpp_qsv which has a detail parameter. VPP stands for Video Post Processing and the filter help text simply says “enhancement level” for detail. The potential to do some automatic picture enhancements directly on the GPU piqued my interest so I decided to have a play with it.

Firstly, I grabbed a sample 4k video of a chameleon from pexels because its reasonably close framing of its body and has a fair amount of detail on its scales. Next, I used the same ffmpeg command from my previous post, but with a global_quality (ICQ) value of 30. The idea here is to deliberately degrade the footage so I can see what the detail parameter actually does in terms of trying to enhance the quality. I then generated a series of new videos with a detail level ranging from 0 to 100 in increments of 10 and then calculated VMAF scores for all of them. VMAF is an objective full-reference video quality metric that predicts subjective video quality based on a reference and “distorted” video sequence. The metric can be used to evaluate the quality of different video codecs, encoders, encoding settings etc.

This shows that the detail parameter is doing something but we start to get diminishing returns from a value of about 70 onwards, for much larger increases in file size. I was curious what this had actually done to the resulting video though. Below is a zoomed crop on the eye of the chameleon - Its pretty hard to spot, but the image from ICQ 30 detail 0 is slightly softer compared to the original (which is expected) and the image from ICQ 30 detail 70 is visibly sharper. Whether this means a better experience is up to you, keeping in mind that this is a single frame from a video.

Original

Q30 Detail 0

Q30 Detail 70

Given I’m unlikely to use an ICQ of 30 myself, I was curious how the detail parameter affects the VMAF score for different ICQ levels. Based on my last article I decided that ICQ values of 15, 16 or 17 would be the likely values I’d use, so I figured lets see how these values look for this sample video.

The source video itself is a small 4k video, coming in at only 28MB so I’ve included the figures for the original video in these new graphs as the file size becomes a lot more important.

We can see a similar trend as with the VMAF scores for ICQ 30, but with a larger fall off past a detail value of 70. Given these files should in theory be higher quality to start due to their lower ICQ value, we can see the impact of a higher detail value isn’t as large past 40 or 50. More importantly here, the generated file size seems to be larger than the original from about detail 70 onwards, which means we get no benefit at all other than perhaps a slightly sharper image and wasting extra disk space.

As interesting as these results are, this 28MB sample file is not representative of my use case, so lets re-run the numbers using the same 1080p input file for my previous post with an ICQ of 16, as this is most likely what I’ll be using going forward.

This again shows a similar trend, but with less of an impact at the higher detail levels. The file size however, ends up close to or larger than the original beginning at detail 70 and above which isn’t what I’m personally looking for as I’m trying to find a good balance of quality and file size.

It’s still unclear to me what exactly this parameter does besides adding some sharpening. Based on these results however, if I want to use this in my transcodes it seems like a value somewhere between 30 and 40 should (According to the VMAF scores) enhance my video over a plain transcode, whilst still ultimately reducing the file size. Once again though, remember that quality is subjective and your milage may vary depending on your hardware, source video and settings.

Transcoding with an Intel Arc GPU

Fri, 20 Dec 2024 18:30:51 +0000

I am a bit of an NVIDIA fanboy when it comes to GPUs. This is partly down to being a long time Linux Desktop user (until recently) and them having excellent drivers allowing me to pretty effortlessly get proper hardware acceleration working.

In my lab environment I have an old Nvidia Quadro P400 that’s set up for various transcoding workloads and its been working absolutely great for me despite being an older card. I have been primarily using the h264 codec but I recently moved over to h265 (hevc) mostly out of curiosity, and to see if I could get quality on-par with h264, whilst getting similar file sizes.

Seems simple? Take my existing ffmpeg command (using qmax of 23), and substituting the output codec from h264 to hevc whilst keeping everything else the same, This gets me a file encoded with hevc thats similar (ever so slightly larger) than what I’d have with h264 and comparable quality. I can pixel peep a few frames to see the differences but I could also calculate a VMAF score for these two outputs (between 0 and 100) to give a subjective quality score compared to the original source file. Doing so, the hevc encoded output scores slightly higher for an ever so slightly larger file. Anecdotal posts on the internet suggest a score between 94 and 96 is acceptable, but higher is better. Ultimately its up to you what you consider acceptable quality.

This is all well and good, but where does Intel Arc come into this?

I managed to find an Intel Arc A310 for pretty cheap and figured I’d grab one to throw into my lab. On paper this is a much newer card so should have decent performance for encoding - and it also natively supports AV1 acceleration if I ever wanted to experiment with that in the future. My goal is to find a good balance of quality, file size, speed and compatability.

The codecs used for NVIDIA and Intel Quick Sync (QSV) don’t accept all the same inputs, so using the new GPU is not as simple as just replacing hevc_nvenc codec with hevc_qsv. This shouldn’t be much a problem though as they are different pieces of hardware so will naturally do things differently.

ICQ (Intelligent Constant Quality) parameters for the Arc GPU are controlled using -global_quality instead of -qmax and I picked a starting number of 15 as it gave me a file size slightly larger than NVENC’s h264 whilst 16 gives a smaller file using my test input.

In order to validate our subjective quality, the VMAF score should decrease as the ICQ number increases so I tested all the way from 15 through 19. With ICQ, lower numbers are better quality and larger files, whilst larger numbers are worse quality, but smaller files. Since using qmax 23 for NVENC was larger than h264, I also included qmax 24 in my results.

Looking at these, using an ICQ value of 15, 16 or 17 gives us supposedly a superior quality file, but only 16 and 17 result in a smaller file than we get with NVENC qmax 23. All of these resulting tests are still significantly smaller than the original untranscoded file, and only two are larger than the h264 transcoded file using the P400, so in order to get the best balance of quality and size it seems like a value of 16 is optimal, at least for my hardware and use case.

Quality is subjective and all results will depend on the source and what you find personally acceptable, but the encoding improvements from using newer hardware really shows here - I can get a better quality hevc file, with a smaller footprint and do it faster (The A310 is almost twice as fast as the P400) so using this in my lab moving forward seems like a no-brainer.

Results will of course vary depending on what input you use. I only tested this with a single file for writing this post, but I have seen similar results for a variety of other inputs.

For completeness, these are the ffmpeg commands I used (ffmpeg version 7.1). I am by no means an expert with ffmpeg, so there are probably plenty of ways to further tune this but for now I’m happy with my results.

# NVENC
# Adjust qmax to control ICQ and file size
 ffmpeg -v quiet -loglevel error -stats -y \
 -hwaccel cuda \
 -hwaccel_output_format cuda \
 -i "input.mkv" \
 -fps_mode passthrough \
 -c:s copy \
 -c:a copy \
 -c:v hevc_nvenc \
 -map 0 \
 -dn \
 -preset medium \
 -profile:v main \
 -spatial_aq:v 1 \
 -rc-lookahead 32 \
 -rc vbr \
 -qmin:v 16 -qmax:v 23 \
 -vtag hvc1 \
 -max_muxing_queue_size 4096 \
 -f matroska \
 "output.mkv"

# Intel QuickSync
# Adjust global_quality to control ICQ and file size
 ffmpeg -v quiet -loglevel error -stats -y \
 -hwaccel qsv \
 -hwaccel_output_format qsv \
 -i "input.mkv" \
 -fps_mode passthrough \
 -c:s copy \
 -c:a copy \
 -c:v hevc_qsv \
 -map 0 \
 -dn \
 -preset medium \
 -profile:v main \
 -look_ahead 1 \
 -look_ahead_depth 99 \
 -global_quality 16 \
 -extbrc 1 \
 -vtag hvc1 \
 -max_muxing_queue_size 4096 \
 -f matroska \
 "output.mkv"

# Calculating VMAF (requires Netflix VMAF library)
# Uses CPU, Adjust n_threads for your CPU
 ffmpeg -y \
 -i "generated_ouput.mkv" \
 -i "input.mkv" \
 -lavfi libvmaf=n_subsample=5:n_threads=16 -f null -

Auto mount LUKS without a filesystem

Sat, 24 Jun 2023 14:52:58 +0000

One of my friends recently installed a new storage server in our shared lab environment and graciously gave me some storage space on it via iSCSI. I use Proxmox for my personal lab, and I intended to store some non-critical VM disks on this new storage so I could play around with properly using HA (High Availability) with Proxmox. Additionally, I wanted to gain some experience using iSCSI at the same time. While I trust my friend, the storage itself is outside of my control, so I figured it would be good practice to encrypt my data. The general go-to solution for this would seem to be LUKS.

Going into detail on how to set up iSCSI and multipath is out of scope here, but the general idea is that you connect to the storage array via iSCSI, and it then gets exposed to your server as if it were a locally attached block device. From here, you can pretty much do whatever you like. In my case, I want to use it with LVM to store VM disks in Proxmox, so the general layout would be something similar to the image below:

After setting up iSCSI and multipath, my system can see the storage LUN without any problems.

# multipath -ll
san (36589cfc000000c0d01729cc32c52ecc4) dm-9 TrueNAS,iSCSI Disk
size=2.0T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=15 status=active
|- 3:0:0:0 sdb 8:16 active ready running
`- 4:0:0:0 sdc 8:32 active ready running

I plan to use LVM on LUKS, which essentially means the LUN in its entirety is encrypted and needs to be correctly decrypted before anything on it can be detected by LVM. I set up /etc/crypttab with the appropriate key files and proceeded to test if it opens correctly without any prompting. This seemed to work as expected (san_luks is just the name I gave it in /etc/crypttab):

# systemctl status systemd-cryptsetup@san_luks.service
● systemd-cryptsetup@san_luks.service - Cryptography Setup for san_luks
Loaded: loaded (/etc/crypttab; generated)
Active: active (exited) since Fri 2023-06-23 20:02:33 BST; 18h ago
Docs: man:crypttab(5)
man:systemd-cryptsetup-generator(8)
man:systemd-cryptsetup@.service(8)
Main PID: 1935 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 48147)
Memory: 0B
CGroup: /system.slice/system-systemd\x2dcryptsetup.slice/systemd-cryptsetup@san_luks.service

Jun 23 20:02:28 pve1 systemd[1]: Starting Cryptography Setup for san_luks...
Jun 23 20:02:28 pve1 systemd-cryptsetup[1935]: Set cipher aes, mode xts-plain64, key size 512 bits for device /dev/disk/by-uuid/4dfbd655-258b-42f8-b1a7-94155bdf1465.
Jun 23 20:02:33 pve1 systemd[1]: Finished Cryptography Setup for san_luks.

I proceeded to reboot, and… it didn’t open the volume. I confirmed that iSCSI and multipath were fine, but the volume just didn’t open on boot.

I tried enabling the cryptsetup service:

# systemctl enable systemd-cryptsetup@san_luks.service
Failed to enable unit: Unit /run/systemd/generator/systemd-cryptsetup@san_luks.service is transient or generated.

I tried all sorts of arguments in /etc/crypttab, but nothing seemed to work.

After a fair amount of digging, it became apparent that the typical use-case for this kind of setup is to open an encrypted volume and then mount a filesystem that resides on it. All the articles I could find were geared towards this and hinted that you need to have crypttab set to automatically open the encrypted volume (which I did), but you also need a corresponding entry in /etc/fstab that references it. This reference, when systemd tries to mount it, actually triggers cryptsetup, and everything just works nicely.

This is a problem because I am not mounting a filesystem.

Update: See addendum below

Ultimately, I settled on a dirty workaround to ensure my encrypted volume was opened correctly on system boot. I had already confirmed that it works without any prompts from me, so I created a new systemd unit file that should run after multipath is started and set a requirement for my unit to trigger systemd-cryptsetup@san_luks.service.

This should run after multipath is loaded, so I have access to the block device it creates, and then trigger cryptsetup. The unit file runs once at boot and basically does nothing. I’m sure there’s a nicer way to do this, but for me, this worked.

# cat /etc/systemd/system/luks_auto_open.service
[Unit]
Description=LUKS Auto open
After=multipathd.service
Requires=systemd-cryptsetup@san_luks.service

[Service]
Type=oneshot
ExecStart=/usr/bin/echo
User=root

[Install]
WantedBy=multi-user.target

I don’t like this workaround at all, but it gets the job done and allows me to continue setting up the new storage for use in Proxmox.

I suspect there’s a more elegant way to do this that doesn’t involve a makeshift unit file, but my Google-fu has failed me.

Addendum

As it turns out, there is indeed a better way to do this. My friend Dataforce also ran into this problem and refused to accept there wasn’t a better way to do this.

“I’m convinced there must be a better more correct way”. 😄

He was so perturbed at my solution that he actually spent some more time than I did doing research and found the answer:

systemctl enable remote-cryptsetup.target

Initial testing seems to actually do what we want, rather than having to use my artisanal unit file. Check out his page for more details

A Stratum 1 Time Server - The Basics

Sat, 12 Feb 2022 16:31:06 +0000

A while ago, I decided to play around with trying to create something cheap that would be able to synchronise its time by using GPS instead of over the internet - essentially building my own Stratum 1 time server. I intend this to be a series of articles about NTP and time servers, but let’s start with the basics…

What is a Stratum 1 Time Server?

The lower the “Stratum” number the more accurate the time it should be able to tell. Usually Stratum 1 time servers are directly connected to an accurate source of time such as (but not limited to) Atomic, GPS or even Radio clocks. The stratum of the machine which itself is directly interfacing with the accurate time source would be “Stratum 0”. For each additional layer of NTP servers added, the stratum increases by 1.

So if the server that’s connected directly to the time source is stratum 0, it would appear to any clients who want to synchronise with it as a stratum 1 source. If someone then tries to synchronise to one of those, they would see a stratum 2 source and so on. The image on the right illustrates the top-down increase in stratum and was taken from Wikipedia.

Generally the higher the stratum number, the less-accurate its view of time is. A stratum ranges from stratum 1 all the way to stratum 15 indicating the time is getting further away from a reference clock the higher you go. Stratum 16 indicates the clock is pretty far from the reference and should be considered un-synchronised and should not be used.

Why do we need time servers?

Every computer out there - be it a desktop, laptop, server or otherwise usually need to keep track of the time one way or the other. Computers usually have dedicated hardware to keep track of the time (notable exceptions are Raspberry Pis), some of which are more accurate than others but ultimately each one is different.

To demonstrate how computers can’t keep track of accurate time on their own, I rigged up 2 machines and synchronised their clocks to the same time server. Once initially synchronised I simply left them to run for 8 hours to see how much their view of time has drifted away from what the time server suggests is correct. This test was completed without ntpd creating a drift file (more on that later)

One thing note here is that the Pis clock is running faster than the reference clock, whereas the laptop is actually running slower.

This clearly shows that over just 8 hours, there’s a significant discrepancy between the two machine’s local clocks vs what the time actually is. Obviously the longer they remain un-synchronised, the larger the drift becomes.

How do I deal with time drift?

On Linux, running ntpd has the advantage of having some sophisticated features and algorithms behind it designed to keep your local clock as accurate as possible.

Once ntpd has been running for a while (and has been able to successfully synchronise time with an external source) it’s able to estimate the inherent drift of your local clock and compensate for it accordingly.

The drift itself is called “frequency error”. Calculating it is a process that continuously happens in the background, which ultimately results in the generation of a “drift file”. This file is used to compensate for drift when ntpd starts up next so it doesn’t have to go through the initial learning process again.

Once it had determined the frequency error for my two test machines, I ran the same test we did earlier to see how well ntpd coped with clock drift.

As you can see from the above graph (kept at the same scale), we’ve easily managed to keep drift on both machines to within 0.1s of our reference clock. You’ll notice however that both clocks are now running faster than our reference…

Environmental conditions affect your computers ability to track time, as does system load and even power fluctuations from your utility provider. This means that if you were to monitor the frequency error reported by ntpd, you’d see that it can fluctuate quite a bit; You’ll likely never be able to get a stable frequency error so will likely never get 100% perfect time-keeping without help, but its a damn good start and a large improvement over having no clock discipline at all!

So, where do I start?

Most default installs for ntpd come pre-configured with servers from the NTP Pool Project so as long as you’re running ntpd then you should be OK.

You can check the current status of ntpd by querying the local machine for its list of peers by using ntpq -pn

# ntpq -pn
 remote refid st t when poll reach delay offset jitter
==============================================================================
*85.199.214.99 .GPS. 1 u 32 64 377 8.406 -0.147 0.073
-138.68.151.69 194.117.9.136 3 u 38 1024 377 5.835 2.372 15.680
+129.250.35.250 249.224.99.213 2 u 1052 1024 377 6.587 0.666 0.139
+80.87.131.131 80.87.128.222 4 u 442 1024 377 22.322 -0.635 0.270

This output shows some useful information so lets work through it (Taken from http://nlug.ml1.co.uk/2012/01/ntpq-p-output/831 where more detailed descriptions of this output can be found)

remote - The remote peer or server being synced to
refid - Where or what the remote peer or server is itself synchronised to
st - The remote peer or server Stratum (from our point of view)
t - Type (u: unicast or manycast, b: broadcast or multicast, l: local reference clock, s: symmetric peer, A: manycast server, B: broadcast server, M: multicast server)
when - When last polled (seconds ago, “h” hours ago, or “d” days ago)
poll - Polling frequency
reach - An 8-bit left-shift shift register value recording polls (bit set = successful, bit reset = fail) displayed in octal
delay - Round trip communication delay to the remote peer or server (milliseconds)
offset - Mean offset (phase) in the times reported between this local host and the remote peer or server (milliseconds)
jitter - Mean deviation (jitter) in the time reported for that remote peer or server (miliseconds)

What does all that ACTUALLY mean?

Looking at what we have above - we’re using 85.199.214.99 as our primary time source (signified by the *). That server is getting its time direct from a GPS clock and so is a stratum 1 server. Its reported time differs from what we think the current time is by only 0.147ms (we’re running faster than it).

The stratum 2 and 4 servers are also pretty close with their offset so are potential candidates to synchronise time with if the current stratum 1 server goes offline. ntpd will automatically choose which server to synchronise from and will also weed out ones that are seemingly reporting bad time by comparing it with the other servers loaded. If you’re using a configuration which makes use of the NTP pool then the servers that appear will likely be different each time you start ntpd.

A big thanks goes to the volunteers at the NTP Pool Project for making it possible without running your own infrastructure!

This article should hopefully have given you a bit more insight as to why time servers are needed. In the future I plan to explore setting up a local NTP server to use with a view of getting our own GPS timing device hooked up and working.

Ubiquiti EdgeRouter IPsec performance

Wed, 29 Aug 2018 17:11:16 +0000

I’ve been working on setting up a lab environment for myself and decided to pick up a couple of Ubiquiti’s EdgeRouter 4 routers to act as my core routing devices, along with an EdgeRouter X to act as my out-of-band router. These new devices, along with an EdgeRouter Lite loaned to me for this article by Dom at LoveServers, puts me in a reasonably good position to do some performance testing between these different models of EdgeMAX routers. I thought a good place to start would be to compare how well these different models perform in terms of IPsec throughput and overall CPU usage at the same time.

Part of my lab setup will involve provisioning a couple of IPsec tunnels between the lab and my home network. So whilst this article will mainly focus on IPsec, I will be including some general observations/comparisons between the devices too which may be informative to some people.

Methodology

According to Ubiquiti’s data sheets, the EdgeRouter 4 should be the beefier device, so it seemed logical to use the pair I have to establish a base number which I can compare the other two models against. The physical layout I’ve used for the testing is pretty straightforward in that it’s two Dell R210ii servers directly attached to a router each, with another link between the two routers, as illustrated below.

The routers themselves are initially at the factory default settings other than rudimentary interface configs, some static routes, hardware offloading (more on that later) and the configuration components needed to establish an encrypted GRE tunnel using IPsec. Unfortunately, as I am limited by the overall performance for just one of the EdgeRouter 4’s, I wont really know if the throughput is capped by encryption or decryption performance; So for the purposes of this test, I will assume the value I establish will be the same each way for the EdgeRouter 4. I may explore this in the future once I have something that can push more data than these devices are capable of, which should allow me to measure them independently.

However, as these devices should, in theory, out-perform the other models, I should be able to discern encryption and decryption throughput separately for the EdgeRouter Lite and EdgeRouter X

I referenced this article from Ubiquiti’s support center which, at the time of writing, details both the encryption and hashing algorithms that are supported by the different offload engines contained within my test devices - specifically for firmware v1.10.5.

I thus settled on these settings for the bulk of my testing which are supported by both the Cavium (ER4 and ERL) and MediaTek (ERX) offload engines. Each device was rebooted when any change to its hardware offload settings were made, just to ensure it loaded correctly.


Phase 1 - Internet Key Exchange	Phase 2 - Encapsulating Security Payload
Key Exchange: IKEv2 Encryption Algo: AES256 Hash Algo: SHA1 Diffie-Hellman Group: 14 (2048 bit) Lifetime: 86400	Mode: Tunnel Encryption Algo: AES128 Hash Algo: SHA1 Perfect Forward Secrecy: Enabled Lifetime: 3600

Establishing a Baseline

For each of the tests, regardless of router configuration, I will be useing iperf to measure throughput between servers and will keep the settings at basically default - run via TCP, for 30s and display output every 1s (In CSV format so I can compile the results).

Server2# iperf -s
Server1# iperf -c 172.26.2.2 -x CSV -y c -t 30 -i 1

As for measuring CPU usage, I popped a script in /tmp on both routers being tested that would calculate CPU usage based on /proc/stat and output to console with a timestamp so I could correlate between tests.

Once I had the initial test plan sorted out, I made sure that all relevant hardware offloading was enabled and started by measuring if the devices can actually forward traffic at 1 Gbps. This was a simple test using iperf from Server 1 over to Server 2 via the routers, without any IPsec or GRE configuration in place. Just pure packet forwarding via static routes.

The routers are ordered from left to right; Most expensive to least expensive. EdgeRouter 4, EdgeRouter Lite and EdgRouter X respectively and thankfully, it seems that all routers are more than capable of forwarding packets at basically line rate (See image to the left).

Whilst I expected nothing less, I wanted to make sure that they were capable of actually forwarding packets at gigabit speeds - I’ve seen some routers that do a lot worse! What is more interesting about this first test seems to lie with the CPU usage of the devices whilst pushing packets.

Both the EdgeRouter 4 and EdgeRouter Lite use SoC’s from Cavium and use a “not insignificant” amount of CPU when just routing, whereas the EdgeRouter X uses a MediaTek based SoC which hardly makes a dent.

From observing what’s going on from the PoV of the device, most of the usage on the Cavium based devices appears to comes from soft interrupts so, I’d expect this is simply down to the different manufacturers hardware offloading methods which probably explains the surprisingly low CPU usage for the EdgeRouter X.

Whilst this is certainly an interesting point to see this early on, I would be curious as to how the CPU in the X performs with other tasks that cant really be offloaded as easily. I may test this in more detail based on a more traditional home/office type test with QoS and ACLs etc or even some dynamic routing enabled to try to tax the CPU a bit more.

For now however, that’s out of scope of this article.

Initial Testing

At the time of writing this, there wasn’t really much in the way of documented figures for IPsec performance on the EdgeRouter 4 that I could find, so I figured - I have the kit, lets test it!

The initial test was to measure throughput between two EdgeRouter 4s, and use that as a base to compare with the older models as they’re also quite popular. As mentioned previously; I am testing between just these two devices and I cant tell if the performance I see in my results is due to hitting a ceiling on encryption or decryption, so for the remainder of this article I will assume both figures are the same for the ER4.

Whilst the testing was done using the IPsec configuration outlined earlier, I did also test using AES256 for both Phase 1 and 2. As you might expect, throughput was a little less when using AES256 due to the increase in computation required. I decided to stick with AES128 as the primary choice for the remainder of the tests.

AES256 is a better choice, generally speaking for encryption, however the added compute required didn’t seem worth it to me in conjunction with the relatively short key lifetime’s, AES128 should be secure enough for my purposes.

I did also dabble with MD5 as the hashing algorithm, but in all of my tests it performed worse than SHA1 - I simply chalk this up to the offload engine being optimised for SHA1 vs MD5.

If you’re looking for guidance on what configuration to use for IPsec, a good read is the NCSC guidelines for IPsec along with this page which explains why you might choose one type over the other.

As you can see, the EdgeRouter 4 performs pretty well in this test. CPU usage for both tests were around the 50% mark.

Initial Observations

Something I did note during my tests but couldn’t quite explain is that sometimes, throughput between the EdgeRouter 4s would drop by about 100 Mbps along with a CPU usage drop from 50% utilization, down to 35-40%. I haven’t been able to figure this one out definitively however I am guessing its a side-effect of the interaction between the CPU and its co-processor for offloading - possibly something to do with power states. It would only manifest every 1 in 10 tests or so and apply consistently for that flow.

Because I was consistently getting around 440-450 Mbps however, I went with an average of those results as the value for the ER4.

Another curious observation came about when playing around with the different hardware offload modes. For all of the tests I compiled results with all hardware offloading enabled - but out of interest, I decided to disable ALL hardware offloading EXCEPT for the IPsec module and in all cases I got better throughput (about 5% more) out of the EdgeRouter 4 with a consistent 50% CPU usage, like before.

To me, this result is somewhat unexpected. I would have thought that offloading as much as possible to the offload engine would give better results but that does not appear to be the case.

I am only speculating but I think this may be to do with packets going back and forth from the co-processor unit in the Cavium chip. For example, A packet may come into the device for forwarding - get offloaded and then need to come back out of the offload engine to be processed further before being GRE encapsulated (and offloaded) and then ultimately encrypted (again, IPsec offloaded)? I am curious to see how this behaviour would affect overall performance of the device when its doing other CPU related tasks and not just IPsec encryption. However that’s out of scope for this article.

Testing the EdgeRouter Lite

Now that we have some figures for the EdgeRouter 4, I can move on to testing one of the other models. The logic here is that the EdgeRouter 4 is the more capable device by far, so by putting one of the (in theory) less powerful devices in place of one of the EdgeRouter 4s, the result I get would be capped by the performance of that device, and thus we measure its performance.

Using this logic, I can push traffic through the EdgeRouter Lite as the first hop, thus testing its encryption performance, and vice versa, having the EdgeRouter Lite as the last hop, testing its decryption performance.

The EdgeRouter Lite clearly suffers with its weaker CPU here - It was pretty much maxed out through the testing. This would probably perform OK if you don’t have a lot of bandwidth to play with, but doesn’t seem to do well much past 100 Mbit.

As observed with the ER4, curiously I see higher throughput through the EdgeRouter Lite with all but IPsec offloading disabled (about 10-12% improvement) whilst CPU usage is about the same.

Testing the EdgeRouter X

Given the result observed earlier for the raw throughput test of the EdgeRouter X, I was very interested to see how this one would turn out. This particular model of EdgeRouter is based on a different SoC manufacturer than the other two devices (MediaTek, vs Cavium respectively) so has a different hardware offload engine. This even more apparent in the configuration of the router and how you enable hardware offloading.

For the Cavium devices, we have a choice to enable specific features as/where needed (Forwarding has to be offloaded for any of the others to work, except for IPsec)

set system offload ipv4 forwarding enable
set system offload ipv4 gre enable
set system offload ipv4 pppoe enable
set system offload ipv4 vlan enable

set system offload ipv6 forwarding enable
set system offload ipv6 pppoe enable
set system offload ipv6 vlan enable

set system offload ipsec enable

Whereas for the EdgeRouter X’s MediaTek system, we simply have the choice of

set system offload hwnat enable
set system offload ipsec enable

So, using the same IPsec configuration settings for the EdgeRouter 4 and EdgeRouter Lite tests (detailed above), this router performed surprisingly well - Better than I had initially expected.

There’s a noticeable improvement in throughput and CPU usage on the ERX over the ERL which is quite surprising given the price point of the two models.

Interestingly however, unlike the two Cavium devices, the EdgeRouter X loses encryption performance with all other offloading disabled but gains some on decryption. Whilst the other two devices CPU usage remained pretty consistent, there is a noticeable change on the ERX for all offload vs IPsec only offloading.

This just reinforces the fact that not all offload engines are equal - even within devices under the same brand. That being said however the EdgeRouter X seems to be a very capable device and I do look forward to a time when I can compare it to the EdgeRouter Lite in a more “real-world” scenario.

Bandwidth-capped Test

The above tests do well to illustrate how well each model of router perform when unrestricted and allowed to try to push as much as they possibly can. I opted to do one final test and limit bandwidth to 100Mbit. I decided to do this on each server by forcing their NICs to 100M FDX. The aim of this test is to illustrate how each routers compare to each other when given the same workload.

All of the routers managed line rate (100M) and as the above shows, really reinforces how much better the EdgeRouter X is at handling IPsec over the EdgeRouter Lite. Interestingly in the capped test, the ERL used more CPU on decryption which is the opposite to earlier when it was pushing as much as it possibly can.

Conclusion

I originally set out to determine what throughput I could expect from these various devices at each end of an IPsec tunnel to allow for a more informed decision as to how to structure my network. The EdgeRouter 4 is pretty much what I expected it to be - in that it’s a very capable device and looks like it will be more then capable as a core router for my use case.

Despite the image this article paints for the EdgeRouter Lite, it is still a very capable device. It’s a router that’s aimed at home or small office users and the likelihood they have an internet connection that requires higher IPsec throughput is pretty low. Whilst the device itself isnt the cheapest around, the features it comes with certainly make it an attractive prospect for more advanced users who want to get more out of their Router than something like a BT Home Hub.

The EdgeRouter X on the other hand is a truly intriguing device! On the face of things it looks like it should perform better than the Lite at most things - however I will likely be comparing these in a more direct fashion in the future. Whilst the EdgeRouter X does indeed perform well, It will not be suitable as a drop-in replacement for the Lite for all situations - namely where dealing with high packet counts are required. The EdgeRouter Lite is rated at 1 million Packets-Per-Second (PPS) at 64 bytes in size whereas the EdgeRouter X is only rated for 260,000.

With that being said though, the EdgeRouter X is still a very good candidate for enthusiast users or light deployments - especially given its PoE passthrough feature.

I will leave you with a final graph that shows each router’s IPsec throughput side-by-side to highlight the differences between them.

It’ll be interesting to see if Ubiquiti can improve on these numbers in the future through software updates, or if the limitations are purely with the offloading hardware.

Dual-stacking Proxmox Web UI (pveproxy)

Wed, 20 Jun 2018 20:49:59 +0000

As part of my recent (and ongoing) project to implement native IPv6 on my own infrastructure (except at home… I’m looking at you Hyperoptic), I decided to try to dual-stack as much as possible so that when I have IPv6 connectivity, services would prefer that over IPv4, without making things unavailable.

As it turns out, Proxmox’s Web interface (pveproxy) doesn’t listen on the IPv6 address family by default. This stumped me for a little while, but its pretty simple to fix when you know whats going on.

This post is going to spend most of its time explaining why this happens rather than the fix. If you’re just here to see how to do it, check below.

Dual-Stacking pveproxy - aka “The Fix”

pveproxy seems to rely on the contents of /etc/hosts in order to figure out what address family to use when binding port 8006 (the default GUI port). In order to change this we need to edit our /etc/hosts file and change the line which identifies our Proxmox host

We start with (for example)

10.0.0.100 pvehost.some.domain pvehost pvelocalhost

And we should change it to

::ffff:10.0.0.100 pvehost.some.domain pvehost pvelocalhost

All we’ve done here is add ::ffff: before the IP address of our node. This tells the OS that its an IPv6 address - but its a special IPv4-mapped-IPv6 address. Restart the pveproxy service and you should now see it binds to an IPv6 socket

tcp6 0 0 :::8006 :::* LISTEN -

Viola - The Proxmox Web interface should work on both IPv4 and IPv6.

How pveproxy chooses which family to use…

By default, pveproxy decides to bind to an IPv4 family socket. By changing the family to v6, enables it to work for both families, primarily because of how the OS is configured.

Specifically, this depends on the value of bindv6only in sysctl. If this is enabled then the socket would only work for IPv6 addresses and applications would have to specifically bind to both v4 and v6 address families independently, depending on which they want to support.

Leaving bindv6only disabled (default) is my preferred choice here as this would affect more than just Proxmox for me.

In order to figure this one out, I had to dig into the supplied perl modules for Proxmox. At the time of writing, I was using Proxmox 5.2 for my testing. Because I’m looking at the source for the modules, I wont quote line numbers as they will likely change over time. I will however link to the corresponding git commits.

If we start with the module for pveproxy - /usr/share/perl5/PVE/Service/pveproxy.pm

(Git: 2d3d4cf7ff410a04c6fe0579f59dde4fec1e56e3)

 my $family = PVE::Tools::get_host_address_family($self->{nodename});
 my $socket = $self->create_reusable_socket(8006, undef, $family);

It’s pretty clear here that it gets the address family for something, and then creates a socket bound to port 8006 using that family. Tracing back to where nodename actually gets set, we end up looking at /usr/share/perl5/PVE/INotify.pm

(Git: 0b9ea4adc3e078a47369cd97bc6c271a45f16870))

sub nodename {
 return $cached_nodename if $cached_nodename;
 my ($sysname, $nodename) = POSIX::uname();
 $nodename =~ s/\..*$//; # strip domain part, if any
 die "unable to read node name\n" if !$nodename;
 $cached_nodename = $nodename;
 return $cached_nodename;
}

My understanding of what the above snippet does (POSIX::uname) is that it in-fact grabs the same values as would be visible from uname on command line. Specifically the “kernel-name” and “nodename” arguments

root@pvehost:~# uname -sn
Linux pvehost

root@pvehost:~# uname --kernel-name --nodename
Linux pvehost

From here, a host lookup is performed against the nodename value. On a fresh install this is typically an IPv4 address, so pveproxy will only ever bind to the IPv4 address family. The change to the host entry made earlier now technically makes this an IPv6 address, thus the family chosen is IPv6.

Personally, I’d like to have a bit more control over how pveproxy binds in the future.

Ethereum Mining with NVIDIA on Linux

Thu, 06 Jul 2017 00:05:52 +0000

Bitcoin has been getting a lot of attention in the media lately; primarily due to its association with recent high profile ransomware incidents. Bitcoin was first introduced back in 2009 as a digital currency and has been steadily gaining traction since, increasing almost 140% in value so far this year!

Bitcoin however, is not the only player in the game; Ethereum is another popular digital currency, currently sitting at 2nd place in total market value and is getting a lot of activity on currency exchanges.

An incentive to help keep the underlying “network” up and running (for both Ethereum and Bitcoin, amongst others) is a piece of software which can be used to validate transactions on the network and issue rewards for doing so. This is otherwise known as Mining.

My current Gaming PC seems to grab me a reasonable amount of Ethereum Tokens on 2+ year old hardware. Certainly enough to make me want to play around with actually mining some.

TLDR

Skip right to the overclocking commands here. Otherwise, continue reading :)

Why use Linux?

The mining program I am using is currently a Fork of a Fork from the original C++ based version developed by the Ethereum Foundation. In its FAQ there seem to be known issues with NVIDIA, WDDM and Windows 10 which cause lower-than expected performance of NVIDIA cards when mining. A supposed fix is to use an older set of NVIDIA drivers along with a different build of the mining software specifically built for CUDA 6.5.

The machine I am using is my Gaming PC (which requires Windows), so I’m not prepared to jump through those hoops - running older drivers isn’t always conducive to a good gaming experience. My graphics card is NVIDIA and I’ve always had reasonable experiences using their Linux drivers so giving this a go on Linux seemed like a natural fit.

My hardware

I’ve not yet built myself a dedicated mining rig but these are the specs I’m working with:

Gigabyte H67N-USB3-B3 Motherboard
Intel Core i7-2600 CPU @ 3.40GHz
16GB RAM
NVIDIA GeForce GTX 970 - GM204 (GV-N970IXOC-4GD)
Current Driver Version: 375.66

Getting Started

In order to start mining Ethereum, you’ll first need a working Linux installation - My distro of choice is Ubuntu (16.04 LTS) Xenial.

When first installing and running Ubuntu I did have to make use of nomodeset in grub before anything would actually display on my screen - This seems to be card specific but it might help you get going! See here for more info about nomodeset.

Once you’re set up, you’ll need to install NVIDIA drivers. You can obtain Linux specific ones from NVIDIA directly and install those. As I’m using Ubuntu however, there should be some available in repos - apt-get install nvidia-375 at the time of writing this.

Establishing a baseline

Before we make any tweaks, we should establish what our baseline is on default settings. Keep in mind that this is specific to me and my hardware - your baseline may well be different depending on what equipment you use.

The version of ethminer I am using comes with a built in benchmark mode. To make these tests a fair comparison between tweaks, I’ve tested everything against block 3969748 (A random block I chose which represents the current difficulty of Ethash).

Running the benchmark for 10 trials, at 5 seconds each gives me an average hash rate of 18.31MH/s at the current difficulty. This is my baseline.

Enabling experimental features

By default, the NVIDIA drivers do not let you change any Experimental features (manual control of things) out of the box. In order to enable these, we need to add an option to our xorg.conf file. Before we make any changes here, you should back up your current config file if one exists. In my case this is /etc/X11/xorg.conf but your location may vary.

Once you know where your config file is, we need to add NVIDIA’s Coolbits option to the Devices section in the config. Using 28 in my case enables all experimental features - Even if we dont use them all :)

Section "Device"
 Identifier "Device0"
 Driver "nvidia"
 Option "Coolbits" "28"
 VendorName "NVIDIA Corporation"
EndSection

If you’re using multiple GPUs, you need to add the Option to each individual Device section in your config.

A simple way to see if it worked is to (after a reboot) open nvidia-settings and look at the Thermal Settings tab; You should see fan control options now available:

Persistence Mode

Before we do any actual tweaks, we should enable Persistence Mode to ensure our changes stay active even after we close any applications that might be using our card.

The nvidia-smi utility allows you to control multiple GPUs with its -i option.

nvidia-smi -i 0 -pm 1
nvidia-smi -i 1 -pm 1
nvidia-smi -i 2 -pm 1
etc

So enabling Persistence Mode on my machine would be:

simon@kirika:~$ sudo nvidia-smi -i 0 -pm 1
Enabled persistence mode for GPU 0000:01:00.0.
All done.

Power Limits

One of the tweaks we have access to is the ability to change power limits for our card. This is also controlled via nvidia-smi command line tool.

Lets look at what power limit options we have available:

simon@kirika:~$ nvidia-smi -i 0 -q -d power

==============NVSMI LOG==============

Timestamp : Mon Jul 3 18:11:28 2017
Driver Version : 375.66

Attached GPUs : 1
GPU 0000:01:00.0
 Power Readings
 Power Management : Supported
 Power Draw : 12.90 W
 Power Limit : 160.00 W
 Default Power Limit : 160.00 W
 Enforced Power Limit : 160.00 W
 Min Power Limit : 100.00 W
 Max Power Limit : 180.00 W
 Power Samples
 Duration : 78.01 sec
 Number of Samples : 119
 Max : 61.22 W
 Min : 12.71 W
 Avg : 14.83 W

On this initial query, I can see that the default power limit for my card is 160 watts and that the maximum I can set it to is 180 watts.

Lets set it to 180w

simon@kirika:~$ sudo nvidia-smi -i 0 -pl 180
Power limit for GPU 0000:01:00.0 was set to 180.00 W from 160.00 W.
All done.

More power should mean better performance, Right? - Nope.

Performance States

I performed a benchmark at each power level my card supports, with a 5w increment between them.

The graph above shows what I’d expect when we decrease our power limit below 160w; A steady decrease in hashes the lower the supplied power goes. At the top end of the scale though, I don’t actually gain any performance improvements what-so-ever.

After a bit of reading around on forums - This is because the mining application is providing a compute workload to the card, rather than something more graphical in nature. The default clock speeds for “Applications” appears lower than what the card is capable of and thus it gets capped into P2 Performance State; Hence the flatline on the graph.

To view what our current Application Clock is set to, we can take a look using nvidia-smi once again, and looking for Application Clocks and Default Application Clocks.

simon@kirika:~$ nvidia-smi -i 0 -q -d clock

==============NVSMI LOG==============

Timestamp : Wed Jul 5 00:00:38 2017
Driver Version : 375.66

Attached GPUs : 1
GPU 0000:01:00.0
 Clocks
 Graphics : 1240 MHz
 SM : 1240 MHz
 Memory : 3004 MHz
 Video : 1141 MHz
 Applications Clocks
 Graphics : 1076 MHz
 Memory : 3505 MHz
 Default Applications Clocks
 Graphics : 1076 MHz
 Memory : 3505 MHz
 Max Clocks
 Graphics : 1418 MHz
 SM : 1418 MHz
 Memory : 3505 MHz
 Video : 1304 MHz
 SM Clock Samples
 Duration : 547.35 sec
 Number of Samples : 100
 Max : 1476 MHz
 Min : 1240 MHz
 Avg : 1422 MHz
 Memory Clock Samples
 Duration : 547.35 sec
 Number of Samples : 100
 Max : 3505 MHz
 Min : 3004 MHz
 Avg : 3491 MHz
 Clock Policy
 Auto Boost : N/A
 Auto Boost Default : N/A

This output is also useful in that it shows what our current maximum clock speeds can be set to. It’s clear in the output above that our max clock differ from what Applications are permitted to use.

Armed with these values, I can proceed to change my Application Clocks to allow the GPU to clock higher when performing compute workloads. This higher performance state is known as P0 Performance State.

simon@kirika:~$ sudo nvidia-smi -i 0 -ac 3505,1418
Applications clocks set to "(MEM 3505, SM 1418)" for GPU 0000:01:00.0
All done.

Lets run our benchmark again:

You can just make out that the top-end of our graph is no longer a flatline which suggests we’re now using the full range of clock speeds available on the card. Albeit with little gain… But, I’m sure I can do better!

Overclocking

The final stages of tweaking are whats known as overclocking. The values I’ve chosen here are very much specific to my hardware. You might be able to get a more stable overclock, or have to use lower values - It entirely depends on your own hardware.

In order to achieve this, I’ll be using nvidia-settings which requires a working X server to be able to apply changes to our card. We tell nvidia-settings which X server to attach to with the -c :0 option and in my examples, we’re using display :0

We can also apply changes to multiple GPUs by changing the ID number in the command:

nvidia-settings -c :0 -a [gpu:0]/GPUGraphicsClockOffset[3]=0
nvidia-settings -c :0 -a [gpu:1]/GPUGraphicsClockOffset[3]=0
nvidia-settings -c :0 -a [gpu:2]/GPUGraphicsClockOffset[3]=0

Figuring out how large an overclock your card supports will take a bit of trial, error and a lot of reboots - I started with 100MHz and increased slowly until the card became unstable/crashed. Once you know a stable number, increase it in smaller increments until you start having issues.

My card took a stable overclock at 240MHz. If I go as high as 250MHz, CUDA would sometimes fail to initialize, so make sure you test each value properly.

simon@kirika:~$ nvidia-settings -c :0 -a [gpu:0]/GPUGraphicsClockOffset[3]=240

 Attribute 'GPUGraphicsClockOffset' (kirika:0[gpu:0]) assigned value 240.

For the sake of being thorough, you can also apply an offset to Memory. The comments above still apply here with finding a stable value. In my case, an offset of 1,100MHz was stable, although this actually seemed to result in a frequency change of only 550MHz (which is half of 1,100). In my testing however, a Memory overclock had negligible impact on my results, but your mileage may vary so its still worth giving it a shot…

nvidia-settings -c :0 -a [gpu:0]/GPUMemoryTransferRateOffset[3]=1100

Once you know a good value for both of these, we can now look at adjusting the Application Clocks mentioned earlier to accommodate the new clock speeds for compute workloads.

Check the max values using nvidia-smi -i 0 -q -d clock like before, and set the new clocks accordingly.

simon@kirika:~$ sudo nvidia-smi -ac 4055,1658
Applications clocks set to "(MEM 4055, SM 1658)" for GPU 0000:01:00.0
All done

With my card overclocked and stable, I did another round of benchmarks and you can clearly see the improvement compared to default settings. I mentioned previously that the memory overclock had a negligible impact on my results so I’ve omitted those from the graph.

Conclusion

I’ve managed to improve my hash rate by 18.4% - from 18.31MH/s to 21.68MH/s, by overclocking my GPU and increasing the power limit available to the card. Power consumption of the card has increased by approximately 12.5% when at 180 watts (instead of the default 160w), but it’s still worth me running at the higher power threshold.

If however, I wanted to keep power usage at similar (or lower) levels to what we started with, I can limit to 160w and still achieve 21.06MH/s with the overclock, which is still a 15% improvement over default settings! This also has the added advantage of keeping GPU temperatures under control.

For convenience, here are all the changes I’ve made in order to get this improvement (which includes some fan control options to keep on top of cooling):

# Edit X Config to add coolbits
# Reboot
# Make sure X is running

# Enable persistence mode
sudo nvidia-smi -i 0 -pm 1

# Increase our Power Limit from 160 up to 180
sudo nvidia-smi -i 0 -pl 180

# Ensure fan speed is controllable and set it to 55%
# (You can use whatever value you like for fan speed)
nvidia-settings -c :0 \
-a [gpu:0]/GPUFanControlState=1 \
-a [fan:0]/GPUTargetFanSpeed=55 \

# Overclock GPU/RAM
nvidia-settings -c :0 \
-a [gpu:0]/GPUGraphicsClockOffset[3]=240 \
-a [gpu:0]/GPUMemoryTransferRateOffset[3]=1100

# Apply new clocks
sudo nvidia-smi -ac 4055,1658

The nvidia-settings commands could be merged into one, but I kept them separate for clarity.

Shoehorning Proxmox into mdadm RAID1

Sun, 12 Mar 2017 01:04:30 +0000

My home server has always been somewhat of a Frankenstein’s Monster when it comes to hardware. Until about 2 years ago, it consisted of a large JBOD array of several different drive sizes with no kind of redundancy what-so-ever - for both OS and Data disks. That all changed though when I decided to invest in some 3TB disks and build a “proper” RAID array for my data; Which coincidentally was around the same time my data storage array lost a drive resulting in about 800GB of data-loss - I was unable to source a decent hardware RAID card at the time so I went with Software RAID (mdadm) instead.

The chassis which houses my home server happens to be an old X-Case model that has 10 internal bays, a couple of 5.25" for Optical drives and a 5.25" to 3.5" caddy that seems to be designed for a floppy drive (it has a cut-out at the front for access). I use the floppy slot to house my OS disk and the rest are for my data disks which are all in RAID6 for redundancy. This gives me a total of 11 hard drives in this chassis in its default configuration.

When I rebuilt my server to make use of Proxmox, I put in a known-working drive as the OS drive; It was an old drive at the time but didn’t have much in the way of power on hours so I figured it’d last me a while. Here I am a few years later and the drive is still going, but SMART data is showing some figures that’re making me worried it’s going to start having problems soon so I set about replacing it.

Step 1 - Replace the problem drive

Sure, I could have just bought a new drive and re-installed my server from scratch, but frankly I couldn’t be bothered with all the niggly applications and configurations I’d need to get right to make it all work nicely again.

Besides, where’s the fun in that?

I bought a 3.5" to Dual 2.5" adaptor off eBay (In-Win 3RAMZF078700) which, as you might guess, should nicely house a couple of 2.5" drives where my old 3.5" OS disk used to live. Having already made use of mdadm on this system, and now having two drives, I should be able to use them in RAID1 for my OS disk.

I chose to get two HGST Travelstar 7K1000 drives because a friend uses a lot of them and has (so far) had no problems with them. They also spin at 7200RPM so should get reasonable performance in RAID1 given I’m limited to essentially a single drive’s performance over more conventional 2.5" drives which operate at 5400RPM.

There were a couple of things I needed to consider before embarking on this project though - most of which were potential physical constraints inside the chassis itself.

Do I have enough SATA data ports?

My SuperMicro X8DTL-3F motherboard has 6 SATA ports and 8 SAS ports. I learned the hard way that you can only boot from the SATA ports, and that the on-board LSI SAS controller does not support drives larger than 2TB. For my configuration this leaves me with just the 6 on board SATA ports so I invested in an LSI 9211-8i HBA (It’s actually a cross-flashed Dell H200A - http://blog.michael.kuron-germany.de/2014/11/crossflashing-dell-perc-h200-to-lsi-9211-8i/comment-page-1/)

The 9211-8i in theory supports 8 drives - combine the 6 SATA ports on the motherboard with the 8 on the HBA and I have theoretically got 14 ports available to use with my 2 boot drives (RAID1) and 10 data drives (RAID6)

I can only fit 11 disks in the chassis (12 if you count the 2.5" disks instead) so I should have enough.

Do I have enough SATA power connectors?

My PSU is an EVGA 650 GQ which has 9 native SATA power ports and a further 3 molex connectors to which I can attach some 2xSATA splitters. I use one of the molex connectors for cooling though so I only get 4 SATA connectors from molex and 9 from native. This should be enough for my purposes.

Will the dual 2.5" bracket even fit where a 3.5" disk would normally go?

The HGST Travelstar disks come in at 9.5mm in height. The disk that sits on the bottom of the bracket has enough clearance for a 12mm drive, but the top one was a potential problem as it could sit on the bracket taller than a standard 3.5" drive.

I’d fitted the 2.5" drives and stood the bracket next to two different 3.5" models and as you can see, its about the same height. It is slightly taller than the drive on the right (WD800JD) But is about the same height as the one on the left (WD10EADS)

I do have open-air above my drives in this chassis so height restrictions are not a problem for me; However if you wanted to use this bracket in a more conventional 3.5" bay, you should not have any issues using two 9.5mm drives.

The placement of these drives in the “floppy” bay unfortunately means there is no airflow from case fans so I did have to consider the gap between the drives to allow for cooling - which as you can see should be ample clearance.

Step 2 - Cloning the original

As my plan was to go from a single 3.5" drive to two 2.5" drives, that does beg the question as to how I get the data onto the smaller drives. I could have skipped this step and jumped right into Step 4 to assemble the RAID array using both the old and one of the new drives, to later replace the old drive with the remaining new one.

Instead, I opted to use a tool called CloneZilla which will clone the original drive onto the new one (although you could have just used dd) whilst leaving the original untouched should I need to revert (or re-clone) it at a later date. This was a personal choice but I like the idea of having the safety net should anything go wrong further down the line then, at least I couldstart again if needed.

First we connect one of the new HGST drives up and temporarily plug in the old OS drive so that we can clone it. Next I booted Clonezilla and followed the on-screen instructions clone the drives.

You might need to zero the destination disk first so that everything clones over nicely.

It’s very important that you make sure your destination drive is correct because CloneZilla will overwite anything thats already there

Once the clone is complete, you should be able to disconnect the old drive and boot from the new one. If all goes well then your system should look the same from the OS’s point of view other than the fact you’re now on a larger disk (In my case anyway, I went from 160GB to 1TB).

Step 3 - Fixing the partitions on the larger disk

Before we delve too deep into this, I realize my partitions are not standard for Proxmox these days in that I have a /boot partition outside of the main root partition (for me /dev/sdd2 is /boot). Nonetheless this should not affect the overall process if your partitions are different - you just need to skip the parts that’re specific to me and adjust according to your configuration.

Naturally your mileage may vary depending on how your system is set up. Don’t just blindly paste commands into a terminal as you could cause irreversible data-loss :) - Make sure you understand what you’re doing!

Now that we’ve a cloned the old OS disk onto one of the new (larger) disks, I thought now would be a good time to fix the primary data partition (where the “pve” LV lives) so that it takes up the entirety of the disk. I went from a 160GB to 1TB drive so to do this I simply loaded parted and took a look at the partitions whilst telling it to display sector count instead of the default - It showed me a warning about the GPT size being wrong and offered to correct it for me.

root@yuzuki:~# parted /dev/sdd
GNU Parted 3.2
Using /dev/sdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit s
(parted) print free
Warning: Not all of the space available to /dev/sdd appears to be used, you can fix the GPT
to use all of the space (an extra 1640943360 blocks) or continue with the current setting?
Fix/Ignore? Fix
Model: ATA HGST HTS721010A9 (scsi)
Disk /dev/sdd: 1953525168s
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name Flags
 34s 2047s 2014s Free Space
 1 2048s 4095s 2048s primary bios_grub
 2 4096s 1048575s 1044480s ext3 primary boot, esp
 3 1048576s 312580095s 311531520s primary lvm
 312580096s 1953525134s 1640945039s Free Space

Here we can see the current starting sector for our last partition (partition 3, aka /dev/sdd3) is 1048576 and the last usable sector of free space is 1953525134. We should simply be able to remove the partition table (It wont remove data) and then re-create it with a larger size. As long as the starting position is the same then everything should line-up.

(parted) rm 3
Error: Partition(s) 3 on /dev/sdd have been written, but we have been unable to inform the
kernel of the change, probably because it/they are in use. As a result, the old partition(s)
will remain in use. You should reboot now before making
further changes.
Ignore/Cancel? i
(parted) mkpart primary 1048576 1953525134
(parted) toggle 3 lvm
(parted) print
Model: ATA HGST HTS721010A9 (scsi)
Disk /dev/sdd: 1953525168s
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name Flags
 1 2048s 4095s 2048s primary bios_grub
 2 4096s 1048575s 1044480s ext3 primary boot, esp
 3 1048576s 1953525134s 1952476559s primary lvm

(parted) quit
Information: You may need to update /etc/fstab.

Finally we need to make sure that LVM is aware of the new disk size lest we forget about it later on

root@yuzuki:~# pvdisplay /dev/sdd3
 --- Physical volume ---
 PV Name /dev/sdd3
 VG Name pve
 PV Size 148.55 GiB / not usable 3.00 MiB
 Allocatable yes
 PE Size 4.00 MiB
 Total PE 38028
 Free PE 4095
 Allocated PE 33933
 PV UUID 6p74nl-SxZp-4xu7-j3lH-ACsg-Hs9I-5rNwpq

root@yuzuki:~# pvresize /dev/sdd3
 Physical volume "/dev/sdd3" changed
 1 physical volume(s) resized / 0 physical volume(s) not resized

root@yuzuki:~# pvdisplay /dev/sdd3
 --- Physical volume ---
 PV Name /dev/sdd3
 VG Name pve
 PV Size 931.01 GiB / not usable 312.50 KiB
 Allocatable yes
 PE Size 4.00 MiB
 Total PE 238339
 Free PE 204406
 Allocated PE 33933
 PV UUID 6p74nl-SxZp-4xu7-j3lH-ACsg-Hs9I-5rNwpq

If you didn’t want to implement RAID1 then this is where you’d stop - We’ve moved our OS to a new disk without reinstalling.

I however do want to implement RAID1 so in Step 4 I’ll walk through how to prepare the second RAID member and begin the process of moving towards having a complete array.

Step 4 - Preparing for RAID

The purpose of this article is to document how I implemented software RAID1 on a live system without having to reinstall. In the previous steps, I took a little detour in that I cloned the original disk onto one of the new replacement disks. Primarily this was to preserve the original data should anything not go according to plan. The following steps are where you’d start if you wanted to skip the cloning process and work on the source disk without a “spare” copy of your data.

It’s important that you identify which companion disk you intend to use for your array. A useful tool to use here would be smartctl. With this tool you can query each disk until you find which ones you want to work with (Especially useful if you have a lot of disks).

root@yuzuki:~# smartctl -i /dev/sdd
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.4.35-2-pve] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: HGST Travelstar 7K1000
Device Model: HGST HTS721010A9E630
Serial Number: JR1000********
LU WWN Device Id: 5 000cca 8c8d62953
Firmware Version: JB0OA3U0
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 6
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Mon Mar 6 21:16:45 2017 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

In my case, /dev/sdd is the non-RAID disk which currently houses my OS and /dev/sde is the blank disk I want to initialize my array onto. Firstly we need to copy the current partition table from /dev/sdd to /dev/sde. As the partition table is GPT, we should use sgdisk to do this.

root@yuzuki:~# sgdisk -R=/dev/sde /dev/sdd
The operation has completed successfully.

Once this command has completed, we should now change the type code for /dev/sde2 and /dev/sde3 to reflect that these partitions are members of software RAID. You can do this with any partition tool that supports GPT, but in my case I’ll use sgdisk to change the types.

root@yuzuki:~# sgdisk -t 2:fd00 /dev/sde
The operation has completed successfully.

root@yuzuki:~# sgdisk -t 3:fd00 /dev/sde
The operation has completed successfully.

This should nicely result in /dev/sde having the same partition structure as /dev/sdd (but with both partition types as Linux RAID)

root@yuzuki:~# fdisk -l /dev/sde

Disk /dev/sde: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: A4EF094E-5546-4DC1-A262-2B08B9C586B1

Device Start End Sectors Size Type
/dev/sde1 2048 4095 2048 1M BIOS boot
/dev/sde2 4096 1048575 1044480 510M Linux RAID
/dev/sde3 1048576 1953525134 1952476559 931G Linux RAID

Step 5 - Initialize RAID

We’re now at the stage where both our disks should be nearly identically partitioned and we can actually start creating our arrays. We’ll first do this in a degraded state which will allow us to move data around.

In my case /dev/sdd2 is /boot and /dev/sdd3 houses my LVM data (ultimately containing / and swap).

I already have /dev/md0 as my primary data storage RAID, so /dev/md1 will be for /boot and /dev/md2 for everything else.

Lets start by creating both mdadm devices, but using the currently empty disk (/dev/sde) - I chose to ignore the notice displayed as my version of grub has support for mdadm out of the box.

root@yuzuki:~# mdadm --create /dev/md1 --level=1 --raid-disks=2 missing /dev/sde2
mdadm: Note: this array has metadata at the start and
 may not be suitable as a boot device. If you plan to
 store '/boot' on this device please ensure that
 your boot-loader understands md/v1.x metadata, or use
 --metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.

root@yuzuki:~# mdadm --create /dev/md2 --level=1 --raid-disks=2 missing /dev/sde3
mdadm: Note: this array has metadata at the start and
 may not be suitable as a boot device. If you plan to
 store '/boot' on this device please ensure that
 your boot-loader understands md/v1.x metadata, or use
 --metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.

Next we need to ensure that the new arrays are started at boot. Typically this is done by adding them to mdadm.conf. The location of this file may vary depending on your OS but on debian its in /etc/mdadm. I simply take a look at the 2 arrays and add them to the end of the config file

root@yuzuki:~# mdadm --detail --scan /dev/md{1,2} >> /etc/mdadm/mdadm.conf

We should now be in a position where you have /dev/md1 and /dev/md2 and they should both start automatically on-boot. Next we’ll look at making use of them!

Step 6 - Configure grub and move /boot

We should now have our degraded arrays which we can now immediately start to use. I’ll begin by creating a new file system on /dev/md1 for use with /boot and then move data to it. Obviously if your /boot is not a separate partition but is included in your root partition then you don’t need to worry about this part.

root@yuzuki:~# mkfs.ext4 /dev/md1
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 521920 1k blocks and 130560 inodes
Filesystem UUID: 37e651bc-ffc1-41d8-a9c7-99ca9273b829
Superblock backups stored on blocks:
 8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

root@yuzuki:~# mkdir /mnt/tmp
root@yuzuki:~# mount /dev/md1 /mnt/tmp/
root@yuzuki:~# cp -ax /boot/* /mnt/tmp/
root@yuzuki:~# umount /mnt/tmp
root@yuzuki:~# rmdir /mnt/tmp

Once all this is done, lets edit /etc/fstab to tell our system to mount /boot from the new array. Your fstab will be different to mine, but find where /boot is mounted and change it to /dev/md1 (I kept my original UUID around just in-case)

root@yuzuki:~# cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
#UUID=f6f3e14e-a2fe-468f-80ee-8824dab51e36 /boot ext3 defaults 0 1

/dev/md1 /boot ext4 defaults 0 1
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/data/wdred /mnt/storage ext4 defaults,barrier=0 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

It’s critical you make sure there are no errors here, getting this wrong could result in an un-bootable system!

At this point I chose to reboot my system to ensure it’s booting correctly and running /boot from the correct place

root@yuzuki:~# mount | grep boot
/dev/md1 on /boot type ext4 (rw,relatime,stripe=4,data=ordered)

mdadm needs to be told to start all arrays on-boot (As we intend to host / in an array) so edit /etc/default/mdadm and change INITRDSTART to ‘all’ (Mine was set to ’none’ by default)

root@yuzuki:~# grep ^INITRDSTART /etc/default/mdadm
INITRDSTART='all'

Followed by some grub and initrd changes

root@yuzuki:~# echo 'GRUB_DISABLE_LINUX_UUID=true' >> /etc/default/grub
root@yuzuki:~# echo raid1 >> /etc/modules
root@yuzuki:~# echo raid1 >> /etc/initramfs-tools/modules

And then make sure grub (boot loader) exists on the 2 new RAID disks followed by a grub update and RAM disk update

root@yuzuki:~# grub-install /dev/sdd
Installing for i386-pc platform.
Installation finished. No error reported.
root@yuzuki:~# grub-install /dev/sde
Installing for i386-pc platform.
Installation finished. No error reported.

root@yuzuki:~# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.4.35-2-pve
Found initrd image: /boot/initrd.img-4.4.35-2-pve
Found memtest86+ image: /memtest86+.bin
Found memtest86+ multiboot image: /memtest86+_multiboot.bin
done

root@yuzuki:~# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-4.4.35-2-pve

Now we’re in the final stretch!

Step 7 - Assembling the arrays

If everything has gone to plan, our system should now be running /boot from /dev/md1 which means the old /boot partition on /dev/sdd2 is no longer needed. We should set its type to Linux RAID and add it to our array

root@yuzuki:~# sgdisk -t 2:fd00 /dev/sdd
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.

root@yuzuki:~# mdadm --add /dev/md1 /dev/sdd2
mdadm: added /dev/sdd2

root@yuzuki:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md2 : active raid1 sde3[1]
 976107200 blocks super 1.2 [2/1] [_U]
 bitmap: 1/8 pages [4KB], 65536KB chunk

md1 : active raid1 sdd2[2] sde2[1]
 521920 blocks super 1.2 [2/1] [_U]
 [===========>.........] recovery = 55.2% (288896/521920) finish=0.0min speed=144448K/sec

md0 : active raid6 sdh1[0] sdc1[6] sda1[7] sdg1[5] sdf1[4] sdi1[2] sdb1[1]
 14650670080 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/7] [UUUUUUU]

unused devices: <none>

And viola, we now have (when its recovered) a working software RAID for /boot

Next we’ll add /dev/md2 to LVM and make the space available to our pve volume group so that we can simply pvmove the remaining data to our RAID array.

root@yuzuki:~# pvcreate /dev/md2
 Physical volume "/dev/md2" successfully created
root@yuzuki:~# pvdisplay
 --- Physical volume ---
 PV Name /dev/sdd3
 VG Name pve
 PV Size 931.01 GiB / not usable 312.50 KiB
 Allocatable yes
 PE Size 4.00 MiB
 Total PE 238339
 Free PE 204406
 Allocated PE 33933
 PV UUID 6p74nl-SxZp-4xu7-j3lH-ACsg-Hs9I-5rNwpq

 "/dev/md2" is a new physical volume of "930.89 GiB"
 --- NEW Physical volume ---
 PV Name /dev/md2
 VG Name
 PV Size 930.89 GiB
 Allocatable NO
 PE Size 0
 Total PE 0
 Free PE 0
 Allocated PE 0
 PV UUID PzVPqV-Rn1V-ASA5-Q4Ni-cXhS-LUCu-QZUTiC


root@yuzuki:~# vgextend pve /dev/md2
 Volume group "pve" successfully extended

You might notice that /dev/sdd3 and /dev/md2 aren’t the same size according to LVM - I suspect this is because of RAID metadata overheads. Thankfully because I am moving from smaller to larger disks, I have enough Free PE that I can move without issue. If however you have no Free PE’s and your disks are not equal in size like shown above, you might need to shrink one of your Logical Volumes and fix the file system later.

I’m happy that I can proceed here so I move the contents of /dev/sdd3 to /dev/md2 within LVM

root@yuzuki:~# pvmove /dev/sdd3 /dev/md2
 /dev/sdd3: Moved: 100.0%

The pvmove itself will take a while depending on how much data there is to move (Allocated PE) but once that is done, all we have left to do is remove the old disk from LVM

root@yuzuki:~# vgreduce pve /dev/sdd3
 Removed "/dev/sdd3" from volume group "pve"

root@yuzuki:~# pvremove /dev/sdd3
 Labels on physical volume "/dev/sdd3" successfully wiped

And then to add the old partition to our corresponding mdadm array

root@yuzuki:~# mdadm --add /dev/md2 /dev/sdd3
mdadm: added /dev/sdd3

root@yuzuki:~# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md2 : active raid1 sdd3[2] sde3[1]
 976107200 blocks super 1.2 [2/1] [_U]
 [>....................] recovery = 1.3% (12725760/976107200) finish=133.5min speed=120174K/sec
 bitmap: 2/8 pages [8KB], 65536KB chunk

md1 : active raid1 sdd2[2] sde2[1]
 521920 blocks super 1.2 [2/2] [UU]

md0 : active raid6 sdh1[0] sdb1[6] sda1[7] sdg1[5] sdf1[4] sdi1[2] sdc1[1]
 14650670080 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/7] [UUUUUUU]

unused devices: <none>

A final reboot to make sure everything comes up correctly and if so, its just a matter of waiting for RAID recovery to complete.

Routed Subnets, rp_filter and arp_ignore

Fri, 23 Sep 2016 20:40:25 +0000

Over the past couple of years, I’ve been quite accustomed to my ISP providing me with an IP block (typically a /29) instead of just a single static IP, but recently I’ve switched to Hyperoptic for their FTTP offering (Because, Gigabit!). Unfortunately, they seem to only offer single IPs for residential service and require you to be on their business package which costs £££ if you want more IPs.

To get around this I decided to try to tunnel some IPs home - My setup is rather complex but I’m going to skip over the specifics for most of it and focus on the issue I was facing - lets ignore how the IPs route around the rest of my network and ultimately end up at the router for now and just look at the rather simplified view below.

The Problem

The image shows that I essentially have a router which has two NICs, one connected to the public internet, and one connected to a switch within my network. The second NIC has two IPs assigned to it as follows:

10.0.2.51/24 which is my internal private subnet
172.26.0.9/29 which is the subnet I want to tunnel/route to my server (172.x for the sake of this document)

Traffic would come into Router from an upstream device with a destination of 172.26.0.13. As the router has a route to a subnet in which that IP resides (by virtue of it having an IP in that subnet), it knows that IP should be reachable directly via our NIC2 interface and so ARPs for 172.26.0.13 out of it. Because both interfaces on Server and NIC2 on Router all sit in the same broadcast domain, my server happily replies to the ARP request (out of both interfaces - more on this later) so the router now forwards traffic down the wire to my server; Except there’s no reply… but only sometimes.

After much head scratching, I noticed that when I do sometimes get a correct response from 172.26.0.13, the ARP table for Router was showing the MAC address for NIC1 on Server. When I don’t get a reply (usually after the ARP entry expires) it instead shows the MAC for NIC2 - But having the MAC for NIC2 is what I’d expect, so whats going on?

The Explanation

After spending a good amount of time throwing arbitrary queries into Google and cussing to myself, I eventually caved and asked my Network Engineer for a flat-mate, Dataforce, for a fresh set of eyes.

We both spent a little while talking through the setup and testing various things and after a while took a look at a sysctl setting for rp_filter (Reverse Path Filter). Consulting kernel documentation at https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt, it says the following for rp_filter:

rp_filter - INTEGER
	0 - No source validation.
	1 - Strict mode as defined in RFC3704 Strict Reverse Path
	 Each incoming packet is tested against the FIB and if the interface
	 is not the best reverse path the packet check will fail.
	 By default failed packets are discarded.
	2 - Loose mode as defined in RFC3704 Loose Reverse Path
	 Each incoming packet's source address is also tested against the FIB
	 and if the source address is not reachable via any interface
	 the packet check will fail.

	Current recommended practice in RFC3704 is to enable strict mode
	to prevent IP spoofing from DDos attacks. If using asymmetric routing
	or other complicated routing, then loose mode is recommended.

If I check the current value for Server, it is indeed STRICT (1)

root@server:~# cat /proc/sys/net/ipv4/conf/all/rp_filter
1

This implies that when a packet is received on an interface, the source IP is checked against the local routing table to see if the route in which the server would normally talk to the source IP, is the same interface at which the packet was received. My server’s default route (0.0.0.0) is out via NIC1, so traffic that ultimately comes from the internet via my tunnelled subnet would arrive on NIC2, but rp_filter would match against NIC1 - thus the check fails and the packet is dropped as invalid.

The Solution

Based on the documentation above, I either want to set rp_filter to be either 0 (NONE) or 2 (LOOSE). This is done via sysctl and the value can be changed whilst live in one of two ways:

root@server:~# sysctl -w net.ipv4.conf.all.rp_filter=2
net.ipv4.conf.all.rp_filter = 2

root@server:~# echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter

Using either of these methods will only change the value until the server is rebooted. The value reverts to whatever is defined in sysctl.conf on boot so to make this change permanent, either edit your config, or add the following line if its not already defined:

root@server:~# echo "net.ipv4.conf.all.rp_filter = 2" >> /etc/sysctl.conf

Changing this to LOOSE (2) means that when a packet is received on an interface, rp_filter will check the route table to see if the source IP is reachable via any interface and if so, accept it (because it matches against 0.0.0.0). If I were to set this to NONE (0), it outright wouldn’t do any verification checks so would just work normally regardless - the choice as to which you use is up to you.

Bonus Issues!

Now, if you recall I mentioned earlier that without this change, it sometimes worked and sometimes didn’t. This is a whole different issue altogether!

When Router sends an ARP request, its a broadcast, so Server will see it on both NIC1 and NIC2 as they are both within the same broadcast domain. By default Server will respond to the ARP request out of both interfaces because it has the requested IP on one of its local interfaces. So, sometimes when Router ARPs for 172.26.0.13, the reply that leaves NIC1 on Server arrives back at Router first, so Router sends packets to NIC1.

Using the scenario above with rp_filter set STRICT, packets arrive into NIC1 and the source would be reachable via NIC1 so everything just kinda works. However when ARP replies from NIC2 arrive back at Router first, we are back to Server dropping packets as invalid because of rp_filter being STRICT.

Now that we understand rp_filter a bit better and have changed it to LOOSE, we should probably make sure that Server only responds to ARP on the interface with the IP being requested. Whilst everything seemingly just works without doing this, we should make sure the flow of traffic is predictable and not randomly switching between interfaces based on which ARP reply arrives back first; so lets take a look back at the kernel documentation linked above and focus on arp_ignore

arp_ignore - INTEGER
	Define different modes for sending replies in response to
	received ARP requests that resolve local target IP addresses:
	0 - (default): reply for any local target IP address, configured
	on any interface
	1 - reply only if the target IP address is local address
	configured on the incoming interface
	2 - reply only if the target IP address is local address
	configured on the incoming interface and both with the
	sender's IP address are part from same subnet on this interface
	3 - do not reply for local addresses configured with scope host,
	only resolutions for global and link addresses are replied
	4-7 - reserved
	8 - do not reply for all local addresses

	The max value from conf/{all,interface}/arp_ignore is used
	when ARP request is received on the {interface}

So as above, lets set the value that best suits to what we need and make it persistent, in this case I chose 2

root@server:~# echo 2 > /proc/sys/net/ipv4/conf/all/arp_ignore
root@server:~# echo "net.ipv4.conf.all.arp_ignore = 2" >> /etc/sysctl.conf

We’ve now ensured that ARP replies are sent out of only the interface which has the IP being requested. Reverse Path Filter will no longer drop packets based on which interface the source is reachable and I can now talk to my server via my tunnelled IPs and everyone’s happy! :)

*Icons provided by https://icons8.com*

Odd behaviour with /etc/mtab being a regular file

Wed, 13 Jul 2016 19:53:35 +0000

One of our customers had recently requested a Bare-Metal Restore (BMR) of one of their servers; which is a pretty routine task for us. However, upon bringing the restored server up I’d noticed some odd behaviour with some of its services, notably snmpd.

Our monitoring successfully polls most metrics that we look for, however fails on getting disk statistics and eventually snmpd just starts timing out. Using snmpbulkwalk I could see that I was getting a response right up until midway through the HOST-RESOURCES MIB. It did look to be hanging on mount points and once snmpbulkwalk had timed out, I couldn’t get a successful response from snmpd again. This was also seemingly affecting MariaDB, preventing it from starting, amongst other things.

I tried to run df to check on disk usage/inodes, but this also hung. From previous experience this usually means one of the mount points is broken so I ran strace df which indicated what the problem was:

stat("/proc/sys/fs/binfmt_misc", ^C
Process 2961 detached

You can see from the above that the process stalled whilst trying to stat /proc/sys/fs/binfmt_misc. After a bit of Googling, a “quick and simple” fix for this was to un-mount and remount it. Note that a mount -o remount didn’t work and similarly just hung.

I’d found the command to mount this again at https://www.kernel.org/doc/Documentation/binfmt_misc.txt

umount -f -l /proc/sys/fs/binfmt_misc
mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc

This is all well and good, but I want a more permanent fix. What was causing this to fail consistently whenever the system boots?

I suspect this relates to the systemd-binfmt service, so should be handled pretty early in our boot process. Checking journalctl to see what’s going on with systemd yields the following error which jumped out at me:

Jul 12 08:08:01 myserver1.example.com systemd[1]: /etc/mtab is not a symlink or not pointing to
/proc/self/mounts. This is not supported anymore. Please make sure to replace this file by a
symlink to avoid incorrect or misleading mount(8) output.

Interesting…

Taking a look at /etc/mtab, this was indeed a regular file instead of a symlink. Being the likely cause for the observed mount issues (due to the system thinking things are mounted when in-fact they are not), this was removed and re-added as the correct symlink.

[root@myserver1 ~]# ls -al /etc/mtab
lrwxrwxrwx 1 root root 17 Jul 12 08:46 /etc/mtab -> /proc/self/mounts

The server was then rebooted and everything seemed to come back up without any issues.

This server was seemingly working fine before the BMR, but had only been up 3 months or so. I’ve checked other servers deployed at the same time with the same template and they are not affected which rules out our deployment system. I’ve also tested our BMR process again (Now that /etc/mtab has been fixed) but it comes back correctly, as a symlink.

The backup from which the BMR was originally performed has since unfortunately cycled out of retention so I can’t do much more debugging from that; This does mean that at the moment I’m not sure how this change came about for this particular server but I’m glad I’ve got to the bottom of its strange behaviour.

Growing my Linux RAID5 array

Sat, 09 Jul 2016 20:00:11 +0000

Recently I’ve needed to expand my RAID5 array by adding another disk. My experience with mdadm since starting to use it back in 2009 was for RAID1 only so I’ve never really needed to actually expand an array and so I thought I’d document the process here for future reference.

This process should in theory work for RAID4 and RAID6 personailities too, not just RAID5.

My array consists of 4x 3TB WD Red NAS drives in RAID5 for a total of ~9TB of space. I’m not going to cover creating of the array here because its mostly the same as RAID1, this instead will cover how to add a new disk to it and subsequent growing of LVM and ext4 to make use of the extra space. You can see how everything is stacked together in the image below.

There are a number of steps involved in expanding my array and I’ll be going through each one but loosely speaking they are:

Create a partition on the new drive to be inserted

Add the new drive to my array
Grow the array
Add the free space to LVM
Grow the logical volume
Resize ext4 to make use of the new space

I will admit, I had to do a little bit of brushing up on my mdadm and partitioning skills and I’d found these two sites useful:

https://raid.wiki.kernel.org/index.php/Growing

http://www.cyberciti.biz/tips/fdisk-unable-to-create-partition-greater-2tb.html

For the purposes of this article though, I’ve already installed the new disk into the chassis and made sure that my system can see the new drive before doing any further work; Lets take a look at my array before doing anything to it - You can see here I have about 9TB of space

root@yuzuki:~# df -h /dev/mapper/data-wdred
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/data-wdred 8.2T 7.5T 678G 92% /mnt/storage

So lets begin…

Partition the new drive using parted

I’d normally use fdisk for this because its what I am most familiar with, however it seems that you can’t create a partition greater than 2TB in size using it so I’ve had to use parted for this. The disk is a blank disk so your output may vary from the below if you’re working with something that is already partitioned, but first we create a GPT partition and set it to be 3TB in size (as we want to use the whole disk), we then check what parted thinks exists on the disk just to make sure everything is correct and thats the disk partitioned

root@yuzuki:~# parted /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt
Warning: The existing disk label on /dev/sda will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? Yes
(parted) unit TB
(parted) mkpart primary 0 3
(parted) toggle 1 raid
(parted) print
Model: ATA WDC WD30EFRX-68E (scsi)
Disk /dev/sda: 3.00TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name Flags
 1 0.00TB 3.00TB 3.00TB primary  raid

(parted) quit
Information: You may need to update /etc/fstab.

I set the “type” flag on the partition to be Linux RAID - Again this was out of habbit and I’m not really sure if it affects auto scanning of the array these days.

That should be everything for partitioning done - You should now see in your devices /dev/sda and /dev/sda1

Grow the array using Linux RAID (mdadm)

Now that we’ve partitioned the new disk to be used, we need to add it to our raid5 array. This part is simple enough and once I’d added it, I detailed the array and you can see the new disk as a spare near the bottom

root@yuzuki:~# mdadm --add /dev/md0 /dev/sda1
mdadm: added /dev/sda1

root@yuzuki:~# mdadm --detail /dev/md0
/dev/md0:
 Version : 1.2
 Creation Time : Wed Sep 9 00:42:04 2015
 Raid Level : raid5
 Array Size : 8790402048 (8383.18 GiB 9001.37 GB)
 Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
 Raid Devices : 4
 Total Devices : 5
 Persistence : Superblock is persistent

 Update Time : Sun Jul 3 13:42:19 2016
 State : clean
 Active Devices : 4
Working Devices : 5
 Failed Devices : 0
 Spare Devices : 1

 Layout : left-symmetric
 Chunk Size : 512K

 Name : yuzuki:0 (local to host yuzuki)
 UUID : 85414a00:04a22cb3:8cd5656f:8e86f737
 Events : 57988

 Number Major Minor RaidDevice State
 0 8 81 0 active sync /dev/sdf1
 1 8 33 1 active sync /dev/sdc1
 2 8 65 2 active sync /dev/sde1
 4 8 49 3 active sync /dev/sdd1

 5 8 1 - spare /dev/sda1

Great! If we left this as-is now then this new disk would simply be automatically used to rebuild the array if one of the current active disks fail, essentially RAID5 with a hot-spare. I however want to incorporate this disk into my array to be able to grow capacity.

I read that at the start of the growing process there’s a critical section where if power is interrupted this data can become lost which would kill the entire array so It’s recommended to back this up onto a different disk/array for recovery purposes. Thankfully I didn’t need to use this

Growing is also fairly straightforward in that you tell the array how many active drives you want (in my case 5) and which array to change and everything else is done in the background.

root@yuzuki:~# mdadm --grow --raid-devices=5 --backup-file=/home/simon/grow.bak /dev/md0
mdadm: Need to backup 6144K of critical section..

Actually growing the array is the most time consuming part of this process; Overall this next task took 21 hours to complete on my rig.

root@yuzuki:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sda1[5] sdf1[0] sdd1[4] sde1[2] sdc1[1]
 8790402048 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
 [>....................] reshape = 0.0% (402624/2930134016) finish=1333.9min speed=36602K/sec

unused devices: <none>

root@yuzuki:~# mdadm --detail /dev/md0
/dev/md0:
 Version : 1.2
 Creation Time : Wed Sep 9 00:42:04 2015
 Raid Level : raid5
 Array Size : 8790402048 (8383.18 GiB 9001.37 GB)
 Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
 Raid Devices : 5
 Total Devices : 5
 Persistence : Superblock is persistent

 Update Time : Sun Jul 3 13:46:39 2016
 State : active, reshaping
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
 Spare Devices : 0

 Layout : left-symmetric
 Chunk Size : 512K

 Reshape Status : 0% complete
 Delta Devices : 1, (4->5)

 Name : yuzuki:0 (local to host yuzuki)
 UUID : 85414a00:04a22cb3:8cd5656f:8e86f737
 Events : 58026

 Number Major Minor RaidDevice State
 0 8 81 0 active sync /dev/sdf1
 1 8 33 1 active sync /dev/sdc1
 2 8 65 2 active sync /dev/sde1
 4 8 49 3 active sync /dev/sdd1
 5 8 1 4 active sync /dev/sda1

As you can see here, the reshape has started and is giving me an estimate of ~22 hours - The estimate and speed will vary throughout the process and you can control the speed at which it works (if you need to lower it to reduce disk IO for some reason) via /proc/sys/dev/raid/speed_limit_min and /proc/sys/dev/raid/speed_limit_max. You can also see here that I no longer have a “spare” disk too; This means I’ll have 4 drives worth of capacity with 1 used for parity data, so when complete, I should have 12TB worth of storage.

Once the reshape has completed we can move onto the next stage.

Growing LVM

Now that our array itself has grown, we need to make LVM aware of the free space before we can use it and we do this simply by issuing a pvresize command on the block device mdadm uses for exposing our array, after which we can see that 2.7TiB is registered as free for both the Physical Volume and in the Volume Group too.

root@yuzuki:~# pvresize /dev/md0
 Physical volume "/dev/md0" changed
 1 physical volume(s) resized / 0 physical volume(s) not resized

root@yuzuki:~# pvscan
 PV /dev/sdb3 VG pve lvm2 [148.55 GiB / 16.00 GiB free]
 PV /dev/md0 VG data lvm2 [10.92 TiB / 2.73 TiB free]
 Total: 2 [11.06 TiB] / in use: 2 [11.06 TiB] / in no VG: 0 [0 ]

root@yuzuki:~# vgdisplay data
 --- Volume group ---
 VG Name data
 System ID
 Format lvm2
 Metadata Areas 1
 Metadata Sequence No 3
 VG Access read/write
 VG Status resizable
 MAX LV 0
 Cur LV 1
 Open LV 1
 Max PV 0
 Cur PV 1
 Act PV 1
 VG Size 10.92 TiB
 PE Size 4.00 MiB
 Total PE 2861458
 Alloc PE / Size 2146093 / 8.19 TiB
 Free PE / Size 715365 / 2.73 TiB
 VG UUID XT2yMd-iyVD-4u0e-5AG8-7GED-kcfd-ebLjNL

All thats left now is to expand the Logical Volume which I want to use all available free space in the Volume Group:

root@yuzuki:~# lvextend -l +100%FREE /dev/mapper/data-wdred /dev/md0
 Size of logical volume data/wdred changed from 8.19 TiB (2146093 extents) to 10.92 TiB (2861458 extents).
 Logical volume wdred successfully resized

Now we’re in the home stretch!

Expanding ext4 file system

The final leg of this task is to expand the file system that sits on top of all this.

I opted to do this online which means it ran without a consistency check - If you wanted to check it first you’d need to un-mount the file system in order to run e2fsck. I originally expected this to take a fairly long time but it turns out it took less than a minute to complete.

The image to the right illustrates where we’re at in the process - we’ve added the new disk to our RAID array and expanded LVM to fit, this last step will be to grow our ext4 file system to boot.

Naturally you don’t have to use ext4, this is just my preference however it is at this stage that your path could differ from mine if you’re not using ext4 (or ext3) but the process should be similar if your chosen file System supports growing.

root@yuzuki:~# time resize2fs -p /dev/mapper/data-wdred
resize2fs 1.42.12 (29-Aug-2014)
Filesystem at /dev/mapper/data-wdred is mounted on /mnt/storage; on-line resizing required
old_desc_blocks = 524, new_desc_blocks = 699
The filesystem on /dev/mapper/data-wdred is now 2930132992 (4k) blocks long.

real 0m35.221s
user 0m0.008s
sys 0m29.460s

And that’s it, My storage array has now grown! Overall the process took me about 22-23 hours because I kept getting distracted in-between but the majority of this time was waiting for mdadm to grow the array. We can compare output of df from before and after

Before

root@yuzuki:~# df -h /dev/mapper/data-wdred
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/data-wdred 8.2T 7.5T 678G 92% /mnt/storage

After

root@yuzuki:~# df -h /dev/mapper/data-wdred
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/data-wdred 11T 7.5T 3.4T 69% /mnt/storage

And I still have space for another 5 drives in my chassis :)

*HDD Icons provided by https://icons8.com*

Slow DNS resolving using bind9 as caching resolver

Sun, 26 Jun 2016 02:30:46 +0000

I currently have 4 DNS servers across my estate and until recently these were all configured to forward all queries to Google DNS (8.8.8.8). I ended up having an issue with Google caching an undesired record value so I opted to change my DNS servers so that they no longer forward queries elsewhere, but instead try to answer it themselves; Doing this gives me slightly more control over my DNS cache.

As I use named (bind9) this was a pretty trivial change - Simply remove the forwarders { 8.8.8.8; }; clause in my configuration and that should be that.

During my post-change testing though I’d noticed that resolution was taking significantly longer for un-cached queries than I’d expect (microsoft.gointeract.io is only used to illustrate my issue):

root@ns3:~# rndc flush && dig microsoft.gointeract.io

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> microsoft.gointeract.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40048
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 9

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;microsoft.gointeract.io. IN A

;; ANSWER SECTION:
microsoft.gointeract.io. 300 IN CNAME interact-utm.cloudapp.net.
interact-utm.cloudapp.net. 60 IN A 23.101.130.247

;; AUTHORITY SECTION:
cloudapp.net. 172799 IN NS prd3.azuredns-cloud.net.
cloudapp.net. 172799 IN NS prd1.azuredns-cloud.net.
cloudapp.net. 172799 IN NS prd2.azuredns-cloud.net.
cloudapp.net. 172799 IN NS prd4.azuredns-cloud.net.

;; ADDITIONAL SECTION:
prd1.azuredns-cloud.net. 172799 IN A 204.79.195.43
prd1.azuredns-cloud.net. 172799 IN AAAA 2a01:111:2005:5::5
prd2.azuredns-cloud.net. 172799 IN A 65.55.117.43
prd2.azuredns-cloud.net. 172799 IN AAAA 2a01:111:2006:c::5
prd3.azuredns-cloud.net. 172799 IN A 204.79.195.48
prd3.azuredns-cloud.net. 172799 IN AAAA 2a01:111:2020:3::5
prd4.azuredns-cloud.net. 172799 IN A 65.55.117.48
prd4.azuredns-cloud.net. 172799 IN AAAA 2a01:111:2032:1::5

;; Query time: 4502 msec
;; SERVER: 10.0.2.13#53(10.0.2.13)
;; WHEN: Sat Jun 25 17:08:38 BST 2016
;; MSG SIZE rcvd: 374

The entire query took 4.5 seconds which seems unusually long. I was seeing similar Query times for my other DNS servers too and they’re located in other facilities dotted around the country so I set about trying to find out why.

Using tcpdump I could quickly see a potential cause for the delay…

17:08:35.729948 IP6 fe80::6432:36ff:fe38:3536.20251 > 2a01:111:2005:5::5.53: 1759% [1au] A? interact-utm.cloudapp.net. (54)
17:08:36.530082 IP6 fe80::6432:36ff:fe38:3536.21384 > 2a01:111:2020:3::5.53: 33150% [1au] A? interact-utm.cloudapp.net. (54)
17:08:37.330127 IP6 fe80::6432:36ff:fe38:3536.22815 > 2a01:111:2006:c::5.53: 49009% [1au] A? interact-utm.cloudapp.net. (54)
17:08:38.130220 IP6 fe80::6432:36ff:fe38:3536.4519 > 2a01:111:2032:1::5.53: 3339% [1au] A? interact-utm.cloudapp.net. (54)
17:08:38.930305 IP 46.xx.xx.xx.1965 > 204.79.195.43.53: 13538% [1au] A? interact-utm.cloudapp.net. (54)
17:08:38.937824 IP 204.79.195.43.53 > 46.xx.xx.xx.1965: 13538*- 1/0/1 A 23.101.130.247 (70)

This shows my server sending queries to an IPv6 address using its link-local address (IPv4 obfuscated).

I do not have IPv6 configured for anything yet
The link-local address isn’t public routable

Here we can see named is waiting for a short timeout period before moving onto the next address to query - It eventually gets a response on its IPv4 address. Each IPv6 address in the recursion chain will add more delay.

By default named works with both IPv4 and IPv6, but it seems that becuase my system *seems* IPv6 capable, it tries to use it regardless which causes a delay in resolution. As I don’t yet have working IPv6 connectivity everywhere, I opted to disable IPv6 in named itself for the time being.

Looking at named manual, simply invoking the daemon with -4 should do the trick. As I am running Ubuntu, I added it to named’s defaults:

root@ns3:~# cat /etc/default/bind9
# run resolvconf?
RESOLVCONF=no

# startup options for the server
OPTIONS="-u bind -4"

Once added, restart named and you should see much improved resolving times:

root@ns3:~# rndc flush && dig microsoft.gointeract.io

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> microsoft.gointeract.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42883
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 9

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;microsoft.gointeract.io. IN A

;; ANSWER SECTION:
microsoft.gointeract.io. 300 IN CNAME interact-utm.cloudapp.net.
interact-utm.cloudapp.net. 60 IN A 23.101.130.247

;; AUTHORITY SECTION:
cloudapp.net. 172800 IN NS prd4.azuredns-cloud.net.
cloudapp.net. 172800 IN NS prd1.azuredns-cloud.net.
cloudapp.net. 172800 IN NS prd2.azuredns-cloud.net.
cloudapp.net. 172800 IN NS prd3.azuredns-cloud.net.

;; ADDITIONAL SECTION:
prd1.azuredns-cloud.net. 172800 IN A 204.79.195.43
prd1.azuredns-cloud.net. 172800 IN AAAA 2a01:111:2005:5::5
prd2.azuredns-cloud.net. 172800 IN A 65.55.117.43
prd2.azuredns-cloud.net. 172800 IN AAAA 2a01:111:2006:c::5
prd3.azuredns-cloud.net. 172800 IN A 204.79.195.48
prd3.azuredns-cloud.net. 172800 IN AAAA 2a01:111:2020:3::5
prd4.azuredns-cloud.net. 172800 IN A 65.55.117.48
prd4.azuredns-cloud.net. 172800 IN AAAA 2a01:111:2032:1::5

;; Query time: 231 msec
;; SERVER: 10.0.2.13#53(10.0.2.13)
;; WHEN: Sat Jun 25 17:10:29 BST 2016
;; MSG SIZE rcvd: 374

Much better :)

PHP Handlers and PHP 5 vs 7

Sat, 11 Jun 2016 22:39:47 +0000

The idea for this post actually came from observing performance after moving this site over to WordPress (and to a lesser extent, my other website, General Photography). I use Puppet orchestration for my servers and I made the mistake of trying to convert everything to puppet manifests and also get things working how I wanted without fully understanding how puppet works at the same time; That however is another story. This does means that my puppet manifests are not very flexible currently and I use suphp as my main PHP handler just because for me, at the time, it was easier.

That being said - I’m fully aware of the performance issues suphp exhibits because my Observium install used it at first - it now uses mod_php because loading all the graphs was noticably slow. I thought I’d try to quantify the performance differences between suphp and mod_php and decided I should do all 3 common PHP handlers. Whilst I was gathering metrics for this I decided It’d be useful to include PHP7 results too.

The Environment

I tried to ensure my testing was fair so I’ll detail my setup. Firstly I cloned the container my website runs on 3 times. One so I can test suphp, one for mod_php and one for php-fpm.

I will be performing these tests against different instances of my website which is based on WordPress 4.5.2

The first container was already set to go as I use suphp already - it was an exact clone of my website container. The second container I’d removed suphp and installed mod_php and for the final container, I’d removed suphp and installed mod_fastcgi and also set up php-fpm. The intial round of testing was done on these 3 containers which were then upgraded to PHP7 and the same round of testing done again.

I made sure the containers were running on the same host as my main website container to ensure similar performance as my live site and for results to be comparable to each other. They all use the same database container too so thats one less thing to worry about.

Host Server Spec

CPU: Intel(R) Xeon(R) CPU L3360 @ 2.83GHz 4C/4T

RAM: 8GB DDR2 PC2-6400 (800 MHz)

HDD: 2x 1TB WD-Black (WD1002FAEX) in RAID1

OS: Proxmox 4.2-5/7cf09667

Kernel: 4.4.6-1-pve

Container Spec

Contianer Type: LXC

OS: Ubuntu 14.04.1 (Yes, I know it needs updating!)

RAM: 1GB

CPU Limit: 2

CPU Units: 1024

Software Versions

Apache Version: 2.4.7

PHP 5.5.9-1ubuntu4.17

PHP 7.0.7-4+deb.sury.org~trusty+1

WordPress 4.5.2

MySQL: 5.5.41-0ubuntu0.14.04.1 (Although this is hosted in its own container which remains the same through the testing)

Configuration

Apache Mode: mpm_prefork

Apache MaxClients: 20

Apache ServerLimit: 20

Apache MaxRequestsPerChild: 4000

php-fpm Process Manager: dynamic

php-fpm max_children: 20

The Testing

I first started by using curl to send 20 sequential GET requests to index.php via Apache and timing how long it takes to send me the generated contents of the page and then averaged it. For comparison purposes I also recorded the length of time PHP took to generate the page by invoking it on command line via php-cgi -f index.php, again taking an average.

You can see from the chart above that suphp takes longer to return results than just invoking on the command line but this makes sense because Apache has to process the request and spawn a PHP process to fully deal with it along with network overheads between my test client and test server. PHP7’s performance enhancements are visible here too with all handlers using PHP7 resulting in faster loading times. For both PHP5.5 and PHP7, mod_php and php-fpm seem to share similar performance.

The configuration section above is important for our next test. I limited the number of processes that could spawn so as to not take out my container by putting it under too high load specifically for mod_php - Whilst you can tweak those options to suit your liking/environment, each handler have different things to consider - primarily how much RAM they consume.

For both suphp and php-fpm handlers, they offload PHP to another process. For the particular modules I had configured in Apache, on average each process for both suphp and php-fpm tests seemed to hover around 7.5MB - for mod_php however things are a bit different. mod_php is a module that’s loaded by Apache which allows it to interpret PHP itself without having to pass it off to another process, which is by far one of the reasons mod_php performs so well however this does mean that for each Apache process created, it has to load this module every time which makes this a rather RAM heavy approach - During testing of mod_php I’d clocked average Apache RAM usage at around 54MB per process so for all 20 processes running, that’d all but saturate RAM on my container.

php-fpm is definately the way to go if RAM is your concern. Whilst it does maintain a separate pool of processes ready to process PHP, its not constantly spawning/killing processes like suphp does and doesnt have to contend with other items in RAM like Apache does making it leaner.

We’ve seen above how each handler performs just by processing a simple request, lets take a look what they look like under a bit of load…

In order to generate some load, I’d used a tool called siege in benchmark mode. Simply put, you point it at a URL (or a list of URLs), tell it how many concurrent “users” you want and give it some criteria for how long to run. In my case I used

siege -t1m -b -c150 https://www.simonmott.co.uk/

which would simulate 150 concurrent users hitting my website for 1 minute. Once a “user” has sent and received a successful (or failed) request, it immediately sends a new one. Testing in this method illustrates how quickly each handler can actually deal with requests (and shows off the performance improvements for PHP7 quite nicely).

To illustrate how each handler manages to cope with requests I plotted average, longest and shortest response times which shows especially how suphp, along with the low hit-rate, just struggles to keep up because of having to constantly spawn new processes, on top of that, because the processes are short lived (for the duration of the request), they do not benefit from any kind of opcode caching as mod_php and php-fpm do.

Conclusion

Another major pitfall of mod_php aside from its RAM consumption is that all PHP it processes is done so as the user Apache runs as which isn’t ideal for a multi-site environment because all sites need to be accessed (and modified, depending on your application) by your Apache user. This simply means in a multi-site environment that if someone compromises one of your websites, they can potentially glean database username/passwords for other sites accessible by this user and/or other sensitive information depending on your setup.

suphp is probably one of the easiest ways to afford more security over your websites because each process it spawns to deal with php, is done so as the user that owns the php file its trying to execute. This means that as long as your file permissions are sane, you shouldn’t be able to read file contents of other users but you just take a major performance hit because of the way it works.

This leaves php-fpm. This by far is the best way to go if you put the time into configuring it. This handler allows pools of processes to run as specific users so grants you the same level of security as suphp, but it does so by keeping processes around much longer than suphp which means it benefits from not having the overheads of spawning new processes for each request, and can make use of opcode caching to further enhance performance along with better resource management in general.

The performance improvements for PHP7 are not something to ignore either - In the case of this WordPress blog, simple page loading time was 2x faster via both mod_php and php-fpm using the same settings. Furthermore as shown above, PHP7 was able to handle 3x more requests in 1 minute with better average loading times than the same settings using PHP5.5.

For me, the clear winner here is php-fpm, no matter which version of PHP you decide to roll

Caching with Apache's mod_cache and mod_deflate

Mon, 06 Jun 2016 18:21:49 +0000

I currently work in the hosting industry and as part of my job I have to deal with WordPress on a regular basis and as such I’ve seen how appalling it can be sometimes with regards to page loading times. Given that my content on here doesn’t change all too often, it makes sense to spend the time generating the content only once and serving it to meet multiple requests for the same document. In order to do this one would usually employ some form of caching.

WordPress itself does have the option for multiple caching plugins which integrate nicely but in my experience they still have to pass the request off to a PHP handler which means some processing is still needed to accommodate the request (albeit less than a full page load). For the purposes of this article though, I’m going to assume a generic framework as this technique is not specific to WordPress at all.

There’re a number of software solutions to sit in front of your web server which act as a cache and some of the more notable ones are nginx and varnish and each have their pros and cons (As far as I know, Varnish can’t do native SSL termination yet). For this post though, I’ll be making use of some Apache modules to do the same thing. This was mostly an experiment for myself to see what I could do with it - I’ve found some limitations which as of yet I haven’t been able to work around for lack of knowledge on the inner workings of Apache, but for now It does the job.

I’ll probably be doing an article on varnish and/or nginx in the future, but for now lets focus on mod_cache and mod_cache_disk.

Lets start by taking a benchmark of how long the homepage takes to load without cache. The following will hit my homepage 5 times with a 1 second delay between each request

simon@kinmu:~$ i=0; while [ $i -lt 5 ]; do time -p curl "https://www.simonmott.co.uk/" > /dev/null; sleep 1; i=$[$i+1]; done 2>&1 | grep real | awk '{print $2}' | awk '{avg += ($1 - avg) / NR;} END {print "Average: " avg "s";}'
Average: 0.696s

This gives us an average of 0.696s to load the page… This is the time to beat :)

Before we begin - I encourage you to read the Apache documentation for mod_cache and mod_cache_disk - it will make a lot more sense if you do!

mod_cache - The config

I’ll start by showing you my current config - I’ll then go through each option and comment as to why I made the decision to use that particular setting/option. This snippet of config lives in my VirtualHost directive for my website and I suspect a lot of my comments will essentially be copy/pasted from apache.org

CacheQuickHandler off

CacheRoot /var/cache/apache2/mod_cache_disk/simonmott.co.uk
CacheEnable disk /
CacheDirLevels 2
CacheDirLength 1
CacheMaxFileSize 2000000

CacheIgnoreNoLastMod On
CacheDefaultExpire 7200
CacheIgnoreCacheControl On
CacheLastModifiedFactor 0.5
CacheIgnoreHeaders Set-Cookie Cookie
CacheHeader on

CacheLock on

CacheDisable /wp-admin
CacheDisable /wp-login.php
CacheDisable /wp-cron.php

SetOutputFilter CACHE
AddOutputFilterByType DEFLATE text/html text/plain text/css application/javascript application/rss+xml text/xml image/svg+xml

The config explained

OK, we’ve got the config, now lets go through it

CacheQuickHandler off

The CacheQuickHandler directive controls the phase in which the cache is handled.

In the default enabled configuration, the cache operates within the quick handler phase. This phase short circuits the majority of server processing, and represents the most performant mode of operation for a typical server. The cache bolts onto the front of the server, and the majority of server processing is avoided.

When disabled, the cache operates as a normal handler, and is subject to the full set of phases when handling a server request. While this mode is slower than the default, it allows the cache to be used in cases where full processing is required, such as when content is subject to authorization

We currently don’t want the cache to be processed “quickly” for reasons we’ll explain later ;)

CacheRoot /var/cache/apache2/mod_cache_disk/simonmott.co.uk

This option simply defines where on disk the cache will be stored - The default for my installation is /var/cache/apache2/mod_cache_disk but I opted to split the cache by vhost so that I have slightly finer control (For example, I can completely clear the cache for my website by removing the contents of /var/cache/apache2/mod_cache_disk/simonmott.co.uk which shouldn’t affect the cache for another website)

CacheEnable disk /

This option simply enables the cache for any URL under this domain. If you wanted to only cache for simonmott.co.uk/test then you would use “/test” instead of “/”

CacheDirLevels 2
CacheDirLength 1
CacheMaxFileSize 2000000 # <-- This is 2MB

This set of options controls how many files can be stored on disk and the max-file size that can be committed to cache. CacheDirLevels specifies how many levels of subdirectory there should be, and CacheDirLength specifies how many characters should be in each directory. With the example settings given above, the hash would be turned into a filename prefix as /var/cache/apache2/mod_cache_disk/simonmott.co.uk/x/y/TGxSMO2b68mBCykqkp1w.

The overall aim of this technique is to reduce the number of subdirectories or files that may be in a particular directory, as most file-systems slow down as this number increases. With setting of “1” for CacheDirLength there can at most be 64 subdirectories at any particular level. With a setting of 2 there can be 64 * 64 subdirectories, and so on. Unless you have a good reason not to, using a setting of “1” for CacheDirLength is recommended.

Setting CacheDirLevels depends on how many files you anticipate to store in the cache. With the setting of “2” used in the above example, a grand total of 4096 subdirectories can ultimately be created. With 1 million files cached, this works out at roughly 245 cached URLs per directory. CacheMaxFileSize controls the max size of a file that can be stored (in bytes).

CacheIgnoreNoLastMod On
CacheDefaultExpire 7200

The CacheIgnoreNoLastMod directive provides a way to specify that documents without last-modified dates should be considered for caching, even without a last-modified date. If neither a last-modified date nor an expiry date are provided with the document then the value specified by the CacheDefaultExpire directive will be used to generate an expiration date.

CacheIgnoreCacheControl On

CacheIgnoreCacheControl On tells the server to attempt to serve the resource from the cache even if the request from a client contains no-cache header value

CacheLastModifiedFactor 0.5

In the event that a document does not provide an expiry date but does provide a last-modified date, an expiry date can be calculated based on the time since the document was last modified with the CacheLastModifiedFactor directive

CacheIgnoreHeaders Set-Cookie Cookie

CacheIgnoreHeaders specifies additional HTTP headers that should not to be stored in the cache. For example, it makes sense in some cases to prevent cookies from being stored in the cache.

CacheHeader on

When the CacheHeader directive is switched on, an X-Cache header will be added to the response with the cache status of this response.

CacheLock on

The CacheLock directive enables the thundering herd lock for the given URL space.

CacheDisable /wp-admin
CacheDisable /wp-login.php
CacheDisable /wp-cron.php

The CacheDisable directive instructs mod_cache to not cache urls at or below url-string. The values I’ve chosen here ARE specific to WordPress

The last two lines in my config for SetOutputFilter and AddOutputFilterByType is where things get interesting and this one requires some more in-depth explanation…

mod_deflate and the Vary header

Because of where mod_cache initially sits, if you enable mod_deflate and your visitor sends appropriate “Accept-Encoding” headers then mod_deflate will dutifully compress the output for you, but this happens before mod_cache saves it. This isn’t a big problem because mod_deflate adds a “Vary: accept-encoding” header to the output, thus mod_cache will store a separate cached copy of your content for each variation of the “Accept-Encoding” header it encounters. This method does mean that ALL processing is cached - dynamic content generated by PHP is saved, so no need to process the request again and CPU time to compress the output has already been done which to some is ideal and you can stop here without adding the last two lines above.

I however don’t like the idea of multiple copies of the same page - If I visit a page on my Desktop using Chrome, I want the same page when I visit on my Phone using Chrome. The problem here is that Chrome for desktop sends Accept-Encoding: gzip, sdch whereas Chrome for Android sends Accept-Encoding: gzip, sdch, br which is a different variation of the header, thus they get two separate copies processed and cached, despite having an already cached copy that is of suitable encoding. Something to also consider is the vector for a DoS by filling my server with files based on lots of different Accept-Encoding headers

Depending on how popular your website is, you could end up with hundreds of copies for a single page cached because of the differing headers and for my site, that just doesn’t sit right with me. Documentation here suggests you can defer mod_deflate until after your content has been saved in the cache, however this doesn’t seem to work - It spits an error about duplicate “CACHE” filters and just continues as normal, compressing then caching.

[cache:debug] [pid 9497] mod_cache.c(1718): [client 10.0.0.20:40156] AH00777: cache: CACHE filter was added twice, or was added where the cache has been bypassed and will be ignored: /

I have a work-around, but it does have a caveat which I’ll get to shortly - I have also approached Apache httpd mailing lists, but as of yet I don’t have a solid fix for this but for now the workaround is:

SetOutputFilter CACHE
AddOutputFilterByType DEFLATE text/html text/plain text/css application/javascript application/rss+xml text/xml image/svg+xml

To the best of my knowledge (as the docs are a bit sketchy here), SetOutputFilter will force all output to be passed through the CACHE filter; Doing this here ensures content is saved to our cache first and we then conditionally pass through DEFLATE for text-based content (As that generally compresses well). Sounds exactly like what I want, right?

Well… Not quite.

It seems that doing this combination of output filters means that content is actually being saved into our cache before mod_expires has chance to add correct cache-control headers and specific expires headers. This just means that all content is being cached for the duration of CacheDefaultExpire, instead of what I try to set via mod_expires. The headers do eventually get added though so at least our visitors browsers will cache content correctly :)

For the time being I can live with this behaviour, but as mentioned earlier, I will probably be looking to move my web stack over to nginx so this will suffice until then.

Conclusion

So, we have caching all set up and working, but is it making a difference? Well - Once a page is requested, but its not cached (a MISS), the request will process as normal and return output to the visitor. Subsequent requests are served from cache so no processing is needed.

Once all this was set up, I visited my website which generated the page and stored it in cache - My average page loading time from earlier was 0.696s, so lets see what it is now its in cache

simon@kinmu:~$ i=0; while [ $i -lt 5 ]; do time -p curl "https://www.simonmott.co.uk/" > /dev/null; sleep 1; i=$[$i+1]; done 2>&1 | grep real | awk '{print $2}' | awk '{avg += ($1 - avg) / NR;} END {print "Average: " avg "s";}'
Average: 0.036s

Thats an impressive 19x improvement on average page loading time!

I’d say its working nicely ;)

Full screen Flash Video on Linux

Mon, 05 Aug 2013 00:01:23 +0000

So here we have it - I’ve finally got around to tackling this issue which has been bugging me for a while.

When watching a flash video in full screen on dual monitors, as soon as you go off to do something else you will soon notice that your video is no longer full screen! How annoying.

The way to fix this is to edit the flash player binary. Firstly you need a hex editor - I use “ghex”

$ sudo apt-get install ghex

You can of course use your hex editor of choice.

Now that we have that sorted, its time to find our flash binary! Start playing a flash video and then see which plugin is open.

$ lsof -n | grep -i flash
 firefox 4655 simon mem REG 8,1 17418724 1966425 /usr/lib/flashplugin-installer/libflashplayer.so

Open this file with your hex editor and look for the following string:

_NET_ACTIVE_WINDOW

Once you have this found, change any of the letters - In my case I changed it to:

__ET_ACTIVE_WINDOW

Save and re-open your browser and you should be able to watch full screen video in flash player whilst doing other things!

This will need to be re-applied every time you update flash.

A final note about this modification:

Chrome (not Chromium) uses its own built in flash plugin called PepperFlash which doesn’t have “_NET_ACTIVE_WINDOW”, so if you want this fix to work with Google Chrome, enter “chrome://plugins/” in the url bar, then click “Details” on the right, scroll to the Adobe Flash Player plugin and 2 versions should be listed; disable the one that has “/opt/google/chrome/PepperFlash/libpepflashplayer.so” as its path. Doing this, Google Chrome will use the system Adobe Flash Player and not the built-in Google Chrome Adobe Flash Player.

Enjoy

UN2400 Mobile Broadband on Ubuntu 11.10

Sun, 11 Mar 2012 09:09:32 +0000

I have a HP Compaq Mini 311c-1030SA netbook with ION. I decided to encrypt it the other day just on the off chance it was stolen that way I would be happy that none of my data would be lost.

It had the orignal OS (Windows XP Home) and Kubuntu Linux in a dual boot configuration. Windows encrypted nicely with TrueCrypt and I decided to start fresh with Linux as the version I had installed was fairly old and as far as I am aware there is no way to on the fly encrypt a currently installed distro.

Lets just say that due to a slip of the hand I somewhat destroyed my encrypted Windows partition! (Foolishly, one of the first things I did when i got the netbook a few years ago was remove the recovery partition for more HDD space too)

This isn’t really a big problem for me as I hardly ever used the Windows install. Here is the clincher though, the netbook has a Qualcomm UN2400 Mobile Broadband chip which requires the firmware to be uploaded to the chip based off what country/carrier you plan on using. This firmware and some of the ppp configuration scripts are installed onto the Windows partition… which I had just destroyed!

Getting the firmware was not a problem as I downloaded the driver from the HP website and extracted the .msi file to get what I needed. For convenience the firmware can be obtained here.

Now that I have the firmware images on my netbook again I can follow the guide written by my good friend, Dataforce. This can be located on his website, blog.dataforce.org.uk

To save you all some time here is what I did to get mine working (Mostly taken from Dataforce’s website):

# Install gobi-loader
 sudo apt-get install gobi-loader

# Create missing directory
 sudo mkdir -p /lib/firmware/gobi

# Use full paths for gobi_loader
 sudo sed -i 's@"gobi_loader@"/lib/udev/gobi_loader@' /lib/udev/rules.d/60-gobi.rules

# Copy firmware to the device
 sudo cp /home/simon/HP/UMTS/amss.mbn /lib/firmware/gobi/
 sudo cp /home/simon/HP/UMTS/apps.mbn /lib/firmware/gobi/
 sudo cp /home/simon/HP/6/uqcn.mbn /lib/firmware/gobi/

# Stop modemmanager corrupting the firmware whilst it is being uploaded by blacklisting the non-modem version of the device
 echo 'ATTRS{idVendor}=="03f0", ATTRS{idProduct}=="241d", ENV{ID_MM_DEVICE_IGNORE}="1"' >> /lib/udev/rules.d/77-mm-usb-device-blacklist-custom.rules

As a breif explanation I use the image in the “6” folder as it is listed on HP’s Support Pages as “Generic UTMS (Europe)” however I suggest you take a look and choose whats best for you.

Restart your machine and check you have the correct number of ttyUSB devices

$ ls /dev/ttyUSB*
 /dev/ttyUSB0 /dev/ttyUSB1 /dev/ttyUSB2

The hard part is done! All that is left is, in my case at least, to use the build in KDE NetworkManager check that the “Mobile Broadband” tab is no longer disabled and follow the wizard for creating a new mobile broadband connection for your contry/carrier.

VPN Bonding

Mon, 05 Mar 2012 13:44:22 +0000

This page details step by step how I accomplished aggregating two separate internet connections into one, seemingly single, connection to the Internet. This “single” connection has a greater bandwidth capacity that either of the individual connections could yield. As a side affect this also nicely doubles up as a redundant connection. This method can be used for achieving link aggregation for both home and business users alike for a fraction of the cost of commercial connections / aggregation units available that do the same thing and in theory could be expanded to as many links you like however be warned, the more links there are, the more overheads there will be with tunnelling; thus potentially noticeable drops in speed.

For this demonstration I am using two Virtual Machines via VirtualBox. This enables me to experiment with without cutting my servers off from remote administration. Both machines are running Ubuntu 10.04.3 LTS (Lucid) however there is no reason this wont work for other distributions as long as you modify the steps accordingly.

Just to get this out of the way. I am not responsible for anything you might break. Be cautious when playing around with routing tables/network interfaces especially with remote servers as it could potentially cause loss of connectivity which might not be a simple fix.

Requirements

Firstly, we will cover the requirements needed for this demonstration:

Linux bonding driver on both machines (see Initial Setup)
Server with a fat internet connection. I have a server in a Data Center with 100Mbit link that I use.
The remote server ideally needs to have two or more IP addresses accessible (see routing for details why)
Two or more Internet connections at the site you wish to create the aggregated connection
Server at the site you wish to aggregate that you can use as a Gateway device

*NB, This article is now quite old. I’ve heard that newer versions of ifenslave break this now*

How It Works

The idea behind this is quite simple and has been around for a long time. Think of it in terms of plumbing. You have a narrow pipe from your house (Your ISP), to the drain outside (The Internet). This pipe can only carry so much water (Information) to its destination at any given moment. If you add another pipe and link them together you are essentially allowing both pipes between them to carry more to the destination, thus increasing throughput.

The only initial problem with this is that each of the connections to your ISP have their own routes and IP addresses. You cant simply have multiple connections to one ISP and have it magically work (Unless your ISP supports Channel Bonding and will set it up for you). What this guide aims to do (continuing from the analogy above), is wrap both pipes together with tape so that from inside the house and outside, they appear as one and carry more down tham then a single pipe on its own.

The solution is to create a VPN tunnel to your fat pipe endpoint via each connection. Once the tunnels have been bonded it will act as one interface. Your remote fat pipe server will appear as one IP address but all traffic to that new IP address is being split down both links and vice versa. Once at the other end, as its VPN tunnelled, your server will see it as originating from the same IP inside your network, regardless of the route it takes. This then enables you to route it to the Internet and back with ease.

Diagram

This diagram summarizes the setup we are trying to achieve:

Initial Setup

First off we’ll install the bonding driver onto both servers. This is nice and simple: (Note # represents a command as root)

# apt-get install ifenslave

As we want to aggregate the multiple connections into one seemingly single link, I will be using Bonding mode 1, aka balace-rr. To enable this, we need to edit the /etc/network/interfaces file and add the following:

Note that my local subnet is 10.0.0.0/8 and so it is clear when I am using the tunnel, my VPN subnet will be 172.26.0.0/30

local-server# pico /etc/network/interfaces

#This should be added to the config file
 iface bond0 inet static
 address 172.26.0.2
 netmask 255.255.255.252
 bond-slaves tun0 tun1
 bond_mode balance-rr

And the remote server:

remote-server# pico /etc/network/interfaces

#This should be added to the config file
 iface bond0 inet static
 address 172.26.0.1
 netmask 255.255.255.252
 bond-slaves tun0 tun1
 bond_mode balance-rr

As a brief explanation, balance-rr does not load balance connections but actually load balances the packets.

This might seem backwards to some people but when all is said and done, we will be forwarding all internet traffic to 172.26.0.1 thus that would be our gateway. As a rule of thumb gateway’s are always the first IP in the subnet… (Usually)

Routing

For this to work, we need to ensure that we have separate connections to our endpoint. These separate connections need to travel through each internet connection. For arguments sake, I have two ADSL connections and thus two routers on my network - 10.0.0.1 and 10.0.0.254.

My fat pipe endpoint also needs to have two IP addresses. For this example - 87.117.217.27 and 87.117.217.44. These will enable me to ensure that my tunnels are routed via separate ISPs.

On my local server I need to do the following:

local-server# ip route add 87.117.217.27 via 10.0.0.1 dev eth0
local-server# ip route add 87.117.217.44 via 10.0.0.254 dev eth0

Note: If you want to aggregate more than two connections, using this method you would need the same number of remote IP addresses as you do Internet connections to ensure that traffic is sent via each interface.

Please be aware that this is by no means an elegant solution…

Creating VPN Tunnels - Using SSH

I originally decided to use SSH tunnels when I was playing around with bonding as both of my servers already had OpenSSH installed which meant no messing about installing/configuring OpenVPN (SSH is so versatile!). The only caveat to this method is that it requires you to allow SSH Logins via Root. It is also worth noting that using SSH Tunnels for the VPN connections essentially means you are tunnelling TCP over TCP which can potentially have a lot of overheads, especially via SSH due to the encryption and compression should you use it. See this article for some more information.

At this stage I’d like to point out that whilst I did use SSH, you can potentially use any VPN/Tunnelling technology such as OpenVPN - as long as you get a tun/tap interface that you can use with the bonding driver. That is however out of the scope of this article. If you will be using another technology or protocol to create your tun or tap devices then you can skip ahead to Bonding the links

As mentioned at the top of this page, my end point is in a Data Center. It is however a VM running on my server. Primarily this was to prevent me from losing access to my machine if I broke things however it also adds the security of, if the machine is compromised, the intruder doesn’t have access to any of my services.

Never the less, we will secure it as much as possible… You need to edit your /etc/ssh/sshd_config file on your Remote Server and change/add the following:

PermitRootLogin without-password
PermitTunnel yes
PasswordAuthentication no

Once that is done, add some keys from your local server to your remote server so that you can ssh to the root account. More information on SSH Keys can be found here or simply by googling.

Once you have confirmed that you can SSH to the root account of the remote server from the root account of your local server you are ready to go.

We will be making use of the -w parameter of ssh. This will create a tunX interface on both ends that we will later bond together.

local-server# ssh -NTCf -w 0:0 87.117.217.27
local-server# ssh -NTCf -w 1:1 87.117.217.44

-N Do not execute a remote command. This is useful for just forwarding ports (or in this case tunnels).
-T Disable pseudo-tty allocation.
-C Requests compression of all data (including stdin, stdout, stderr, and data for forwarded X11 and TCP connections). Compress can speed up or potentially slow down your speed, so try with and without.
-f Requests ssh to go to background just before command execution.

If this is successful, ifconfig -a on both machines should reveal a tun0 and tun1 interface!

# ifconfig -a
 tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
 POINTOPOINT NOARP MULTICAST MTU:1500 Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500
 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

tun1 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
 POINTOPOINT NOARP MULTICAST MTU:1500 Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500
 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Great! Now we can proceed to bonding them.

NOTE: It might be advisable at this time to check netstat to ensure that you have connections coming in from separate ISPs:

remote-server# netstat -an | grep ":22" | grep ESTABLISHED
 tcp 0 48 87.117.217.27:22 66.118.190.24:54284 ESTABLISHED
 tcp 0 0 87.117.217.27:22 78.33.121.161:50165 ESTABLISHED
 tcp 0 0 87.117.217.44:22 92.40.207.249:56158 ESTABLISHED

The bold addresses are my tunnels and the other is a terminal I have open to execute netstat. The command I used shows only connections on port 22 (default ssh port) and will only show Established ones. Looks like we are good to go!

Bonding the links

This part is relatively easy now all the heavy lifting is done. As mentioned above, we have created a bond0 interface on both servers that have two (or potentially more) slaves, tun0 and tun1. To bring this interface up we simply execute the following on both servers

This should be done on both servers:

# ifup bond0
 # ifconfig bond0
 bond0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
 inet addr:172.26.0.X Bcast:172.26.0.3 Mask:255.255.255.252
 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
 RX packets:41 errors:0 dropped:0 overruns:0 frame:0
 TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:8054 (8.0 KB) TX bytes:168 (168.0 B)

Presto! Both interfaces are now up and running. You can test to ensure it is working by trying to ping each other

# ping 172.26.0.1
 PING 172.26.0.1 (172.26.0.1) 56(84) bytes of data.
 64 bytes from 172.26.0.1: icmp_seq=1 ttl=64 time=28.3 ms
 64 bytes from 172.26.0.1: icmp_seq=2 ttl=64 time=28.2 ms
 64 bytes from 172.26.0.1: icmp_seq=3 ttl=64 time=25.9 ms

Next you ned to remove the default gateway of your local server (if there is one) and tell it to use the remote server’s VPN address as its gateway instead:

local-server# ip route del 0.0.0.0/0
local-server# ip route add 0.0.0.0/0 via 172.26.0.1 dev bond0

All that is left is to enable ipv4 forwarding on both servers and add a route back to your internal network on the remote server and then you should be all set for using the local server as a gateway for the internet!

To enable:

This should be done on both servers:

# sysctl net.ipv4.ip_forward=1

To enable permanently:

This should be done on both servers:

# echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf

And then to tell the remote server how to get back to our network so that our requests get answered:

remote-server# ip route add 10.0.0.0/8 via 172.26.0.2 dev bond0

A pinch of iptables hackery:

remote-server# iptables -A FORWARD -o eth0 -i bond0 -s 10.0.0.0/8 -m conntrack --ctstate NEW -j ACCEPT
remote-server# iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
remote-server# iptables -A POSTROUTING -t nat -o eth0 -j MASQUERADE

Your local server should now be able to access the world wide web. Note that this server should, in this example at least, have an IP in the 10.0.0.0/8 subnet. That is the IP you use for your gateway on all your other machines to allow them to use the aggregated link.

I will at a later stage document automating the whole process as much as I can and have some form of connection monitoring in case the ADSL links stop working. Watch this space!

NAT

If everything was done correctly, your bonded link should be up and running. Your remote server should be able to “see” your local server via its VPN IP address and vice versa! If this is the case then most of the leg work has been done for you.

All outgoing connections to the internet from your local server should now be appearing as if they are from your remote server’s IP address, in my example, 87.117.217.27. (You can quickly test this by using whatismyip.com). Lets say that you want to access a web page that you have set up on the local server that gives statistics about your setup. We will use iptables to acheive this:

We are assuming that your remote server connects to the internet via eth0

remote-server# iptables -A PREROUTING -t nat -i eth0 -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.26.0.2:80

Once this is done, any connection to http://87.117.217.27 should be transparently forwarded down your bonded link to the other end which should serve your web page.

This could also be expanded to, for example, allow Windows Remote Desktop to one of your internal computers to be accessible via the bonded link, or any service inside your LAN to be accessible from the publicly routable address of your remote server

remote-server# iptables -A PREROUTING -t nat -i eth0 -p tcp -m tcp --dport 3389 -j DNAT --to-destination 10.0.0.60:3389

NOTE: For the example above for forwarding RDP connections, the PC you are forwarding to will need to have its gateway set to the 10.0.0.0/8 address of the machine hosting the bonded link else it wont know where to send the responses to.

I found Karl Rupp’s NAT page to be helpful in explaining NAT with iptables!

Office 2010, KMS Activation and Domain Relationships

Tue, 28 Feb 2012 21:33:51 +0000

Whilst this seems trivial to some people, not only did this issue affect my office, but one of our branches in India too so I figure its worth noting somewhere!

My current employer has many operating companies (OpCo’s) dotted around the world, all under one brand. Because of this structure, our OpCo in the UK has a two way trust relationship between our domain and the parent company’s. For arguments sake, lets call them UKNET and CENTRALNET.

At the end of last year we were told to use a Microsoft KMS server hosted on CENTRALNET to install Office 2010 on some of our workstations. Easy.. or so we thought.

On UKNET we have our own DNS servers and for the SRV type that we need, they were infact misconfigured. This lead to our Office installs refusing to activate!

The easiest way to check it is working correctly is on a PC on UKNET (adapt to meet your own needs of course):

C:\>nslookup -type=srv _vlmcs._tcp.example.com
 Server: dns.example.com
 Address: 192.168.1.1

Non-authoritative answer:
 _vlmcs._tcp.example.com SRV service location:
 priority = 0
 weight = 0
 port = 1688
 svr hostname = kms.example.com

In the above snippet, example.com is the FQDN for your domain. A result like the one above implies that your machine is being told where to go for the KMS activation and as long as you can communicate with kms.example.com, you are good to go! (Cue trusty telnet!!)

If however, you are not getting any results from your DNS server, you can either change your DNS settings to one that does give the correct result or simply tell Office where to go to activate!

For this, I will just be letting Office know where to go. You need to complete your install and then on the target machine run:

cscript "%ProgramFiles%\Microsoft Office\Office14\ospp.vbs" /sethst:kms.example.com

Note that Office14 is version specific and is in fact Office 2010. No doubt this figure will change for newer versions but for now this works.

Once the above command has completed successfully simply launch an application from the suite and check its activation status. If all went well, you should have a nice activated version of office!

LAN Events!

Fri, 22 Apr 2011 23:14:29 +0000

For those of you that know me fairly well, you would know that over the last 5-6 years i have attended a number of LAN events hosted by MultiPlay.co.uk. Whilst I decided to take a break from the event last year we have jumped back on the band wagon again and are here for this years Spring event, i42.

To be truthful the group of us that agreed to attend this year decided this would be a meet up primarily aimed at coding DMDirc and as yet that coding remains to be seen. For the time being we have succumbed to the usual shenanigans that befalls a LAN event - Lots of junk food, lots of video games, lots of random shouting, “Hax!” or “ROFL!” (Pronounced roffle) as a brief example and general other tom foolery.

I must admit all things aside i do enjoy the atmosphere shared at a LAN especially when you are surrounded by friends

Me, myself and I

Tue, 22 Mar 2011 06:49:13 +0000

As you have have discovered, I’m not especially big on blogging. I lead quite a mundane (or at least I think so) life compared to some other people I know. That said I feel I should share at least something new!

I had some advise from a few of my closest peers to which the “jist” was the need to not sit on my arse and at least try to better myself. As of a few days ago I made the decision to look into the possibility of gaining some professional certifications. I am looking to start a career in the IT industry and, partly because of where I live (and my unwillingness to just get up and move) and my lack of industrial experience … its not going very well.

I feel pursuing these certifications will not only aid me in the career department (eventually) but the knowledge and mindset of “I did that” would boost my view of my own self worth and improve me as a person.

Admittedly this is all talk currently. As some of you know, most of the time my “bark” has no “bite” to put it bluntly.

I am hoping to achieve the following certifications:

Microsoft Certified Technology Specialist (MCTS)

Windows Server 2008 Network Infrastructure Configuration (70-642)

Windows Server 2008 Active Directory Configuration (70-640)

CompTIA Linux+ (Exams LX0-101 and LX0-102) - with the aim of furthering this to LPIC-1

"Nexus One - Web meets Phone"

Wed, 21 Apr 2010 20:46:21 +0000

On Monday my shiny new Nexus One phone finally arrived!

The phone was ordered on 14th April but with all the recent disruptions to air traffic caused by the icelandic volcano, Eyjafjallajökull, delivery of my phone was delayed.

Overall I am really impressed with the phone. It’s a lot more zippy than my older G1 and looks one hell of a lot nicer too. The UI is pretty similar to the G1’s Android 1.6 but there are some significant differences. Most notably are the 2 extra home screens, giving the phone a total of 5 screens instead of 3. The AMOLED is sharp, bright and most importantly, larger which makes a nice change as it is a pleasure to look at. The phone is nicely weighted (not too heavy, not too light) and feels sturdy.

All this was expected though, given how long Android has been available on mobile devices and I think that Google and HTC have had ample time to perfect on the design. On the note of the design, one thing I (and probably many others) consider to be a flaw is the SDcard slot. You have to remove the battery to get at the card which is… not useful.

On the plus side however, I found the voice recognition very accurate with no training what so ever. I’ve spent a few hours crazily talking to my phone…

If you have any questions, please feel free to contact me (when the contact form works) or simply ask Google to lend its wisdom.

Logical Volume Manager - LVM

Thu, 11 Mar 2010 20:25:03 +0000

This page covers some of the basics of using LVM. Please ensure that you back up any important data on any had drive or partition you wish to use for LVM.

I take no responsibility for any data loss if you choose to use any of the commands on this page

Installation

For the purposes of this guide, I am using Ubuntu Hardy 8.04 and the package manager apt but this should work with any installation of LVM

First off, you need to install LVM. For my install it was as simple as:

# apt-get install lvm2

Once this is complete you should have a working install of LVM. To check, look for the following binaries

$ which lvcreate
 /sbin/lvcreate
 $ which pvcreate
 /sbin/pvcreate
 $ which vgcreate
 /sbin/vgcreate

If you do not have the three binaries above then you need to double check you installed LVM correctly.

Setup

The very first thing you need to do (After backing up any data that needs to be backed up of course) is decide which disks or partitions you wish to use. Once this is done we need to set the correct type of partition for use with LVM. In my example, I will be using a partition that takes up an entire disk.

To set up the partition we use fdisk. As root we need to:

# fdisk /dev/sda

The number of cylinders for this disk is set to 1125.
 There is nothing wrong with that, but this is larger than 1024,
 and could in certain setups cause problems with:
 1) software that runs at boot time (e.g., old versions of LILO)
 2) booting and partitioning software from other OSs
 (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help):

If you already have any partitions on this disk you should remove them by typing d and selecting each partition as they are listed.

Once this is done, we need to create a new partition

Command (m for help): n
 Command action
 e extended
 p primary partition (1-4)
 p
 Partition number (1-4): 1
 First cylinder (1-1125, default 1):
 Using default value 1
 Last cylinder or +size or +sizeM or +sizeK (1-1125, default 1125):
 Using default value 1125

Command (m for help):

And then we change its type to Linux LVM

Command (m for help): t
 Selected partition 1
 Hex code (type L to list codes): 8e
 Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help):

To finalize the partitioning, we type w:

Command (m for help): w
 The partition table has been altered!

Calling ioctl() to re-read partition table.
 Syncing disks.

This step will need to be repeated for every disk (or partition) you wish to use in your LVM Array.

pvcreate

Once all your disks have been partitioned you need to write LVM Superblock headers into them, this is done with the pvcreate command:

# pvcreate /dev/sda1 /dev/sdb1
 Physical volume "/dev/sda1" successfully created
 Physical volume "/dev/sdb1" successfully created

Thats it, you now have all the nessecery information for LVM Arrays in the superblocks of the disks. This will be amended at a later stage automatically by LVM to include the UUIDs for each disk/partition in the event of an array failure.

vgcreate

Now that is overwith we need to create a volume group to house the logical disk. In this example I am going to use a volume group of VolGrp001 and a logical disk name of LogVol001

# vgcreate VolGrp001 /dev/sda1 /dev/sdb1
 Volume group "VolGrp001" successfully created

lvcreate

Now we need to create the new logical volume and add it to the newly created volume group.

Whilst there are many more options available with these commands I’ve omitted most of them for ease. Feel free to have a play around, just take note to be careful as if you mess something up you could lose all the data you have on the hard drives you are trying to use in an LVM Array.

To create the new array we use the following (the flag after LogVol001 is a lowercase L):

# lvcreate --name LogVol001 -l 100%VG VolGrp001
 Rounding up size to full physical extent 2793.96 GB
 Logical volume "LogVol001" created

Thats it, all the hard work of setting up LVM is done. All that is left is to make it contain a usable format and then stand in awe as you have a huge, seemingly single disk to store all your data

In most cases, I would imagine you want to use the newly created LVM Device for data storage. My Choice of filesystem is ext3 however you can use any you like along with any settings you choose.

To create a useable ext3 format on the LVM Device we use mkfs.ext3:

# mkfs.ext3 -m 0 /dev/VolGrp001/LogVol001

The above command might take some time depending on how large the array is and how well your disks perform. It’s worth noting that the -m 0 option gives you some more space. By default ext3 reserves 5% of the total space for the super user. If you are using this to store data the super user would not need this

Once this is done you can now happily mount the new file system and start using it (/mnt/lvm is my mount point of choice):

# mount /dev/VolGrp001/LogVol001 /mnt/lvm

Extending LVM Devices

So, you’ve had your LVM Device up and running for a while and it’s starting to get full. You can happily add extra disks/partitions to your LVM device.

Firstly, as with creating a new LVM Array you need to partition the disk to Linux LVM. This is done with the pvcreate command:

# pvcreate /dev/sdc1
 Physical volume "/dev/sdc1" successfully created

The next tast will be to add the new disk or partition to the same Volume Group as the Logical volume we want to extend. To do this we use the vgextend

WARNING: It is recommended that before you make any modifications to a logical volume that you unmount it first

# vgextend VolGrp001 /dev/sdc1

/dev/sdc1 has now been added to the volume group. Now we need to extend the size of the logical volume (the same as earlier, this flag is a lowercase L)

# lvextend -l +100%FREE /dev/VolGrp001/LogVol001 /dev/sdc1

The above adds 931.32GB of usable space to the Logical Volume and tells it where this space actually is, /dev/sdc1. The next task is to resize the partition to fill the Device. It is the same as if you were resizing an ext3 patition on a physical disk. First we check the file system for errors (Make sure it is unmounted first) and then we resize the partition to take up the entirety of the disk space available

# e2fsck -f /dev/VolGrp001/LogVol001
# resize2fs /dev/VolGrp001/LogVol001

Again, this make take some time depending on the size of the file system and how well your hard drives perform. Once all this is done however, you’re ready to go. Re mount the file system and it should have increased by 931.32GB or whatever you increased it by.

Java Development

Sun, 24 Jan 2010 18:45:05 +0000

Yesterday, I took my first official steps at attempting to learn Java. It has been on the back of my mind for a long time (since the days of Visual Basic 6) but I have just never settled into it; Yes, I’ve tinkered, but never anything serious.

Thanks to some of my long time friends I now have a place to develop my skills and a place to get assistance as and when I need it. Since early 2007 they have been on-off developing an alternate IRC client for Linux that happens to be cross platform with a wide selection of operating systems due to it being Java based. In my opinion their client has far surpassed any IRC client that I’ve ever come across. If you don’t agree, keep watching, you soon will!

The client can be located at DMDirc.com

The project has since moved to Github so check it out here

Happy New Year

Fri, 01 Jan 2010 13:23:22 +0000

Happy new year.

This year has been full of interesting things for all of us and today is the start of yet another. I hope you’re all not too stuffed on turkey as it all goes back to normal Monday, wouldn’t want you falling down the stairs as you cant support yourself! There is lots to be done this year.

OpenSSH

Sat, 05 Sep 2009 23:49:45 +0000

Here you will find some tips and tricks for OpenSSH that I found useful. Note that the instructions on this page do not cover all the capabilities of OpenSSH and I advise you to use these commands at your own risk.

SSH Aliases

If, like me, you have lots of servers and find it quite tedious to continually type “ssh admin@server1.co.uk” for all your servers, Aliases might be a god send!

To add an alias is really simple. Below is an example of an alias that shortens “ssh admin@server1.co.uk” to “ssh s1” (This can be further shortened with bash aliases which, at present, are not covered here).

To do this, put the following in /home/simon/.ssh/config:

Host s1
 HostName server1.co.uk
 User admin

For a full and complete list of config options see “man ssh_config” on your server

Keys

SSH keys allow you greater security whilst at the same time giving you easier access to machines you regularly need to SSH to. Keys come in pairs, you have a Public key and a Private key, Which as suggested, Private keys are, well, pivate and Public keys are public.

To generate a key pair, using your local machine type the following:

$ ssh-keygen -t rsa -b 2048

Doing this will ask you to enter a pass phrase. At this stage it might be worth noting, if you provide a pass phrase, the key will not be usable for passwordless logins with SSH (e.g executing remote commands). If you want security, enter a pass phrase. It is recomended this be a sentence with a mix of words, numbers and symbols such as “!”.

If you want to use this key for passwordless logins just simply press Return on your keyboard.

Once this is done you should be told that the key has been created successfully and you should also now have the location of the key pair

Your identification has been saved in /home/simon/.ssh/id_rsa.
 Your public key has been saved in /home/simon/.ssh/id_rsa.pub.
 The key fingerprint is:
 91:c3:7f:d7:af:d1:e5:69:b4:fa:07:1f:17:8b:90:fe simon@shinobu

It is recomended that you never put your id_rsa anywhere that cannot be trusted. Whilst in most cases it is only you that has access to your home folder, Superusers such as root can also.

To install a key onto a server you want to ssh to, simply put the contents of id_rsa.pub into /home/simon/.ssh/authorized_keys and change its permissions to 600 (rw——-)

We have a handy built in script called ssh-copy-id which does all that for you. Assuming your public key is in /hom/simon/.ssh/id_rsa.pub then just type the following into a console:

$ ssh-copy-id -i /home/simon/.ssh/id_rsa.pub admin@yourserver.com

You should now be prompted for your password. Once successfully logged in you will see something similar to:

Now try logging into the machine, with "ssh 'admin@yourserver.com'", and check in:

 .ssh/authorized_keys

 to make sure we haven't added extra keys that you weren't expecting.

The above copies the contents of your id_rsa.pub to your server, appends its contents to your authorized_keys file (or creates one if needed) and changes its permissions

Once this has been done try to SSH to your server and if you created the key with a pass phrase, you should be prompted for it. If you created a phrase-less key then you should be allowed into your server without any prompt.

If you fail to log in with the correct key then the OpenSSH daemon will, by default, revert to password authentication. If you do not want this to happen, please see below

Agent and Agent Forwarding

Whilst having keys with pass phrases are more secure than standard passwords, it can be extremely monotonous to have to type it in every time you want to do anything that requires SSH especially if the phrase is a long sentance with symbols. To avoid this, we use ssh-agent

In order for us to be able to use SSH and not type a pass-phrase in we need to add our identity (see keys) to our agent session. This needs to be done every time your desktop is started.

$ ssh-add
 Identity added: /home/simon/.ssh/id_rsa (/home/simon/.ssh/id_rsa)

Once this is done, you will only need to type your pass phrase in once. You can SSH to your server and log out as many times as you want, it will only ask for the phrase once!

Agent Forwarding can be useful when you need to SSH to a server to work on. Lets say you need to then copy a file from that server to another, but the other server only allows access with SSH keys. You can forward your key to the new server without having to open a complete new SSH connection from your local machine. To do this all you need to do is add your identity to ssh-agent (see above) and add a flag to your SSH command:

$ ssh -A admin@yourserver.com

The -A flag tells SSH to forward your identity which is stored on the server (assuming the server has Agent forwarding enabled). If keys are set up correctly you should be allowed access to this server without hiccup. Whilst connected to this server you can test SSH forwarding by trying to SSH to another server that has your Public key (see keys).

$ ssh admin@yourotherserver.com

If everything is set up correctly then you should now be on yourotherserver.com without having to type any passwords!

SECURITY NOTE: Using agent forwarding can leave you open to “Agent Hijacking”. Whilst this does not give anyone your Key it does leave the doors open for them to use your Forwarded information to access servers you would normally have access to with your key. For more info please read this article: Security Issues with Key Agents

Iggy Woes

Fri, 04 Sep 2009 07:22:27 +0000

I have a pet Iguana called Iggy. Its a female Green Iguana and over the weekend she broke her leg, badly. (See image to the right)

I was in Milton Keynes over the bank holiday weekend with the #MDBot Group (Quakenet). We arrived on Saturday at about midday and we left the next day in the afternoon. Milton Keynes is, on average about 2.5 to 3 hours away from me so leaving at 15:15 meant I wouldn’t get home till early evening.

When I eventually arrived home after having a relaxing 1.5 hour bus journey from Milton Keynes Central to Birmingham International, my pet Iguana did her usual routine of coming over to me to say hello which, in her world means she wants me to make a fuss of her and usually give her some food. After I had spent 5 or so minutes rubbing some dead skin off her back and prepairing some cress for her to munch on, I sat at my computer and proceeded to catch up on some of the TV shows I had missed over the weekend. After a short while I noticed Iggy was climbing down onto the floor (Which isnt unusual, she likes a good wander), as she walked around the side of the desk I saw her leg was the size of a rugby ball…

As this was a Sunday Evening, taking her to our local vet was out of the question. They charge £90 to look at anything out of hours, which doesn’t include the cost of any extras involved such as X-rays. I was stuck as this would also apply for the Monday which was a bank holiday. Some of you may wonder why I didn’t take her and spend the £90. Quite frankly, I cant afford it. My Iguana was moving around fine, she was keeping her weight off her leg so I decided I would do what I could to restrict her movement and to take her to the vet during normal working hours.

I managed to get an appointment for earlier today at a vet that deals with reptiles, Rose Cottage in Runcorn. After an examination and being advised what the issue was, I obtained the X-ray image above. As you can see the break is a relatively nasty one and is broken into three pieces. I am a bit disappointed that it wasn’t something less serious, as god only knows what will happen if the break doesn’t start to heal properly in the next 4 weeks.

Once I have taken her back to the vet to have it looked at again I will post another update, just in case anyone is interested!

My Website

Sun, 02 Aug 2009 22:48:36 +0000

Well, just in case you didn’t realize, this is my website. For the most part it is about myself or things I enjoy. I will use this site to host facts/statistics about me and also things that will, in the future, help me with things I may have forgotten (for example, config files, command arguments for things like iptables etc).

If you find any of these pages useful please feel free to link to them. If you are sad and have no life like me feel free to subscribe to my RSS Feed.

If you have any suggestions please feel free to pop them over to me via the contact form (when it works) or alternately via IRC in #agari on Quakenet.

Thats about it for introductions, more to come (soon I hope!)