Dilbert.com Comic Downloader in bash

I've made some code to download most Dilbert.com comics.

It can be found here:

https://github.com/richardskumat/misc/tree/master/dilbert

This should keep working until the site's layout changes a lot.

There are some exceptions where fixed links are provided, since the image host returns 500 error code for some images.

There are 10020 comics since 16th April 1989 to 20th September 2016.

The comic on the 31st August was the 10000th comic.

The 10020 images take a total of 1.2GB disk space.

There are several other downloaders according a quick web search.

SSH with multiple jump hosts

Gentoo's wiki page has an excellent about multiple jump hosts.

https://wiki.gentoo.org/wiki/SSH_jump_host

To ~/.ssh/config:

Host *+*
  ProxyCommand ssh $(echo %h | sed 's/+[^+]*$//;s/\([^+%%]*\)%%\([^+]*\)$/\2 -l \1/;s/:/ -p /') exec nc -w1 $(echo %h | sed 's/^.*+//;/:/!s/$/ %p/;s/:/ /')

Usage can be seen on the wiki.

I used the following to access a virtual Pfsense instance over a Debian vm over the Proxmox host:

ssh proxmox-host-alias+ip-of-debian-vm+ip-of-pfsense-vm -l root

Fix for Proxmox v4 "You do not have a valid subscription for this server" notification

Proxmox Logo
This is quite annoying on servers where you have no paid subscription and the web interface will annoy people on every login.

The files responsible for this are in:

/usr/share/pve-manager/ext6/pvemanagerlib.js:    noSubKeyHtml: 'You do not have a valid subscription for this server. Please visit < a target="_blank" href="http://www.proxmox.com/products/proxmox-ve/subscription-service-plans" >www.proxmox.com< /a> to get a list of available options.',
/usr/share/pve-manager/touch/pvemanager-mobile.js:    noSubKeyHtml: 'You do not have a valid subscription for this server. Please visit < a target="_blank" href="http://www.proxmox.com/products/proxmox-ve/subscription-service-plans">www.proxmox.com< /a> to get a list of available options.',

This can be disabled with:

sed -i.bak "s/data.status\ !==\ 'Active'/false/g" /usr/share/pve-manager/ext6/pvemanagerlib.js
sed -i.bak "s/data.status\ !==\ 'Active'/false/g" /usr/share/pve-manager/touch/pvemanager-mobile.js 

on the cli.

Running sed -i.bak "etc" will leave a backup file with .bak extension.

This was originally based on James Coyle's page.

If you're already logged in, the browser's cache will need to be cleared and the login page refreshed.

Changing the user agent on Firefox

The user agent part is a text identifier browsers send to websites so sites can serve better pages to visitors.

Most popular sites take user agent detection too far, and starts spamming people with ads(especially on mobile), pop-ups, redirects and other annoying things.

Most of these annoyances can be prevented by using a good adblocker(Ublock Origin can filter out visible parts of sites, eg images or cookie notices, which is very useful), disabling javascript and host file blocking on a rooted phone(eg shell code for downloading bad hosts list from someonewhocares.org).

Both the mobile and desktop versions of Firefox respect the general.useragent.override string value in about:config. This doesn't exist by default, so needs to be added.

Once general.useragent.override has been created in about:config, a good value can be assigned by looking at this site for examples.

Since those examples are outdated, version numbers need to be updated as well.

Sites like Google.com will annoy people to install Chrome or update it, so having a "fresh" Chrome user agent and disabling javascript on Google's sites make them more usable. Unless of course they'll stop rendering google.com without javascript.

Wikipedia and the Chrome dev blog has a list of recent Chrome versions to look at.

Duckduckgo provides a good response to what's my user agent requests at https://duckduckgo.com/?q=my+user+agent.

Nginx: blocking a GET request if it contains a certain word

An interesting thing I've noticed is wordpress gives away author usernames in the URL on the author pages.

Example:

https://site.com/author/username

The problem is username is also the login name for that "author".

WordPress also redirects to author pages if someone starts querying for author ids. The url in question that returns/redirects to the site.com/author/username is https://site.com/?author=number. Number can be replaced with integers, like 0, 1, 2, 3 and so on.

WordPress incrementally increases author ids in its database, so newer authors on a site will have higher ids, while administrators may have 1, 2 or 3 for example.

If a matching author id is found in the requested URL than wordpress redirects the visitor to the author page of the author and exposes the login name of the author in URL.

The following snippet placed in a server {} block should return forbidden/403 pages for request URLs containing the author string. This will also block pages or posts containing the word author in the URL.

The snippet is:

if ($request_uri ~ "author") {
        return 403;
}

Genisoimage to make a custom Windows ISO

The code is the following:


genisoimage -b boot/etfsboot.com -no-emul-boot -iso-level 2 -udf -J -l -D -N -joliet-long -relaxed-filenames -o windows.iso extracted-windows-folder

-o windows.iso = the output iso file

extracted-windows-folder = The folder extracted from the original ISO

On Debian, the following are needed:

  1. Windows ISO file
  2. genisoimage and p7zip-full apt packages

The original Windows ISO file needs to be extracted with 7z.

Example:

7z x original-windows.iso

On EL, 7z might be called 7za, but x is still used for extracting.

The extracted-windows-folder as a result can have content added, such as the Virtio drivers.

The output ISO files haven't been tested on systems with UEFI boots.

Shell script for modifying Xfce workspace count from CLI

Xfce logoLink:

https://github.com/richardskumat/misc/blob/master/xfce/xfws

Usage:

bash -x xfws +2

bash -x xfws -2

bash -x xfws 6

$0 = /path/to/filename.sh

$0 +5 - increase xfce workspace count by 5. Eg from 2 to 7.

$0 -2 - decrease xfce workspace count by 2. Eg from 7 to 5.

$0 8 - set xfce workspace count to 8. Eg from 5 to 8.

Code:

#!/bin/bash
# modify xfce workspace count
# xfconf-query -c xfwm4 -p /general/workspace_count

if [ -z $1 ];then
echo "First arg was empty. Exiting."
cat << EOF
Usage of $0:

$0 +5 - increase xfce workspace count by 5. Eg from 2 to 7.

$0 -2 - decrease xfce workspace count by 2. Eg from 7 to 5.

$0 8 - set xfce workspace count to 8. Eg from 5 to 8.
EOF
exit
fi


count="$(xfconf-query -c xfwm4 -p /general/workspace_count)"
# modifier to new count
newcountplus()
{
xfconf-query -c xfwm4 -p /general/workspace_count -s ${newstring}
}

newcountminus()
{
xfconf-query -c xfwm4 -p /general/workspace_count -s ${newstring2}
}

newcountnormal()
{
xfconf-query -c xfwm4 -p /general/workspace_count -s $1
}

# try to filter out + and - from the first arguement with cut
string="$(echo $1 | cut -c2-3)"

if [ -z ${string} ];then
string=0
fi

# check for python2
python --version &> /tmp/py2
# if python 2 string isn't found in /tmp/py2 then try python2 executable
if ! grep -qs "Python 2" /tmp/py2;then
rm /tmp/py2
python2 --version &> /tmp/py2
fi

# check for python3
python3 --version &> /tmp/py3

if grep -qs "Python 2" /tmp/py2;then
py2ok="yes"
rm /tmp/py2
fi

if grep -qs "Python 3" /tmp/py3;then
py3ok="yes"
rm /tmp/py3
fi

# modify $str with python to an integer(eg 2 or 5)
# use python3 first
if [ "${py3ok}" = yes ];then
newstring="$(python3 -c "num = int('${string}');print(num + ${count})")"
newstring2="$(python3 -c "num = int('${string}');print(${count} - num)")"
else

# else use python2 if it exists
if [ "${py2ok}" = yes ];then
newstring="$(python -c "num = int("${string}");print num + ${count}")"
newstring2="$(python -c "num = int('${string}');print ${count} - num")"
fi

fi

case $1 in
+*)
# plus increase
newcountplus
exit
;;
-*)
# minus decrease
newcountminus
exit
;;
*)
# else modify to new count
newcountnormal $1
;;
esac

Bash Personal Journal

I've written a short script for maintaining a personal journal with bash and an editor(eg nano or vim).

The code is available on Github.

There is a short readme.md file for explanation.

The script itself is here(won't be updated for a while):

#!/bin/bash
# Write my journal
# Variables
rpath="$HOME/tmp/journal"
year="$(date +%Y)"
month="$(date +%m)"
day="$(date +%d)"
# if rpath/month/date doesn't exist, create it
if [ ! -d "${rpath}/${year}/${month}" ];then
mkdir -p "${rpath}/${year}/${month}"
fi
# if today's journal file doesn't exist, create it
if [ ! -e "${rpath}/${year}/${month}/${day}" ];then
touch "${rpath}/${year}/${month}/${day}"
fi
# Write the created date(y/m/d) and file path to today's journal file
# if it's empty
if [ ! -s "${rpath}/${year}/${month}/${day}" ];then
cat <<-EOF >> "${rpath}/${year}/${month}/${day}"
# Created on ${year}/${month}/${day}
# this file was located at ${rpath}/${year}/${month}/${day}
EOF
fi
# Select either nano or vi as $EDITOR
if [ -e "$(which nano)" ];then
EDITOR="$(which nano)";
else
EDITOR="$(which vi)";
fi
# Write the journal
$EDITOR "${rpath}/${year}/${month}/${day}"

The readme file's content

I've created a short bash script to write to journal you maintain on the cli with your editor.

It maintains a hiercharchy of $journalroot/year/month/day.

day is journal file itself.

$journalroot is $rpath in the script.

I've called the script "journal" and put into a directory in my $PATH, so calling journal from bash, creates the year/month directories if not present, then the journal file itself.

There's an $EDITOR value inside, which tries nano first, then vi.

It also writes the date and location it was located in, if the journal file is empty, otherwise it doesn't do that.

Change rpath and/or delete editor part.

Centos 7 i686(32 bit) container

RHEL/Centos 7 is officially 64 bit only, which is good enough for companies/most people.

I'm on low end hardware, so need 32 bit images.

Centos 7 32 bit image

CentOSCentos 7 32 bit builds are in the AltArch section in https://wiki.centos.org/Download.

There's also an ARM build for Raspberry Pis, however I found that Centos 7 ARM build didn't support the Realtek wifi plug(Manufacturer: Realtek, 802.11n WLAN Adapter) my Raspi 2B has.

I got myself the minimal ISO, http://mirror.centos.org/altarch/7/isos/i386/CentOS-7-i386-Minimal-1511.iso, installed it in virtualbox and mounted the vdi file via qemu-nbd.

Using the Centos 7 i686 VDI file as a container

From that point, systemd-nspawn can use it as a "container". It also works as a chroot.

The vdi image boots up with systemd-nspawn, using the -b(boot) flag.

I'd recommend aliasing nspawn to a shorter command, like npw.

I assume LXC could also run it.

This image is obviously a full OS, not a barely functional container image you can get via debootstrap for example.

I forgot to change the FS to ext4 from xfs during Centos 7 installation, but that causes no issues with using the vdi as a container, at least on the backports 4.6 kernel on Debian 8.

Debian Instructions for mounting the VDI

Required packages needed for mounting the vdi, on a host with LUKS LVM:

kpartx qemu-utils

Install required packages

apt-get install kpartx qemu-utils -y

Create a dev device using qemu-nbd

qemu-nbd -c /dev/nbd0 /path/to/centos7.vdi

Create a partition map for the host of the vdi

Both my host and stock installations of Centos 7 use lvm.

kpartx -a /dev/nbd0

Activate the lvm volumes of the vdi file

vgchange -ay

Check for the VDI's lvm partition presence

lvscan

Output should be similar to:

root@stuff:/# lvscan
ACTIVE '/dev/cl/swap' [stuff GiB] inherit
ACTIVE '/dev/cl/root' [stuff GiB] inherit
ACTIVE '/dev/deb-vg/root' [stuff GiB] inherit
ACTIVE '/dev/deb-vg/home' [stuff GiB] inherit
ACTIVE '/dev/deb-vg/swap' [stuff GiB] inherit

Make a directory for mounting the / of the vdi

mkdir /mnt/c7

Mount the root partition of the VDI

The lvm group was called cl I guess, so vdi volumes were mapped to /dev/cl/{root,swap}.

mount /dev/cl/root /mnt/c7

Test the container

systemd-nspawn -D /mnt/c7

If you get a shell, it means you can chroot into it.

Also worth setting up your administrator user and root password at this point.

Then see if the image is nspawn "bootable":

systemd-nspawn -bD /mnt/c7

Making a copy of the VDI image

Now it's time to make a copy of the mounted image at /mnt/c7.

cp -aR /mnt/c7 /path/to/save/point

Again, we can test with nspawn to make sure our copy works.

systemd-nspawn -D /path/to/save/point

If it works okay, we should get a similar screen as:

centos7 container

Unmounting the VDI and deactivating the LVM group/volumes

First, we need to unmount the VDI root.

umount /mnt/c7

Output of lvscan at this point:

root@debian:/# lvscan
ACTIVE '/dev/cl/swap' [0 GiB] inherit
ACTIVE '/dev/cl/root' [0 GiB] inherit
ACTIVE '/dev/debian-vg/root' [0 GiB] inherit
ACTIVE '/dev/debian-vg/home' [0 GiB] inherit
ACTIVE '/dev/debian-vg/swap' [0 GiB] inherit

Then we need to deactivate the LVM group called cl in this case.

vgchange -an cl

Result:

0 logical volume(s) in volume group "cl" now active

Output of lvscan:

inactive '/dev/cl/swap' [0 GiB] inherit
inactive '/dev/cl/root' [0 GiB] inherit
ACTIVE '/dev/debian-vg/root' [0 GiB] inherit
ACTIVE '/dev/debian-vg/home' [0 GiB] inherit
ACTIVE '/dev/debian-vg/swap' [0 GiB] inherit

We need then tell kpartx to remove the partition mapping of the VDI file.

kpartx -d /dev/nbd0

Lvscan output then is:

ACTIVE '/dev/debian-vg/root' [0 GiB] inherit
ACTIVE '/dev/debian-vg/home' [0 GiB] inherit
ACTIVE '/dev/debian-vg/swap' [0 GiB] inherit

You can now remove the nbd0 device with qemu-nbd:

qemu-nbd -d /dev/nbd0

Removing /dev/nbd0 before deactivating the LVM of the VDI and removing the partition mapping would result in similar errors:

/dev/mapper/nbd0p1: read failed after 0 of 4096 at 524222464: Input/output error

/dev/mapper/nbd0p1: read failed after 0 of 4096 at 0: Input/output error

Useful links:

http://www.aossama.com/mount-virtualbox-vdi-image-debian/

https://centos.org