Javascript Asyncronous coding in
Posted: 2022-08-10 18:49:54 by Alasdair Keyes
Being a (primarily) back-end developer, my use of Javascript has been fairly limited. Mostly using jQuery and some basic raw Javascript in the olden-days for frontend validation, basic animation and alerts etc.
I've been writing a project in Quasar https://quasar.dev/ and delving further into the mysteries of async programming.
I was struggling to understand some documentation I found online. After some digging, it turns out that through various versions of ECMAScript, the concept of asynchronous programming has evolved and changed, giving you multiple ways of doing things with callbacks, promises and async/await.
I found this video on Youtube https://www.youtube.com/watch?v=PoRJizFvM7s which although only 24 minutes long gives a fantastic introduction to how each iteration of callbacks/promises/async/await has been developed, how they work and how to code against them.
It gives and intro with examples and that was all I need to get it straight in my head and make some real progress. It's a must-watch for anyone starting to get involved with Javascript other than simple libraries like jQuery.
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
Setting remote IP on Laravel controller tests
Posted: 2022-08-07 12:45:15 by Alasdair Keyes
When building a Laravel website you might want to create allow/block lists based on user's IP or from GeoIp information. This is easy enough using geoip2/geoip2 (https://packagist.org/packages/geoip2/geoip2) but how do you test your code is working correctly with specific IP addresses when writing your functional/integration tests?
At the beginning of a test that requires a custom IP you can add $this->serverVariables = ['REMOTE_ADDR' => '1.2.3.4']; and this will be what your controller sees in the IlluminateHttpRequest object.
public funtion testGeoIpFunctionality(): void
{
$this->serverVariables = ['REMOTE_ADDR' => '1.2.3.4'];
$response = $this->get('/website/endpoint');
$response->assertStatus(200);
...
// other assertions
...
}
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
"They" are watching... even quicker than before
Posted: 2022-05-06 20:45:08 by Alasdair Keyes
I wrote a blog post in 2019 about the website of a newly registered domain getting visited by a bot within 5 hours of the website coming online. You can read the article here - Security first, "they" are watching.
In short, I had surmised that the Certificate Transparency logs were being monitored to discover new sites so they could be scanned for vulnerabilities before an admin had a chance to harden the website.
I read an article today (https://portswigger.net/daily-swig/wordpress-sites-getting-hacked-within-seconds-of-tls-certificates-being-issued) which looks as if this premonition has come to pass. Wordpress websites are apparently getting hacked 'within seconds' of the TLS certificates being issue.
It looks like the logs are being tailed and visited much quicker than before... from 5 hours 3 years ago to <1 minute today.
I've steered clear of Wordpress for years now and often advise my clients to do the same. Although the usability and extensibility of Wordpress is fantastic, the scope for vulnerabilities in both plugins and the core code is too great to rely on. If you do run it, assess if you really need it for a public facing site and if you don't, add IP or Basic Authentication restrictions to your webserver config to restrict access to only those who need it.
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
Reducing dependencies and expanding Laravel Blade
Posted: 2022-05-05 08:05:06 by Alasdair Keyes
I've recently spent a couple of days moving my site to the latest version of Laravel. There were some problems upgrading through such a large number of major versions which I'll likely do a blog post about later.
Whilst I was re-adding my Composer dependencies I was looking at what I really needed. My blog posts are written in markdown and stored in a the database, the Laravel template engine, Blade, converts them from Markdown to HTML. For this task I was using the parsedown/laravel plugin (https://packagist.org/packages/parsedown/laravel)which uses erusev/parsedown (https://packagist.org/packages/erusev/parsedown)underneath to do the actual markdown processing.
I try to minimise dependencies used for two reasons,
- Reduced complexity in the codebase
- Reduced attack vectors either from attacks directly against my site or supply chain attacks through PHP's Composer system.
Whilst browsing through the Laravel Docs I noticed that they have an inbuilt Str::markdown (https://laravel.com/docs/9.x/helpers#method-str-markdown) helper which might allow me to do the same thing. Under the hood it uses Commonmark from the PHP League (https://commonmark.thephpleague.com/)
I used a couple of custom options on Parsedown, which I needed to be sure worked with the Laravel version.
$parseDown = Parsedown::instance();
$parseDown->setUrlsLinked(false);
$parseDown->setMarkupEscaped(false);
setUrlsLinked: false means that URLs aren't automatically converted into a href links and setMarkupEscaped: false means that I can include HTML markup in my blog posts if I desire.
After reading through the Common mark docs I the relative options were allow_unsafe_links: true and html_input: allow flag and I'd be set. Although these are defaults for Commonmark, I want to explicitly declare them in case defaults change in future.
I only used markdown in my templates and Parsedown automatically adds a blade directive of @parsedown("# Markdown Title") which I made use of. My first task was to create a Blade directive to process markdown in my templates, I decided on the name processMarkdown()
I created app/Providers/CustomBladeFunctionProvider.
<?php
declare(strict_types=1);
namespace App\Providers;
use Illuminate\Support\Facades\Blade;
use Illuminate\Support\ServiceProvider;
use Illuminate\Support\Str;
class CustomBladeFunctionProvider extends ServiceProvider
{
/**
* Bootstrap the application services.
*
* @return void
*/
public function boot(): void
{
// Provide @processMarkdown
Blade::directive('processMarkdown', function ($parameter) {
return "<?= rtrim(Str::markdown($parameter, [ 'allow_unsafe_links' => true, 'html_input' => 'allow' ])); ?>";
});
}
}
Then added this to my providers in config/app.php.
...
'providers' => [
...
App\Providers\CustomBladeFunctionProvider::class,
...
],
...
Then all I needed to do was update my templates from
@parsedown($blogPost->body)
@processMarkdown($blogPost->body)
And then remove the parsedown/laravel dependency to slim down my codebase.
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
Language Transfer for learning a new language
Posted: 2022-04-13 12:21:46 by Alasdair Keyes
I've recently tried to get back into learning German and I have started using the Language Transfer website.
Language Transfer is run by one chap, Mihalis who teaches a range of languages, French, Italian, German, Spanish, Greek, Turkish, Arabic and even Swahili to English speakers. His concept for learning languages is to understand what shared parts of English are similar (or transferred) across into the language you are learning and use that as a base to get a fast grounding.
I've been using Duolingo on and off for a while, but often become frustrated in it's lack of explanation as to why certain aspects of the language are the way they are. Language Transfer really adds to it by teaching common rules as to how to construct sentences in the given language, how verbs congugate, nouns pluralise, etc.
If you're learning any languages, I highly recommend checking the site to see if he is teaching your language and I think you'll find it a great help.
https://www.languagetransfer.org/
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
Mounting an encrypted ZFS dataset at boot on Debian 11 Bullseye
Posted: 2021-11-09 16:43:36 by Alasdair Keyes
I've recently got myself another HP Microserver which has space for 4 disks so I decided setup Debian 11 on one disk and use the other three to create a ZFS zpool for data storage.
The last time I'd experimented with ZFS on Linux (ZoL) on a virtual machine, encryption wasn't available, but it is now so I enabled if for my dataset. This is fine when the dataset is created, as it will auto-mount, but it doesn't auto-mount on reboot as it's encrypted.
It turns out ZFS handles the process of obtaining the encryption key and mounting the volume as two distinct processes. This means that when the ZFS mount service starts, it will skip mounting the encrypted volume because there is no key available to it.
The Linux standard dm-crypt/LUKS encryption requires you to update /etc/crypttab with each encrypted volume on the system and it will prompt for a password at boot time. ZFS does have the ability to use a file as the encryption key, but as I already have to enter a password for the OS drive, I was looking for do the same for the ZFS dataset.
After some investigation I found the solution on the Arch Linux Wiki (https://wiki.archlinux.org/title/ZFS#Native_encryption). They provide a snippet for a systemd service file that can be set to run before the ZFS mount service to ask for the encryption keys.
It did require tweaking as the path to the ZFS binary is different on Debian. In short, create the file /etc/systemd/system/zfs-load-key.service with the following content...
[Unit]
Description=Load ZFS encryption keys
DefaultDependencies=no
After=zfs-import.target
Before=zfs-mount.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/zfs load-key -a
StandardInput=tty-force
[Install]
WantedBy=zfs-mount.service
Once that is done run the following commands to refresh systemd with the new service and then set it to run on boot.
systemctl daemon-reload
systemctl enable zfs-load-key.service
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
LXC Debian containers and unknown GPG signing keys
Posted: 2021-06-04 10:14:27 by Alasdair Keyes
I needed to create a Debian Buster LXC container on my laptop and when running the following LXC create command I received the following error
# lxc-create -t debian -n testcontainer -- -r buster
debootstrap is /usr/sbin/debootstrap
Checking cache download in /var/cache/lxc/debian/rootfs-buster-amd64 ...
gpg: key 7638D0442B90D010: 4 signatures not checked due to missing keys
gpg: key 7638D0442B90D010: "Debian Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>" not changed
gpg: Total number processed: 1
gpg: unchanged: 1
Downloading debian minimal ...
I: Retrieving InRelease
I: Checking Release signature
E: Release signed by unknown key (key id DCC9EFBF77E11517)
The specified keyring /var/cache/lxc/debian/archive-key.gpg may be incorrect or out of date.
You can find the latest Debian release key at https://ftp-master.debian.org/keys.html
Failed to download the rootfs, aborting.
Failed to download 'debian base'
failed to install debian
lxc-create: testcontainer: lxccontainer.c: create_run_template: 1626 Failed to create container from template
lxc-create: testcontainer: tools/lxc_create.c: main: 319 Failed to create container testcontainer
This is telling me that the key used to sign the Debian release is unknown to LXC. It also shows that LXC is using the file /var/cache/lxc/debian/archive-key.gpg as the GPG keyring.
We can check the keys listed in that keyring with the following command. As a break down, this is running the regular gpg utility, but the --no-default-keyring and --keyring arguments are telling gpg to manage just the keyring file that LXC is using.
# gpg --no-default-keyring --keyring /var/cache/lxc/debian/archive-key.gpg --list-key
/var/cache/lxc/debian/archive-key.gpg
-------------------------------------
pub rsa4096 2014-11-21 [SC] [expires: 2022-11-19]
126C0D24BD8A2942CC7DF8AC7638D0442B90D010
uid [ unknown] Debian Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
Which shows it only has the key for Debian 8 - Jessie...
To get the latest version we need to check that the key listed in the error is a valid Debian key, otherwise we could be opening ourselves up to downloading malicious files.
Visiting https://ftp-master.debian.org/keys.html shows that the GPG key with fingerprint DCC9EFBF77E11517 listed in the error is the valid Debian 10 Buster release key.
Now that we're satisfied that nothing shady is going on, we can import the key to the keyring.
Download the key from the Debian site...
# wget "https://ftp-master.debian.org/keys/release-10.asc"
...
2021-06-04 10:51:53 (35.6 MB/s) - ‘release-10.asc’ saved [1200/1200]
Then import into the keyring...
# gpg --no-default-keyring --keyring /var/cache/lxc/debian/archive-key.gpg --import release-10.asc
gpg: key DCC9EFBF77E11517: public key "Debian Stable Release Key (10/buster) <debian-release@lists.debian.org>" imported
gpg: Total number processed: 1
gpg: imported: 1
Running the --list-key command we ran before shows the new key in the the LXC keyring
# gpg --no-default-keyring --keyring /var/cache/lxc/debian/archive-key.gpg --list-key
/var/cache/lxc/debian/archive-key.gpg
-------------------------------------
pub rsa4096 2014-11-21 [SC] [expires: 2022-11-19]
126C0D24BD8A2942CC7DF8AC7638D0442B90D010
uid [ unknown] Debian Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
pub rsa4096 2019-02-05 [SC] [expires: 2027-02-03]
6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517
uid [ unknown] Debian Stable Release Key (10/buster) <debian-release@lists.debian.org>
We can now run the create container command...
# lxc-create -t debian -n akeyescouk -- -r buster
debootstrap is /usr/sbin/debootstrap
Checking cache download in /var/cache/lxc/debian/rootfs-buster-amd64 ...
gpg: key 7638D0442B90D010: 4 signatures not checked due to missing keys
gpg: key 7638D0442B90D010: "Debian Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>" not changed
gpg: Total number processed: 1
gpg: unchanged: 1
Downloading debian minimal ...
I: Retrieving InRelease
I: Checking Release signature
I: Valid Release signature (key id 6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517)
I: Retrieving Packages
I: Validating Packages
...
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
PHPUnit 9.x coverage reporting
Posted: 2020-11-24 13:25:46 by Alasdair Keyes
PHPUnit 9.x coverage reporting
I started a new Laravel project today and used the latest Laravel 8.x release. After installation I go through and update a few things such as adding in phpmd, phpcs, laravel-debugbar and also setup PHPUnit code coverage reports that I can hook into gitlab's code coverage reporting tools.
After making the changes to my phpunit.xml file I was greeted with the following error
PHPUnit 9.4.3 by Sebastian Bergmann and contributors.
Warning - The configuration file did not pass validation!
The following problems have been detected:
Line 29:
- Element 'log': This element is not expected.
Test results may not be as expected.
.. 2 / 2 (100%)
Time: 00:00.386, Memory: 30.00 MB
OK (2 tests, 2 assertions)
Line 29 is part of the <logging> block I added in for coverage reporting.
<phpunit ....>
<logging>
<log type="coverage-text" target="php://stdout" showUncoveredFiles="true"/>
<log type="coverage-html" target="build/logs/html/" showUncoveredFiles="true"/>
</logging>
</phpunit>
After reading through the documentation for PHPUnit 9 (which is what is pulled in with Composer for Laravel 8) this is changed from logging to report and is now under the testsuites tag and has an changed syntax.
<phpunit ....>
<testsuites processUncoveredFiles="true">
...
<report>
<text outputFile="php://stdout"/ showUncoveredFiles="true">
<html outputDirectory="build/logs/html/"/>
</report>
<testsuites>
</phpunit>
I'm probably not going to be the only one caught out by this, so I thought it warranted a post.
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
PHP Docker Image and opcache
Posted: 2020-07-25 12:19:25 by Alasdair Keyes
I've recently started working on a new project using NGINX, PHP 7.4, Redis, PostgreSQL and Laravel 7.
As it's a new project I thought I would Dockerise it from the start. After configuring my docker-compose.yml file I built the environment and installed Laravel7 with the Laravel Debugbar (https://packagist.org/packages/barryvdh/laravel-debugbar)
I noticed that the bootstrapping of the basic Laravel App was taking over 100ms. I ran the config cache config:cache and it barely made any difference.
This didn't seem right to me but I had a number of variables and I was unsure of where to start looking... or if it was a problem at all
- This was my first use of Laravel 7 - Maybe it had just got slow since Version 5?
- This was using Docker - Maybe my laptop just wasn't too speedy or Docker itself was adding an overhead?
Thankfully the first part of my investigation found me a solution. I created a phpinfo() page on an existing Laravel 5 setup and on the Docker container. It turns out that the opcache isn't enabled on the Docker image by default.
Adding the following RUN statement to my DockerFile sorted the issue
docker-php-ext-install opcache
After restarting the container, the app bootstraps in 25-30ms. I'm unsure why opcache isn't enabled by default, I can't think of any problems it would cause and I would imagine in over 99% of situations users would want it on... no one wants slow PHP.
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
The Freelance Developer Podcast
Posted: 2020-03-24 23:35:20 by Alasdair Keyes
I happened to chance across a post on LinkedIn from an old colleague of mine who has started a new podcast about freelance development. If you've got some time and you're either a contractor or looking to contract in the future, it's worth a listen.
https://www.thefreelancedeveloper.co.uk/
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
© Alasdair Keyes
IT Consultancy Services
I'm now available for IT consultancy and software development services - Cloudee LTD.
Happy user of Digital Ocean (Affiliate link)
Version:master-2c7f8ada3f
