Need: I keep bringing up Kubernetes clusters / destroying them as a part of my learning of Kubernetes. When using Kubespray, I observed the required packages were getting downloaded from the internet. Though I have a decent 300 Mbps internet connectivity, I decided to check if I can have a local repository mirror in my lab.
Searching the internet figured out it was not a complex task to set up. There were two main steps involved. Downloading the repo – around 240G (at the time of typing out this post for Ubuntu 20.04) and the second one configuring either proftpd or apache2 to frontend and serve the files/requests. ( Thanks to https://www.linuxtechi.com/setup-local-apt-repository-server-ubuntu/ ).
Note: Both the steps could be done in a single VM, but I decided to have two different VMs – just a personal choice and no specific technical reason.
Instantiate a VM in which the repository would be downloaded – 400G storage allocated.
Install apt-mirror
apt install -y apt-mirror
Edit the contents of /etc/apt/mirror.list – changed occurrences of ‘artful’ to ‘focal’, my requirement and uncommented backports repository.
####### config ##################
#
# set base_path /var/spool/apt-mirror
#
# set mirror_path $base_path/mirror
# set skel_path $base_path/skel
# set var_path $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch <running host architecture>
# set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads 40
set _tilde 0
#
############# end config ##############
deb http://archive.ubuntu.com/ubuntu focal main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu focal-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse
#deb http://archive.ubuntu.com/ubuntu focal-proposed main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu focal main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu focal-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse
#deb-src http://archive.ubuntu.com/ubuntu focal-proposed main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse
clean http://archive.ubuntu.com/ubuntu
Download or mirror the repository locally.
The first-time download would take some time depending on the speed of internet connectivity (around 240 G to download).
# nohup apt-mirror
The repository gets downloaded at /var/spool/apt-mirror. It was recommended to add a corn job that would periodically execute apt-mirror and update the repository locally. I preferred to update the same as and when required. Subsequent downloads or executing apt-mirror does not take much time – depends on the delta updates available.
Download CNF directories
In Ubuntu 20.04 LTS, apt-mirror does not sync the CNF directory and its files, so we have to manually download and copy the folder and its files. Not sure if there would be delta updates to these CNF directories. Decided to modify the script from the site by adding the reference to backports repository and retaining only for amd64 architecture.
Contents of download_cnf.sh
#!/bin/bash
for p in "${1:-focal}"{,-{security,updates,backports}}\
/{main,restricted,universe,multiverse};
do >&2 echo "${p}"
wget -q -c -r -np -R "index.html*" "http://archive.ubuntu.com/ubuntu/dists/${p}/cnf/Commands-amd64.xz"
done
Executing the script results in a directory “archive.ubuntu.com” getting created with the required files.
Configure mirror server in the lab
Add DNS entry for “mirror.datachronicles.net” in the DNS server, and instantiate a VM, Ubuntu 20.04, 400G disk.
Install apache2
apt install -y apache2
Create a document (in this case package repository) directory
mkdir -p /var/www/html/ubuntu
Set the ownership to www-data
chown www-data:www-data /var/www/html/ubuntu
Update the site configuration (/etc/apache2/sites-enabled/datachronicles.net), As I have wild card certificate for the domain, configuring SSL too)
<VirtualHost *:80>
ServerAdmin admin@datachronicles.net
DocumentRoot /var/www/html/ubuntu
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
<VirtualHost _default_:443>
ServerAdmin admin@datachronicles.net
DocumentRoot /var/www/html/ubuntu
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
SSLEngine on
SSLCertificateFile /var/www/html/ubuntu/certs/datachronicles.crt
SSLCertificateKeyFile /var/www/html/ubuntu/certs/privatekey.key
SSLCertificateChainFile /var/www/html/ubuntu/certs/datachronicles-chain.crt
<FilesMatch "\.(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars
</FilesMatch>
<Directory /usr/lib/cgi-bin>
SSLOptions +StdEnvVars
</Directory>
</VirtualHost>
Enable SSL and restart apache2
a2enmod ssl
systemctl restart apache2
Now from the other VM (where the repository was download need to update the document folder with the repository) – Created an “update.sh” script for the same. Note : mirror gets resolved to the IP of the mirror VM – DNS services updated internally in the lab.
#!/bin/bash
apt-mirror
rsync -av /var/spool/apt-mirror/var root@mirror:/var/www/html/ubuntu/
rsync -av /var/spool/apt-mirror/skel root@mirror:/var/www/html/ubuntu/
rsync -av /var/spool/apt-mirror/mirror root@mirror:/var/www/html/ubuntu/
rm -rf /root/archive.ubuntu.com
/root/cnf.sh
scp -r archive.ubuntu.com root@mirror:/var/www/html/ubuntu/mirror/
rm -rf /root/archive.ubuntu.com
Updated the base image used with the contents of /etc/apt/sources.list referring to the local mirror. Spin up a test VM and test
root@test:~# cat /etc/apt/sources.list
deb http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal main restricted
deb http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal-updates main restricted
deb http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal universe
deb http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal-updates universe
deb http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal multiverse
deb http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal-updates multiverse
deb http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse
deb http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal-security main restricted
deb http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal-security universe
deb http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal-security multiverse
root@test:~# apt update
Hit:1 http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://mirror.datachronicles.net/mirror/archive.ubuntu.com/ubuntu focal-security InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
root@test:~#
That’s it!!!!