Introduction
Debian 12 Bookworm is a versatile and stable Linux distribution, widely used for servers, desktops, and other applications. One of the key tools for containerization is Podman, an alternative to Docker that offers a rootless mode and better security.
Overview
What
Podman is a daemonless container engine for developing, managing, and running OCI containers on your Linux system. It is a drop-in replacement for Docker.
Who
Podman is ideal for system administrators, developers, and DevOps professionals who need to manage containers efficiently and securely without requiring root access.
Where
Podman can be installed on any Debian 12 Bookworm system, whether it's a personal workstation or a production server environment.
When
Podman should be installed when setting up a new server or when migrating from other containerization solutions like Docker, especially if enhanced security is a requirement.
Why
Podman offers several advantages and disadvantages as highlighted in the table below:
| Pros | Cons |
|---|---|
| Rootless operation enhances security. | Learning curve for those familiar with Docker. |
| Compatible with Docker CLI commands. | Some Docker-specific features may not be available. |
| Daemonless architecture reduces resource usage. | Less community support compared to Docker. |
How
Step-by-Step Installation of Podman on Debian 12 Bookworm:
| Step 1 | Update the package index:(`...sudo apt update`) |
| Step 2 | Install Podman:(`...sudo apt install -y podman`) |
| Step 3 | Verify the installation:(`...podman --version`) |
Consequences
Using Debian 12 Bookworm with KVM can have several consequences:
| Positive |
|
| Negative |
|
Conclusion
By installing Podman on Debian 12 Bookworm, you gain a powerful and secure containerization tool. Its advantages in security, compatibility with Docker, and reduced resource usage make it an excellent choice for container management.
Install and Configure
Install Podman that is Container management tool.
Step [1]Install Podman.
root@bizantum:~# apt -y install podman
Step [2]Download an official image and create a Container and output the words [Welcome to the Podman World] inside the Container.
# download the official image
root@bizantum:~# podman pull debian
Resolved "debian" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/debian:latest...
Getting image source signatures
Copying blob bba7bb10d5ba done
Copying config 49081a1edb done
Writing manifest to image destination
Storing signatures
49081a1edb0b55df1967387e4c234add2d3f8ef0dc1f4953e7eaf552dc761c5a
# run echo inside a container
root@bizantum:~# podman run debian /bin/echo "Welcome to the Podman World"
Welcome to the Podman World
Step [3] Connect to the interactive session of a Container with [i] and [t] option like follows. If [exit] from the Container session, the process of a Container finishes.
root@bizantum:~# podman run -it debian /bin/bash
root@5c8e3406597c:/# # connected
root@5c8e3406597c:/# uname -a
Linux 5c8e3406597c 6.1.0-9-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08) x86_64 GNU/Linux
root@5c8e3406597c:/# exit
exit
root@bizantum:~# # come back
Step [4]If you'd like to run a Container as a Daemon, add [d] option.
root@bizantum:~# podman run -itd debian /bin/bash
26652f4918eb615f5c4e4466bd7dc91bed381330e52ebdd880cdbe05b0c101b5
# show podman processes
root@bizantum:~# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
26652f4918eb docker.io/library/debian:latest /bin/bash 23 seconds ago Up 24 seconds ago epic_wescoff
# attach to container session
root@bizantum:~# podman exec -it 26652f4918eb /bin/bash
root@26652f4918eb:/# # connected
root@26652f4918eb:/# exit
# stop container process
# * if force stop, specify [podman kill ***]
root@bizantum:~# podman stop 26652f4918eb
root@bizantum:~# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Add Container Images
Add new Container images you modified settings.
Step [1]For example, update an official image with installing [apache2] and add it as a new Container image.
# show container images
root@bizantum:~# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/library/debian latest 49081a1edb0b 8 days ago 121 MB
# run a container and install [apache2]
root@bizantum:~# podman run debian /bin/bash -c "apt-get update; apt-get -y install apache2"
root@bizantum:~# podman ps -a | tail -1
ec2ab97e352a docker.io/library/debian:latest /bin/bash -c apt-... About a minute ago Exited (0) About a minute ago adoring_kalam
# add the image that [apache2] was installed
root@bizantum:~# podman commit ec2ab97e352a bizantum.lab/debian-apache2
Getting image source signatures
Copying blob 332b199f36eb skipped: already exists
Copying blob 622e3a22918b done
Copying config f047c6f28c done
Writing manifest to image destination
Storing signatures
f047c6f28c56a68c2881a7a2a7a24850d38e0c1c5ca59e6a632e34cf3d60e723
# show container images
root@bizantum:~# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
bizantum.lab/debian-apache2 latest f047c6f28c56 20 seconds ago 260 MB
docker.io/library/debian latest 49081a1edb0b 8 days ago 121 MB
# confirm [apache2] to run a container
root@bizantum:~# podman run bizantum.lab/debian-apache2 /usr/bin/whereis apache2
apache2: /usr/sbin/apache2 /usr/lib/apache2 /etc/apache2 /usr/share/apache2 /usr/share/man/man8/apache2.8.gz
Access to Services on Containers
If you'd like to access to services like HTTP or SSH that are running on Containers as daemons, Configure like follows.
Step [1]For example, use a Container that [apache2] is installed.
root@bizantum:~# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
bizantum.lab/debian-apache2 latest f047c6f28c56 3 minutes ago 260 MB
docker.io/library/debian latest 49081a1edb0b 8 days ago 121 MB
# run a container and also start [apache2]
# map with [-p xxx:xxx] to [(Host Port):(Container Port)]
root@bizantum:~# podman run -dt -p 8081:80 bizantum.lab/debian-apache2 /usr/sbin/apachectl -D FOREGROUND
25d79bdb048396b868a0b31a913abcbd8fe464f09f6c51a1eefb9f4f7426b6c0
root@bizantum:~# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
25d79bdb0483 bizantum.lab/debian-apache2:latest /usr/sbin/apachec... 11 seconds ago Up 11 seconds ago 0.0.0.0:8081->80/tcp beautiful_buck
# create a test page
root@bizantum:~# podman exec 25d79bdb0483 /bin/bash -c 'echo "Apache2 on Podman Container" > /var/www/html/index.html'
# verify accesses
root@bizantum:~# curl localhost:8081
Apache2 on Podman Container
# also possible to access via container network
root@bizantum:~# podman inspect -l | grep \"IPAddress
"IPAddress": "10.88.0.7",
"IPAddress": "10.88.0.7",
root@bizantum:~# curl 10.88.0.7
Apache2 on Podman Container
Use Dockerfile
Use Dockerfile and create Container images automatically. It is also useful for configuration management for Container images.
Step [1]For example, Create a Dockerfile that Nginx is installed and started.
root@bizantum:~# vi Dockerfile
# create new
FROM debian
MAINTAINER ServerWorld <admin@bizantum.lab>
RUN apt-get update
RUN apt-get -y install nginx
RUN echo "Dockerfile Test on Nginx" > /var/www/html/index.html
EXPOSE 80
CMD ["/usr/sbin/nginx", "-g", "daemon off;"]
# build image ⇒ podman build -t [image name]:[tag] .
root@bizantum:~# podman build -t bizantum.lab/debian-nginx:latest .
STEP 1/7: FROM debian
STEP 2/7: MAINTAINER ServerWorld <admin@bizantum.lab>
--> 00970b06f91
STEP 3/7: RUN apt-get update
.....
.....
COMMIT bizantum.lab/debian-nginx:latest
--> e2938e55cf7
Successfully tagged bizantum.lab/debian-nginx:latest
e2938e55cf7cbd06ac0c38b967476b4897f464ff74fc16f88cb4f0c30fe1895e
root@bizantum:~# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
bizantum.lab/debian-nginx latest e2938e55cf7c About a minute ago 158 MB
bizantum.lab/debian-apache2 latest f047c6f28c56 8 minutes ago 260 MB
docker.io/library/debian latest 49081a1edb0b 8 days ago 121 MB
# run container
root@bizantum:~# podman run -d -p 80:80 bizantum.lab/debian-nginx
fd506a88790a2a0c3cdda70f3b5d9be3041b145a0df0bfcdd6d393cce319ca5c
root@bizantum:~# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fd506a88790a bizantum.lab/debian-nginx:latest /usr/sbin/nginx -... 11 seconds ago Up 11 seconds ago 0.0.0.0:80->80/tcp festive_clarke
# verify accesses
root@bizantum:~# curl localhost
Dockerfile Test on Nginx
# also possible to access via container network
root@bizantum:~# podman inspect -l | grep \"IPAddress
"IPAddress": "10.88.0.11",
"IPAddress": "10.88.0.11",
root@bizantum:~# curl 10.88.0.11
Dockerfile Test on Nginx
The format of Dockerfile is [INSTRUCTION arguments] . Refer to the following description for INSTRUCTION.
| INSTRUCTION | DESCRIPTION |
|---|---|
| FROM | It sets the Base Image for subsequent instructions. |
| MAINTAINER | It sets the Author field of the generated images. |
| RUN | It will execute any commands when Docker image will be created. |
| CMD | It will execute any commands when Docker container will be executed. |
| ENTRYPOINT | It will execute any commands when Docker container will be executed. |
| LABEL | It adds metadata to an image. |
| EXPOSE | It informs Docker that the container will listen on the specified network ports at runtime. |
| ENV | It sets the environment variable. |
| ADD | It copies new files, directories or remote file URLs. |
| COPY | It copies new files or directories. The differences of [ADD] are that it's impossible to specify remote URL and also it will not extract archive files automatically. |
| VOLUME | It creates a mount point with the specified name and marks it as holding externally mounted volumes from native host or other containers. |
| USER | It sets the user name or UID. |
| WORKDIR | It sets the working directory. |
Use External Storage
When a Container is removed, data in it are also lost, so it's necessary to use external storage on Containers if you'd like to save your data on Containers.
Step [1]It's possible to mount a directory on Docker Host into Containers.
# create a directory for containers data
root@bizantum:~# mkdir /var/lib/containers/disk01
root@bizantum:~# echo "persistent storage" >> /var/lib/containers/disk01/testfile.txt
# run a Container with mounting the directory above on [/mnt]
root@bizantum:~# podman run -it -v /var/lib/containers/disk01:/mnt debian /bin/bash
root@bc45ab289b96:/# df -hT /mnt
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/debian--vg-root ext4 77G 1.9G 72G 3% /mnt
root@bc45ab289b96:/# cat /mnt/testfile.txt
persistent storage
Step [2]It's also possible to configure external storage by Podman Data Volume command.
# create [volume01] volume
root@bizantum:~# podman volume create volume01
volume01
# display volume list
root@bizantum:~# podman volume ls
DRIVER VOLUME NAME
local volume01
# display details of [volume01]
root@bizantum:~# podman volume inspect volume01
[
{
"Name": "volume01",
"Driver": "local",
"Mountpoint": "/var/lib/containers/storage/volumes/volume01/_data",
"CreatedAt": "2023-06-20T18:49:09.789666889-05:00",
"Labels": {},
"Scope": "local",
"Options": {},
"MountCount": 0,
"NeedsCopyUp": true,
"NeedsChown": true
}
]
# run a container with mounting [volume01] to [/mnt] on container
root@bizantum:~# podman run -it -v volume01:/mnt debian
root@122855d1d6ce:/# df -hT /mnt
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/debian--vg-root ext4 77G 1.9G 72G 3% /mnt
root@122855d1d6ce:/# echo "Podman Volume test" > /mnt/testfile.txt
root@122855d1d6ce:/# exit
root@bizantum:~# cat /var/lib/containers/storage/volumes/volume01/_data/testfile.txt
Podman Volume test
# possible to mount from other containers
root@bizantum:~# podman run -v volume01:/var/volume01 debian cat /var/volume01/testfile.txt
Podman Volume test
# to remove volumes, do like follows
root@bizantum:~# podman volume rm volume01
Error: volume volume01 is being used by the following container(s): 122855d1d6cee008c6399704e2d459f9331d0186646e7b90640e14e64b37fe1c, 9309271fe8fafd9a281006daa6587ff336aadf72c3dcb9cc61ece75b8b4d573e: volume is being used
# if some containers are using the volume you'd like to remove like above,
# it needs to remove target containers before removing a volume
root@bizantum:~# podman rm 122855d1d6cee008c6399704e2d459f9331d0186646e7b90640e14e64b37fe1c
root@bizantum:~# podman rm 9309271fe8fafd9a281006daa6587ff336aadf72c3dcb9cc61ece75b8b4d573e
root@bizantum:~# podman volume rm volume01
volume01
Use External Storage (NFS)
This is an example to use NFS External Storage.
Step [1]NFS server is required to be running on your local network, refer to here. On this example, configure [/home/nfsshare] directory on [nfs.bizantum.lab] as a shared directory.
Step [2]Create a volume for NFS and use it.
# create [nfs-volume] volume
root@bizantum:~# podman volume create \
--opt type=nfs4 \
--opt o=rw \
--opt device=10.0.0.35:/home/nfsshare nfs-volume
nfs-volume
# display volume list
root@bizantum:~# podman volume ls
DRIVER VOLUME NAME
local nfs-volume
# display details of [nfs-volume]
root@bizantum:~# podman volume inspect nfs-volume
[
{
"Name": "nfs-volume",
"Driver": "local",
"Mountpoint": "/var/lib/containers/storage/volumes/nfs-volume/_data",
"CreatedAt": "2023-06-20T18:58:37.764274613-05:00",
"Labels": {},
"Scope": "local",
"Options": {
"device": "10.0.0.35:/home/nfsshare",
"o": "rw",
"type": "nfs4"
},
"MountCount": 0,
"NeedsCopyUp": true,
"NeedsChown": true
}
]
# run container with mounting [nfs-volume] to [/nfsshare] on container
root@bizantum:~# podman run -it -v nfs-volume:/nfsshare debian
# verify
root@74d1c42e34b8:/# df -hT /nfsshare
Filesystem Type Size Used Avail Use% Mounted on
10.0.0.35:/home/nfsshare nfs4 28G 1.3G 26G 5% /nfsshare
root@74d1c42e34b8:/# echo "Podman NFS Volume Test" > /nfsshare/testfile.txt
root@74d1c42e34b8:/# cat /nfsshare/testfile.txt
Podman NFS Volume Test
Use Registry
Install Registry to build Private Registry for Podman images. Pull the Registry image and run it. Container images are located under [/var/lib/regstry] on Registry v2 Container, so map to mount [/var/lib/containers/registry] on parent Host for Registry Container to use as Persistent Storage.
Step [1] Configure Registry. This is for the case to use HTTP and no authentication.
root@bizantum:~# podman pull registry:2
root@bizantum:~# mkdir /var/lib/containers/registry
root@bizantum:~# podman run -d -p 5000:5000 \
-v /var/lib/containers/registry:/var/lib/registry \
registry:2
root@bizantum:~# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
72738df86ecd docker.io/library/registry:2 /etc/docker/regis... 9 seconds ago Up 10 seconds ago 0.0.0.0:5000->5000/tcp priceless_nightingale
# verify to push to Registry from localhost
# for HTTP connection, add [--tls-verify=false] option
root@bizantum:~# podman tag debian dlp.bizantum.lab:5000/debian:latest
root@bizantum:~# podman push dlp.bizantum.lab:5000/debian:latest --tls-verify=false
Getting image source signatures
Copying blob 332b199f36eb done
Copying config 49081a1edb done
Writing manifest to image destination
Storing signatures
root@bizantum:~# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
bizantum.lab/debian-nginx latest e2938e55cf7c 19 minutes ago 158 MB
bizantum.lab/debian-apache2 latest f047c6f28c56 27 minutes ago 260 MB
docker.io/library/registry 2 4bb5ea59f8e0 5 days ago 24.6 MB
docker.io/library/debian latest 49081a1edb0b 8 days ago 121 MB
dlp.bizantum.lab:5000/debian latest 49081a1edb0b 8 days ago 121 MB
Step [2]To enable Basic authentication, Configure like follows.
root@bizantum:~# apt -y install apache2-utils
# add users for Registry authentication
root@bizantum:~# htpasswd -Bc /etc/containers/.htpasswd debian
New password:
Re-type new password:
Adding password for user debian
root@bizantum:~# podman run --privileged -d -p 5000:5000 \
-v /var/lib/containers/registry:/var/lib/registry \
-v /etc/containers:/auth \
-e REGISTRY_AUTH=htpasswd \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/.htpasswd \
-e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \
registry:2
# login as a user you added above
root@bizantum:~# podman login dlp.bizantum.lab:5000 --tls-verify=false
Username: debian
Password:
Login Succeeded!
root@bizantum:~# podman tag debian dlp.bizantum.lab:5000/debian2:latest
root@bizantum:~# podman push dlp.bizantum.lab:5000/debian2:latest --tls-verify=false
root@bizantum:~# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
bizantum.lab/debian-nginx latest e2938e55cf7c 22 minutes ago 158 MB
bizantum.lab/debian-apache2 latest f047c6f28c56 29 minutes ago 260 MB
docker.io/library/registry 2 4bb5ea59f8e0 5 days ago 24.6 MB
docker.io/library/debian latest 49081a1edb0b 8 days ago 121 MB
dlp.bizantum.lab:5000/debian latest 49081a1edb0b 8 days ago 121 MB
dlp.bizantum.lab:5000/debian2 latest 49081a1edb0b 8 days ago 121 MB
Step [3]This is for the case you set valid certificate like Let's Encrypt and enable HTTPS connection. This example is based on that certificate were created under the [/etc/letsencrypt] directory.
root@bizantum:~# podman run --privileged -d -p 5000:5000 \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/fullchain.pem \
-e REGISTRY_HTTP_TLS_KEY=/certs/privkey.pem \
-v /etc/letsencrypt/live/dlp.bizantum.lab:/certs \
-v /var/lib/containers/registry:/var/lib/registry \
registry:2
# verify to push to Registry
root@node01:~# podman tag debian dlp.bizantum.lab:5000/debian3:latest
root@node01:~# podman push dlp.bizantum.lab:5000/debian3:latest
root@node01:~# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
bizantum.lab/debian-nginx latest e2938e55cf7c 24 minutes ago 158 MB
bizantum.lab/debian-apache2 latest f047c6f28c56 31 minutes ago 260 MB
docker.io/library/registry 2 4bb5ea59f8e0 5 days ago 24.6 MB
docker.io/library/debian latest 49081a1edb0b 8 days ago 121 MB
dlp.bizantum.lab:5000/debian latest 49081a1edb0b 8 days ago 121 MB
dlp.bizantum.lab:5000/debian2 latest 49081a1edb0b 8 days ago 121 MB
dlp.bizantum.lab:5000/debian3 latest 49081a1edb0b 8 days ago 121 MB
Podman Network
This is the basic usage to configure Podman Network.
Step [1]When running containers without specifying network, default [podman] network is assigned.
# display network list
root@bizantum:~# podman network ls
NETWORK ID NAME DRIVER
2f259bab93aa podman bridge
# display details of [podman]
root@bizantum:~# podman network inspect podman
[
{
"name": "podman",
"id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
"driver": "bridge",
"network_interface": "podman0",
"created": "2023-06-20T19:10:54.257475042-05:00",
"subnets": [
{
"subnet": "10.88.0.0/16",
"gateway": "10.88.0.1"
}
],
"ipv6_enabled": false,
"internal": false,
"dns_enabled": false,
"ipam_options": {
"driver": "host-local"
}
}
]
# [podman] is assigned as container network by default
root@bizantum:~# podman run debian /bin/bash -c "apt-get update; apt-get -y install iproute2; ip route"
.....
.....
default via 10.88.0.1 dev eth0 proto static metric 100
10.88.0.0/16 dev eth0 proto kernel scope link src 10.88.0.20
root@bizantum:~# podman commit $(podman ps -a | tail -1 | awk '{print $1}') bizantum.lab/iproute
Step [2]If you'd like to assign another network, configure like follows.
# create network [network01] with [192.168.100.0/24] subnet
root@bizantum:~# podman network create --subnet 192.168.100.0/24 network01
/etc/cni/net.d/network01.conflist
root@bizantum:~# podman network ls
NETWORK ID NAME DRIVER
e6937d4a3327 network01 bridge
2f259bab93aa podman bridge
# run a container with specifying [network01]
root@bizantum:~# podman run --network network01 bizantum.lab/iproute /bin/bash -c "ip route"
default via 192.168.100.1 dev eth0 proto static metric 100
192.168.100.0/24 dev eth0 proto kernel scope link src 192.168.100.2
# to attach the network to existing running container, set like follows
root@bizantum:~# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2706c69c65a5 bizantum.lab/debian-apache2:latest /usr/sbin/apachec... 6 seconds ago Up 7 seconds ago 0.0.0.0:8081->80/tcp funny_ardinghelli
root@bizantum:~# podman exec 2706c69c65a5 /bin/bash -c "ip route"
default via 10.88.0.1 dev eth0 proto static metric 100
10.88.0.0/16 dev eth0 proto kernel scope link src 10.88.0.21
# attach network to specify an IP address in the subnet
root@bizantum:~# podman network connect network01 2706c69c65a5
root@bizantum:~# podman exec 2706c69c65a5 ip route
default via 192.168.100.1 dev eth1 proto static metric 100
default via 10.88.0.1 dev eth0 proto static metric 100
10.88.0.0/16 dev eth0 proto kernel scope link src 10.88.0.21
192.168.100.0/24 dev eth1 proto kernel scope link src 192.168.100.3
# to disconnect the network, set like follows
root@bizantum:~# podman network disconnect network01 2706c69c65a5
root@bizantum:~# podman exec 2706c69c65a5 ip route
default via 10.88.0.1 dev eth0 proto static metric 100
10.88.0.0/16 dev eth0 proto kernel scope link src 10.88.0.21
Step [3]To remove podman networks, set like follows.
root@bizantum:~# podman network ls
NETWORK ID NAME DRIVER
e6937d4a3327 network01 bridge
2f259bab93aa podman bridge
# remove [network01]
root@bizantum:~# podman network rm network01
Error: "network01" has associated containers with it. Use -f to forcibly delete containers and pods: network is being used
# force remove containers with [-f] option
root@bizantum:~# podman network rm -f network01
network01
Use Docker Command
Install a script named [docker] that emulates the Docker CLI by executes podman commands.
Step [1]Install Podman-docker package.
root@bizantum:~# apt -y install podman-docker
# [docker] command is installed
root@bizantum:~# ll /usr/bin/docker
-rwxr-xr-x 1 root root 163 Nov 13 2022 /usr/bin/docker
# emulates the Docker CLI by executes podman
root@bizantum:~# cat /usr/bin/docker
#!/bin/sh
[ -f /etc/containers/nodocker ] || \
echo "Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg." >&2
exec /usr/bin/podman "$@"
# test [docker] command
root@bizantum:~# docker images
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
REPOSITORY TAG IMAGE ID CREATED SIZE
bizantum.lab/iproute latest e5a651fe23a6 8 minutes ago 157 MB
bizantum.lab/debian-nginx latest e2938e55cf7c 38 minutes ago 158 MB
bizantum.lab/debian-apache2 latest f047c6f28c56 45 minutes ago 260 MB
docker.io/library/registry 2 4bb5ea59f8e0 5 days ago 24.6 MB
docker.io/library/debian latest 49081a1edb0b 8 days ago 121 MB
Use Docker Compose
To Install Docker Compose, it's easy to configure and run multiple containers as a Docker application. Docker Compose is supported on Podman 3.0 or later.
Step [1] Install Podman-docker, refer to here.
Step [2]Install Docker Compose.
root@bizantum:~# apt -y install docker-compose
root@bizantum:~# docker-compose --version
docker-compose version 1.29.2, build unknown
Step [3]For example, Configure an application that has Web and DB services with Docker Compose.
# start podman.socket
root@bizantum:~# systemctl start podman.socket
# define Web service container
root@bizantum:~# vi Dockerfile
FROM debian
MAINTAINER ServerWorld &alt;admin@bizantum.lab>
RUN apt-get update
RUN apt-get -y install nginx
RUN echo "Compose Test on Nginx" > /var/www/html/index.html
EXPOSE 80
CMD ["/usr/sbin/nginx", "-g", "daemon off;"]
# define application configuration
root@bizantum:~# vi docker-compose.yml
version: '3'
services:
db:
image: docker.io/library/mariadb
volumes:
- /var/lib/containers/disk01:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: password
MYSQL_USER: debian
MYSQL_PASSWORD: password
MYSQL_DATABASE: debian_db
ports:
- "3306:3306"
web:
build: .
ports:
- "80:80"
volumes:
- /var/lib/containers/disk02:/usr/share/nginx/html
# build and run
root@bizantum:~# docker-compose up -d
Building web
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
STEP 1/7: FROM debian
STEP 2/7: MAINTAINER ServerWorld &alt;admin@bizantum.lab>
--> Using cache 00970b06f91f398d68bd80a879f505cde9bf7b0326fee60bda2ee0663008fdff
--> 00970b06f91
STEP 3/7: RUN apt-get update
.....
.....
Creating root_db_1 ... done
Creating root_web_1 ... done
root@bizantum:~# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7925b61be272 localhost/root_web:latest /usr/sbin/nginx -... 55 seconds ago Up 55 seconds ago 0.0.0.0:80->80/tcp root_web_1
7230fd252ef6 docker.io/library/mariadb:latest mariadbd 55 seconds ago Up 55 seconds ago 0.0.0.0:3306->3306/tcp root_db_1
# verify accesses
root@bizantum:~# mysql -h 127.0.0.1 -u root -p -e "show variables like 'hostname';"
Enter password:
+---------------+--------------+
| Variable_name | Value |
+---------------+--------------+
| hostname | 7230fd252ef6 |
+---------------+--------------+
root@bizantum:~# mysql -h 127.0.0.1 -u debian -p -e "show databases;"
Enter password:
+--------------------+
| Database |
+--------------------+
| debian_db |
| information_schema |
+--------------------+
root@bizantum:~# curl 127.0.0.1
Compose Test on Nginx
Step [4]Other basic operations of Docker Compose are follows.
# verify state of containers
root@bizantum:~# docker-compose ps
Name Command State Ports
---------------------------------------------------------------------
root_db_1 docker-entrypoint.sh mariadbd Up () :3306->3306/tcp
root_web_1 /usr/sbin/nginx -g daemon off; Up () :80->80/tcp
# show logs of containers
root@bizantum:~# docker-compose logs
Attaching to root_db_1, root_web_1
db_1 | 2023-06-21 00:27:26+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started.
db_1 | 2023-06-21 00:27:26+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
db_1 | 2023-06-21 00:27:26+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started.
db_1 | 2023-06-21 00:27:26+00:00 [Note] [Entrypoint]: Initializing database files
.....
.....
db_1 | 2023-06-21 0:27:31 0 [Note] Server socket created on IP: '::'.
db_1 | 2023-06-21 0:27:31 0 [Note] mariadbd: ready for connections.
db_1 | Version: '11.0.2-MariaDB-1:11.0.2+maria~ubu2204' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
# run any commands inside a container
# container name is just the one set in [docker-compose.yml]
root@bizantum:~# docker-compose exec db /bin/bash
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
root@7230fd252ef6:/#
# stop application and also shutdown all containers
root@bizantum:~# docker-compose stop
Stopping root_db_1 ... done
Stopping root_web_1 ... done
# start a service alone in application
# if set dependency, other container starts
root@bizantum:~# docker-compose up -d web
Starting root_web_1 ... done
root@bizantum:~# docker-compose ps
Name Command State Ports
----------------------------------------------------------------------
root_db_1 docker-entrypoint.sh mariadbd Exit 0 :3306->3306/tcp
root_web_1 /usr/sbin/nginx -g daemon off; Up () :80->80/tcp
# remove all containers in application
# if a container is running, it won't be removed
root@bizantum:~# docker-compose rm
Going to remove root_db_1
Are you sure? [yN] y
Removing root_db_1 ... done
Create Pods
Create Pods like Kubernetes.
Step [1]Create a Pod and add a Container to it.
# create a empty pod
# -p [bind port] -n [pod name]
root@bizantum:~# podman pod create -p 8081:80 -n test-pod
20889920c2fc33fd8fc73cafdf98d23d069b4e33654ca247c9a84fbfd90e4ef1
# show pods
root@bizantum:~# podman pod ls
POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS
20889920c2fc test-pod Created 14 seconds ago 89a87cf6104a 1
# show details of pod
root@bizantum:~# podman pod inspect test-pod
{
"Id": "20889920c2fc33fd8fc73cafdf98d23d069b4e33654ca247c9a84fbfd90e4ef1",
"Name": "test-pod",
"Created": "2023-06-20T19:33:28.99473824-05:00",
"CreateCommand": [
"podman",
"pod",
"create",
"-p",
"8081:80",
"-n",
"test-pod"
],
"ExitPolicy": "continue",
"State": "Created",
"Hostname": "",
"CreateCgroup": true,
"CgroupParent": "machine.slice",
"CgroupPath": "machine.slice/machine-libpod_pod_20889920c2fc33fd8fc73cafdf98d23d069b4e33654ca247c9a84fbfd90e4ef1.slice",
"CreateInfra": true,
"InfraContainerID": "89a87cf6104aabe45e703c43b7ca981f854e49eb16cbd1f335af527f3f1b2c9b",
"InfraConfig": {
"PortBindings": {
"80/tcp": [
{
"HostIp": "",
"HostPort": "8081"
}
]
},
"HostNetwork": false,
"StaticIP": "",
"StaticMAC": "",
"NoManageResolvConf": false,
"DNSServer": null,
"DNSSearch": null,
"DNSOption": null,
"NoManageHosts": false,
"HostAdd": null,
"Networks": [
"podman"
],
"NetworkOptions": null,
"pid_ns": "private",
"userns": "host",
"uts_ns": "private"
},
"SharedNamespaces": [
"ipc",
"net",
"uts"
],
"NumContainers": 1,
"Containers": [
{
"Id": "89a87cf6104aabe45e703c43b7ca981f854e49eb16cbd1f335af527f3f1b2c9b",
"Name": "20889920c2fc-infra",
"State": "created"
}
]
}
root@bizantum:~# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/podman-pause 4.3.1-0 27bd47a0cfe1 About a minute ago 776 kB
localhost/root_web latest 95ff1ec2c79f 7 minutes ago 158 MB
bizantum.lab/iproute latest e5a651fe23a6 20 minutes ago 157 MB
bizantum.lab/debian-nginx latest e2938e55cf7c 50 minutes ago 158 MB
bizantum.lab/debian-apache2 latest f047c6f28c56 57 minutes ago 260 MB
docker.io/library/mariadb latest 99833200524a 4 days ago 410 MB
docker.io/library/registry 2 4bb5ea59f8e0 5 days ago 24.6 MB
docker.io/library/debian latest 49081a1edb0b 8 days ago 121 MB
# run container and add it to pod
root@bizantum:~# podman run -dt --pod test-pod bizantum.lab/debian-nginx
3b83a201ea392b46777e7481eed9831ec8e3f261685946aa2ab7f0aab3af402a
root@bizantum:~# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
89a87cf6104a localhost/podman-pause:4.3.1-0 About a minute ago Up 10 seconds ago 0.0.0.0:8081->80/tcp 20889920c2fc-infra
3b83a201ea39 bizantum.lab/debian-nginx:latest /usr/sbin/nginx -... 9 seconds ago Up 10 seconds ago 0.0.0.0:8081->80/tcp sweet_dijkstra
# verify accesses
root@bizantum:~# curl localhost:8081
Podman Test on Nginx
# stop pod
root@bizantum:~# podman pod stop test-pod
20889920c2fc33fd8fc73cafdf98d23d069b4e33654ca247c9a84fbfd90e4ef1
# remove pod (removed containers all)
root@bizantum:~# podman pod rm test-pod --force
20889920c2fc33fd8fc73cafdf98d23d069b4e33654ca247c9a84fbfd90e4ef1
Step [2]It's possible to create Pod and add Container with one command.
root@bizantum:~# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/podman-pause 4.3.1-0 27bd47a0cfe1 2 minutes ago 776 kB
localhost/root_web latest 95ff1ec2c79f 8 minutes ago 158 MB
bizantum.lab/iproute latest e5a651fe23a6 22 minutes ago 157 MB
bizantum.lab/debian-nginx latest e2938e55cf7c 52 minutes ago 158 MB
bizantum.lab/debian-apache2 latest f047c6f28c56 59 minutes ago 260 MB
docker.io/library/mariadb latest 99833200524a 4 days ago 410 MB
docker.io/library/registry 2 4bb5ea59f8e0 5 days ago 24.6 MB
docker.io/library/debian latest 49081a1edb0b 8 days ago 121 MB
# create a [test_pod2] pod and add [bizantum.lab/debian-nginx] container
root@bizantum:~# podman run -dt --pod new:test-pod2 -p 80:80 -p 3306:3306 bizantum.lab/debian-nginx
8aef8c0efd44a4401341fac197e8d6bda1be3dfb49fbcaab7a3b2ee19b5a62d5
root@bizantum:~# podman pod ls
POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS
598975654d31 test-pod2 Running 13 seconds ago f2ed58888a2f 2
root@bizantum:~# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f2ed58888a2f localhost/podman-pause:4.3.1-0 31 seconds ago Up 32 seconds ago 0.0.0.0:80->80/tcp, 0.0.0.0:3306->3306/tcp 598975654d31-infra
8aef8c0efd44 bizantum.lab/debian-nginx:latest /usr/sbin/nginx -... 31 seconds ago Up 32 seconds ago 0.0.0.0:80->80/tcp, 0.0.0.0:3306->3306/tcp sad_blackwell
# run [mariadb] container and add it to the [test-pod2]
root@bizantum:~# podman run -dt --pod test-pod2 -e MYSQL_ROOT_PASSWORD=Password docker.io/library/mariadb
0e6342196221de2a74c7a5f40e754846717d9e6aede58e12cdef7f100db08278
root@bizantum:~# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f2ed58888a2f localhost/podman-pause:4.3.1-0 About a minute ago Up About a minute ago 0.0.0.0:80->80/tcp, 0.0.0.0:3306->3306/tcp 598975654d31-infra
8aef8c0efd44 bizantum.lab/debian-nginx:latest /usr/sbin/nginx -... About a minute ago Up About a minute ago 0.0.0.0:80->80/tcp, 0.0.0.0:3306->3306/tcp sad_blackwell
0e6342196221 docker.io/library/mariadb:latest mariadbd 12 seconds ago Up 13 seconds ago 0.0.0.0:80->80/tcp, 0.0.0.0:3306->3306/tcp friendly_yalow
root@bizantum:~# curl dlp.bizantum.lab
Dockerfile Test on Nginx
root@bizantum:~# mysql -u root -p -h dlp.bizantum.lab -e "show variables like 'hostname';"
Enter password:
+---------------+-----------+
| Variable_name | Value |
+---------------+-----------+
| hostname | test-pod2 |
+---------------+-----------+
Use by common users
It's possible to use Podman containers by common users.
Step [1]By default, sub UID/GID that are used on user name spaces are assigned to run containers.
# default name spaces number
root@bizantum:~# cat /proc/sys/user/max_user_namespaces
63773
# sub UID/GID mapping file
# 100000 to 165535 (100000 + 65536 - 1) UID are used for running processes in containers on [debian] user
root@bizantum:~# cat /etc/subuid
debian:100000:65536
root@bizantum:~# cat /etc/subgid
debian:100000:65536
# when added new users, sub UID/GID are also added
# n=0, n++
# [start UID/GID = 100000 + (65536 * n)]
# [end UID/GID = (start UID/GID) + 65536 - 1]
root@bizantum:~# adduser bookworm
root@bizantum:~# adduser bullseye
root@bizantum:~# cat /etc/subgid /etc/subgid
debian:100000:65536
bookworm:165536:65536
bullseye:231072:65536
debian:100000:65536
bookworm:165536:65536
bullseye:231072:65536
Step [2]It's possible to run [podman] by common users.
debian@bizantum:~$ podman pull debian
debian@bizantum:~$ podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/library/debian latest 49081a1edb0b 8 days ago 121 MB
debian@bizantum:~$ podman run debian echo "run rootless containers"
run rootless containers
# containers related files are located under the [$HOME/.local] directory
debian@bizantum:~$ ll ~/.local/share/containers/storage
total 40
-rw-r--r-- 1 debian debian 8 Jun 20 19:43 defaultNetworkBackend
drwx------ 2 debian debian 4096 Jun 20 19:43 libpod
drwx------ 2 debian debian 4096 Jun 20 19:43 mounts
drwx------ 2 debian debian 4096 Jun 20 19:43 networks
-rw-r--r-- 1 debian debian 64 Jun 20 19:43 storage.lock
drwx------ 2 debian debian 4096 Jun 20 19:43 tmp
-rw-r--r-- 1 debian debian 0 Jun 20 19:43 userns.lock
drwx------ 3 debian debian 4096 Jun 20 19:43 vfs
drwx------ 3 debian debian 4096 Jun 20 19:43 vfs-containers
drwx------ 3 debian debian 4096 Jun 20 19:43 vfs-images
drwx------ 2 debian debian 4096 Jun 20 19:43 vfs-layers
# possible to create Pods
debian@bizantum:~$ podman pod create -p 8081:80 -n test-pod
debian@bizantum:~$ podman pod ls
POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS
320b3c32a425 test-pod Created 5 seconds ago 28376d9a53e7 1
# for port mapping,
# it's impossible to use less than [1024] ports on Host machine by common users
# possible to use over [1024] ports
debian@bizantum:~$ podman run -d -p 1023:80 bizantum.lab/debian-nginx
Error: rootlessport cannot expose privileged port 1023, you can add 'net.ipv4.ip_unprivileged_port_start=1023' to /etc/sysctl.conf (currently 1024), or choose a larger port number (>= 1024): listen tcp 0.0.0.0:1023: bind: permission denied
debian@bizantum:~$ podman run -d -p 1024:80 bizantum.lab/debian-nginx
debian@bizantum:~$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f56f30ae7a8b bizantum.lab/debian-nginx:latest /usr/sbin/nginx -... 5 seconds ago Up 6 seconds ago 0.0.0.0:1024->80/tcp suspicious_golick
Generate System unit file
It's possible to generate Systemd unit file and set auto-starting for containers.
Step [1]Generate Systemd unit file and Set auto-starting for containers.
root@bizantum:~# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/podman-pause 4.3.1-0 27bd47a0cfe1 14 minutes ago 776 kB
localhost/root_web latest 95ff1ec2c79f 20 minutes ago 158 MB
bizantum.lab/iproute latest e5a651fe23a6 34 minutes ago 157 MB
bizantum.lab/debian-nginx latest e2938e55cf7c About an hour ago 158 MB
bizantum.lab/debian-apache2 latest f047c6f28c56 About an hour ago 260 MB
docker.io/library/mariadb latest 99833200524a 4 days ago 410 MB
docker.io/library/registry 2 4bb5ea59f8e0 5 days ago 24.6 MB
docker.io/library/debian latest 49081a1edb0b 8 days ago 121 MB
# run container
root@bizantum:~# podman run --name debian-nginx -d -p 80:80 bizantum.lab/debian-nginx
root@bizantum:~# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e33db7f8c043 bizantum.lab/debian-nginx:latest /usr/sbin/nginx -... 12 seconds ago Up 12 seconds ago 0.0.0.0:80->80/tcp debian-nginx
# generate Systemd unit file
root@bizantum:~# podman generate systemd --new --files --name debian-nginx
/root/container-debian-nginx.service
root@bizantum:~# cat /root/container-debian-nginx.service
# container-debian-nginx.service
# autogenerated by Podman 4.3.1
# Tue Jun 20 19:49:27 CDT 2023
[Unit]
Description=Podman container-debian-nginx.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm \
-f %t/%n.ctr-id
ExecStart=/usr/bin/podman run \
--cidfile=%t/%n.ctr-id \
--cgroups=no-conmon \
--rm \
--sdnotify=conmon \
--replace \
--name debian-nginx \
-d \
-p 80:80 bizantum.lab/debian-nginx
ExecStop=/usr/bin/podman stop \
--ignore -t 10 \
--cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm \
-f \
--ignore -t 10 \
--cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all
[Install]
WantedBy=default.target
root@bizantum:~# cp /root/container-debian-nginx.service /usr/lib/systemd/system
# enable auto-starting
root@bizantum:~# systemctl enable container-debian-nginx.service
Created symlink /etc/systemd/system/default.target.wants/container-debian-nginx.service → /lib/systemd/system/container-debian-nginx.service.
Step [2]Set auto-starting for pods.
# run Pod
root@bizantum:~# podman run -dt --pod new:nginx-pod -p 80:80 bizantum.lab/debian-nginx
root@bizantum:~# podman pod ls
POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS
3eba77086aa1 nginx-pod Running 4 seconds ago be2100263864 2
# generate Systemd unit file
root@bizantum:~# podman generate systemd --files --name nginx-pod
/root/pod-nginx-pod.service
/root/container-stoic_ganguly.service
root@bizantum:~# cp /root/pod-nginx-pod.service /root/container-stoic_ganguly.service /usr/lib/systemd/system
# enable auto-starting
root@bizantum:~# systemctl enable pod-nginx-pod.service container-stoic_ganguly.service
Created symlink /etc/systemd/system/default.target.wants/pod-nginx-pod.service → /lib/systemd/system/pod-nginx-pod.service. Created symlink /etc/systemd/system/default.target.wants/container-stoic_ganguly.service → /lib/systemd/system/container-stoic_ganguly.service.
Display Container resource usage
You can check percentage of CPU, memory, network I/O for Containers.
Step [1]Check the resource usage of each container.
# display usage statistics with streaming
root@bizantum:~# podman stats
ID NAME CPU % MEM USAGE / LIMIT MEM % NET IO BLOCK IO PIDS CPU TIME AVG CPU %
72738df86ecd priceless_nightingale 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0 0s 0.00%
7cc07ee2f437 upbeat_kilby 0.10% 4.813MB / 16.77GB 0.03% 586B / 656B 11.26MB / 4.096kB 9 12.238ms 0.10%
bc45ab289b96 focused_sammet 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0 0s 0.00%
cbc0fa585f1b strange_varahamihira 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0 0s 0.00%
d5d89f4e6596 zealous_nobel 0.14% 15.25MB / 16.77GB 0.09% 656B / 2.086kB 15.57MB / 8.192kB 56 36.817ms 0.14%
e33db7f8c043 debian-nginx 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0 0s 0.00%
ec2ab97e352a adoring_kalam 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0 0s 0.00%
fd506a88790a festive_clarke 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0 0s 0.00%
# display without streaming
root@bizantum:~# podman stats --no-stream
ID NAME CPU % MEM USAGE / LIMIT MEM % NET IO BLOCK IO PIDS CPU TIME AVG CPU %
72738df86ecd priceless_nightingale 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0 0s 0.00%
7cc07ee2f437 upbeat_kilby 0.01% 4.813MB / 16.77GB 0.03% 866B / 1.356kB 11.26MB / 4.096kB 9 12.238ms 0.01%
bc45ab289b96 focused_sammet 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0 0s 0.00%
cbc0fa585f1b strange_varahamihira 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0 0s 0.00%
d5d89f4e6596 zealous_nobel 0.02% 15.25MB / 16.77GB 0.09% 866B / 2.786kB 15.57MB / 8.192kB 56 39.762ms 0.02%
e33db7f8c043 debian-nginx 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0 0s 0.00%
ec2ab97e352a adoring_kalam 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0 0s 0.00%
fd506a88790a festive_clarke 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0 0s 0.00%
# display for a specific container
root@bizantum:~# podman stats 7cc07ee2f437 --no-stream
ID NAME CPU % MEM USAGE / LIMIT MEM % NET IO BLOCK IO PIDS CPU TIME AVG CPU %
7cc07ee2f437 upbeat_kilby 0.00% 4.813MB / 16.77GB 0.03% 936B / 1.566kB 11.26MB / 4.096kB 9 12.238ms 0.00%
# specify time interval for streaming display (sec)
# * default --interval is 5 sec
root@bizantum:~# podman stats 7cc07ee2f437 --interval 10
ID NAME CPU % MEM USAGE / LIMIT MEM % NET IO BLOCK IO PIDS CPU TIME AVG CPU %
7cc07ee2f437 upbeat_kilby 0.00% 4.813MB / 16.77GB 0.03% 1.006kB / 1.776kB 11.26MB / 4.096kB 9 12.238ms 0.00%
# display with specific format
root@bizantum:~# podman stats --no-stream --format "table {{.ID}} {{.CPUPerc}} {{.MemPerc}}"
ID CPU % MEM %
72738df86ecd 0.00% 0.00%
7cc07ee2f437 0.00% 0.03%
bc45ab289b96 0.00% 0.00%
cbc0fa585f1b 0.00% 0.00%
d5d89f4e6596 0.00% 0.09%
e33db7f8c043 0.00% 0.00%
ec2ab97e352a 0.00% 0.00%
fd506a88790a 0.00% 0.00%
- Get link
- X
- Other Apps
Bizantum
Fajar Vicky
Posts
Comments
Post a Comment
Thank you for your comment! We appreciate your feedback, feel free to check out more of our articles.
Best regards, Bizantum Blog Team.