Linuxdev Documentation

No title

Linuxdev

Boot up Linux development env using Vagrant

	Docker Desktop	Linuxdev with Vagrant
Docker hub account	_{signin is required if you want to stop checking updates}	_{✅login to Docker hub is not necessary}
Control over the Docker Engine Version	_{Should use latest above v3, unless the user ignore the updates}	_{✅have the controll for docker-ce (community edition)}
Licensing	_{free for small business and personal, but auto-checking-update}	_{✅free of charge (w/o VB extension pack)}
Network setup	_{Host network}	_{✅Can Choose . Bridge(host) . NAT + Host-only}
Performance	_{. Hyper-v VM; might be slightly better but it likely consumes more memory}	_{. Virtualbox VM; not so bad . Has alternatives (Hyper-v/VMWare/Virtualbox)}
Environment	_{Slightly different environment between . Windows WSL2 (Custom distro integration required for Docker host access) . Windows VM (w/o Docker host acess) . Mac VM (w/o Docker host acess)}	_{✅Common Linux environment available in Mac/Windows . Full Linux VM will be provided as a Docker host (Files can be shared natively in Linux host) . Docker engine can be also accessed from the host OS (docker and docker-compose client installation required)}
Installation	_{Installer provided}	_{Install scripts provided}
Starting Docker	_{Autostart configurable}	_{Startup script is provided both for Windows and Mac}
Clients	_{Provided along with the Docker Desktop installer}	_{Install scripts provided}
Config	_{✅Configurable in GUI}	_{Configurable in script}
Docker storage	_{Configurable size}	_{✅Configurable 2k block size with a lot of inodes}
Logs/status	_{✅GUI provided}	_{Only available in CLI}
Limiting memory usuage	_{.wslconfig file for WSL2/GUI for VM}	_{.env file/VirtualBox GUI}

⚠ Note for Windows VBS (Virtualization-based security or Device Guard)

Some recent windows update enables VBS and it will turn on hyper-v and that would make Vrtualbox slow and unstable.

If systeminfo command shows following at the bottom, it’s good to use Virtualbox.

Hyper-V Requirements:      VM Monitor Mode Extensions: Yes
                           Virtualization Enabled In Firmware: Yes
                           Second Level Address Translation: Yes
                           Data Execution Prevention Available: Yes

Pease follow he issue for further guidance.

Why?

Docker is necessary for developing nowadays. But if you are not using Linux as the OS, it requires VM for Docker engine.

I was using Linux in Virtualbox for many years and found that was quite nice and had no problem for using docker in it. And personally satisfied with the performance of docker in VM In the other hand, Docker Desktop, I found couple of issue with using Docker Desktop in Windows recently.

What I didn't like about WSL/Docker Desktop

Memory usage is keep growing upto VM memory limitation
some file was missing for binding into docker
Additional settings to ssh in to the linux
ssh connection env was a bit different (not sure) when used in terminal
X client was blocked by Windows Firewall
not sure what it was but felt heavy

Some known Docker Desktop for Windows issue

Watching files in the host is not working
Hyper-v is not returning unused memory frequently so the host can be struggling with memory.

So it’s generally recommended to use WSL2 Ubunu but I would rather use VM

Linuxdev with Vagrant (VirtualBox)

Virtualbox seems to be able to run many containers more stable and does not apply much pressure to the host memory. (compared to WSL2)
Generally there are less issues in developing in Linux than Windows (Same reason to use WSL2)
Full configurable linux host (docker)
USB ports can be used directly in VM (Note: USB3 in VB is available in the extension pack, which is free for only personal users)

Also for Mac Users

Not only for the Windows users, there would be some advantage for Mac users as well

Easily backup/cleanup VM/dev env
The dev env is a sandbox
A reference environment for Linux env
Run linux GUI apps using X server and tunneling
Docker is running Linux VM already, why not?
Multiple network interfaces available

And if you are familiar with the Linux command line, using Linux makes more senses than using Docker Desktop.

Share VM when you need to bootcamp to windows
Access to docker host
Unlimited access to host(Linux VM) root directory
Faster host(Linux VM) volume sharing
Free

Performance comparison

For comparison, I tried one of my project which builds multiple docker images with node.js and filling up 20GB of docker storage.

Building includes installing node packages and webpack. Docker build kit was used for cache mounting.

Env	Build Time	Note
Linuxdev (Windows host) 6GB, i7 4 cores	24m 12s
WSL2 (Docker Descktop) 16GB, i7 6 cores (no .wslconfig)	19m 18s	spent up to 13GB
WSL2 (Docker Descktop) 8GB, i7 6 cores (with .wslconfig)	18m 22s
Linuxdev (Windows host) 8GB , i7 6 cores	✅ 14m 39s
Mac OSX (Docker Descktop) 8GB, i7 6 cores	19m 37s	First build failed after 1 hour
Linuxdev (Windows host) 8GB , i7 4 cores	21m 13s

This was just an instance of the build. Just to let you know. As you noticed, Linuxdev was the best in some situation.

Setting up the Linuxdev environment

⚠ Note for Windows users

The script will disable Hyper-v (WSL2) and replace with VM and you can also use Docker from the host OS

Running scripts

Unzip or git clone this repo
- https://github.com/kennyhyun/linuxdev/archive/refs/heads/main.zip
Run setup scripts
- Windows
  1. Install host dependencies and dev tools
    - Open powershell as Admin and run
      - Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass: enables ps1 for this powersell process
      - setup.ps1
    - This might require rebooting.
  2. Bootup vagrant with provision
    - Open terminal (git bash) and run bootstrap.sh
- Mac
  1. Install host dependencies and dev tools
    - Open teminal and run setup.sh
  2. Bootup vagrant with provision
    - continue to run bootstrap.sh
Wait until bootstrap.sh does
- input username to use in the VM
- Install Debian Linux 10.10
- Install Docker v20.10.8 (configurable in config/env_var.sh; remove VERSION if you want the latest)
- Install docker-compose (v1.29.2; configurable in .env __VM__COMPOSE_VERSION)
- Create a user with UID 1000 and sudoer
- ohmyzsh
- expose samba share, Projects
- generate id_rsa key and show public key
- add ssh config for linuxdev
- generate docker certificates and set bash variables

It’s okay to repeat this bootstrap script if something went wrong.

After finished bootstraping

Copy and paste ssh public key to use in Github and where ever it’s required.

Now you can ssh into Linux dev env

ssh linuxdev

and you can also run any linux commands from the host terminal

ssh linuxdev -t echo hello from VM

For Windows Terminal, there is also a profile generated for the machine.

An additional external configuration dotfiles project like this can be added

If DOTFILES_REPO has been defined in .env, it clones the repo to ~/dotfiles and try to run

bootstrap*
init*
install*
setup*

any of files which is executable.

Demo

setup.ps1 (Windows Powershell script; Use setup.sh for Mac)

bootstrap.sh (Mac; You can also use it in Git bash in Windows Terminal)

Packages managed by Linuxdev

Packages covered by setup (host)

vagrant
virtualbox
vscode
git
Windows Terminal (Windows)
iterm2 (Mac)
gnu-sed (Mac)
and so on…

Packages covered by bootstrap (VM)

docker (installed by vagrant provision)
docker-compose
python3-pip
git
zsh
oh-my-zsh
vim-gtk (for vim-python3)
dnsutils
and so on…

Tips

Configuring VM

Installing and using docker clients

This vm provides docker connection in 2376 port. If you have docker client and set the env vars you can use docker from the host like Docker Desktop.

You can also install docker and docker-compose by running install-docker-clients script.

.bashrc or .zshrc has DOCKER_HOST and required variables for Mac/Git-bash and docker_env.bat will set variables in the command terminal in Windows.

Bootstrap installes the required variables automaticaly so you can use docker straightaway.

Additional Goals

Share virtualbox env across Bootcamp
Provide recommended fonts with Powerline patch

License

MIT License

Configuring VM

Fri, 15 Oct 2021 00:00:00 GMT

Configuring the VM

You can create .env to customize the VM environment. Bootstrap interactively inputs the required values with some default values.

Interactive Variables

They have default values and are asked during bootstrap.sh and saved so it’s not asked again.

Name

Machine name

NAME=awesome-name

This will rename the machine name in VirtualBox. run vagrant reload to apply when updated.

It will also set the hostname of VM

Cpus and memory

CPUS=4
MEMORY=8192

This will adjust cpus and memory, run vagrant reload to apply when updated.

Vagrant Username

EXPAND_DISK_GB=10

Dotfiles Repo

DOTFILES_REPO=https://github.com/kennyhyun/dotfiles.git

This will make bootstrap continue to set VM environment using this external script repo

** This has no default variable

Optional Variables

it’s not saved automatically but you can manually add by editing .env file

Expand disk size

EXPAND_DISK_GB=10

It’s using 60GB of disk image but it’s dynamically allocated. It’s is great in most case but when the disk space is expanded, the VM performance will be deteriorated.

This will expand the disk during bootstrap. And you will have some slowness on the VM for a while but would not be slow while using the VM afterwhile.

This should be setup before running bootstrap. Or you can retry after removing /dummy

Docker lib disk

DOCKER_DISK_SIZE_GB=45

This creates a dedicated docker disk and mount to /var/lib/docker

This uses a Fixed size disk image for the performance, so please check the free space before setting this. There is a startup script to check empty disk partition and format to utilise as a docker disk. If you want to change the size, shutdown the VM and remove the image and delete existing, and change this and start the VM. Please note any data in the container will be gone with the previous disk image.

Please note that creating a fixed size image can take a few minutes, but maybe longer in Mac (like an hour). Please be patience.

This has no default value so it uses the dynamic sized system disk image (maximum 60GB).

If you had some data left in the system disk docker libs, you can see that by 1. stop docker, 2. unmounting /var/lib/docker, 3. start docker again. You can also delete that after unmounting if you don’t need that any more.

docker-compose version

COMPOSE_VERSION=1.29.2

Downloads docker-compose from https://github.com/docker/compose/releases/download/\${dc_version}/docker-compose-$(uname -s)-$(uname -m)

You can add shared directory by envionment variables. You will need to vagrat reload

HOST_PATHS=~/Projects,C:\ProgramData

If you use docker in the host OS and binding volumes, the source volume should be in the HOST_PATHS

If the path does not exist, it creates a blank directory and shares

Setting VM environment variables

If you want to set VM environment variables by the host .env file, use __VM__ prefix

__VM__VERSION=1.2.3

Will set

VERSION=1.2.3

in the VM

** __VM__ZSH_THEME will set the omz theme in the VM. Please try your favorite one

https://github.com/ohmyzsh/ohmyzsh/wiki/Themes

Setting VM hosts entries

If you want to add some hosts entry from the host .env file, use __VMHOSTS__ prefix

__VMHOSTS__127_0_0_1=somehost

Note that underscores (_) will be replaced to ..

Will addd

127.0.0.1 somehost

to /etc/hosts in the VM

If you need multiple enrties with the same IP address, you will need some suffix to distinguish enrties

__VMHOSTS__127_0_0_1=somehost
__VMHOSTS__127_0_0_1_a=anotherhost
__VMHOSTS__127_0_0_1_t=theotherhost

Anything after 4th underscore is ignored

Fonts

FONT_URLS=https://www.fontsquirrel.com/fonts/download/camingocode
PATCHED_FONT_URLS=https://github.com/tonsky/FiraCode/releases/download/5.2/Fira_Code_v5.2.zip

will download url and install those automatically.

you can use urls for ttfs or zip files separated with comma

It patches font for powerline with nerd fonts.

For already patched fonts, it can be used in PATCHED_FONT_URLS

Trouble shooting and advanced usages

Fri, 15 Oct 2021 00:00:00 GMT

Trouble shooting and advanced usages

Want to change the host only network IP

192.168.99.123 is hardcoded in Vagrantfile so you can edit

  config.vm.network "private_network", ip: "192.168.99.123"

Want to destroy the VM but use the docker lib storage in the new VM

vagrant destory will remove all the VM storage attached.

To avoid deletion of the storage attached,

Shutdown the VM first with vagrant halt
dettach the disk using Virtualbox
1. Open Oracle VM VirtualBox

2. In Tools > Media, identify docker.xx.vdi in the right pane, which is used in the VM 3. right click and click Release 3. destroy the VM running ./destroy.sh in linuxdev dir 4. bootstrap.sh 5. run sudo /root/docker.disk.sh /dev/sdb1 to attach the storage

Want to create another VM

Clone this repo in the other directory and use different machine name during bootstrap.sh

You will need to manage DOCKER_xxx variables manually

Creating machine failed

Find %USERPROFILE%\VirualBox VMs\<machine_name>\Logs and try to delete. If it cannot be deleted, see the Task Manager and

End tasks for VirtualBox Headless Frontend and try to remove the Logs directory and try again.

OSX Big Sur

VirtualBox fails to add host-only network when it’s not allowed in the Security & Privacy

Without this, creating VM using Vagrant won’t work.

Version 11.4?

I had some issue with granting the Virtualbox kernel extension, which was keep asking the permission after rebooting. Allowing actually was not working.

The issue was fixed I upgrade OSX to 16.

If you have 11.4, please consider upgrade to 11.6 or the latest.

Use ext4 partition on Windows 10

This would be just an instace how to use ext4 partitions

USB external disks can be mounted directly to the Linux on VM and if you have VirtualBox Extension pack, USB3 is also supported.

The following instruction is mounting external disk to VirtualBox without the extension pack.

Mount entire system disk to Virutalbox

create vmdk file for the physical drive

Windows Disk Management will show the external disks with the number like 0, 1, 2

if you have the only system drive, it will be 0. so the find the number of the disk which you want to mount.

And run following command in Powershell (admin) with modifying the number 2 to the number of your disk

VBoxManage internalcommands createrawvmdk -filename "$env:USERPROFILE\VirtualBox VMs\external-disk-2.vmdk" -rawdisk \\.\PhysicalDrive2

add the vmdk to your virtualbox.

  config.vm.provider "virtualbox" do |vb|
...
    disk_filename = (ENV['USERPROFILE'] || "") + "/VirtualBox VMs/ext4t2.vmdk"
    vb.customize ['storageattach', :id, '--storagectl', 'SATA Controller', '--port', 3, '--type', 'hdd', '--medium', disk_filename]
  end

You will need Administrator privileges to starth the VM.

Shutdown all the running VirtualBox VM and kill close Virtualbox and kill all running VBox related tasks. and runt vagrant reload command as Admnistrator.

When the VM is running, you can check the partitions by

ls -la /dev/disk/by-uuid/

or by root (sudo su -)

fdisk -l

mount the partition by mount /dev/disk/by-uuid/4fcc4aba-dd74-4212-b6f0-154c49c69242 /mnt/external-disk-2 or add that to /etc/fstab

And you can use startup_as_admin.bat in the scripts directory to start the VM as administrator

Tips

Sun, 26 Sep 2021 00:00:00 GMT

Tips

Useful Commands

vagrant halt to shut down the VM
vagrant up to turn on the VM
vagrant reload apply .env settings like MEMORY, CPUS with rebooting vm
./destory.sh to destroy the VM and start from scratch

If you want to repeat from scratch for some reason, you can run ./destroy.sh and retry bootstrap.sh.

Docker

docker is available and you will see the samba container running for the VM

Please use install-docker-clients script if you don’t have docker clients installed.

Vagrant Manager would be nice to have. Try setup.ps1 with -withvagrantmanager

Mapping network drive from the machine

Virtualbox machine has IP of 192.168.99.123 by default and it shares Projects directory so Host machine can see the files in it.

\\192.168.99.123\Projects

** Windows git global config should have filemode turned off

Setup parameters

.\setup.ps1 -nodevtools

./setup.sh --no-devtools

will skip installing git(for Mac), vscode, and terminal.

--no-{vscode,git,vagrant,virtualbox,...} is also available

** git should be required for Windows to run git-bash

basic os setting

-withosconfig will update registry for

Disable Secure Desktop (UAC Dimming)
Set active hour (8am to 2am)
Show hidden files and extensions
Disable Windows Update

You can also run separately by scripts/basic-config.ps1

Docker Storage

Docker tend to use many small files especially for node.js projects

If the main storage has not enough inodes, docker can fail because of the disk space. You can check that df -h has some free space but df -hi shows a low free space.

BTW, You can prune unused file by following docker command but it would rebuild required files soon.

docker system prune --volumes

This vgrantfile has additional space file of 40GB and it can be configured by DOCKER_DISK_SIZE_GB=40

Details For Windows 10 users

setup.ps1

:warning: Note that this script will disable WSL2(Hyper-V).

Please backup any required files before running. Docker will be still available by this VM

Right click windows menu and click Windows Powershell (Admin)

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass

Run the setup script in the directory of this repo

\Users\xxx\linuxdev\setup.ps1

** Running setup script again will check updates and install if newer version found

bootstrap.sh

Open Windows Terminal for Gitbash or just Git Bash

In linuxdev dir (this repo)

./bootstrap.sh

This will create virtualbox machine and bootup and config

Tips

Sun, 26 Sep 2021 00:00:00 GMT

팁

유용한 명령

vagrant halt 로 VM을 종료합니다
vagrant up 으로 VM을 시작합니다
vagrant reload 로 MEMORY, CPUS와 같은 .env 세팅을 적용하고 재부팅 합니다
./destory.sh 로 VM을 파기하고 생성전의 상태로 돌아갑니다.

여러가지 이유로 처음부터 시작하고 싶다면, ./destroy.sh 한 후 다시 bootstrap.sh 명령을 실행하면 됩니다.

도커

VM안에서 도커가 이용가능하고 VM을 위해 작동중인 samba 컨테이너를 확인할 수 있습니다.

호스트쪽에서도 도커를 사용할 수 있습니다.

도커 실행파일이 설치되어있지 않으면 install-docker-clients 스크립트를 이용해 도커 명령어를 설치할 수 있습니다

Vagrant 매니저 도 사용하시면 좋습니다.

VM으로부터 네트웍 드라이브 매핑

Virtualbox 머신은 디폴트로 IP어드레스 192.168.99.123 가 설정됩니다. 그리고 유저 홈의 Projects 디렉토리를 공유하고 있습니다. 호스트 머신은 VM내의 Projects 디렉토리를 매핑할 수 있습니다.

\\192.168.99.123\Projects

** 윈도우쪽에 설치된 git을 사용하는 경우는 global config의 filemode를 off로 설정하여야 합니다.

Setup 명령 파라메터

.\setup.ps1 -nodevtools

또는

./setup.sh --no-devtools

는 git(for Mac), vscode, terminal 등의 설치를 건너뜁니다.

--no-{vscode,git,vagrant,virtualbox,...} 옵션도 사용 할 수 있습니다

** 윈도우에서 git은 gitbash 때문에 필수입니다.

basic os setting

-withosconfig 로 다음과 같이 필수 레지스트리를 수정 합니다

Secure Desktop (UAC Dimming) 비활성
active hour 설정 (오전 8시 부터 오전 2시)
숨김파일 및 확장자 표시
Windows Update 비활성

You can also run separately by scripts/basic-config.ps1

도커 저장소

도커는 특히 node.js 프로젝트를 사용하는 경우 많은 작은 파일을 사용하게 됩니다.

만약 주 디스크 용량이 inode 갯수가 부족한 경우는 도커 명령이 실패할 수 있습니다. 만약 그러한 상황일 경우 df -h명령으로는 공간이 좀 남아있지만, df -hi명령으로는 여유가 없는 것을 확인할 수 있을 것입니다.

그럴 때에 보통 도커 커맨드로 사용하지 않는 파일들을 지워서 공간을 확보할 수 있지만 곳 다시 용량부족으로 문제가 될 수 있습니다.

docker system prune --volumes

본 Vagrantfile은 DOCKER_DISK_SIZE_GB=40 설정으로 40GB용량의 도커만의 고유 디스크 저장소를 확보 할 수 있습니다.

윈도우10 사용자를 위한 추가 설명

setup.ps1

:warning: 본 스크립트는 WSL2(Hyper-V)를 비활성 합니다.

실행전에 WSL쪽에서 필요한 파일들을 백업하세요. 도커는 VM을 통해서 다시 이용가능합니다.

윈도우 메뉴버튼을 오른클릭 하여 윈도우 파워쉘 (관리자)를 클릭합니다.

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass

본 리포지터리의 폴더 (linuxdev)의 셋업 스크립트를 실행합니다.

\Users\xxx\linuxdev\setup.ps1

** 셋업 스크립트를 다시 실행하게 되면 최신버전을 확인하고 새 버전이 있으면 설치합니다.

bootstrap.sh

Windows Terminal의 Gitbash를 열거나 Git Bash창을 엽니다.

linuxdev 디렉토리 (본 리포지터리) 에서

./bootstrap.sh

위 명령은 virtualbox 머신을 생성하고 부팅시키고 자동 설정을 수행합니다.

Linuxdev Documentation

No title

Linuxdev

⚠ Note for Windows VBS (Virtualization-based security or Device Guard)

Why?

Some known Docker Desktop for Windows issue

Linuxdev with Vagrant (VirtualBox)

Also for Mac Users

Performance comparison

Setting up the Linuxdev environment

⚠ Note for Windows users

Running scripts

After finished bootstraping

Demo

Packages managed by Linuxdev

Packages covered by setup (host)

Packages covered by bootstrap (VM)

Tips

Configuring VM

Installing and using docker clients

Additional Goals

License

Configuring VM

Configuring the VM

Interactive Variables

Name

Cpus and memory

Vagrant Username

Dotfiles Repo

Optional Variables

Expand disk size

Docker lib disk

docker-compose version

Share host directories

Setting VM environment variables

Setting VM hosts entries

Fonts

Trouble shooting and advanced usages

Trouble shooting and advanced usages

Want to change the host only network IP

Want to destroy the VM but use the docker lib storage in the new VM

Want to create another VM

Creating machine failed

OSX Big Sur

Version 11.4?

Use ext4 partition on Windows 10

Mount entire system disk to Virutalbox

create vmdk file for the physical drive

Tips

Tips

Useful Commands

Docker

Mapping network drive from the machine

Setup parameters

basic os setting

Docker Storage

Details For Windows 10 users

setup.ps1

bootstrap.sh

Tips

팁

유용한 명령

도커

VM으로부터 네트웍 드라이브 매핑

Setup 명령 파라메터

basic os setting

도커 저장소

윈도우10 사용자를 위한 추가 설명

setup.ps1

bootstrap.sh