Memory cannot be added later. What should the host keep?

Keep two ledgers

The processor has 32 cores and 64 threads. Memory is chosen at order as 128GB or 256GB of DDR4-3200 ECC RDIMM. The two sizes are two machines. After installation, no module can be added. Threads can be shared across guests over time. Memory pages do not appear because a guest was overcommitted. Two ledgers come first: one for threads, one for fixed RAM. Both start by subtracting what the host keeps.

Disks are a third ledger. The default is 2×1TB NVMe, with no hardware RAID, each disk direct-attach. How the system disk and the guest disks split is in how Proxmox splits the two direct-attach NVMe disks. A correct split does not turn 128GB into 256GB.

Bandwidth and addresses do not change the memory tier. How an extra address counts with the included one is in how extra IPv4 addresses count with the included one. This note only settles cores and RAM.

Threads the host keeps

Sixty-four threads come from 32 physical cores, two SMT threads on each core. One SMT thread is a partial core. A database, a compile job, or a guest that itself runs virtual machines takes budget in physical cores. Guests that mostly wait on the network can share threads.

The host keeps a set of threads for the web interface, networking, and storage. With no ZFS, keep 4 threads, which is 2 cores. If any pool uses ZFS, keep 8 threads, which is 4 cores, so scrub and checksum work still has room. Proxmox has no switch named host reserve. Every virtual machine’s CPU affinity avoids those threads. With no ZFS, guests use 4-63 and stay off 0–3. With ZFS, guests use 8-63 and stay off 0–7. Thread numbers run from 0 to 63. A guest left unrestricted still occupies the reserved threads, so every guest gets the same affinity. Host processes may still use every thread.

vCPUs that are busy at the same time stay inside the threads that remain: 60 with no ZFS, 56 with ZFS. Site guests may sum past that, because they do not all run full. Any guest that stays busy counts its vCPUs as cores out of 32. That cap is 30 cores with no ZFS and 28 with ZFS. Both memory tiers use the same processor. The 256GB machine does not add cores.

A database with 8 vCPUs counts as 8 cores. With no ZFS the host already holds 2 cores, so 22 cores remain for other guests that stay busy. Ten sites at 2 vCPUs each sum to 20 threads and can sit beside that database. A database at 16 vCPUs plus four guests at 8 busy vCPUs each sums to 48 cores, past 30. Reduce the busy vCPUs. The 256GB machine still has 32 cores.

After install, lscpu shows the NUMA node count. This 7543P is a single-socket processor, and the node count can still be more than one. When it is, keep a database guest’s memory and vCPUs on one node. A single node needs no extra placement.

RAM the host keeps

Proxmox documentation says to leave at least 1GB free for the host when assigning guest memory. That is the floor. This machine runs several guests, and the node also holds the interface, logs, and page cache. The budget leaves another 8GB for the node, assigned to no guest.

ZFS ARC is separate, and the real cap is read first. When the installer places the system on ZFS, Proxmox VE 8.1 and later write an ARC cap of about 10 percent of RAM, and not above 16GB, into /etc/modprobe.d/zfs.conf. Ten percent of 128GB is about 12.8GB, under the cap. Ten percent of 256GB is over 16GB, so the cap is 16GB. That cap stays on the host side. Early VE 8 media, and a pool created after install, may have no cap.

A pool created later falls back to the ZFS default, about half of RAM. From ZFS 2.3 the default can reach 62.5 percent. On 128GB that removes about 64GB before any guest exists. Set the cap, then create guests.

Read c_max in /proc/spl/kstat/zfs/arcstats. The unit is bytes. Divide by 1024 three times for GiB. If the file is missing, ZFS is not loaded and ARC is 0. To set the cap, write zfs_arc_max in zfs.conf. The value is GiB times 1024 cubed. For about 10 percent of 128GB, 12GiB is 12884901888. On the 256GB machine the ARC cap is 16GiB, 17179869184. On both sizes that cap sits above the default ARC minimum, about 1/32 of RAM, so this one line is enough. Then run update-initramfs -u and reboot the node. Do not create guests on the new number before the reboot.

The hard cap on fixed guest RAM is physical RAM, minus the ARC cap, minus 8GB.

Ballooning does not add a memory module. Proxmox lends RAM only when minimum memory is below the maximum, and only while host use stays under the target, 80 percent by default. It takes the RAM back when the host is short. A database, and any guest that itself runs virtual machines, sets minimum equal to maximum so cache is not reclaimed. Site guests may keep a gap. Shares stay at the default of 1000. KSM only merges identical pages. It does not turn 128GB into 256GB. Leave KSM out of the budget.

Plans that fit in 128GB

The comparisons below use no ZFS and a hard cap of 120GB. Fixed RAM means minimum equals maximum. Site vCPUs are not counted as always busy.

All four stay at or under 120GB, and the busy cores stay under 30. Slack is more than 8GB, so with no ZFS they can stay on 128GB. More sites still fit when each one is small and they are not all busy together. The limit in these rows is the RAM sum. Cores still have room.

Eight busy cores plus the host’s 2 cores are far from 32. On ZFS the hard cap is about 107GB. The 96GB row still has about 11GB left and can stay on 128GB. The 104GB row has about 3GB left, which is tight, so that plan moves to 256GB.

Plans that take 256GB

The first three rows pass 120GB, so 128GB with no ZFS cannot hold them. The fourth row is 112GB. It does not pass 120GB, but only 8GB remains, so it is tight and moves to 256GB. With no ZFS the 256GB hard cap is 248GB. All four rows fit, and each has more than 8GB left.

Twenty guests at 8GB, with 2 vCPUs each, sum to 40 threads. That still sits inside the 60 threads left. The limit is RAM. A database that wants 128GB fixed leaves no room for the node’s 8GB on the 128GB machine. The 256GB cap of 248GB holds it, with 120GB left.

A 64GB database plus a 48GB nested host sums to 112GB. Against 120GB only 8GB remains, so the plan moves to 256GB. If the system disk is already ZFS, the cap is about 107GB and 112GB is already over it. Count the nested host’s 48GB once, as a whole, on the physical machine. Do not add only the small guests inside it, and do not add them a second time.

Both tiers use one AMD EPYC 7543P, 32 cores / 64 threads, 2.8–3.7 GHz, 256MB L3. Memory is DDR4-3200 ECC RDIMM. 128GB is $462 a month. 256GB is $595 a month. The difference is $133. The 256GB tier adds memory. The core count stays 32. The machines are in Tokyo, Japan. Proxmox VE 8 and 9 are available with no OS charge. The default disks are 2×1TB NVMe, no hardware RAID, each direct-attach, and usable space is the sum. Bandwidth is 250M, 1G, or 2G, at $0, $255, or $745 a month, unmetered, and it does not change the RAM ledger. One IPv4 is included. Extra addresses are $2 each, up to 255 more. The amount due is the monthly total times the number of months. Monthly, quarterly, semiannual, and annual terms have no discount. There is no setup fee. The node is managed over SSH.

A guest that runs guests

Nested virtualization is for a guest that itself runs KVM or another Proxmox. Ordinary sites and databases do not need it. After it is on, inner guests spend the outer guest’s RAM and vCPUs, and the outer guest spends the physical machine. All three levels are subtracted.

Confirm svm on the host

On the node, run grep -m1 svm /proc/cpuinfo. Output means the processor has exposed AMD’s svm flag. With no output, inner guests will not get hardware assist. Stop the plan there.

Read the nested parameter

Run cat /sys/module/kvm_amd/parameters/nested. A value of 1 means the host side is already on. A value of 0 means every guest is shut down first. Create or append /etc/modprobe.d/kvm-amd.conf with options kvm-amd nested=1. The module name in that line uses a hyphen. The sysfs path uses an underscore. Both names are the same module. If the file already has lines, append. Do not overwrite it. Then reboot the node.

The outer guest uses host CPU

On the node, run qm set 101 --cpu host. Replace 101 with the outer guest’s id. The same control is Hardware, Processors, Type: host. After the change, shut the guest down from the node and start it again so QEMU is created with host CPU. A reboot inside the guest OS does not apply the new type.

After the guest is up, run the svm check inside it. Output means inner guests can use hardware assist. An outer guest with 8 vCPUs gives every inner guest those same 8, not extra physical cores. Three inner guests at 4 vCPUs each sum to 12, which is already past the outer 8. The inner layer is oversubscribed. The physical 32 cores do not grow.

Subtract the outer RAM once

An outer guest given 48GB must hold its inner fixed RAM plus what the outer system keeps, inside that 48GB. Three inner guests at 8GB sum to 24GB. The outer system keeps 8GB. 48GB covers that. One more inner guest at 32GB makes 24+32+8 = 64, past 48. Enlarge the outer guest first. That enlargement comes back out of the physical budget: 48GB becoming 64GB is subtracted again from the physical cap.

Place that layer on the earlier table. With no ZFS and a 120GB cap, a 64GB database plus this 48GB outer guest sums to 112GB, with 8GB left, so the plan moves to 256GB. The 24GB inside the outer guest is already inside the 48GB. Nested virtualization suits a lab and a few small guests inside one outer guest. Do not pack the inner layer at the density of the physical machine. RAM already given to the outer guest cannot also be given to other guests on the node.

Close the ledger before ordering

The budget uses guests that are on together, not every guest that ran once during the week. A compile guest that is off during the day stays out of the daytime sum. If it runs with the database at night, the night set is a second sum. Both sets must fit the hard cap, with more than 8GB left, before 128GB is the order. A daytime-only sum that ignores the night set runs out of RAM after install, and no module can be added.

1. Decide whether the system disk is ZFS. If it is, include the ARC cap. If it is not, ARC is 0. A pool created later gets a cap of about 10 percent, and not above 16GB, before the rest of the math.
2. The node keeps 8GB. With no ZFS, guest affinity avoids 0–3. With ZFS, it avoids 0–7.
3. Add the fixed RAM of guests that run together. A nested host is counted once, as a whole. Inner guests are not added again.
4. The sum stays on 128GB only when it is within the hard cap and the cap minus the sum is more than 8GB. Otherwise order 256GB.
5. Add vCPUs that stay busy, as cores. The cap is 30 with no ZFS and 28 with ZFS. Past that, neither memory tier adds cores.
6. Choose Proxmox VE 8 or 9. Split system and guest disks as in the disk note. After SSH to the node, check svm, the nested parameter, and ARC with the commands above.

One daytime set: a database fixed at 64GB and 8 vCPUs, minimum also 64GB; four apps at 8GB; two sites at 4GB. The RAM sum is 104GB. With no ZFS the cap is 120GB, so 16GB remains, which is more than 8GB, and the order stays at 128GB. The database is 8 cores, the host is 2, and the apps and sites are not always busy. On ZFS the cap is about 107GB, about 3GB remains, and the order moves to 256GB. If the database itself is fixed at 128GB, that one guest is already over 120GB, so the order is 256GB. With no ZFS the 256GB cap is 248GB. A 128GB database plus the original 40GB of apps and sites sums to 168GB, with 80GB left.

The core ledger is the same on both tiers. The RAM ledger is not. If the fixed-RAM sum and the slack do not fit 128GB, the order is the 256GB machine.