A review of QEMU 5.0

I will review QEMU 5 by using it to boot the Manjaro Linux ISO on Mac OS Catalina.

Introduction to QEMU

QEMU (short for quick emulator) is an emulator that you can use to run different operating systems on your machine. It is Open Source software licensed under the GPLv2 and runs on Linux, Mac OS and Windows. See the downloads page for options on how to install QEMU.

Version 5.0.0 was released on April 28th 2020 and has a long list of changes.

Install QEMU on Mac OS

Homebrew is a package manager for Mac OS. Visit Homebrew’s homepage for more information, including how to install it.

Installing QEMU is then a single command:

brew install qemu

Download the Manjaro ISO

Manjaro is based on Arch Linux and is a great Linux distribution. QEMU makes it easy to try Manjaro on another OS. Download the latest ISO for the Desktop Environment (DE) you’d like to try. The DE you use is a personal choice and depends on your preference.

For this review I decided to go with Manjaro Gnome.

Create a QEMU Harddisk file

I created a directory for the VM, and under that directory created a QEMU harddisk file with a size of 20GB (typically enough to test out almost any Linux distro).

qemu-img create -f qcow2 manjaro-gnome.qcow2 20G

Booting Manjaro in a QEMU VM

I’ve found it useful to create a startup script for each VM. This script starts QEMU with a list of optimal settings.

qemu-system-x86_64 \
  -m 2048 \
  -usb \
  -device usb-tablet \
  -cdrom "/path/to/manjaro-gnome-20.0.2-200531-linux56.iso" \
  -drive file=./manjaro-gnome.qcow2,if=virtio \
  -accel hvf \
  -smp 4 \
  -show-cursor \
  -vga virtio \
  -full-screen

A description of each setting:

-m: The amount of memory in KB. 1GB is the recommended minimum, while 2GB is more than enough if you have the RAM to spare.

-usb: Enable the VM to use the physical machine’s USB devices.

-device: usb-tablet

-cdrom: the path to the ISO file to mount.

-accel: use HVF (Mac OS’ hypervisor framework).

-smp: multiprocessor support: more cores equates to better performance. I recommend setting this to the full number of cores available, which can always be reduced if you need heavy processing on the host while the guest is running.

-show-cursor: show the guest mouse cursor.

-vga: display driver to use.

-full-screen: use fullscreen mode.

An interesting blog article on the available display drivers: https://www.kraxel.org/blog/2019/09/display-devices-in-qemu/

Booting the Manjaro ISO with QEMU

Start QEMU using the shell script created earlier:

./run-qemu.sh

The virtual machine should start and boot to the provided ISO. The boot screen is displayed:

Select Boot to start Manjaro from the ISO.

Once Manjaro has started you can either try the system directly, or click on Launch Installer to install the system on the virtual harddisk.

Conclusion

QEMU 5 is a very capable virtual machine emulator. This review was performed on Mac OS Catalina, but Windows and Linux are also supported hosts. QEMU has many more options that you can experiment with, for example, emulating a different CPU architecture than what the host provides.

Don’t be afraid to try a higher SMP setting for better performance. You can even specify the total number of cores available on the host and QEMU will handle it gracefully. Memory of 1G is the minimum recommended setting for a desktop OS.

Nim v1.2.2 released

Click the banner below to read the official blog post:

This release contains bug fixes only. This includes 6 fixes for ARC (memory manager), which was previously not considered ready for serious use due to stability issues.

ARC uses referencing counting instead garbage collection (which most other memory management options for Nim implement). According to the Nim docs this allows for “deterministic performance for hard realtime systems”. ARC was introduced in v1.2.

Another noteworthy fix is for the Nim documentation generator (nim doc) which was broken in v1.2.

Nim v1.2 and Zero Copy

Allocating and copying memory once isn’t slow, but when in a loop, or anything else that leads to millions+ of iterations you will find that performance degrades. Zero copy is the technique of referring to memory, or a segment of memory, without making a copy of it.

One example is a string slice where you want to view/compare only a portion of the string. Another example is reading a block of memory from a database or socket, and referring to fields within that block without making a copy each time.

The benefit of using zero copy techniques is that it can lead to huge gains in performance. The downside is that it can lead to bugs that are difficult to track down. For example if the original memory being referenced is freed but you continue to use the reference, or if the reference points to the wrong memory address.

C and C++ are two languages where referencing memory by using pointers is very common. Tools such as valgrind can help to identify memory related bugs, but not in all cases.

Right now Nim (v1.2) only supports this technique by use of the openarray proc. However work is ongoing to support better functionality for zero copy, see:

Related links: [RFC] View/slice #12 and let/var/openArray inside objects/containers #178.

Nim Concurrency and Parallelism

Introduction

Nim supports both threads and async/await for concurrency. New libraries are also expanding the choices developers have for concurrency and parallelism.

Parallel and Spawn

According to the Nim v1.2 manual, the new preferred way to achieve parallelism in Nim is to use parallel and/or spawn, found in the threadpool module. Spawn passes a specified task (a procedure) to the threadpool. Parallel creates a section to execute a block in parallel.

Links:

Nim in Action

Notable chapters:

  • Chapter 3: has a chat application that uses async.
  • Chapter 6: concurreny and parallelism (threads).

Example code: https://github.com/dom96/nim-in-action-code

Threads

In Nim threads are used as in most languages. The typical approach is that you create a number of threads using createThread() and wait for all of them to finish using joinThreads().

Threads are useful for when you can easily divide work up across a known number. However, knowing the number of threads to use can be difficult. The number of cores in the system can be used as an indicator, but the assumption is that these cores are free for your program to use.

Using threads can be difficult if memory is shared between threads, and that memory is modified by multiple threads. This can introduce bugs that are difficult to detect and fix. You can use locks to handle this. A lock is often called a mutex is other languages.

Links:

Async/await

Async/await is useful for when you want the main thread to keep running, but start work asynchronously, and later wait for the result. This pattern is especially useful in networking and database calls, where you are typically waiting for IO, but your program can keep working while waiting for the result of the IO.

Links:

Weave

Weave is a multi-threading runtime: https://github.com/mratsim/weave.

Features:

  • Message-passing based.
  • Scalable to available cores.
  • Fast and low-overhead.
  • Based on futures similar to async/await for concurrency.

Note the current limitations: “Weave has not been tested with GC-ed types. Pass a pointer around or use Nim channels which are GC-aware. If it works, a heads-up would be valuable. This might improve with Nim ARC/new runtime.”

Actor Model

Nim doesn’t have an actors framework/library that is maintained at present.

Parallelism in the Database

Most databases allow for parallelism, especially when working with set operators like UNION or when querying partitions. Details of database parallelism are out of scope for this article, but worth keeping in mind when performance tuning a system that uses a database.

Open Source WYSIWYG Web Editors

Web WYSIWYG (what you see is what you get) editors are useful for editing content on websites. The idea is that you can see what your content will look like while editing it. Such editors are typically seen in Content Management Systems (CMSs).

Open Source, with Price Plans

Open Source, for free

Read: Books on Javascript

Draft PostgreSQL 13 Release Notes

A draft release of the PostgreSQL 13 release notes is available. Notable changes include:

The pg_bench utility is one way to measure the performance of a new version of PostgreSQL (it has been improved to benchmark partitions). Your actual systems that run on PostgreSQL would need to be benchmarked separately.

See also: PostgreSQL Books

Aliases in Nim

Nim doesn’t have an alias keyword, but you can still create aliases using templates. The syntax is:

alias_name: untyped = name_to_create_alias_for

Here’s an example:

template settings: untyped = my_object_1.my_logging_container.all_settings

settings.name = "test"
settings.log_level = "debug"
settings.flush_frequency = 1

he use case here is that you’re working with a complex, nested object, and you want to use an alias to reduce the amount of code on the screen without a loss of performance. Using an alias gives you exactly that.

See also: Nim Books

Using PostgreSQL with Django on a Mac

PostgreSQL on Mac via Brew

  • A quick and easy way of using PostgreSQL on Mac OS is to install it via Brew.
  • pgAdmin 4 provides a web interface for administering PostgreSQL and can be installed on a local developer machine.

    Django Migrations are Optional

    Django migrations are optional, and if you have difficulties with migrations there is a manual alternative, which can be preferable for some developers:

  • Create a schema.sql file which maintains all tables for your schema that you create (skip the tables automatically created by Django for user management).
  • Add schema.sql to your version control for the project.
  • Create tables manually, or with pgAdmin 4. With pgAdmin 4 you create right-click a table and generate the create table SQL script.
  • Turn off migrations for each model in Django:
class MyTable(models.Model):
    my_table_id = models.AutoField(primary_key=True)
    name = models.CharField(max_length=256, blank=False)
    tag = models.CharField(max_length=256, blank=False)

    class Meta:
        managed = False
        db_table = 'account'
  • Disable migrations for each app that doesn’t use them in settings.py (as well as for the existing entries) in MIGRATION_MODULES, for example:
# Migration modules

MIGRATION_MODULES = {
    'my_app': None,

    'auth': None,
    'account': None,
    'contenttypes': None,
    'default': None,
    'sessions': None,

    'core': None,
    'profiles': None,
    'snippets': None,
    'scaffold_templates': None,
}

See also: Django Books

Avoiding Nim’s Gotchas

This article is written for Nim v1.2.

A lack of libraries

Nim doesn’t have the large selection of libraries that C/C++ and Python have, but you can use libraries from those languages in Nim.

Projects that help at a high-level to make this easier:

  • For C/C++ libraries use nimterop.
  • For Python modules use nimpy (also included in nimporter).

    Limited code error messages

    Nim’s error messages for invalid code can be limited/misleading. This is because the compiler reports on the immediate problem, often at a low-level, without hinting at the actual cause.

  • Not including a standard module when using its features: for example the tables or options module, but using tables or options in your code.
  • Not making a function public when calling it from another module: currently the compiler gives an ‘undeclared identifier’ error.
  • Not making a field public when calling it from another module: currently the compiler gives an ‘undeclared field’ error.
  • Not using get() when trying to get the value of an optional field of an object: current the compiler gives an ‘undeclared field’ error if you leave out the get() call.

Limited compiler error messages

  • On many Linux or Mac environments the current directory (.) is not in the path. Nim expects the current directory to be in the path or you can’t easily include your modules in your project (unless you use relative paths). This can be corrected by adding the the current directory to the path when calling the compiler. Use -p:. or –path:. when calling nim c.

See also: Nim Books

Pros and Cons of Nim

Last updated: 29th May 2020 (corrections). See also: discussion on Hacker News.

6 Reasons to Use Nim

Nim is a programming language that has a syntax similar to Python but is compiled to binary code.

See also: Nim Books

1. Friendly Syntax

Nim has a syntax that is similar to Python.

2. High Performance

Nim is much faster than Python (not when comparing C libraries wrapped for Python of course). Nim code compiles first to C then to binaries, with C-like performance always a possibility.

3. Statically Typing

Nim has a statically typed system, unlike Python where types can only be specified optionally. A statically typed language gives better information to the compiler for better optimizations.

4. Managed Memory

Memory leaks are not a concern as Nim uses one of several available garbage collectors. The new ARC option works with reference counting instead of a GC.

5. Package Manager

Nimble is a package manager which makes installing Nim libraries easy.

6. Version 1.0 Milestone

Nim has seen low levels of adoption, but this may be due to version 1.0 taking a long time to be released. Now released in September 2019 Nim has a good foundation for applications and libraries to build on.

Nim has good reasons to use it, but also has some disadvantages. These are likely to improve over the next few years.

See also: Nim Books

4 Disadvantages of using Nim

1. Small Community

Nim has a small following, which means few jobs are available, and the libraries and frameworks for Nim are lacking. C, C++ and Python libraries can be utilized, however, helping to bridge the gap.

2. Better concurrency/parallelism support needed

Nim supports threads, channels and async. However in general concurrency/parallelism needs more work.
Related article: an overview of the current state of Nim’s concurrency/parallelism features.

3. Unhelpful error messages

The Nim compiler often gives unhelpful error messages.

4. Web Frameworks are Lacking

Many languages became popular due to well developed web frameworks (e.g. Ruby has Rails, Elixir has Phoenix and Crystal has Amber, Lucky and others). Nim has Jester which still needs further development and documentation. There is also Karax which is a SPA web framework.