Application Porting

The Including Programs in Redox page gives an example to port/modify a pure Rust program, in this page we explain the advanced way to port pure Rust programs, mixed Rust programs (Rust and C/C++ libraries, for example), C/C++ programs and others.

(Before reading this page you must read the Build System page)

Recipe

A recipe is how we call a software port on Redox, this section explain the recipe configuration and details to consider.

Create a folder at cookbook/recipes/program-category with a file named as recipe.toml inside, we will modify this file to fit the program needs.

  • Recipe creation from terminal with GNU Nano:
cd ~/tryredox/redox

mkdir cookbook/recipes/program-category/program-name

nano cookbook/recipes/program-category/program-name/recipe.toml

Recipe Configuration Example

The recipe configuration (recipe.toml) example below contain all supported recipe options. Adapt for your script, program, library or data files.

TOML sections and data types can also be mentioned using the section-name.data-type-name format for easier explanation and better explanation writing.

[source]
git = "repository-link" # source.git data type
upstream = "repository-link" # source.upstream data type
branch = "branch-name" # source.branch data type
rev = "version-tag" # source.rev data type
tar = "tarball-link.tar.gz" # source.tar data type
blake3 = "source-hash" # source.blake3 data type
patches = [ # source.patches data type
    "patch1.patch",
    "patch2.patch",
]
same_as = "../program-name" # source.same_as data type
script = """ # source.script data type
insert your script here
"""
[build]
template = "build-system" # build.template data type
cargoflags = "--option-name" # build.cargoflags data type
configureflags = [ # build.configureflags data type
    "OPTION1=text",
    "OPTION2=text",
]
cmakeflags = [ # build.cmakeflags data type
    "-DOPTION1=text",
    "-DOPTION2=text",
]
mesonflags = [ # build.mesonflags data type
    "-Doption1=text",
    "-Doption2=text",
]
dependencies = [ # build.dependencies data type
    "library1",
    "library2",
]
script = """ # build.script data type
# Uncomment the following if the package can be dynamically linked
#DYNAMIC_INIT
insert your script here
"""
[package]
dependencies = [ # package.dependencies data type
    "runtime-dependency1",
    "runtime-dependency2",
]
  • [source] - Section for data types that manage the program source (only remove it if you have a source folder)
  • source.git - Git repository of the program (can be removed if a Git repository is not used), you can comment out it to not allow Cookbook to force a git pull or change the active branch to master or main
  • source.upstream - If you are using a fork of the program source with patches add the program upstream source here (can be removed if the upstream source is being used on the git data type)
  • source.branch - Program version Git branch or patched Git branch (can be removed if using a tarball or the master or main Git branches are being used)
  • source.rev - Git tag or commit hash of the latest stable version or last working commit of the program (can be removed if you are using a tarball or waiting Rust library version updates)
  • source.tar - Program source tarball (can be removed if a tarball is not used)
  • source.blake3 - Program source tarball BLAKE3 hash, can be generated using the b3sum tool, install with the cargo install b3sum command (can be removed if using a Git repository or under porting)
  • source.patches - Data type to load .patch files (can be removed if patch files aren't used)
  • "patch1.patch", - The patch file name (can be removed if the patches data type above is not present)
  • source.same_as - Insert the folder of other recipe to make a symbolic link to the source folder of other recipe, useful if you want modularity with synchronization
  • source.script - Data type used when you need to change the build system configuration (to regenerate the GNU Autotools configuration, for example)
  • [build] - Section for data types that manage the program compilation and packaging
  • build.template - Insert the program build system (cargo for Rust programs, configure for programs using GNU Autotools and custom for advanced porting with - custom commands)
  • build.cargoflags - Data type for Cargo flags (string)
  • build.configureflags - Data type for GNU Autotools flags (array)
  • build.cmakeflags - Data type for CMake flags (array)
  • build.mesonflags - Data type for Meson flags (array)
  • build.dependencies - Data type to add dynamically or statically linked library dependencies
  • "library1", - Library recipe name (can be removed if the dependencies data type above is not present)
  • build.script - Data type to load the custom commands for compilation and packaging
  • [package] - Section for data types that manage the program package
  • package.dependencies - Data type to add tools, interpreters or "data files" recipes to be installed by the package manager or build system installer
  • "runtime-dependency1", - Tool, interpreter or data recipe names (can be removed if the dependencies data type above is not present)

Quick Recipe Template

This is a recipe template for a quick porting workflow.

#TODO not compiled or tested
[source]
git = "repository-link"
rev = "version-tag"
tar = "tarball-link"
[build]
template = "build-system"
dependencies = [
    "library1",
]

You can quickly copy and paste this template on each recipe.toml, that way you spent less time writting and has less chances for typos.

  • If your program use a tarball, you can quickly remove the git and rev data types.
  • If your program use a Git repository, you can quickly remove the tar data type.
  • If you don't need to pin a Git tag or commit hash for the latest stable release or last working commit, you can quickly remove the rev data type.
  • If the program don't need C, C++ or patched Rust dependencies, you can quickly remove the dependencies = [] section.

After the #TODO comment you will write your current porting status.

Cookbook

The GCC and LLVM compiler frontends on Linux use the Linux target triplet by default, it will create Linux ELF binaries that don't work on Redox because it can't undertstand them.

Part of this process is to use glibc (GNU C Standard Library) which don't support Redox system calls, to make the compiler use relibc (Redox C Standard Library) Cookbook need to tell the build system of the program or library to use it, it's done with environment variables and target/platform flags for the Redox target.

Cookbook have build system templates to avoid custom commands for cross-compilation, but it's not always possible because some build systems or programs aren't standardized or adapted for cross-compilation.

(Build systems have different methods to enable cross-compilation and pass a different C standard library to the compiler, you will need to read their documentation, program/library specific configuration or hack them)

Cross Compiler

Cookbook use Rust/GCC forks to do cross-compilation of recipes (programs) with relibc to any supported CPU architecture, you can check our cross-compilers on GitLab (GCC, LLVM, Rust and their pre-compiled binaries).

Cross Compilation

The Cookbook default compilation type is cross-compilation because it reduces the requirements to run programs on Redox and allow us to do Redox development from Linux and other Unix-like systems.

By default Cookbook use the CPU architecture of your host system but you can change it easily on your .config file (ARCH? environment variable).

  • Don't use a hardcoded CPU architecture in the script data types of your recipe.toml, it breaks cross-compilation with a different CPU architecture is used by the build system.
  • All recipes must use our cross-compilers, a Cookbook template does this automatically but it's not always possible, read the build system configuration of your program/library to find these options or patch the configuration files.

Templates

A recipe template is the build system of the program or library supported by Cookbook.

  • template = "cargo" - Build with Cargo using cross-compilation and static linking (Rust programs with one package in the Cargo workspace).
  • template = "configure" - Build with GNU Autotools/GNU Make using cross-compilation and dynamic linking.
  • template = "cmake" - Build with CMake using cross-compilation and dynamic linking.
  • template = "meson" - Build with Meson using cross-compilation and dynamic linking.
  • template = "remote" - Download the remote Redox package of the recipe if available in the package server
  • template = "custom" - Run your commands on the script = field and build (Any build system or installation process).

The script = field runs any terminal command supported by GNU Bash, it's important if the build system of the program don't support cross-compilation or need custom options to work on Redox (you can't use the build.script data type if the custom template is not used).

Each template (except custom) script supports build flags, you can add flags as an array of strings:

  • cargoflags = "foo"
  • configureflags = [ "foo" ]
  • cmakeflags = [ "foo" ]
  • mesonflags = [ "foo" ]

To find the supported Cookbook Bash functions, look the recipes using a script = field on their recipe.toml or read the source code.

Cases

  • Programs using the Cargo build system have a Cargo.toml file
  • Programs using the GNU Autotools build system have a configure or autogen.sh file in the source tarball
  • Programs using the CMake build system have a CMakeLists.txt file
  • Programs using the Meson build system have a meson.build file

Metapackages

Metapackages are packages without any files, just dependencies. Use the following recipe example to create a metapackage:

[package]
dependencies = [
    "package1",
    "package2",
]

Cookbook - Custom Template

The custom template enable the build.script = data type to be used, this data type will run any command supported by the GNU Bash shell. The shell script will be wrapped with Bash functions and variables to aid the build script. The wrapper can be found in this Cookbook source file.

  • Script example
[build]
script = """
first-command
second-command
"""

The script section starts at the location of the ${COOKBOOK_BUILD} environment variable (recipe-name/target/$TARGET/build). This ${COOKBOOK_BUILD} will be an empty folder, while recipe sources are in ${COOKBOOK_SOURCE}. It is expected that the build script will not modify anything in ${COOKBOOK_SOURCE}, otherwise, please use the source.script = data type.

Functions

Each template has a Bash function to be used in the script data type when you need to customize the template configuration to fix the program compilation or enable/disable features.

  • cookbook_cargo - Bash function of the cargo template
  • cookbook_configure - Bash function of the configure template
  • cookbook_cmake - Bash function of the cmake template
  • cookbook_meson - Bash function of the meson template
  • DYNAMIC_INIT - Bash function to configure the recipe to be dynamically linked
  • DYNAMIC_STATIC_INIT - Bash function to configure the recipe to be both statically and dynamically linked (library recipe only)

Environment Variables

These variables available in the script:

  • ${TARGET} - Redox compiler triple target ($ARCH-unknown-redox)

  • ${GNU_TARGET} - Redox compiler triple target (GNU variant)

  • ${COOKBOOK_MAKE_JOBS} - Total CPU threads available

  • ${COOKBOOK_RECIPE} - Recipe folder.

  • ${COOKBOOK_ROOT} - The Cookbook directory.

  • ${COOKBOOK_SOURCE} - The source folder at recipe-name/source (program source).

  • ${COOKBOOK_SYSROOT} - The sysroot folder at recipe-name/target/$TARGET/sysroot (library sources).

  • ${COOKBOOK_BUILD} - The build folder at recipe-name/target/$TARGET/build (recipe build system).

  • ${COOKBOOK_STAGE} - The stage folder at recipe-name/target/$TARGET/stage (recipe binaries).

  • For RISC-V, ${TARGET} and ${GNU_TARGET} is riscv64gc-unknown-redox and riscv64-unknown-redox, usually you want ${TARGET} unless the script requires a GNU target triple.

  • To get $ARCH, you need to add ARCH="${TARGET%%-*}" to the beginning of the script.

There are more variables depending on the build script that you are using.

We recommend that you use path environment variables with the " symbol to clean any invalid characters (like spaces) on the path, spaces are interpreted as command separators and will break the path.

Example:

"${VARIABLE_NAME}"

If you have a folder inside the variable folder you can call it with:

"${VARIABLE_NAME}"/folder-name

Or

"${VARIABLE_NAME}/folder-name"

Quick Template

You can quickly copy these environment variables from this section.

"${COOKBOOK_SOURCE}/"

"${COOKBOOK_BUILD}/"

"${COOKBOOK_SYSROOT}/"

"${COOKBOOK_STAGE}/"

Packaging Behavior

Cookbook download the recipe sources on the source folder (recipe-name/source), copy the contents of this folder to the build folder (recipe-name/target/$TARGET/build), build the sources and move the binaries to the stage folder (recipe-name/target/$TARGET/stage).

If your recipe has library dependencies, it will copy the library source and linker objects to the sysroot folder to be used by the build folder.

  • Moving the program files to the Redox filesystem

The "${COOKBOOK_STAGE}"/ path is used to specify where the recipe files will be stored in the Redox filesystem, in most cases /usr/bin and /usr/lib.

You can see path examples for most customized recipes below:

"${COOKBOOK_STAGE}"/ # The root of the Redox build system
"${COOKBOOK_STAGE}"/usr/bin # The folder where all system-wide executables go
"${COOKBOOK_STAGE}"/usr/lib # The folder where all system-wide static and shared library objects go

GNU Autotools script

Use this script if the program or library needs to be compiled with custom options

  • Configure with dynamic linking
script = """
DYNAMIC_INIT
COOKBOOK_CONFIGURE_FLAGS+=(
    --option1
    --option2
)
cookbook_configure
"""
  • GNU Make without Configure
script = """
DYNAMIC_INIT
COOKBOOK_CONFIGURE_FLAGS+=(
    --option1
    --option2
)
COOKBOOK_CONFIGURE="true"

rsync -av --delete "${COOKBOOK_SOURCE}/" ./
cookbook_configure
"""

Definition of cookbook_configure is roughly:

function cookbook_configure {
    "${COOKBOOK_CONFIGURE}" "${COOKBOOK_CONFIGURE_FLAGS[@]}" "$@"
    "${COOKBOOK_MAKE}" -j "${COOKBOOK_MAKE_JOBS}"
    "${COOKBOOK_MAKE}" install DESTDIR="${COOKBOOK_STAGE}"
}

GNU Autotools configuration script

Sometimes the program tarball or repository is lacking the configure script, so you will need to generate this script.

  • Add the following code below the [source] section
script = """
DYNAMIC_INIT
autotools_recursive_regenerate
"""

CMake script

Use this script for programs using the CMake build system, more CMake options can be added with a -D before them, the customization of CMake compilation is very easy.

  • CMake using dynamic linking
script = """
DYNAMIC_INIT
COOKBOOK_CMAKE_FLAGS+=(
    -DOPTION1=text
    -DOPTION2=text
)
cookbook_cmake
"""
  • CMake inside a subfolder
script = """
DYNAMIC_INIT
COOKBOOK_CMAKE_FLAGS+=(
    -DOPTION1=text
    -DOPTION2=text
)
cookbook_cmake "${COOKBOOK_SOURCE}"/subfolder
"""

Definition of cookbook_cmake is roughly:

function cookbook_cmake {
    "${COOKBOOK_CMAKE}" "${COOKBOOK_SOURCE}" \
        "${COOKBOOK_CMAKE_FLAGS[@]}" \
        "$@"

    "${COOKBOOK_NINJA}" -j"${COOKBOOK_MAKE_JOBS}"
    DESTDIR="${COOKBOOK_STAGE}" "${COOKBOOK_NINJA}" install -j"${COOKBOOK_MAKE_JOBS}"
}

Meson script

Use this script for programs using the Meson build system, more Meson options can be added with a -D before them, the customization of Meson compilation is very easy.

Keep in mind that some programs and libraries need more configuration to work.

  • Meson using dynamic linking
script = """
DYNAMIC_INIT
COOKBOOK_MESON_FLAGS+=(
    -Doption1
    -Doption2
)
cookbook_meson
"""
  • Meson inside a subfolder
script = """
DYNAMIC_INIT
COOKBOOK_MESON_FLAGS+=(
    -Doption1
    -Doption2
)
cookbook_meson "${COOKBOOK_SOURCE}"/subfolder
"""

Cargo script

Use this script if you need to customize the cookbook_cargo function.

script = """
COOKBOOK_CARGO_FLAGS=(
    --bin foo
)
PACKAGE_PATH="subfolder" cookbook_cargo "${COOKBOOK_CARGO_FLAGS[@]}"
"""

If the project is roughly a simple Cargo project then cookbook_cargo is all that you need.

script = """
cookbook_cargo
"""

Analyze the source code of a Rust program

Rust programs and libraries use the Cargo.toml configuration file to configure the build system and source code.

While packaging Rust programs you need to know where the main executable is located in the Cargo project, to do this you need to verify the Cargo.toml files of the project.

A Rust program can have one or more Cargo packages to build, read the common assumptions below:

  • Most Rust programs with a src folder use one Cargo package, thus you can use the cargo template.
  • Most Rust programs with multiple Cargo packages name the main package with the name of the program.

Beyond these common source code organization, there are special cases.

  • In some Rust programs the Cargo.toml file contains one of these data types:
[[bin]]
name = "executable-name"
[[lib]]
name = "library-object-name"

The [[bin]] is what you need, the program executable is built by this Cargo package.

But some programs don't have the [[bin]] and [[lib]] data types, for these cases you need to see the source code files, in most cases at the src folder.

  • The file named main.rs contains the program executable code.
  • The file named lib.rs contains the library object code (ignore it).

(Some Rust programs use packages instead of example files for examples, to discover that see if the "examples" folder has .rs files (examples files) or folders with Cargo.toml files inside (packages) )

Cargo packages command example

This command is used for Rust programs that use package folders inside the repository for compilation, you need to use the name on the name field below the [package] section of the Cargo.toml file inside the package folder (generally using the same name of the program).

(This will fix the "found virtual manifest instead of package manifest" error)

script = """
cookbook_cargo_packages program-name
"""

(You can use cookbook_cargo_packages program1 program2 if it's more than one package)

Cargo package with flags

If you need a script for a package with flags (customization), you can use this script:

script = """
package=package-name
"${COOKBOOK_CARGO}" build \
            --manifest-path "${COOKBOOK_SOURCE}/Cargo.toml" \
            --package "${package}" \
            --release \
            --add-your-flag-here
        mkdir -pv "${COOKBOOK_STAGE}/usr/bin"
        cp -v \
            "target/${TARGET}/release/${package}" \
            "${COOKBOOK_STAGE}/usr/bin/${package}"
"""
  • The package-name after package= is where you will insert the Cargo package name of your program.
  • The --add-your-flag-here will be replaced by the program flag.

Cargo bins script example

Some Rust programs use bins instead of packages to build, to build them you can use this script:

script = """
binary=bin-name
"${COOKBOOK_CARGO}" build \
            --manifest-path "${COOKBOOK_SOURCE}/Cargo.toml" \
            --bin "${binary}" \
            --release \
            --add-your-flag-here
        mkdir -pv "${COOKBOOK_STAGE}/usr/bin"
        cp -v \
            "target/${TARGET}/release/${binary}" \
            "${COOKBOOK_STAGE}/usr/bin/${binary}"
"""
  • The bin-name after binary= is where you will insert the Cargo package name of your program.
  • The --add-your-flag-here will be replaced by the program flags.

Cargo flags command example

Some Rust programs have flags for customization, you can find them below the [features] section in the Cargo.toml file.

script = """
cookbook_cargo --features flag-name
"""

Disable the default Cargo flags

It's common that some flag of the program doesn't work on Redox, if you don't want to spend much time testing flags that work and don't work, you can disable all of them to see if the most basic featureset of the program works with this script:

script = """
cookbook_cargo --no-default-features
"""

Enable all Cargo flags

If you want to enable all flags of the program, use:

script = """
cookbook_cargo --all-features
"""

Cargo profiles command example

This script is used for Rust programs using Cargo profiles.

script = """
cookbook_cargo --profile profile-name
"""

Cargo examples command example

This script is used for examples on Rust programs.

script = """
cookbook_cargo_examples example-name
"""

(You can use cookbook_cargo_examples example1 example2 if it's more than one example)

Cargo examples with flags

This script is used for Cargo examples with flags.

script = """
recipe="$(basename "${COOKBOOK_RECIPE}")"
    for example in example1 example2
    do
        "${COOKBOOK_CARGO}" build \
            --manifest-path "${COOKBOOK_SOURCE}/${PACKAGE_PATH}/Cargo.toml" \
            --example "${example}" \
            --release \
            --add-your-flag-here
        mkdir -pv "${COOKBOOK_STAGE}/usr/bin"
        cp -v \
            "target/${TARGET}/${build_type}/examples/${example}" \
            "${COOKBOOK_STAGE}/usr/bin/${recipe}_${example}"
    done
"""

(Replace the example1 item and others with the example names, if the program has only one example you can remove the example2 item)

Rename binaries

Some programs or examples use generic names for their executable files which could cause conflicts in the package installation process, to avoid this use the following command after the compilation or installation commands:

mv "${COOKBOOK_STAGE}/usr/bin/binary-name" "${COOKBOOK_STAGE}/usr/bin/new-binary-name"
  • Duplicated names

Some recipes for Rust programs can duplicate the program name in the executable (name_name), you can also use the command above to fix these cases.

Change the active source code folder

Sometimes a program don't store the source code on the root of the Git repository, but in a subfolder.

For these cases you need to change the directory of the ${COOKBOOK_SOURCE} environment variable in the beginning of the build.script data type, to do this add the following command on your recipe script:

COOKBOOK_SOURCE="${COOKBOOK_SOURCE}/subfolder-name"
  • An example for a Rust program:
script = """
COOKBOOK_SOURCE="${COOKBOOK_SOURCE}/subfolder-name"
cookbook_cargo
"""

Configuration Files

Some programs require to setup configuration files from the source code or tarball, to setup them use the following script example:

[build]
template = "custom"
script = """
cookbook function or custom build system commands
mkdir -pv "${COOKBOOK_STAGE}"/usr/share # create the /usr/share folder inside the package
cp -rv "${COOKBOOK_SOURCE}"/configuration-file "${COOKBOOK_STAGE}"/usr/share # copy the configuration file from the program source code to the package
"""

Modify the script above to your needs.

Script-based programs

Read the following scripts to package interpreted programs.

Adapted scripts

This script is for scripts adapted to be packaged, they contain shebangs and renamed the file to remove the script extension.

(Some programs and libraries need more configuration to work)

  • One script
script = """
mkdir -pv "${COOKBOOK_STAGE}"/usr/bin
cp "${COOKBOOK_SOURCE}"/script-name "${COOKBOOK_STAGE}"/usr/bin/script-name
chmod a+x "${COOKBOOK_STAGE}"/usr/bin/script-name
"""

This script will move the script from the source folder to the stage folder and mark it as executable to be packaged.

(Probably you need to mark it as executable, we don't know if all scripts carry executable permission)

  • Multiple scripts
script = """
mkdir -pv "${COOKBOOK_STAGE}"/usr/bin
cp "${COOKBOOK_SOURCE}"/* "${COOKBOOK_STAGE}"/usr/bin
chmod a+x "${COOKBOOK_STAGE}"/usr/bin/*
"""

This script will move the scripts from the source folder to the stage folder and mark them as executable to be packaged.

Non-adapted scripts

You need to use the following script examples for scripts not adapted for packaging, you need to add shebangs, rename the file to remove the script extension (.py) and mark as executable (chmod a+x).

(Some programs and libraries need more configuration to work)

  • Python script example
script = """
mkdir -pv "${COOKBOOK_STAGE}"/usr/bin
cp "${COOKBOOK_SOURCE}"/script-name.py "${COOKBOOK_STAGE}"/usr/bin/script-name
chmod a+x "${COOKBOOK_STAGE}"/usr/bin/script-name
"""

(Rename the "script-name" parts with your script name and the .py extension for your script programming language extension if needed)

This script will rename your script name, make it executable and package.

  • Multiple scripts
script = """
mkdir -pv "${COOKBOOK_STAGE}"/usr/bin
for script in "${COOKBOOK_SOURCE}"/*
do
  shortname=`basename "$script" ".py"`
  cp -v "$script" "${COOKBOOK_STAGE}"/usr/bin/"$shortname"
  chmod a+x "${COOKBOOK_STAGE}"/usr/bin/"$shortname"
done
"""

This script will rename all scripts to remove the .py extension, mark all scripts as executable and package.

  • Shebang

It's the magic behind executable scripts as it make the system interpret the script as an common executable, if your script doesn't have a shebang on the beginning it can't be launched like an conventional compiled program executable.

To allow this use the following script:

script = """
mkdir -pv "${COOKBOOK_STAGE}"/usr/bin
cp "${COOKBOOK_SOURCE}"/script-name.py "${COOKBOOK_STAGE}"/usr/bin/script-name
sed -i '1 i\#!/usr/bin/env python3' "${COOKBOOK_STAGE}"/usr/bin/script-name
chmod a+x "${COOKBOOK_STAGE}"/usr/bin/script-name
"""

The sed -i '1 i\#!/usr/bin/env python3' "${COOKBOOK_STAGE}"/usr/bin/script-name command will add the shebang on the beginning of your script.

The python3 is the script interpreter in this case, use bash or lua or whatever interpreter is appropriate for your case.

There are many combinations for these script examples: you can download scripts without the [source] section, make customized installations, etc.

Dynamically Linked Programs

The DYNAMIC_INIT acts as a marker that indicates the recipe can be dynamically linked. It automatically sets LDFLAGS and RUSTFLAGS based on the preferred linkage. See the environment variables section under configuration settings for more information.

In most cases if you want to use dynamic linking for a recipe just prepend DYNAMIC_INIT in the recipe script. Depending on the recipe, this should suffice. However, sometimes you may need to regenerate the GNU Autotools configuration, which you can do by invoking the autotools_recursive_regenerate helper function after DYNAMIC_INIT (See the examples below). This is to make sure the build system uses our libtool fork. In other cases, more recipe-specific modification may be required.

Example

# <...snip...>

[build]
template = "custom"
script = """
+DYNAMIC_INIT
cookbook_configure
"""

# <...snip...>
[source]
+script = """
+DYNAMIC_INIT
+autotools_recursive_regenerate
+"""

[build]
template = "custom"
script = """
+DYNAMIC_INIT
+cookbook_configure
"""

Dynamically linked programs depend on shared libraries at runtime. To include these libraries, you must add them in the build.dependencies data type.

Example

# <...snip...>

[build]
dependencies = [
    "libmpc",
    "libgmp",
]

Troubleshooting

  • Why the dynamic linker (ld.so) is not finding my library?

Set LD_DEBUG=all and re-run the program. It will show you where library objects are being found and loaded, as well as the library search paths. You probably forgot to add a library in the build.dependencies list. You can also use patchelf on your host or on Redox to display all DT_NEEDED entries of an object (patchelf --print-needed <path>). It is available by default in the desktop variant.

Sources

Tarballs

Tarballs are the most easy way to build a C/C++ program or library because the build system is already configured (GNU Autotools is the most used), while being more fast to download and process than big Git repositories if shallow clone is unused (the system don't need to process Git deltas).

Your recipe.toml will have the following content:

[source]
tar = "tarball-link"

Copy the tarball link and paste in the tarball-link field.

Only use official tarballs, GitHub auto-generate tarballs for each new release or tag of the program, but they aren't static (break the checksum) and don't verify the archive integrity.

You can find the official tarballs in the release announcement assets with the program name and ending with tar.gz or tar.xz (their URLs contain "releases" instead of "archive"), while unstable tarballs can be found on the "Source code" buttons (their URLs contain "archive").

  • In most cases they are created using the GNU Tar tool.
  • Avoid files containing the names "linux" and "x86_64" on GitHub, they are pre-built binaries for some operating system and CPU architecture, not source code.
  • Some programs require Git submodules to work, you can't use tarballs if the official tarball don't bundle the submodules.
  • Archives with tar.xz and tar.bz2 are preferred as they tend to have a higher compression level, thus smaller file size.

Build System

In most cases the tarballs use GNU Autotools to build, it's common that the tarball method of compilation is not well documented, causing confusion on new packagers.

To investigate, you can do the following things:

  • Build with the configure template and see if it works (sometimes you need to use some flag or customize)
  • Search the Git repository of the program or library for autogen.sh and configure.ac files, it means that support for GNU Autotools is available, when some tarball is created, it comes with a configure file inside, this file doesn't exist on the Git repository and you need to create it by running the autogen.sh script.
  • Sometimes these files are available but GNU Autotools is deprecated (because it's old), we recommend that you use the supported build system (CMake or Meson in most cases).

Sometimes it's hard to find the official tarball of some software, as each project website organization is different.

To help on this process, the Arch Linux packages and AUR are the most easy repositories to find tarball links in the configuration of packages.

  • Arch Linux packages: Search for your program, open the program page, see the "Package Actions" category on the top right position and click on the "Source Files" button, a GitLab page will open, open the .SRCINFO and search for the tarball link on the "source" fields of the file.

See the nano package example.

  • AUR: Search for your program, open the program page, go to the "Sources" section on the end of the package details.

Git Repositories

Some programs don't offer official tarballs for releases, thus you need to use their Git repository and pin the tag or commit hash of the latest stable version or last working commit.

Your recipe.toml will have the following content:

[source]
git = "repository-link"
rev = "version-tag"

GitHub release

Each GitHub release has a tag or commit hash, you will use it to pin the lastest stable version of the program to keep code stability.

Example:

  • Open the Rust 1.74 release announcement
  • The tag is 1.74.0 and the commit hash is 79e9716c980570bfd1f666e3b16ac583f0168962 and is shortened as 79e9716

GitLab release commit hash

Each GitLab release has a tag or commit hash, you will use it to pin the lastest stable version of the program to keep code stability.

Example:

Dependencies

A program dependency can be a library (a program that offer functions to some program), a runtime (a program that satisfy some program dependency when it's executed) or a build tool (a program to configure/build some program).

Most C, C++ and Rust programs place build tools/runtime together with development libraries (packages with -dev suffix) in their build instructions documentation.

Example:

sudo apt-get install cmake libssl-dev

The cmake package is the build system (build tool) while the libssl-dev package is the library (OpenSSL) linker objects (.a and .so files), the Debian package system bundle shared/static objects on their -dev packages (other Linux distributions just bundle shared objects).

You would need to create a recipe of the libssl-dev package and add in the build.dependencies data type of your recipe.toml file, while the cmake package would need to be installed on your system.

Dependencies added in the build.dependencies data type can be statically linked (if the DYNAMIC_INIT function is not used) or dynamically linked (if the DYNAMIC_INIT function is used), while dependencies added in the package.dependencies data type will be installed by the build system installer or package manager.

Mixed Rust programs have crates ending with -sys to use bundled or system C/C++ libraries.

If you want an easy way to find dependencies, see the Debian stable packages list.

You can search them with Ctrl+F, all package names are clickable and their websites is available on the right-side of the package description/details.

  • We recommend to use the FreeBSD dependencies of the program if available because Linux dependencies tend to contain Linux-specific kernel features not available on Redox (unfortunately the FreeBSD package naming policy don't separate library objects/interpreters from build tools in all cases, thus you need to know or search each item to know if it's a library, interpreter or build tool)
  • Debian packages are the most easy way to find dependencies because they are the most used by software developers to describe "Build Instructions" dependencies.
  • Don't use the .deb packages to create recipes, they are adapted for the Debian environment.
  • The Debian naming policy use dashes as separators in packages with custom options (program or library variant) enabled (program-variant), check the source package to be sure
  • The recipe PATH environment variable only use build tools at /usr/bin, it don't read the /usr/lib and /include folders because the Linux library objects don't work on Redox.
  • Don't add build tools in the build.dependencies data type, check the Debian and Arch Linux meta-packages for a common reference of build tools.
  • The compiler will build the development libraries as .a files (objects for static linking) or .so files (objects for dynamic linking), the .a files will be mixed in the final binary while the .so files will be installed out of the binary (stored on the /lib directory of the system).
  • Linux distributions add a number after the .so files to avoid conflicts on the /usr/lib folder when packages use different API versions of the same library, for example: library-name.so.6.
  • You need to know this information because each software is different, the major reason is the "Build Instructions" organization of each program.

If you have questions about program dependencies, feel free to ask us on the Chat.

Bundled Libraries

Some programs have bundled libraries, using CMake or a Python script, the most common case is using CMake (emulators do this in most cases).

The reason for this can be control over library versions to avoid compilation/runtime errors or a patched library with optimizations for specific tasks of the program.

In some cases some bundled library needs a Redox patch, if not it will give a compilation or runtime error.

Most programs using CMake will try to detect the system libraries on the build environment, if not they will use the bundled libraries.

The "system libraries" on this case is the recipes specified on the build.dependencies = [] section of your recipe.toml.

To determine if you need to use a Redox recipe as dependency check if you find a .patch file on the recipe folder or if the recipe.toml has a git = field pointing to the Redox GitLab, if not you can probably use the bundled libraries without problems.

Generally programs with CMake use a -DUSE_SYSTEM flag to enable the "system libraries" behavior.

Environment Variables

Sometimes specify the library recipe on the dependencies = [] section is not enough, some build systems have environment variables to receive a custom path for external libraries.

When you add a library on your recipe.toml the Cookbook will copy the library source code to the sysroot folder at cookbook/recipes/your-category/recipe-name/target/your-target, this folder has an environment variable that can be used inside the script = field on your recipe.toml.

Example:

script = """
export OPENSSL_DIR="${COOKBOOK_SYSROOT}"
cookbook_cargo
"""

The export will active the OPENSSL_DIR variable on the environment, this variable is implemented by the program build system, it's a way to specify the custom OpenSSL path to the program's build system, as you can see, when the òpenssl recipe is added to the dependencies = [] section its sources go to the sysroot folder.

Now the program build system is satisfied with the OpenSSL sources, the cookbook_cargo function calls Cargo to build it.

Programs using CMake don't use environment variables but an option, see this example:

script = """
COOKBOOK_CMAKE_FLAGS+=(
    -DOPENSSL_ROOT_DIR="${COOKBOOK_SYSROOT}"
)
cookbook_cmake
"""

On this example the -DOPENSSL_ROOT_DIR option will have the custom OpenSSL path.

Submodules

In some programs or libraries you can't use tarballs because they don't carry the necessary Git submodules of the program (most common in GitHub generated tarballs), on these cases you will need to use the Git repository or the commit of the last stable release (Cookbook download the submodules automatically).

To identify if the program use Git submodules, check if it have external folders to other repository (they appear with a commit hash on the right side) or the existence of a .gitmodules file.

Follow these steps to use the last stable version of the program when Git submodules are necessary:

  • Open the program/library Git repository.
  • Check the "Releases" or "Tags" buttons, in most cases the program have a stable release at "Releases".
  • In both pages the commit hash of the stable release will be the first item of the announcement below the version number.
  • Copy the repository link/release commit and paste on your recipe.toml, for example:
git = "your-repository-link"
rev = "your-release-commit"

If the last stable release is years behind, we recommend that you ignore it and use the Git repository to download/build bug fixes sent after this old version, if you are concerned about the program upstream breaking the recipe, you can use the commit of the last successful CI test.

Configuration

The determine the program dependencies you can use Arch Linux and Gentoo as reference.

  • The build instructions of C/C++ programs tend to mix necessary and optional dependencies together.
  • Most Rust programs have build instructions focused on Linux and force some dependencies, some crates could not need them to work, investigate which crates the program is using.
  • Some programs and libraries have bad documentation, lack build instructions or explain the dependencies, for these cases you will need to read third-party sources or examine the build system.

Arch Linux and AUR are the most simple references because they separate the build tools from runtimes and build dependencies, thus you make less mistakes.

They also have less expanded packages, while on Debian is common to have highly expanded programs and libraries, sometimes causing confusion.

(An expanded package is when most or all optional dependencies are enabled)

But Arch Linux is not clear about the optional feature flags and minimum dependencies to build and execute a program.

Using Gentoo as reference you can learn how to make the most minimum Redox port and increase your chances to make it work on Redox.

But Gentoo modify the feature flags of their packages to be used by their package system, thus you should use the FreeBSD Ports.

Arch Linux and AUR

Each package page of some program has a "Dependencies" section on the package details, see the items below:

  • dependency-name - Build or runtime dependencies, they lack the () symbol (required to make the program build and execute)
  • dependency-name (make) - Build tools (required to build the program)
  • dependency-name (optional) - Programs or libraries to expand the program functionality

See the firefox package, for example.

Gentoo

The Gentoo distribution does a wonderful job to document many programs and libraries, like source code location, dependencies, feature flags, cross-compilation and context.

It's the most complete reference for advanced packaging of programs, you can search the Gentoo packages on the Gentoo Packages website.

To start you need to read the Gentoo documentation page to learn advanced packaging and some problems.

The "Dependencies" section of a Gentoo package will show a table with the following categories:

  • BDEPEND - Host build tools (don't add them on the dependencies = [] section of your recipe.toml)
  • DEPEND - These dependencies are necessary to build the program
  • RDEPEND - These dependencies are necessary to execute the program, can be mandatory or optional
  • PDEPEND - Optional dependencies (customization)

The complex classification of Gentoo allow the packager to easily make a minimum build of a program on Redox, it's important because some optional dependencies can use APIs from the Linux kernel not present on Redox.

Thus the best approach is to know the minimum necessary to make the program work on Redox and expand from that.

Build Tools

Add missing recipe build tools in the podman/redox-base-containerfile file (for Podman builds) or install them on your system (for Native builds).

The podman/redox-base-containerfile file and native_bootstrap.sh script covers the build tools required by recipes on the demo.toml filesystem configuration.

Feature Flags

The program/library build systems offer flags to enable/disable features, it will increase the chance to make them work on Redox by disabling Linux-specific or unsupported features/libraries.

Sometimes you need to read the build system configuration to find important or all flags that weren't documented by the program.

Cargo

You can find the feature flags below the [features] section in the Cargo.toml file.

GNU Autotools

You can find the feature flags in the INSTALL or configure files.

CMake

You can find the feature flags in the CMakeLists.txt file.

Meson

You can find the feature flags in the meson_options file.

FreeBSD Reference

If you can't find the program build system flags the FreeBSD port Makefiles are the best reference for feature flags to Redox as they tend to disable Linux-specific features and are adapted to cross-compilation, increasing the program/library compatiblity with non-Linux systems.

(You need to disable the program/library's build system tests to make cross-compilation work)

(Use the "Go to file" button to search for the software name)

Building/Testing The Program

Tip: If you want to avoid problems not related to Redox install the program dependencies and build to your system first (if packages for your Unix-like distribution aren't available search for Debian/Ubuntu equivalents).

To build your recipe, run:

make r.recipe-name

If you get an error read the log and determine if it is one of the following problems:

  • Missing build tools
  • Cross-compilation configuration problem
  • Lack of Redox patches
  • Missing C, POSIX or Linux library functions in relibc

Use this command to log any possible errors on your terminal output:

make r.recipe-name 2>&1 | tee recipe-name.log

If the compilation was successful the recipe can be installed in the QEMU image and tested inside of Redox to find possible runtime errors or crashes.

  • To temporarily install the recipe to your QEMU image run make p.recipe-name
  • To permanently install the recipe to your QEMU image add your recipe name (recipe-name = {}) below the last item in the [packages] section of your TOML config at config/your-cpu-arch/your-config.toml and run make image

To test your recipe inside of Redox with Orbital, run:

make qemu

If you only want to test in the Redox terminal interface, run:

make qemu gpu=no

Update crates

Sometimes the Cargo.toml and Cargo.lock of some Rust programs can hold a crate versions lacking Redox support or a broken Redox code path (changes on code that make the target OS fail), this will give you an error during the recipe compilation.

  • The reason of fixed crate versions is explained on the Cargo FAQ.

To fix this you will need to update the crates of your recipe after the first compilation and build it again, see the ways to do it below.

(Bump a crate version on Cargo.toml can break some part of the source code, on this case the program needs a source code patch to use the updated API of the crate)

One or more crates

In maintained Rust programs you just need to update some crates to have Redox support (because they frequently update the crate versions), this will avoid random breaks on the dependency chain of the program (due to ABI changes) thus you can update one or more crates to reduce the chance of breaks.

We recommend that you do this based on the errors you get during the compilation, this method is recommended for maintained programs.

  • Expose the Redox build system environment variables to the current shell, go to the source folder of your recipe and update the crates, example:
make env

cd cookbook/recipes/your-category/recipe-name/source

cargo update -p crate1 crate2

cd -

make r.recipe-name

If you still get the error, run:

make cr.recipe-name

All crates

Most unmaintained Rust programs carry very old crate versions with lacking/broken Redox support, this method will update all crates of the dependency chain to the latest possible version based on the Cargo.toml configuration.

Be aware that some crates break the API stability frequently and make the programs stop to work, that's why you must try the "One crate" method first.

  • This method can fix locked crate versions on the dependency tree, if these locked crate versions don't change you need to bump the version of the crates locking the crate version, you will edit them in the Cargo.toml and run cargo update again (API breaks are expected).

(Also good to test the latest improvements of the libraries)

  • Expose the Redox build system environment variables to the current shell, go to the source folder of your recipe and update the crates, example:
make env

cd cookbook/recipes/your-category/recipe-name/source

cargo update

cd -

make r.recipe-name

If you still get the error, run:

make cr.recipe-name

Verify the dependency tree

If you use the above methods but the program is still using old crate versions, see this section:

Patch crates

Redox forks

It's possible that some not ported crate have a Redox fork with patches, you can search the crate name on the Redox GitLab, generally the Redox patches stay in the redox branch or redox-version branch that follow the crate version.

To use this Redox fork on your Rust program, add this text on the end of the Cargo.toml in the program source code:

[patch.crates-io]
crate-name = { git = "repository-link", branch = "redox" }

It will make Cargo replace the patched crate in the entire dependency chain, after that, run:

make r.recipe-name

Or (if the above doesn't work)

make cr.recipe-name

Or

make env

cd cookbook/recipes/your-category/recipe-name/source

cargo update -p crate-name

cd -

make r.recipe-name

If you still get the error, run:

make cr.recipe-name

Local patches

If you want to patch some crate offline with your patches, add this text on the Cargo.toml of the program:

[patch.crates-io]
crate-name = { path = "patched-crate-folder" }

It will make Cargo replace the crate based on this folder in the program source code - cookbook/recipes/your-category/your-recipe/source/patched-crate-folder (you don't need to manually create this folder if you git clone the crate source code on the program source directory)

Inside this folder you will apply the patches on the crate source and rebuild the recipe.

Cleanup

If you have some problems (outdated recipe), try to run these commands:

  • This command will delete your old recipe binary/source.
make c.recipe-name u.recipe-name
  • This command will delete your recipe binary/source and build (fresh build).
make ucr.recipe-name

Search Text on Recipes

To speed up your porting workflow you can use the grep tool to search the recipe configuration:

cd cookbook/recipes

grep -rnwi "text" --include "recipe.toml"

This command will search all match texts in the recipe.toml files of each recipe folder.

Search for functions on relibc

Sometimes your program is not building because relibc lack the necessary functions, to verify if they are implemented run the following commands:

cd relibc

grep -nrw "function-name" --include "*.rs"

You will insert the function name in function-name

Create a BLAKE3 hash for your recipe

You need to create a BLAKE3 hash of your recipe tarball if you want to merge it on upstream, to do this you can use the b3sum tool that can be installed from crates.io with the cargo install b3sum command.

After the first run of the make r.recipe-name command, run these commands:

b3sum cookbook/recipes/your-category/recipe-name/source.tar

It will print the generated BLAKE3 hash, copy and paste on the blake3 = field of your recipe.toml

Verify the size of your package

To verify the size of your package use this command:

ls -1sh cookbook/recipes/your-category/recipe-name/target/your-target

See the size of the stage.pkgar and stage.tar.gz files.

Submitting MRs

If you want to add your recipe on Cookbook to become a Redox package on the build server, read the package policy.

After this you can submit your merge request with proper category, dependencies and comments.