OMF386 Linker

Introduction

I am planning to create a linker for the OMF386 object format. This object format was used by Intel in the 1980s for their i386 toolchain (ASM386, PLM386, BLD386, BND386, MAP386, LIB386). It is not to be confused with OMF-386, which is an extension of the original ISIS-II 8086 OMF. OMF-386 is much more common and supported widely. OMF386 is a completely different format that is very elusive. OMF-386 is record based, like OMF. OMF386 is not record based but instead section based, like most other object formats. They are totally incompatible.

The only company I have found that appears to offer a OMF386 linker is TenAsys, a spin-off of RadiSys, the owner of Intel's Multibus division and therefore iRMX, Intel's real time operating system. I contacted TenAsys years back about the linker and never got a response. Looking up OMF386 or BND386 only gives documentation and some people complaining about how they can't find the tools or specification.

However, I have recently found a paper from 1995 titled Analysis Of Object File Formats For Embedded Systems. This paper contains the specification and structure of OMF386. From this I will create a linker that will be able to serve as a substitute for BND386.

Unique Features of OMF386

OMF386 is an object format that allows for the unique features of the i386 to be used. Few other object formats support these features. Among these features are segmentation and gates.

Segmentation

Segmentation is a memory protection system introduced with the i286. Segmentation works using segments: regions of memory. A segment has a base and a limit. The base segment is the address in memory where the segment starts. The limit is how large the segment is in bytes or 4096-byte pages starting with the 386.

Segments are described using descriptors which reside in either the Global Descriptor Table (GDT) or Local Descriptor Table (LDT). The GDT applies to the entire CPU at all times but the LDT can be changed automatically using hardware multitasking (rarely used today). These descriptors contain fields for the base, limit, flags, and access control. Segments can be marked as being for data, code, or system use. When marked as data, segments can also be made read-only or read/write. Code segments are always read-only. System segments are special and will be explained later.

Segement descriptors have a Descriptor Privilege Level (DPL) describing what privilege level the CPU must be running at to access the segment. x86 has 4 privilege levels when using segmentation, 0 is the most privileged, 3 is the least privileged. Today, only ring 0 and 3 are used in operating systems (paging only supports these two privilege levels).

Segments are accessed in code using the segment registers DS, CS, SS, ES, FS, and GS. CS is the code segment. DS is the data segment and will be used for data accesses unless a segment override prefix is used. SS is the stack segment and is used when performing stack operations (PUSH, POP, etc.). ES is used in string operations for the destination. FS and GS are extra registers added with the i386 only accessible using segment override prefixes. Segment registers can be loaded using MOV or POP. However, they cannot be loaded from an immediate value using MOV, only from a GPR or memory.

Segmentation, as you can see, is a very different memory model than the typical paging virtual memory system used today. This makes it incompatible with most object formats because, for example, those formats don't understand segments or how to relocate them. Segmentation support is one of the features that makes OMF386 unique.

Loaders

As well as creating a linker/binder, I need to create loaders as well. I am thinking of making two loaders. One loader runs from DOS and uses something like DOS/4GW to configure memory, and another loader that boots, loads the image, relocates segments, and configures the GDT, IDT, LDT, etc. before passing over control to the program.