20151222 - Random Holiday 2015
Google Photos = Actually Working (edit)
Minus has basically imploded loosing a lot of the image content on this blog, unfortunately Google Photos is
Vintage Interactive Realtime Operating System
Taking part of my Christmas vacation time to reboot my personal console x86-64 OS project. I've rebooted this project more times than I'm willing to disclose on this blog. Hopefully this reboot will stick long enough to build something real with it. Otherwise it's been great practice, and the evolution of ideas has certainly been enlightening. Here is a shot of the binary data for the boot sector in the editor,
My goal is to rewind to the principals of the Commodore 64, just applied to modern x86-64 hardware and evolved down a different path. In my case to build a free OS with open source code which boots into a programming interface, which fully exposes the machine, and provides some minimal interface to access hardware including modern AMD GPUs. With a feature list which is the inverse of a modern operating system: single-user, single-tasking, no paging, no memory protection ... the application gets total control over the machine, free to use parallel CPU cores and GPU(s) for whatever it wants. Ultimately manifesting as a thumb drive which can be plugged into a x86-64 machine with compatible hardware and boot the OS to run 100% from RAM, to run software.
A Tale of Two Languages
C64 provided Basic at boot, I'm doing something vastly different. Booting into an editor which is a marriage between a spreadsheet, hex editor, raw memory view, debugging tool, interactive live programming environment, annotated sourceless binary editor with automatic relink on edit, and a forth-like language. Effectively I've embedded a forth-like language in a sourceless binary framework. The editor runs in a virtualized console which can easily be embedded inside an application. The editor shows 8 32-bit "cells" of memory per line (half cacheline), with a pair of rows per cell. Top row has a 10-character annotation (with automatic green highlight if the cell is referenced in the binary), the botton row shows the data formatted based on a 4-bit tag stored in the annotation. Note the screen shot showing the boot sector was hand assembled 8086 (so is embedded data), built from bits of NASM code chunks then disassembled (it's not showing any of the embedded language source). Tags are as follows,
unsigned 32-bit word
signed 32-bit word
four unsigned 8-bit bytes
32-bit floating point value
-----
unsigned 32-bit word with live update
signed 32-bit word with live update
four unsigned 8-bit bytes with live update
32-bit floating point value with live update
-----
32-bit absolute memory address
32-bit relative memory address [RIP+imm32]
toe language (subset of x86-64 with 32-bit padded opcodes)
ear language (forth-like language, encoded in binary form)
-----
5-character word (6-bits per character)
last 3 saved for GPU binary disassembly
Editor designed to edit an annotated copy of the live binary, with a frameword designed to allow realtime update of the live binary as a snapshot of the edited copy. The "with live update" tags mean that the editor copy saves a 32-bit address to the live data in it's copy of the binary (instead of the data itself). This allows for direct edit and visualization of the live data, with ability to still move bits of the binary around in memory.
The "toe" and "ear" tagged cells show editable disassembled x86-64 code in the form of these languages. The "ear" language is a zero-operand forth-like language. Current design,
regs ---- rax = top rcx = temp for shift rbx = 2nd item on data stack, grows up rbp = 4 rdi = points to last written 32-bit item on compile stack bin word name x86-64 meaning -------- ---- ---------------------------- 0389dd03 , add ebx,ebp; mov [rbx],eax; dd2b3e3e \ sub ebx,ebp; c3f2c3f2 ; ret; dd2b0303 add\ add eax,[rbx]; sub ebx,ebp; dd2b0323 and\ and eax,[rbx]; sub ebx,ebp; 07c7fd0c dat# add edi,ebp; mov [rdi],imm; d0ff3e3e cal call rax; 15ff3e3e cal# call [rip+imm]; 058b3e3e get# mov eax,[rip+imm]; 890fc085 jns# test eax,eax; jns imm; 850fc085 jnz# test eax,eax; jnz imm; 880fc085 js# test eax,eax; js imm; 840fc085 jz# test eax,eax; jz imm; c0c73e3e lit# mov eax,imm; 03af0f3e mul imul eax,[rbx]; d8f73e3e neg neg eax; 00401f0f nop nop; d0f73e3e not not eax; dd2b030b or\ or eax,[rbx]; sub ebx,ebp; 05893e3e put# mov [rip+imm],eax; f8d20b8b sar mov ecx,[rbx]; sar eax,cl; e0d20b8b shl mov ecx,[rbx]; shl eax,cl; e8d20b8b shr mov ecx,[rbx]; shr eax,cl; dd2b032b sub\ sub eax,[rbx]; sub ebx,ebp; dd2b0333 xor\ xor eax,[rbx]; sub ebx,ebp;
For symbols the immediate operands are all 32-bit and use "absolute" or "relative" tagged cells following the "ear" tagged cell. Likewise for "dat#" which pushes an immediate on the compile stack, and "lit#" which pushes an immediate data on the data stack, the following cell would have a data tag. The dictionary is directly embedded in the binary, using the edit-time relinking support. No interpretation is done at run-time, only edit-time, as the language is kept in an executable form.
After building so many prototypes which "compile" source to a binary form, I've come to the conclusion that it is many times easier to just keep the source in the binary form, then disassemble and reassemble in the editor on edit. The "toe" subset of x64-64 uses 32-bit opcodes with null "3e" prefix to pad to alignment. Disassembly/reassembly uses a table with,
{opcode form index, 32-bit opcode base, 32-bit 5-character name}
Then under 16 opcode forms, which use a pair of functions, one for disassembly to a common {name,r,rm,s/imm8} form, and the other for assembly. Other than entering data for the table it was trivial engineering work.
A Tale of Two Editors
To fast track this reboot I've built the first editor as a fully non-live prototype of the editor for the OS. This enables me to bring up the OS using an emulator with the standard pass/fail instant debug method (it works if the emulator doesn't enter in the infinite reboot loop). I'm using the same custom VGA font, just different colors because of the bad 256-color palette in xterm. The C based version of the editor was built using the "as a child would do it" method of development over a few days, just start building without any planning, writing code as fast as possible to bring up a prototype. See what works, refine, repeat. It took around 1900 lines total, and produces an insanely large "gcc -Os" compiled binary (optimized for smallest size) at around 160 KB. Later will see how this compares to the editor fully re-written in the "toe/ear" language in the editor itself...