TLDR: I am currently trying to create an operating system for the ti nspire and just hit a (small) milestone, searching for testers in the future
In my current project, called simply OSExt for now (because it should extend the normal ti OS sometime in the future), I deviated from simply writing a program that I wanted to integrate with the OS to wanting to write a whole paralell OS kernel.
The whole kernel-idea started because I hated it when I tried to test a program and the calculator crashed, leaving me waiting for it to reboot, and with no clue why it even crashed in the first place.
This is a massive undertaking, and I may never finish it, but I now have a small testing-version finished, which can run a programm in userspace in its own virtual address space, where it can't do harm to the rest of the system if it goes wrong.
Now I need to implement a proper program loader, a way for the program to use files, allocate new memory, create new processes and communicate with them.
I want to use the newlib library, which is actually already used by ndless, so it shouldn't be that hard to port.
My goal is to make a complete POSIX system, so I can eventually port some open-source software to my os.
If someone wants to help me, or maybe volunteer to test it if it's more mature, I'd be happy. Because I only have an ti nspire CX CAS HW-AA, I can only test the rest in the emulator myself.
If you want to test it in firebird yourself: the github page (https://github.com/nspiredev500/OSExt/tree/slab-allocator-rewrite) or the .tns file (https://github.com/nspiredev500/OSExt/releases/download/v0.3.0/osext.tns)
The current Roadmap:
- implementing a way for programs to allocate memory
- implementing a way for programs to access files, if they have the permission
- implementing POSIX fork and exec to create new processes
- implementing a message system and a pipe system for the programs to communicate
- porting newlib
- implementing a dynamic linker for shared libraries
- making a lcd driver
- making a keypad driver
- making a touchpad driver
- making a desktop environment with a file browser and the ability to execute programs and let them display to the screen
- optimizing memory usage
long-term:
- if I'm not dead already, implementing a usb storage driver (will certainly take much time for this one)
- porting gcc and binutils, so I can develop the kernel on my calculator
- if I'm somehow still alive, integrating it into nLoader to boot from it
I have this shell for debug output:
(https://github.com/nspiredev500/OSExt/raw/slab-allocator-rewrite/kernel%20shell.png)
And here is my proof of going into userspace:
(https://github.com/nspiredev500/OSExt/raw/slab-allocator-rewrite/userspace%20test%201.png)
Here is the address space. All but the first and second entries are my kernel:
(https://github.com/nspiredev500/OSExt/raw/slab-allocator-rewrite/userspace%20address%20space.png)
Nothing groundbreaking, but it now runs a little userspace test on actual hardware.
Now I can start to implement file io, timeslices for processes, IPC and the userspace drivers.
Something a little more visual may be coming in the next weeks, maybe I'm starting with the lcd driver.
It is now able to draw to the screen while the OS is running without any flickering. It does use a bit of performance though.
And weirdly, trying to turn that off while it is installed crashes the calculator, but not the emulator.
I also implemented a miniclock:
(https://github.com/nspiredev500/OSExt/raw/development/images/clock.png)
I have now completed the timer drivers, which means I could now interrupt user mode programs, so nothing can just stay in an endless loop and use all the processor time.
I'm now going to start implementing the system calls for the usermode programs to use.
After that I have to port newlib, and then I can start writing user mode programs, and then I should have something more impressive to present.
This is extremely exciting! I am completely willing to help with testing, but I have an Nspire CX II running fw 5.0 so that might not work. Two questions: is this planned to completely replace the nspire firmware or just work alongside it? Is this planned to replace ndless in the future? Thank you for your work, please keep the project alive it's really exciting!
EDIT:
I just read the GitHub and it answered all my questions lol. Keep up the good work!
I always didn't like the plain black background of the nspire homescreen, so I added basic background image support for windows bmp files.
Currently it just looks for black pixels and replaces them with pixels from your image, but it vanishes when a menu is opened, because it would colour the text, too.
(https://github.com/nspiredev500/OSExt/raw/development/images/walrus%20background.png)
You just have to put a file called "background.bmp" (.tns, but you don't see that on the calculator) in the documents folder.
I hope the walrus isn't copyrighted :)
Quote from: Knucklesfan on April 08, 2020, 11:17:47 PMbut I have an Nspire CX II running fw 5.0 so that might not work.
I want to port it to the CX II, but first I need a new Ndless version to do that.
It will probably be a while until the new version is out.
small update: You can name bmp files "background2.bmp" to "background4.bmp" and get a slideshow of your images.
They change every 15 seconds currently.
Now you can actually set the time instead of only seeing it. I made a small time setting screen, but I needed the touchpad for the arrow keys, and the touchpad driver turned out be a bit hard.
(https://github.com/nspiredev500/OSExt/raw/master/images/time%20screen.png)
Now I will start working on the processes and the scheduler.
I won't be able to work on this for the next 1.5 months, because I have my final exams in this time.
But after that, I have plenty of time.
I am mostly done with my finals now, so I can start working on this project again.
I'm now working on the scheduler and some system calls to let user-mode code read and write files, allocate memory and use the framebuffer.
So soon I should have a graphical "Hello user-mode" working, instead of the firebird debugger showing "usr".
Then I want to make a basic input driver, so I can have a first early userspace console.
I can now load ELF files, relocate them and execute them.
For now the execution is in kernel mode, but this enables me to split functionality into modules and search for them at the start, load them and run them.
This should make the startup faster if you don't need all features, and reduce the file size.
I will split the clock and the background image support into modules.
There will also be a battery charge module, as I was unsatisfied with the 25% reading the OS provides you from the beginning.
I also made a small configuration program that gets compiled with your system compiler.
It asks you what value options should have before the kernel gets compiled.
The options are documented in config/option_ descriptions.txt.
With that I also changed the compiler from my own arm-osext-gcc to a standard arm-none-eabi-gcc (like the one that is included with ndless), to make it easier to compile OSExt.
Now you only need the ndless-sdk and an arm-none-eabi toolchain.
Together this should allow you to actually use the configurability, but I will also release the precompiled modules and a few option configurations.
For more obscure combinations you would have to compile it yourself.
I'll make the 3 modules I talked about, make a small release and then get back to getting userspace working.
The module system is finished.
The clock is now a module, and the background image support too.
There is also a battery charge module, but for some reason it makes the calculator run quite slow and breaks the OS battery charge indicator ??? .
The charset and background images now have to be put into /documents/osext, to reduce the clutter in the documents folder.
The modules have to be put in /documents/modules
At start OSExt tries to load all modules it finds, but there is also a module manager.
You can get to the module manager by pressing and holding ctrl and , .
It displays the currently loaded modules and the memory they occupy.
You can move the selection up and down by pressing the up and down arrows on the touchpad.
The first selection (empty at the start) is for entering a module name.
If you transferred a module to the calculator or uninstalled it, you can type it's name (without .elf.tns), hit enter and it gets loaded.
When you select a module name and hit pi, the module gets uninstalled.
On the GitHub page, you mentioned:
Quote from: nspiredev500making a proper file brower, the integrated one is too slow
Do you know if it's possible to do async file I/O at all? I've written Rust bindings (https://github.com/lights0123/ndless-rs) to ndless, and one thing I've been playing around with is its zero-cost language-level async/await (https://rust-lang.github.io/async-book/01_getting_started/04_async_await_primer.html) support. I have basic support for the timer and keypad which lets you write code that compiles to a state machine (https://github.com/lights0123/ndless-rs/tree/master/ndless-async#example) without having to actually think about it, as if there were threads. Typically, in Linux, this is done with fcntl (https://linux.die.net/man/2/fcntl) and O_NONBLOCK, but ndless doesn't have (expose?) that syscall. Do you know if there's any ability to do that? I don't know how far you're looking to write your OS—if that just can't be done with Nucleus RTOS, is there any way to talk to the flash chip directly any do some processing while waiting for the results, like a modern computer?
Basically, I'm trying to figure out how you're going to make a faster file browser, as I'd assume that it's I/O bound and you can't really speed it up.
Quote from: lights0123 on June 09, 2020, 01:45:42 AMBasically, I'm trying to figure out how you're going to make a faster file browser, as I'd assume that it's I/O bound and you can't really speed it up.
I don't think it's I/O bound, basically it only needs to iterate over the directory entries and stat them.
It should not take that long to do, even on the calculator.
I think it gets slower to open the more files you have (and i have quite a few), even if they are in directories that aren't open. (or that's just my perception, I don't want to delete all my files right now)
It also doesn't cache that information, and reloads every time you open the file explorer.
That is some room for improvement.
I ran a quick test on my calculator, just iterating over the documents folder and stating everything, took only a fraction of a second. I don't know why the document browser takes so long.
Maybe it actually reads the files until EOF to get the size?
Also I'm working on USB support by writing an EHCI driver, writing the device drivers on linux with libusb to test them and then port them to my project.
If I get USB mass storage drivers to work, I can actually do async file I/O. But that will take some time.
Quote from: lights0123 on June 09, 2020, 01:45:42 AMI've written Rust bindings to ndless
That's cool, I thought about using Rust myself, and I even saw your project on github. My first memory manager didn't work, and I thought Rust could help with that. I ended up just doing a rewrite and it seems to work now, but for other things Rust could still be helpful. But I don't know enough about Cargo to do what I need. Maybe you can help me. I would need to compile the code like with GCC's -fPIC or -fPIE flag (https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html). It has to use the global offset table and link into my ELF file.
The global offset table is important, because it self-relocates to a higher base address, and will be mapped like this into every address space created.
I looked at your project now, and it looks great :thumbsup: . In your book you say "Your program will never segfault". Something like this was one of the motivations for my project.
Creating a user-mode wrapper around the OS and the hardware, so no program can crash or brick your calculator.
What timer do you use for your wrapper? I want to remain compatible to as many community projects as possible.
I use the first timer (https://hackspire.org/index.php?title=Memory-mapped_I/O_ports_on_CX#900C0000_-_First_timer) for my millisecond UNIX time, because the OS doesn't use it. If your Rust wrapper also uses the first timer, both cannot work at the same time.
But I can expose timer functions as syscalls, and detecting OSExt is also easy, so you could just do a runtime check.
I remember right now that I also have to make OSExt compatible with the lcd-compatibility feature of ndless 9_9.
Quote from: lights0123 on June 09, 2020, 01:45:42 AMis there any way to talk to the flash chip directly
Ndless exposes a read_nand and write_nand syscall from Nucleus. But you would have to parse the filesystem youself with the information on Hackspire.
I also tried to read the flash chip directly via memory-mapped IO, but it doesn't work right now and I'm very carefull, because I don't want to brick my calculator.
Quote from: lights0123 on June 09, 2020, 01:45:42 AMthis is done with fcntl and O_NONBLOCK
Nucleus writes "initializing POSIX layer" on the uart sometime during boot I think, so it should have the fcntl syscall.
The problem is to know which system call number it has and what arguments it takes.
You could assume normal linux arguments and just fuzz all the system calls in firebird until it works.
Quote from: nspiredev500 on June 10, 2020, 01:58:34 PMQuote from: lights0123 on June 09, 2020, 01:45:42 AMI've written Rust bindings to ndless
That's cool, I thought about using Rust myself, and I even saw your project on github. My first memory manager didn't work, and I thought Rust could help with that. I ended up just doing a rewrite and it seems to work now, but for other things Rust could still be helpful. But I don't know enough about Cargo to do what I need. Maybe you can help me. I would need to compile the code like with GCC's -fPIC or -fPIE flag (https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html). It has to use the global offset table and link into my ELF file.
The global offset table is important, because it self-relocates to a higher base address, and will be mapped like this into every address space created.
I don't know too much about linking, but I'd be glad to help. rustc does compile with PIC by default (https://www.mankier.com/1/rustc#relocation-model):
Quoterelocation-model=[pic,static,dynamic-no-pic]
The relocation model to use. (Default: pic)
Although apparently you can pass arbitrary arguments to LLVM too.
You'd just need to add
[lib]
crate-type = ["staticlib"]
to your Cargo.toml and make sure that you have a lib.rs instead of a main.rs source file. Building will automatically produce a libyourcratename.a that you can link to.
Quote from: nspiredev500 on June 10, 2020, 01:58:34 PMI looked at your project now, and it looks great :thumbsup: . In your book you say "Your program will never segfault". Something like this was one of the motivations for my project.
Creating a user-mode wrapper around the OS and the hardware, so no program can crash or brick your calculator.
What timer do you use for your wrapper? I want to remain compatible to as many community projects as possible.
I use the first timer (https://hackspire.org/index.php?title=Memory-mapped_I/O_ports_on_CX#900C0000_-_First_timer) for my millisecond UNIX time, because the OS doesn't use it. If your Rust wrapper also uses the first timer, both cannot work at the same time.
But I can expose timer functions as syscalls, and detecting OSExt is also easy, so you could just do a runtime check.
I use (https://github.com/lights0123/ndless-rs/blob/master/ndless/src/bindings/timer.rs) the same timer as SDL, which is at 0x900C0000. I still use that timer for sleeping (https://github.com/lights0123/ndless-rs/blob/master/ndless/src/bindings/thread.rs), as I get more precision that the millisecond-accurate msleep Ndless function. I don't ever use the timer at 0x900D0000, although I guess I could switch to it as I don't call Ndless's sleeping functions.
I could switch to your timer syscalls if you provide a way to detect it. I'd still like to get the raw ticks if possible, for the nice 30 μs precision. I don't know how much overhead there is in a syscall—would it be worth it replicating vsyscall or vDSO since you apparently already have separate userspace memory?
Quote from: nspiredev500 on June 10, 2020, 01:58:34 PMQuote from: lights0123 on June 09, 2020, 01:45:42 AMthis is done with fcntl and O_NONBLOCK
Nucleus writes "initializing POSIX layer" on the uart sometime during boot I think, so it should have the fcntl syscall.
The problem is to know which system call number it has and what arguments it takes.
You could assume normal linux arguments and just fuzz all the system calls in firebird until it works.
Any suggestions on how to do that?
Quote from: lights0123 on June 10, 2020, 03:40:02 PMQuote from: nspiredev500 on June 10, 2020, 01:58:34 PMNucleus writes "initializing POSIX layer" on the uart sometime during boot I think, so it should have the fcntl syscall.
The problem is to know which system call number it has and what arguments it takes.
You could assume normal linux arguments and just fuzz all the system calls in firebird until it works.
Any suggestions on how to do that?
You could compile a program that tries to call each syscall number with the specified arguments, and check if it crashes.
I don't know if there is a nspire emulator that can be controlled via console options and gives the UART output on the console, but then it could be optimised.
Here is a list of emulators (https://hackspire.org/index.php?title=Emulators). You could make a script that compiles the program with a syscall number, opens the emulator, checks the output and builds the program with the next system call number.
This is basically the last option you could use if you absolutely need a specific system call.
I think you could better make small blocking I/O calls and check the time between them.
Quote from: lights0123 on June 10, 2020, 03:40:02 PMI could switch to your timer syscalls if you provide a way to detect it. I'd still like to get the raw ticks if possible, for the nice 30 μs precision. I don't know how much overhead there is in a syscall--would it be worth it replicating vsyscall or vDSO since you apparently already have separate userspace memory?
System calls are basically just C functions called by an index in a table. You just have some memory accesses to get the number and save some state, run the function, and then return. There should not be much overhead.
I technically have userspace memory, but nSLD and libndless are hardwired for supervisor mode, as the whole OS runs like that.
Before proper userspace I have to implement at least a simple scheduler, mmap and I/O syscalls. Running Rust (which should already be pretty safe) in usermode isn't worth the overhead and modifications to the libraries.
Detecting OSExt is easy, I use this in OSExt itself to detect if another version is already installed:
register uint32_t control_reg asm("r0");
asm volatile("mrc p15, 0, r0, c1, c0, 0":"=r" (control_reg)::); // reading co-processor register 1
if ((control_reg & (1 << 13)) == (1 << 13)) // if bit 13 is a 1, interrupt vectors are mapped high. I don't think anything other than OSExt does this
{
register uint32_t tt_base asm("r0");
asm volatile("mrc p15, 0, r0, c2, c0, 0":"=r" (tt_base)); // get the translation table base register
tt_base = tt_base & (~ 0x3ff); // discard the first 14 bits, because they don't matter
uint32_t *tt = (uint32_t*) tt_base;
if (tt[(0xe0000000) >> 20] != 0) // if there is something mapped at virtual address 0xe0000000 it should be OSExt, I haven't seen any other program map memory like that
{
// OSExt is installed
}
}
It should be very unlikely that anything else maps the exception vectors high and maps something at 0xe0000000, so this should be reliable.
You can make a variable like bool osext_installed and only do the check once.
I already have a UNIX millisecond function to get the time. I use int64_t for my millisecond time.
I can also use the watchdog timer for sleeping, it has a frequency of 33 MHZ, if you need more precision.
I could write functions to handle sleeping for longer times than the timer.
So you only need getting the time and sleeping?
By the way, in your timer code I noticed the threads only support sleeping, because threads aren't supported by ndless.
I already thought about kernel threads and a separate kernel scheduler.
If you want I can work on thread syscalls.
EDIT: I now have working microsleep and microsecond unix time.
You can specify what functionality you need and I can wrap it in a syscall.
Do you know how to call specific syscalls or do you use the wrappers ndless provides?
I know how to do it in C with inline assembly. It seems Rust supports GCC-style inline assembly, so that's good.
Is there a way to formally specify and API?
Maybe we should move this to a github issue or a separate thread.
Just a small Hotfix release (https://github.com/nspiredev500/OSExt/releases/tag/v0.4.0.1), now OSExt works with the Ndless feature for running programs from HW<W on HW-W+.
I forgot about that, and it crashed when trying to run these programs when OSExt is installed.