KnightOS Updates

u/Lionel Debroux January 25, 2016, 08:47:08 PM

QuoteIt'd also probably be pretty easy to add KnightOS support to TiLP.

I'd think so, all the more:
* recent versions of libticalcs export far more functions than versions before my time as the maintainer did;
* dbus_recv() was recently split to dbus_recv_header() + dbus_recv_data(), according to our earlier discussion, which happened to match a much older feature request from Benjamin Moody. The corresponding commit is still part of the experimental branch for now (this particular commit wasn't updated beyond the commit message for weeks, though), but won't be for too long;
* even if you implemented a brand-new protocol, libticalcs contains support for 4 protocols: 3 official protocols unofficially dubbed DBUS, DUSB and NSP, and libticalcs' specific ROM dumping protocol.

Even in user-space (libusb-win32 / libusb), Windows USB drivers are such a pain for users (filter driver, ever more offensive signing requirements; a Windows expert and libwdi could help ease the pain) that for KnightOS users' sake, even a brand-new code base really ought to be based on at least the drivers provided alongside libticables

u/SirCmpwn January 25, 2016, 08:52:44 PM

Quote from: ben_g on January 25, 2016, 08:39:01 PM
Will we also be able to send files by dropping them on z80e, so that we can load new programs on it while it's running? And will wabbitemu's file transfers work, either with or without an update?
Great work, SirCmpwn!

Well, let me clarify the z80e situation. z80e is presently a debugging tool for KnightOS development. Support for TIOS and Windows and as a general purpose emulator would be great and I would totally review and merge pull requests that add these things, but I'm not personally as interested in working on them as much as I would be interested on helping others work on them. The same goes for integration with tools like Wabbitemu, which is Windows-only. So while eventually someone might add support for sending files to TIOS, don't expect that from me. That being said, I would like to make it possible to mount an emulated calculator's filesystem on a Linux (or perhaps OSX) system and send files through that means, while it's running.

z80e is a great emulator, probably the best, and has a great debugger and great developer experience. I would like to see more people using it, but I need members of the community to step up and help with TIOS and Windows support if that's to be the case.

Quote from: Lionel Debroux on January 25, 2016, 08:47:08 PM
QuoteIt'd also probably be pretty easy to add KnightOS support to TiLP.
I'd think so, all the more:
* recent versions of libticalcs export far more functions than versions before my time as the maintainer did;
* dbus_recv() was recently split to dbus_recv_header() + dbus_recv_data(), according to our earlier discussion, which happened to match a much older feature request from Benjamin Moody. The corresponding commit is still part of the experimental branch for now (this particular commit wasn't updated beyond the commit message for weeks, though), but won't be for too long;
* even if you implemented a brand-new protocol, libticalcs contains support for 4 protocols: 3 official protocols unofficially dubbed DBUS, DUSB and NSP, and libticalcs' specific ROM dumping protocol.

Even in user-space (libusb-win32 / libusb), Windows USB drivers are such a pain for users (filter driver, ever more offensive signing requirements; a Windows expert and libwdi could help ease the pain) that for KnightOS users' sake, even a brand-new code base really ought to be based on at least the drivers provided alongside libticables

We use something similar to the DBus protocol, but with our own protocol on top of that which TiLP would have to implement. Right now I implemented that protocol myself in iofs, but iofs is more proof of concept than shippable code so I imagine that can change in the future. I definitely want to use libticables on all platforms for this (iofs does already use libticables).

u/SirCmpwn January 31, 2016, 09:56:42 PM

I was going to work on I/O support today, but got off track when I started working on improvements to z80e's debugger instead. Now it can load scas object files and show you the source maps (even in the context of a relocatable KnightOS thread)! It also loads up all the symbols for you to debug with. Prior to this change, you'd just see a straight up disassembly during debugging sessions with no context from your code. I also spent some time working on kcc to try and make it possible to view C source code in the z80e debugger like this, but that's going to require more effort than I was willing to put in today (just some refactoring in scas).

u/Max Leiter February 01, 2016, 06:06:43 PM

If you're interested in trying out the z80e updates, I made a PR to update the install-sdk script yesterday. You can find SDK setup instructions here: http://wiki.knightos.org/index.php/Tutorials/General/KnightOS_SDK

u/SirCmpwn February 04, 2016, 04:58:47 AM

u/Max Leiter February 04, 2016, 05:13:59 AM

For anyone curious, the current source for @SirCmpwn 's program is available here: https://gogs.sr.ht/SirCmpwn/iochat

u/Dream of Omnimaga February 04, 2016, 05:15:23 AM

That's a good job

u/SirCmpwn March 03, 2016, 01:59:47 PM

Hey @Lionel Debroux, I'm working more on iofsd and I'm running into some issues.

This is connected to a real calculator. The protocol is DBUS, first the PC sends a packet saying "list /bin". The calc sends a preamble with the number of entries, then that many packets, each with the name and type of one entry. z80e now has support for the link assist and I can confirm that it works perfectly when emulated.

I'm thinking about adding a checksum to each packet to improve the reliability of this. Do you think using libticalc's dbus functions would provide any benefit to reliability as well for the PC side? Also would like to hear your advice in general on this subject.

u/Lionel Debroux March 03, 2016, 04:05:20 PM

QuoteI'm thinking about adding a checksum to each packet to improve the reliability of this.

Yup, on DBUS, you should definitely use a checksum in each packet, like TI's OS for the TI-Z80 and TI-68k series do.

QuoteDo you think using libticalc's dbus functions would provide any benefit to reliability as well for the PC side?

Yup. Using dbus_recv_header() and dbus_recv_data(), which I split out of dbus_recv() a while ago because of the KnightOS use case in this topic (and later found to be a much older idea of Benjamin buried deep down into the todo/wish list), buys you checksum and progress information for the data part of the packet, you don't have to reinvent these on your side

In case you want to define your own commands, I'm now thinking that dbus_recv() could be further generalized by replacing the hard-coded switch statement by:
* a uint8_t * pointing to a list of command IDs considered to yield no data part after the header, plus a uint8_t length for said array;
* a uint8_t * pointing to a list of valid command IDs for a data part, plus a uint8_t length for said array
and binary search through these lists to classify packets.
Even if there's precedent for many-args functions in libticalcs (dusb_cmd_s_var_modify), I'd rather avoid adding a 9-argument variant of dbus_recv(): the lists of allowed command IDs could be passed out of band on a per-handle basis by a new API.
I'll write that idea down into the todo/wish list, for a later development cycle because Benjamin and I are currently working on closing down the current cycle. However, if you feel that such a feature is useful to you, and you'd like to take advantage of it in the near future, we could find an agreement.

u/SirCmpwn March 03, 2016, 04:12:01 PM

Quote from: Lionel Debroux on March 03, 2016, 04:05:20 PM
Yup, on DBUS, you should definitely use a checksum in each packet, like TI's OS for the TI-Z80 and TI-68k series do.

Huh, somehow I didn't understand that this was part of the DBUS protocol. How does that work, protocol-wise? Is there a good spec I should read? How does it handle retries?

Quote from: Lionel Debroux on March 03, 2016, 04:05:20 PM
Yup. Using dbus_recv_header() and dbus_recv_data(), which I split out of dbus_recv() a while ago because of the KnightOS use case in this topic (and later found to be a much older idea of Benjamin buried deep down into the todo/wish list), buys you checksum and progress information for the data part of the packet, you don't have to reinvent these on your side

In case you want to define your own commands, I'm now thinking that dbus_recv() could be further generalized by replacing the hard-coded switch statement by:
* a uint8_t * pointing to a list of command IDs considered to yield no data part after the header, plus a uint8_t length for said array;
* a uint8_t * pointing to a list of valid command IDs for a data part, plus a uint8_t length for said array
and binary search through these lists to classify packets.
Even if there's precedent for many-args functions in libticalcs (dusb_cmd_s_var_modify), I'd rather avoid adding a 9-argument variant of dbus_recv(): the lists of allowed command IDs could be passed out of band on a per-handle basis by a new API.
I'll write that idea down into the todo/wish list, for a later development cycle because Benjamin and I are currently working on closing down the current cycle. However, if you feel that such a feature is useful to you, and you'd like to take advantage of it in the near future, we could find an agreement.

Right now the way the kernel implements DBUS is that it receives the header and then calls a registered handler based on the machine ID, and passes it the header. The handler is expected to return the number of bytes of data the kernel should receive to finish the packet. This design allows you to use the packet length field for arbitrary data so long as your handler recognizes that and tells the kernel to receive 0 bytes (so the kernel never actually reads the length). This is necessary because that's how the TI keyboard sends packets (with scancodes in the length field).

I think I would like to leave the decision to receive more data (and how much data) up to the user of libticalcs, rather than up to libticalcs itself.

u/Lionel Debroux March 03, 2016, 04:44:45 PM

QuoteHuh, somehow I didn't understand that this was part of the DBUS protocol. How does that work, protocol-wise?

Just like it does in files on the computer side: 2 trailing bytes containing the checksum of the data part (not the header part). See https://github.com/debrouxl/tilibs/blob/experimental/libticalcs/trunk/src/dbus_pkt.c .

QuoteIs there a good spec I should read?

The linkguide is less up to date than the source code anyway. See also the header file with machine IDs and command IDs + descriptions thereof, https://github.com/debrouxl/tilibs/blob/experimental/libticalcs/trunk/src/dbus_pkt.h .

QuoteHow does it handle retries?

Good question. TILP tends to yield a full error in such cases, but other clients may wish to behave differently.

QuoteI think I would like to leave the decision to receive more data (and how much data) up to the user of libticalcs, rather than up to libticalcs itself.

Yup, providing clients the ability to make their own decisions by providing independently the two phases of dbus_recv() was the motivation outlined by both yourself, and earlier Benjamin. The corresponding commit currently is the second from the rewritable "experimental" branch on top of the never-rewritten "master" branch. As such, it will pretty much be part of the master branch upon the next forward of that branch, and in fact, I'm unlikely to rewrite such an old commit

The note at the end of my previous post was a wild idea I just had looking at the code, wrote down here and into the wish list.

u/SirCmpwn March 03, 2016, 04:50:18 PM

Quote from: Lionel Debroux on March 03, 2016, 04:44:45 PM
QuoteHuh, somehow I didn't understand that this was part of the DBUS protocol. How does that work, protocol-wise?
Just like it does in files on the computer side: 2 trailing bytes containing the checksum of the data part (not the header part). See https://github.com/debrouxl/tilibs/blob/experimental/libticalcs/trunk/src/dbus_pkt.c .

Ah, that's easy to implement.

Quote from: Lionel Debroux on March 03, 2016, 04:44:45 PM
QuoteHow does it handle retries?
Good question. TILP tends to yield a full error in such cases, but other clients may wish to behave differently.

From what I can tell (see screenshot in previous post), errors are so common that a retry will definitely be necessary. They're so common, in fact, that I'm seriously concerned about things like the header getting mangled frequently, which is difficult to recover from. I'll make up some kind of protocol to handle this. Do you know what I might be doing that could lead to it being so unreliable? I'm just using the link assist. Maybe I should play with the signaling rate.

Quote from: Lionel Debroux on March 03, 2016, 04:44:45 PM
QuoteI think I would like to leave the decision to receive more data (and how much data) up to the user of libticalcs, rather than up to libticalcs itself.
Yup, providing clients the ability to make their own decisions by providing independently the two phases of dbus_recv() was the motivation outlined by both yourself, and earlier Benjamin. The corresponding commit currently is the second from the rewritable "experimental" branch on top of the never-rewritten "master" branch. As such, it will pretty much be part of the master branch upon the next forward of that branch, and in fact, I'm unlikely to rewrite such an old commit
The note at the end of my previous post was a wild idea I just had looking at the code, wrote down here and into the wish list.

Maybe I'm not understanding this. Is dbus_recv/dbus_send shipped with libticalcs right now? Or do I have to build this experimental branch to use it?

u/Lionel Debroux March 03, 2016, 04:58:26 PM

QuoteDo you know what I might be doing that could lead to it being so unreliable?

Nope, I don't know.

QuoteIs dbus_recv/dbus_send shipped with libticalcs right now?

Sure. dbus_recv() and dbus_send() have been directly exported by libticalcs since commit f746457ff6afba7d6211faa009b7594b395c12a2 , Thu Jun 14 06:21:20 2012 +0000, and are therefore part of the libti* corresponding to TILP II 1.17 released 2013/03/31.

QuoteOr do I have to build this experimental branch to use it?

You need to build the experimental branch (for now, soon the master branch) to use dbus_recv_header() and dbus_recv_data(), which is a '2015 change and therefore not part of any released libti* version at the time of this writing.
The usual https://ti-pla.net/tilpinst build script can be requested to build the experimental branch by setting the appropriate environment variable.

u/SirCmpwn March 03, 2016, 06:52:42 PM

Cool, thanks a lot for the help!

u/Max Leiter June 30, 2017, 05:01:03 PM

Figured I'd post what's happened in the past few months:

- SirCmpwn (with a few bugfixes by me) just released SDK v2. Notable features include being able to use local packages and an offline cache.
- Decimal floating point support has landed in the kernel! This was a major issue with KOS until a (seemingly) random developer opened a PR here. While certain math operations haven't been implemented just yet, they're being worked on, and once they are SirC's next side-project will most likely be a functioning calculator. I'll release C bindings for the floating point routines once I have some more free time. You can find the math routines here.
- z80e (z80 emulator in C) now compiles on non-linux platforms. Commit: here. Additionally, @Snektron fixed some errors and improved windows support here.

Last Edit: June 30, 2017, 05:11:44 PM by Max Leiter