Proposing CAS extension: Cue data

Hardware Hacking, Programming and Game Solutions/Cheats
User avatar
robcfg
Posts: 1232
Joined: Sat Apr 04, 2009 10:16 pm
Location: Stockholm, Sweden

Proposing CAS extension: Cue data

Post by robcfg » Fri Aug 04, 2017 10:59 am

Hello everyone!

I've been thinking that it's nice to have CAS files, as they are very convenient, but we are missing some info that would allow to reconstruct the original wave file from a CAS file.

Not to mention, not having to patch games that use custom loaders.

So, this is the idea I've come with. Adding cue data (as bin+cue files for optical media dumps) at the end of the CAS file.

It would be compatible with emulator with no cue support as cue data will be beyond cas data, and would provide necessary data for recreating the original wav file with a good degree of accuracy.

Also, it could be used to store extra data like cheat codes, level codes, or any other useful info that could be accessed within emulators.

Here's an initial spec off the top of my head:

Code: Select all

* CAS data
* '[CUE' - Cue data begin (0x5B435545)

CUE blocks
----------
0xF0 - '0' length in hz (2 bytes)
0xF1 - '1' length in hz (2 bytes)
0x00 - Silence, in miliseconds (2 bytes)
0x0D - Data. Start and end offsets as uint32 (4+4 bytes)
0xEA - External Audio. Like tutorial tapes with embedded audio.
       2 byte string size
       string size bytes with name of external audio file
       4+4 bytes start and end offsets as uint32, could be file offsets or time offsets

Another CUE blocks could include additional data like level passwords, cheats, etc...

* uint32 - offset to cue data begin
* 'CUE]' - Cue data end (footer) (0x4355455D)
What are your thoughts?

sorchard
Posts: 394
Joined: Sat Jun 07, 2014 9:43 pm
Location: Norwich UK

Re: Proposing CAS extension: Cue data

Post by sorchard » Fri Aug 04, 2017 9:43 pm

Hi Rob,

It's an interesting idea. You could add a degree of future proofing by requiring each extension block to specify a length and a type. That way new block types can easily be added and routines can easily skip over the blocks they don't understand, like the WAV file format.

The problem with having a cue marker to identify where the extension data starts is that the marker can appear within the cas stream. For example a native utility that looks for cas file extension data could easily have the magic signature as part of the binary. (There are already one or two examples of native code that read cas files). Maybe there's a standard solution, I don't know, but perhaps there could be some kind of calculated field to validate the cue marker?

Not sure about referring to external files, as they have a habit of going missing, (maybe embed them in blocks?) but there's certainly no harm in having a block type for plain text. That could contain things like a list of related files and loading instructions etc.
Stew

User avatar
robcfg
Posts: 1232
Joined: Sat Apr 04, 2009 10:16 pm
Location: Stockholm, Sweden

Re: Proposing CAS extension: Cue data

Post by robcfg » Fri Aug 04, 2017 10:20 pm

Stew,

thanks for your nice reply!
You could add a degree of future proofing by requiring each extension block to specify a length and a type.
Sure! Like IFF chunks as in WAV files and other formats. The idea crossed my mind but forgot to add it.
The problem with having a cue marker to identify where the extension data starts is that the marker can appear within the cas stream. For example a native utility that looks for cas file extension data could easily have the magic signature as part of the binary. (There are already one or two examples of native code that read cas files). Maybe there's a standard solution, I don't know, but perhaps there could be some kind of calculated field to validate the cue marker?
The idea is to have the data appended at the end of the stream. That way, it won't interfere with the cas stream. I made a custom cas file with cue data at the end of the stream and noticed no problem on XRoar or MAME. Even if you try to CLOAD(M) after the cas stream itself, it won't be recognised by the emulator or the computer.

Also, checking if cue date is present would be quite easy. Just read the las 4 bytes of the cas file and check that they match the 'CUE]' signature. Then, the previous 4 bytes are the offset of the cue data start, which should point to the start signature '[CUE]'. Finally you can parse the cue data chunks.
Not sure about referring to external files, as they have a habit of going missing, (maybe embed them in blocks?) but there's certainly no harm in having a block type for plain text. That could contain things like a list of related files and loading instructions etc.
I thought this would be useful if you have a tape with code and audio interleaved. Of course, the referenced files could be missing but the cas file would still be usable.

Text chunks are a good idea too. Like you said, for storing loading instructions, cheat codes, level passwords or any other info. That way, if you have the cas file, you got all information you need to get it running and it would provide a richer experience when using emulators.

Thank you very much for your feedback!

sixxie
Posts: 1073
Joined: Fri Jul 18, 2008 8:36 am
Location: Hertfordshire
Contact:

Re: Proposing CAS extension: Cue data

Post by sixxie » Sat Aug 05, 2017 3:26 pm

I've long thought a chunked format could handle this.

My only real novel idea concerning it was that chunk type bytes 0x55 and 0xaa could indicated "data until end of file", thus rendering all existing CAS files compatible... ;)

User avatar
robcfg
Posts: 1232
Joined: Sat Apr 04, 2009 10:16 pm
Location: Stockholm, Sweden

Re: Proposing CAS extension: Cue data

Post by robcfg » Sat Aug 05, 2017 11:36 pm

The idea of appending the cue data at the end of the cas file makes them compatible too (well, its really the same file plus extra data), and the extra data is innocuous, as anyway the program won't load anything past it and if you try (did it!), cue data won't be recognised as valid data.

Basically, the extra data will be there for those who want to use it.

sixxie
Posts: 1073
Joined: Fri Jul 18, 2008 8:36 am
Location: Hertfordshire
Contact:

Re: Proposing CAS extension: Cue data

Post by sixxie » Sun Aug 06, 2017 10:31 pm

Annoying, I had a reply CUEd up and the site wanted me to log in again and lost it.

For now though: part implemented (xroar & bin2cas), probably want to change some stuff!

Will probably wrap all internal CAS in synthesised CUE data even when not present as will reduce the amount of special-casing for things like seeking.

sixxie
Posts: 1073
Joined: Fri Jul 18, 2008 8:36 am
Location: Hertfordshire
Contact:

Re: Proposing CAS extension: Cue data

Post by sixxie » Mon Aug 21, 2017 12:56 pm

So I've been working on this. It's sort of usable! The "cascue" branch in both http://www.6809.org.uk/git/xroar.git and http://www.6809.org.uk/git/bin2cas.git contain the work in progress stuff.

bin2cas will generate CAS files containing CUE data if you use the --cue option. It'll be reasonably clever about it: if you generate an autorunning file that contains "fast" sections, it'll generate two versions of the loader. The first at the beginning of the file just loads things slowly, so old emulators will still work. The second at the end of the file turns on fast loading and the CUE data references that instead. Extra leader bytes are still emitted in place of silence in-line, but the CUE data replaces it with the special silence markup.

XRoar will load these adorned CAS files, but won't write them yet. I'd like to make it recognise a few different cycle frequencies automatically...

Format basically as you detailed:

Type (byte)
Size (variable-length uint31)
Data

Types:

0xf0 - timing: uint16 bit0 frequency, uint16 bit1 frequency
0x00 - silence: uint16 duration in ms
0x0d - data: uint32 start offset, uint32 end offset (last byte + 1)

Variable-length uint31 is pretty standard: leading 1 bits in initial byte determine number of additional bytes required, so:

Code: Select all

/* Variable-length uint31 defined as:
 * 7-bit        0nnnnnnn
 * 14-bit       10nnnnnn nnnnnnnn
 * 21-bit       110nnnnn nnnnnnnn nnnnnnnn
 * 28-bit       1110nnnn nnnnnnnn nnnnnnnn nnnnnnnn
 * 31-bit       11110XXX Xnnnnnnn nnnnnnnn nnnnnnnn nnnnnnnn
 */
All surrounded by "[CUE", ..., "CUE]" as you suggest.

I also use undocumented types that define a run of data as a 'block' for which the normal block elements are supplied or generated: leader, sync, type, size, checksum. An ASCII flag rewrites linefeeds as carriage returns on the fly; this lets me define ASCII BASIC loads in terms of CUE data internally. Is something like that worth having in any formal spec?

Attached an example CAS file. AUDIOON:CLOADM to hear the difference between old & new xroar builds...
Attachments
buzzard.cas
Example CAS with CUE data
(25.43 KiB) Downloaded 137 times

User avatar
robcfg
Posts: 1232
Joined: Sat Apr 04, 2009 10:16 pm
Location: Stockholm, Sweden

Re: Proposing CAS extension: Cue data

Post by robcfg » Tue Aug 22, 2017 9:23 am

Hi Ciaran!

From your Buzzard.cas file I get these:

Code: Select all

(00) F0 04 04C6 0998            -> 1222hz, 2456hz
(01) 00 02 01F4                 -> 500 ms silence
(02) 0D 08 0000 638E 0000 655F  -> Data from 0x638E to 0x655F
(03) 00 02 01F4                 -> 500 ms silence
(04) 0D 08 0000 0275 0000 0380  -> Data from 0x0275 to 0x0380
(05) F0 04 06E0 11CE            -> 1760hz, 4558hz
(06) 00 02 01F4                 -> 500 ms silence
(07) 0D 08 0000 03DE 0000 16B5  -> Data from 0x03DE to 0x16B5
(08) F0 04 06E0 11CE            -> 1760hz, 4558hz - Redundant
(09) 00 02 01F4                 -> 500 ms silence
(10) 0D 08 0000 1713 0000 6330  -> Data from 0x1713 to 0x6330
(11) 0000 655f                  -> 0x655F - Start of cue data
This is pretty much spot on from my spec with some minor variations!

I find that the 0xF0 command from line 8 is redundant as the next data block uses the same parameters as the previous one. It doesn't hurt it to be there, but could be removed and the behaviour should be the same.

I also see that you have a different version of the loader at the end of the file, before the cue data, so an emulator that doesn't support cue data will load at standard speed while an emulator that does support it will load at faster speed. Am I right?

I've manually modified the Tube Way Army cas file to add the following cue data:

Code: Select all

(00) F0 04 0480 0960           -> 1152hz, 2400hz
(01) 00 02 03E8                -> 1000 ms silence
(02) 0D 08 0000 0000 0000 0173 -> Data from 0x0000 to 0x0172
(03) 00 02 01F4                -> 500 ms silence
(04) 0D 08 0000 0173 0000 03D6 -> Data from 0x0173 to 0x03D5
(05) 00 02 01F4                -> 500 ms silence
(06) 0D 08 0000 03D6 0000 062E -> Data from 0x03D6 to 0x062D
(07) 00 02 01F4                -> 500 ms silence
(08) 0D 08 0000 062E 0000 0792 -> Data from 0x062E to 0x0791
(09) 00 02 01F4                -> 500 ms silence
(10) 0D 08 0000 0792 0000 8753 -> Data from 0x0792 to 0x8752
(11) 0000 8753                 -> Start of cue data
Could you please test it and tell me if it generates a wav file with the right spaces between blocks?

Cheers,
Rob
Attachments
TWA_Cue.cas
(33.92 KiB) Downloaded 137 times

sixxie
Posts: 1073
Joined: Fri Jul 18, 2008 8:36 am
Location: Hertfordshire
Contact:

Re: Proposing CAS extension: Cue data

Post by sixxie » Tue Aug 22, 2017 12:42 pm

robcfg wrote: This is pretty much spot on from my spec with some minor variations!
Didn't want to go changing things for the hell of it!
I find that the 0xF0 command from line 8 is redundant...
Good point, I've added code to bin2cas.pl to eliminate redundant timing entries.
I also see that you have a different version of the loader at the end of the file, before the cue data, so an emulator that doesn't support cue data will load at standard speed while an emulator that does support it will load at faster speed. Am I right?
Yep, so bin2cas will generate stuff that's good for both.
Could you please test it and tell me if it generates a wav file with the right spaces between blocks?
I've not modified cas2wav to handle it yet, but it sounds good with AUDIO ON:CLOADM!

BTW, I have a "builder" for Tube Way Army that uses bin2cas to very slightly tweak the timings to optimise for its custom loader (the special timings are built in to bin2cas):

http://www.6809.org.uk/git/twa.git

sixxie
Posts: 1073
Joined: Fri Jul 18, 2008 8:36 am
Location: Hertfordshire
Contact:

Re: Proposing CAS extension: Cue data

Post by sixxie » Tue Aug 22, 2017 4:15 pm

I've not modified cas2wav to handle it yet
Have now (mis-spent lunch break ;) )

http://www.6809.org.uk/git/cas2wav.git

Same branch name: cascue

Edit: BTW, I still need to adjust cas2wav so that it better fits cycle widths to samples. At the moment there's no error tracking - whole numbers of samples only - which means odd artifacts in the output.

Post Reply