pioasm: Enhance JSON output with additional program metadata

[gonzalolarralde]

Hello,

My primary goal with this PR is to put json_output in a position where it could be used to derive any other output without having to modify pioasm.

Adds escaping for JSON strings and includes new fields in the output such as pioASMVersion, pioVersion, usedGPIORanges, in/out/setCount configurations, movStatus, fifoConfig, clockDiv, codeBlocks, and langOpts. Also uncomments the instruction output disassembly and ensures proper escaping for symbol names and other string fields.

There is only one spec breaking change: sideset moves to be optional, using the value null when not specified. This change aims to allow consumers have a clear representation of the input. All the other changes are additive.

Please let me know if this direction is aligned with the vision for this output, an alternative could be creating a new raw output to avoid leaking too much information to general consumers of json_output.

Looking forward to your feedback, and appreciate your consideration.

[lurch]

I have no opinion either way, but maybe it'd be helpful to demonstrate (just a PR-comment, not a comment in the code) what an example JSON output might look both before and after your change? 🙂

Pinging also @byteit101 , as the original author of this code in #1394

[gonzalolarralde]

Ah yes, absolutely. Sorry about that miss :)

Here is an example where I try to exercise all the different values. Please note that this is closely following the logic in the c-sdk output, where some declarations are omitted if the values match the default behavior. In cases where not changing a setting lands to the expected result this output will choose not to be explicit.

BTW, I just uploaded a change to BUILD.Bazel so Bazel builds json in a way where it picks up its factory method (same as other outputs), this fixes a bug where bazel builds ignores that output. I can provide (and put on a file) the JSON Schema for this output if that helps. If so, what would be a good location for it?

Examples:

{
	"pioASMVersion": "2.2.1-develop",
	"publicSymbols": {
		"MY_GLOBAL_CONST": 2
	},
	"programs": [
		{
			"name": "all_fields",
			"pioVersion": 1,
			"wrapTarget": 0,
			"wrap": 3,
			"origin": 0,
			"usedGPIORanges": 0,
			"publicSymbols": {
				"MY_CONST": 1
			},
			"publicLabels": {
				"loop": 0
			},
			"in": {"pinCount": 1, "right": true, "autopush": true, "threshold": 8},
			"out": {"pinCount": 1, "right": true, "autopull": true, "threshold": 8},
			"setCount": 1,
			"sideset": {"size": 2, "optional": true, "pindirs": true},
			"movStatus": {"type": "STATUS_TX_LESSTHAN", "n": 4},
			"fifoConfig": "PIO_FIFO_JOIN_TX",
			"clockDiv": {"int": 2, "frac": 128},
			"instructions": [
				{"hex": "F801", "instruction": "set    pins, 1         side 1"},
				{"hex": "5001", "instruction": "in     pins, 1         side 0"},
				{"hex": "7001", "instruction": "out    pins, 1         side 0"},
				{"hex": "0000", "instruction": "jmp    0"}
			],
			"codeBlocks": [
				{
					"lang": "c-sdk",
					"contents": "foo\n"
				},
				{
					"lang": "python",
					"contents": "bar\n"
				}
			],
			"langOpts": [
				{
					"lang": "python",
					"name": "foo",
					"value": "bar"
				}
			]
		}
	]
}

That's the output for this program:

.define public MY_GLOBAL_CONST 2

.program all_fields
.lang_opt python foo=bar
.pio_version 1
.define public MY_CONST 1
.origin 0
.side_set 1 opt pindirs
.in 1 right auto 8
.out 1 right auto 8
.set 1
.clock_div 2.5
.fifo tx
.mov_status txfifo < 4

.wrap_target
public loop:
      set pins, 1 side 1
      in pins, 1 side 0
      out pins, 1 side 0
      jmp loop
.wrap

% c-sdk {
foo
%}

% python {
bar
%}

Here is an example of how I use this output to generate, in this case, the output of compiling @raspberrypi/pico-examples/blob/master/pio/hub75/hub75.pio to swift, and hopefully explains why all this values (including instructions) are helpful:

import CPicoSDK

// This file is autogenerated by pioasm-swift using pioasm version 2.2.1-develop do not edit! //

public enum hub75_data_rgb888 {
    public static let wrap_target: UInt32 = 0
    public static let wrap: UInt32 = 15
    public static let pio_version: UInt8 = 0

    public static let offset_shift1 = 7
    public static let offset_entry_point = 0
    public static let offset_shift0 = 0

    public static nonisolated(unsafe) let instructions: UnsafeBufferPointer<UInt16> = {
        let instructions: [UInt16] = [
                            // shift0:
                            // entry_point:
                0x80A0,     //     pull   block           side 0
                            // .wrap_target
                0x40E1,     //     in     osr, 1          side 0
                0x6068,     //     out    null, 8         side 0
                0x40E1,     //     in     osr, 1          side 0
                0x6068,     //     out    null, 8         side 0
                0x40E1,     //     in     osr, 1          side 0
                0x6060,     //     out    null, 32        side 0
                            // shift1:
                0x80A0,     //     pull   block           side 0
                0x50E1,     //     in     osr, 1          side 1
                0x7068,     //     out    null, 8         side 1
                0x50E1,     //     in     osr, 1          side 1
                0x7068,     //     out    null, 8         side 1
                0x50E1,     //     in     osr, 1          side 1
                0x7060,     //     out    null, 32        side 1
                0x507A,     //     in     null, 26        side 1
                            // .wrap
                0xB016,     //     mov    pins, ::isr     side 1
        ]
        let bufferPointer = UnsafeMutableBufferPointer<UInt16>.allocate(capacity: instructions.count)
        let (_, index) = bufferPointer.initialize(from: instructions)
        assert(index == instructions.endIndex, "Failed to initialize buffer pointer")
        return UnsafeBufferPointer(bufferPointer)
    }()

    #if !PICO_NO_HARDWARE
    public static nonisolated(unsafe) let program_storage = pio_program(
        instructions: instructions.baseAddress!,
        length: 16,
        origin: -1,
        pio_version: Self.pio_version,
        used_gpio_ranges: 0
    )

    public static var program: UnsafePointer<pio_program_t> {
        // This is safe only because program_storage has a static lifetime
        withUnsafePointer(to: program_storage) { $0 }
    }

    public static func get_default_config(offset: UInt32) -> pio_sm_config {
        var c = pio_get_default_sm_config();
        sm_config_set_wrap(&c, offset + Self.wrap_target, offset + Self.wrap)
        sm_config_set_sideset(&c, 1, false, false)
        return c;
    }
    #endif
}

[lurch]

Here is an example of how I use this output to generate...

Interesting, so you're going from PIO -> JSON -> Swift, rather than directly from PIO -> JSON ?

in this case, the output of compiling @raspberrypi/pico-examples/blob/master/pio/hub75/hub75.pio to swift,

It looks like that PIO file has both .program hub75_row and .program hub75_data_rgb888, but your Swift file only has public enum hub75_data_rgb888 ? (Or is that deliberate?)

[gonzalolarralde]

@lurch that's correct! I didn't wanted to jump to the assumption that maintainers would be comfortable with pioasm hosting an output for an unsupported language, specially because there are some nuances in that output to how I'm approaching the packaging for Pico SDK. I'm happy to contribute that as well if folks find it useful.

This change gives everyone a chance to pickup the JSON output and generate their own outputs as needed.
Here is how it looks for me: gonzalolarralde/CPicoSDK@main...pioasm-integration
I needed some harness around pioasm anyway so generating the output from that JSON payload is not a big deal.

RE your 2nd point, correct as well. I decided to bring just a chunk to avoid bloating the comments even more 😅

[lurch]

@lurch that's correct! I didn't wanted to jump to the assumption that maintainers would be comfortable with pioasm hosting an output for an unsupported language

It's obviously entirely up to you how you implement your pipeline (and of course using JSON as an intermediary format is a good stress-test of that format 😉 ), but note that we've also added community-contributed Ada #135 and Go #1604 output formats to pioasm, neither of which are languages directly supported by pico-sdk 🙂

RE your 2nd point, correct as well. I decided to bring just a chunk to avoid bloating the comments even more 😅

😆 👍

[gonzalolarralde]

Sounds great. Let me polish that implementation a little bit and I'll submit a PR with that change.

I'd still consider this change useful, hopefully to create some independence around consuming the output of the compiler, but wonder what's your take.

Another approach, more radical, could be changing all the generators to use the same IR. Sounds impractical in the current context and environment, but would probably make it easier to track all changes and ensure all implementations follow the same logic, as the json output provides a closer-to-1:1 mapping of calls and parameters.

Again, for the current context is too much, just a thought.

[gonzalolarralde]

@lurch quick update, I desisted for now from the idea of bringing support into the tool. I'd like to discuss this further with other folks also working on an approach to make this generic enough if that's fine.

For now this PR is all I have, also referenced the PR where I'm using it in case it helps giving context on how it's used.

I think the most relevant aspect to highlight about this PR, aside from the inclusion of more fields which should not be a breaking change on its own, is the new optional declaration of the field "sideset".

@kilograham thanks for tagging this for 2.2.1.

Please let me know if there's anything I should look into changing or iterating.

Thanks again for your time and consideration!