Adding a destination path for loadable_modules

21 views
Skip to first unread message

John Bauman

unread,
Mar 6, 2024, 11:27:32 PM3/6/24
to [email protected], Jiaming Li, Chase Latta, Puneetha Ramachandra, Suraj Malhotra
Hi all,
The loadable_modules SDK atom is designed to take a list of binaries and other resources, so that they can be included in a package and loaded at runtime. We're currently using this to expose some vulkan layers in the SDK, and it sounds like devicetree will be using this in the future.

We're running into the problem that the schema doesn't allow us to specify a destination path in the package where each file should be located. For the vulkan layers, the binaries must go under lib/ and the resources must go in data/vulkan/explicit_layer.d. This knowledge is hardcoded into the GN SDK generator, but not reflected elsewhere. I don't know what the layout requirements are for devicetree visitors; can someone else chime in on what they are?

I'm thinking we should modify the loadable_module schema to have a destination path for each file, so the Bazel SDK won't have to make assumptions about the layout needed. Any objections?

David Dorwin

unread,
Mar 7, 2024, 12:07:16 AM3/7/24
to John Bauman, [email protected], Jiaming Li, Chase Latta, Puneetha Ramachandra, Suraj Malhotra
Some related observations:
  • Unlike static and shared libraries, there is not a GN template (e.g., //build/cpp/sdk_shared_library.gni) for loadable modules.
    • Instead, //src/lib/vulkan/BUILD.gn does lower level work than possibly any other SDK component.
    • Even though there is currently only one instance, it may be better to have everything in one place.
  • Even with those templates, the IDK build is currently fragile with respect to naming and the checks are insufficient (https://fxbug.dev/323441662), so checking the resulting IDK and LSC will be important if changing anything other than metadata.
  • Chromium's GN SDK may also need to be updated if the schema is changed.

--
All posts must follow the Fuchsia Code of Conduct https://fuchsia.dev/fuchsia-src/CODE_OF_CONDUCT or may be removed.
---
You received this message because you are subscribed to the Google Groups "sdk-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/fuchsia.dev/d/msgid/sdk-dev/CAAr6sCcTHYr66-ktm-cxXB%3DB2Pwf4RO2iM95t8H56_GfgnjXvw%40mail.gmail.com.

John Bauman

unread,
Mar 7, 2024, 12:23:59 AM3/7/24
to David Dorwin, [email protected], Jiaming Li, Chase Latta, Puneetha Ramachandra, Suraj Malhotra
On Wed, Mar 6, 2024 at 4:05 PM David Dorwin <[email protected]> wrote:
Some related observations:
  • Unlike static and shared libraries, there is not a GN template (e.g., //build/cpp/sdk_shared_library.gni) for loadable modules.
    • Instead, //src/lib/vulkan/BUILD.gn does lower level work than possibly any other SDK component.
    • Even though there is currently only one instance, it may be better to have everything in one place.
  • Even with those templates, the IDK build is currently fragile with respect to naming and the checks are insufficient (https://fxbug.dev/323441662), so checking the resulting IDK and LSC will be important if changing anything other than metadata.
  • Chromium's GN SDK may also need to be updated if the schema is changed.
Does Chromium use the GN SDK generated by scripts/sdk/gn/generate.py in the Fuchsia tree, or does it have its own generator?

John Bauman

unread,
Mar 7, 2024, 12:47:59 AM3/7/24
to Chase Latta, David Dorwin, [email protected], Jiaming Li, Puneetha Ramachandra, Suraj Malhotra
That could make sense; I'd still like for there to be support for binaries (so selecting the right artifact based on the target platform) and multiple files per atom (so clients don't need to know the exact structure of the files that need to be included), but renaming this whole concept away from "loadable_module" would work.

On Wed, Mar 6, 2024 at 4:25 PM Chase Latta <[email protected]> wrote:
Would it make more sense for us to add a new IDK type for an artifact that is basically a type erased resource that can be put into a package? IIRC, the vulkan layers contain both json files and shared objects but all the end user needs to care about is where to install them. If we create this new type we don't need to worry about breaking down stream users and we get the benefit of being able to distribute these modules in a way that doesn't make assumptions about how they should be used.

Chase Latta

unread,
Mar 7, 2024, 12:55:05 AM3/7/24
to David Dorwin, John Bauman, [email protected], Jiaming Li, Puneetha Ramachandra, Suraj Malhotra
Would it make more sense for us to add a new IDK type for an artifact that is basically a type erased resource that can be put into a package? IIRC, the vulkan layers contain both json files and shared objects but all the end user needs to care about is where to install them. If we create this new type we don't need to worry about breaking down stream users and we get the benefit of being able to distribute these modules in a way that doesn't make assumptions about how they should be used.

On Wed, Mar 6, 2024 at 1:05 PM David Dorwin <[email protected]> wrote:

David Dorwin

unread,
Mar 7, 2024, 12:55:19 AM3/7/24
to John Bauman, [email protected], Jiaming Li, Chase Latta, Puneetha Ramachandra, Suraj Malhotra
On Wed, Mar 6, 2024 at 1:24 PM John Bauman <[email protected]> wrote:


On Wed, Mar 6, 2024 at 4:05 PM David Dorwin <[email protected]> wrote:
Some related observations:
  • Unlike static and shared libraries, there is not a GN template (e.g., //build/cpp/sdk_shared_library.gni) for loadable modules.
    • Instead, //src/lib/vulkan/BUILD.gn does lower level work than possibly any other SDK component.
    • Even though there is currently only one instance, it may be better to have everything in one place.
  • Even with those templates, the IDK build is currently fragile with respect to naming and the checks are insufficient (https://fxbug.dev/323441662), so checking the resulting IDK and LSC will be important if changing anything other than metadata.
  • Chromium's GN SDK may also need to be updated if the schema is changed.
Does Chromium use the GN SDK generated by scripts/sdk/gn/generate.py in the Fuchsia tree, or does it have its own generator?

Last I heard:
  • The GN SDK in fuchsia.git is deprecated but can't yet be deleted. See https://fxbug.dev/42080529 and its dependencies.
  • Chromium forked the fuchsia.git version and is maintaining it. In the future, this will be the only GN SDK
  • Chromium-based repos are also using Chromium's version (via DEPS I believe).

Puneetha Ramachandra

unread,
Mar 7, 2024, 1:53:01 AM3/7/24
to David Dorwin, John Bauman, [email protected], Jiaming Li, Chase Latta, Suraj Malhotra
As for devicetree, there are a bunch of visitor shared objects which need to be placed under `lib/visitors/` in the driver package. So yes, updating/adding a schema to specify path in package would be useful.

John Bauman

unread,
Mar 8, 2024, 8:09:25 PM3/8/24
to Puneetha Ramachandra, David Dorwin, [email protected], Jiaming Li, Chase Latta, Suraj Malhotra
Ok, I've put up https://fuchsia-review.git.corp.google.com/c/fuchsia/+/1003732 for review, which adds a new "resource" type (name is, of course, subject to change) that has destinations and might be generic beyond "loadable module" usage.

Hunter Freyer

unread,
Mar 8, 2024, 8:42:20 PM3/8/24
to sdk-dev, John Bauman, David Dorwin, [email protected], Jiaming Li, Chase Latta, Suraj Malhotra, Puneetha Ramachandra
Replied on the CL, but I think I'd prefer something more specific. "loadable_modules" was very difficult to understand because it was so generic, when actually there was only one use of it, which was quite specific, and SDKs made use of that knowledge.

Can we just call it `vulkan_layer` or something?

Chase Latta

unread,
Mar 8, 2024, 8:55:35 PM3/8/24
to Hunter Freyer, sdk-dev, John Bauman, David Dorwin, Jiaming Li, Suraj Malhotra, Puneetha Ramachandra
On Fri, Mar 8, 2024 at 9:38 AM Hunter Freyer <[email protected]> wrote:
Replied on the CL, but I think I'd prefer something more specific. "loadable_modules" was very difficult to understand because it was so generic, when actually there was only one use of it, which was quite specific, and SDKs made use of that knowledge.

Can we just call it `vulkan_layer` or something?
I don't think we should call this `vulkan_layer` because we need something exactly like this for devicetree visitors. Each time we add a new type it adds work for each of our SDKs so if we are going to introduce a new type that describes how to put bundle something in a package that needs to be installed at a specific location lets just do that. 

David Dorwin

unread,
Jul 15, 2024, 8:30:33 PM7/15/24
to Chase Latta, Hunter Freyer, sdk-dev, John Bauman, Jiaming Li, Suraj Malhotra, Puneetha Ramachandra
What are the next steps here? loadable_modules is one of the remaining exceptions to Platform Versioning in the SDK, and it doesn't make sense to work on that until we've settled on a path forward here. For example, if we go in another direction, maybe we can just remove loadable_modules.
Reply all
Reply to author
Forward
0 new messages