Add support for Rust v0 symbol mangling scheme #491

cerisier · 2025-01-12T00:40:09Z

Attempt to tackle #474 as per the discussion in the issue thread. Fixes #474.

Until now, demangling was handled by shipping emscripten compiled javascript as a blob and the exact details of how it was compiled had been partially lost, only that it used part of https://github.com/nattofriends/c-filtjs.

In order to support Rust demangling, it has been discussed that we'd take the opportunity to include the source code as part of speedscope and document the process.

This PR follows several guidelines:

Use the same library as c-filtjs was using (GNU libiberty from gcc which is written in pure C).
No additional runtime dependencies.
Keep module loading lazy (since demangling is not always needed)
Optimize for bundle size.

The approach I took is to write a wrapper demangle function in C which routes demangling to the proper demangler.
It uses cplus_demangle_v3 for theItanium C++ ABI and rust_demangle for Rust.
I followed the same logic as llvm::demangle for routing.

This file is then compiled by emscripten into a single JavaScript file embedding the .wasm file.

Improvements

While working on this, I realized that the implementation within llvm support more cases.
I learned about the existence of symbols with up to 4 _ like ____Z which are not handled by libiberty directly.
One improvement could be to use llvm::demangle rather than GNU libiberty and support more cases.

Swift demangling

Swift also has its own mangling scheme and while profilers like Instruments already demangle Swift symbols on macOS, this support is inexistant on Linux. Adn since Swift is being used more and more outside of macOS, it is likely that we'll start to see perf traces of Swift programs recorded on Linux which wouldn't get demangling directly.

This is required because emscripten glue code now uses es6 features like logicalAssignment "??=" which is only supported by newer version of parcel.

jlfwong · 2025-01-12T01:44:54Z

package.json

    "jszip": "3.1.5",
    "pako": "1.0.6",
-    "parcel-bundler": "1.12.4",
+    "parcel": "2.13.3",


Was this necessary as part of this change? I'm not opposed per-se, but I don't like upgrading dependencies as part of PRs that introduce other changes in case they need reverting

Fully understand. I added the explanation in the commit. Sadly the emscripten generated code is a raw JavaScript file that contains features that are not supported by That version of parcel-bundled hence why I had to upgrade it.

This is required because emscripten glue code now uses es6 features
like logicalAssignment "??=" which is only supported by newer version
of parcel.

jlfwong · 2025-01-12T01:45:49Z

src/lib/profile.ts

-        if (!demangleCpp) {
-          demangleCpp = (await demangleCppModule).demangleCpp
+      // This function converts a mangled C++ and Rust name into a human-readable symbol.
+      if (frame.name.startsWith('__Z') || frame.name.startsWith('_R') || frame.name.startsWith('_Z')) {


Hmmm is there any other way of sanely auto-detecting this? __Z is so rare as a prefix it seems unlikely to yield false positives, but _R and _Z seem far more likely and I'd like to avoid incurring this loading cost unless needed

I don't see any other way sadly.

Unless mistaken, _Z is the default mangling prefix and __Z happens mostly on macOS where symbols get an extra _.

I tested with a raw undemangled C++ perf trace on Linux and speedscope didn't demsngle it. I added this as a result but should have made a separate commit to explain.

cerisier · 2025-01-12T20:18:59Z

Referencing rust-lang/rust#60705 for cross visibility :)

src/lib/demangle/demangle.test.ts

jlfwong · 2025-01-13T18:39:17Z

src/lib/profile.ts

+        frame.name.startsWith('_Z')
+      ) {
+        if (!demangle) {
+          const demangleModule = await import('./demangle')


The import was above intentionally at the top of the file to cause eager, non-blocking loading of this module. As written here, the load of this module will only begin after a profile import, which will be much slower.

In this case, I think we want both the import and the loadDemangling to happen eagerly, and then to await the promise in here.

I think the right way of doing that is to eagerly do the import here, and to eagerly create the wasm module in demangle.ts

From what I read, demangle is already imported eagerly by application.tsx no ?

As for demangle.ts, is it fine if I do this ?

diff --git a/src/lib/demangle/demangle.ts b/src/lib/demangle/demangle.ts index b69adeb..70f17a0 100644 --- a/src/lib/demangle/demangle.ts +++ b/src/lib/demangle/demangle.ts @@ -1,11 +1,13 @@ import createWasmDemangleModule from './demangle.wasm' +const wasmDemangleModulePromise = createWasmDemangleModule().then((module) => module) + const cache = new Map<string, string>() export async function loadDemangling(): Promise<(name: string) => string> { // This function converts a mangled C++ name such as "__ZNK7Support6ColorFeqERKS0_" // into a human-readable symbol (in this case "Support::ColorF::==(Support::ColorF&)") - const wasmDemangleModule = await createWasmDemangleModule() + const wasmDemangleModule = await wasmDemangleModulePromise return cached(wasmDemangleModule.wasm_demangle) }

That way, everything is eager from application.tsx.
import ("./demangle") from profile.tsx is awaiting the eager loaded import from application.tsx
and loadDemangling in profile.tsx just await the eager loaded wasmModule triggered by the eager loding of demangle from `application.tsx.

jlfwong · 2025-01-14T07:17:07Z

It looks like there's a bunch of stuff downstream of the parcel upgrade that's otherwise unrelated to this change. I'm going to fix it, then ask you to rebase this on top of that

cerisier · 2025-01-14T17:21:41Z

@jlfwong I have an idea that might save us from upgrade parcel. Basically babelize the generated emscripten code so that it's compatible with whatever javascript version is compatibel with parce-bundler at 1.12.4.

Either from the Makefile directly or as part of the bundling. I'll make a test before suggesting a change here.

jlfwong · 2025-01-14T20:24:07Z

@cerisier that would be appreciated! I'll need to deal with the bundling problems eventually regardless, but would love to land your changes without dealing with that first.

Rather than babel to do this, I might suggest using esbuild in transform mode. You can set the target property or the supported property to achieve this: https://esbuild.github.io/api/#supported

Something like...

esbuild --target=chrome79 --loader:ts emscripten-output.js

cerisier · 2025-01-14T20:25:03Z

Wonderful I'll do that

This reverts commit c7be11d.

cerisier · 2025-01-14T22:35:45Z

@jlfwong done. I reverted your PR about upgrading parcel in the mean time. Let me know if you want to do it on main first and I'll just re-revert it here.

jlfwong · 2025-01-15T00:15:48Z

@cerisier Thanks -- I reverted in main. Can you rebase please to make this PR diff easier to read?

jlfwong · 2025-01-15T10:02:57Z

In the process of trying to fix this problem a different way by switching to esbuild, I realized ran into another challenging requirement: the ability to load things off of file:/// URLs.

This hasn't been an issue to date because speedscope has used neither wasm nor <script type="module" /> anywhere.

Switching to esbuild necessitates module scripts for code splitting, but I can work around it by not using code splitting for the local file:/// builds. This also means that the wasm module introduced in this PR must be inlined as a base64 string for the non-code-split build, since instantiateStreaming and friends will refuse to load from file:/// local builds.

I think this is all resolvable, but a bit of added pain. See #295 for more detail

EDIT: Skimming through the code in this PR, it looks like the wasm file contents are already base64 encoded in here. So, false alarm!

cerisier · 2025-01-15T14:57:22Z

@jlfwong done

cerisier · 2025-01-15T20:01:14Z

Prettier fixed

coveralls · 2025-01-15T20:07:13Z

coverage: 43.35% (-0.02%) from 43.366%
when pulling eb6926d on cerisier:cerisier/rust-demangling
into c04a148 on jlfwong:main.

jlfwong · 2025-01-15T22:59:53Z

Sorry for all the flux -- I landed the change to migrate to esbuild and deployed everything. Can you try rebasing one more time and make sure everything looks good? You can also now remove the steps needed to cross-compile with esbuild to get rid of the ?? since now we're using esbuild as the main bundler here, so parcel 1 is gone entirely.

cerisier · 2025-01-16T17:23:37Z

@jlfwong done !

cerisier · 2025-01-16T19:11:00Z

✅

jlfwong · 2025-01-16T19:46:53Z

Thanks! Will deploy later today

jlfwong · 2025-01-16T22:07:22Z

@cerisier How do I easily create a profile which has symbols mangled in this way? I'd like to test locally before I deploy this to make sure it's working in all the various environments speedscope runs in

cerisier · 2025-01-16T22:13:14Z

I deleted the example trace I was testing this with :(
I'll have to make a new one unless you get it faster.

cerisier · 2025-01-16T22:42:57Z

@jlfwong https://gist.github.com/cerisier/c74eb3e5e55a08a5fdf443b21818db40

jlfwong · 2025-01-17T00:52:18Z

@cerisier Thanks! Was able to test it, everything looks good, and this is now deployed on https://www.speedscope.app/ and on npm as [email protected]

cerisier added 2 commits January 12, 2025 00:08

Upgrade parcel

9560a1b

This is required because emscripten glue code now uses es6 features like logicalAssignment "??=" which is only supported by newer version of parcel.

Use wasm to demangle C++ and Rust

5f7f56e

jlfwong reviewed Jan 12, 2025

View reviewed changes

cerisier added 3 commits January 12, 2025 13:43

lint

1a015f0

move to subfolder

82d7cd1

Add C source code and build instructions

ff9c63f

cerisier marked this pull request as ready for review January 12, 2025 13:59

jlfwong reviewed Jan 13, 2025

View reviewed changes

src/lib/demangle/demangle.test.ts Show resolved Hide resolved

jlfwong reviewed Jan 13, 2025

View reviewed changes

cerisier added 4 commits January 13, 2025 22:15

fallback to import url: for jest compatibility

5f70af5

lint

4e75a98

lint

a3a836c

Add false positive test

09f41d1

This was referenced Jan 14, 2025

Upgrade to parcel 2.13.3 #492

Merged

Unbreak production builds in parcel #493

Closed

cerisier added 2 commits January 14, 2025 16:02

Merge remote-tracking branch 'origin/main' into cerisier/rust-demangling

a517ff5

Properly eagerly import and await

b7a2788

cerisier added 2 commits January 14, 2025 23:33

Revert "Upgrade to parcel 2.13.3 (jlfwong#492)"

6ffa4d3

This reverts commit c7be11d.

Use esbuild on emscripten glue code

5bd2d01

add header to extern post

85c51f9

cerisier added 2 commits January 15, 2025 15:52

Merge remote-tracking branch 'origin/main' into cerisier/rust-demangling

6d2eacd

cleanup

62eafd9

add emscripten version

5b58f86

format

8b31d4e

cerisier added 2 commits January 16, 2025 18:12

Merge remote-tracking branch 'origin/main' into cerisier/rust-demangling

c317bda

Back to esnext

faadf24

clean gitconfig

eb6926d

jlfwong merged commit dd989ea into jlfwong:main Jan 16, 2025
6 checks passed

cerisier deleted the cerisier/rust-demangling branch January 16, 2025 22:12

Add support for Rust v0 symbol mangling scheme #491

Add support for Rust v0 symbol mangling scheme #491

Uh oh!

Conversation

cerisier commented Jan 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Improvements

Swift demangling

Uh oh!

jlfwong Jan 12, 2025

Choose a reason for hiding this comment

Uh oh!

cerisier Jan 12, 2025

Choose a reason for hiding this comment

Uh oh!

cerisier Jan 12, 2025

Choose a reason for hiding this comment

Uh oh!

jlfwong Jan 12, 2025

Choose a reason for hiding this comment

Uh oh!

cerisier Jan 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cerisier commented Jan 12, 2025

Uh oh!

Uh oh!

jlfwong Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cerisier Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jlfwong commented Jan 14, 2025

Uh oh!

cerisier commented Jan 14, 2025

Uh oh!

jlfwong commented Jan 14, 2025

Uh oh!

cerisier commented Jan 14, 2025

Uh oh!

cerisier commented Jan 14, 2025

Uh oh!

jlfwong commented Jan 15, 2025

Uh oh!

jlfwong commented Jan 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cerisier commented Jan 15, 2025

Uh oh!

cerisier commented Jan 15, 2025

Uh oh!

coveralls commented Jan 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jlfwong commented Jan 15, 2025

Uh oh!

cerisier commented Jan 16, 2025

Uh oh!

cerisier commented Jan 16, 2025

Uh oh!

jlfwong commented Jan 16, 2025

Uh oh!

Uh oh!

jlfwong commented Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cerisier commented Jan 16, 2025

Uh oh!

cerisier commented Jan 16, 2025

Uh oh!

jlfwong commented Jan 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

cerisier commented Jan 12, 2025 •

edited

Loading

cerisier Jan 12, 2025 •

edited

Loading

jlfwong Jan 13, 2025 •

edited

Loading

cerisier Jan 13, 2025 •

edited

Loading

jlfwong commented Jan 15, 2025 •

edited

Loading

coveralls commented Jan 15, 2025 •

edited

Loading

jlfwong commented Jan 16, 2025 •

edited

Loading