Skip to content

Starting to add resource metrics on runs of the tool. #60

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
fosterbrereton merged 7 commits into from
Jan 9, 2023
Merged

Starting to add resource metrics on runs of the tool. #60

fosterbrereton merged 7 commits into from
Jan 9, 2023

Conversation

@fosterbrereton
Copy link
Contributor

@fosterbrereton fosterbrereton commented Jan 9, 2023 •
edited
Loading

We are looking to improve our understanding of how ORC utilizes the machine resources during processing, possibly in order to identify tradeoffs we can make in space v. time. This PR adds some initial memory footprint metrics to the end of the ORC run's output, along with a flag to enable it.

Here's an example output:

Resource metrics:
  String pool size / waste:
    0: 32 MiB (33554432) / 2.19 MiB (2292587) / 6.83%
    1: 32 MiB (33554432) / 2.21 MiB (2316317) / 6.90%
    2: 32 MiB (33554432) / 2.20 MiB (2310493) / 6.89%
    3: 32 MiB (33554432) / 2.31 MiB (2420043) / 7.21%
    4: 32 MiB (33554432) / 2.41 MiB (2527218) / 7.53%
    5: 32 MiB (33554432) / 2.13 MiB (2231512) / 6.65%
    6: 32 MiB (33554432) / 2.19 MiB (2292399) / 6.83%
    7: 32 MiB (33554432) / 2.16 MiB (2269894) / 6.76%
    8: 32 MiB (33554432) / 2.13 MiB (2237163) / 6.67%
    9: 32 MiB (33554432) / 2.26 MiB (2370840) / 7.07%
    10: 32 MiB (33554432) / 2.30 MiB (2411155) / 7.19%
    11: 32 MiB (33554432) / 2.32 MiB (2429004) / 7.24%
    12: 32 MiB (33554432) / 2.25 MiB (2355728) / 7.02%
    13: 32 MiB (33554432) / 2.42 MiB (2537523) / 7.56%
    14: 32 MiB (33554432) / 2.30 MiB (2408363) / 7.18%
    15: 32 MiB (33554432) / 2.30 MiB (2409418) / 7.18%
    16: 32 MiB (33554432) / 2.23 MiB (2337468) / 6.97%
    17: 32 MiB (33554432) / 2.43 MiB (2544395) / 7.58%
    18: 32 MiB (33554432) / 2.54 MiB (2664409) / 7.94%
    19: 32 MiB (33554432) / 2.15 MiB (2258360) / 6.73%
    20: 32 MiB (33554432) / 2.40 MiB (2514859) / 7.49%
    21: 32 MiB (33554432) / 2.35 MiB (2464329) / 7.34%
    22: 32 MiB (33554432) / 2.30 MiB (2411696) / 7.19%
    totals: 736 MiB (771751936) / 52.47 MiB (55015173) / 7.13%
  die footprint: 5.31 GiB (5702569584)
leethomason
leethomason approved these changes Jan 9, 2023
View reviewed changes
Copy link
Contributor

@leethomason leethomason left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Nice use of std::array as well.
@fosterbrereton fosterbrereton merged commit 34f16b0 into main Jan 9, 2023
@fosterbrereton fosterbrereton deleted the fosterbrereton/string-pool-reportings branch January 9, 2023 23:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants

@fosterbrereton @leethomason