Skip to content

optimize scalar sampling#58

Merged
daming-lu merged 4 commits intodevelopfrom
optimize_sampling
Jan 4, 2018
Merged

optimize scalar sampling#58
daming-lu merged 4 commits intodevelopfrom
optimize_sampling

Conversation

@daming-lu
Copy link
Collaborator

No description provided.

@daming-lu daming-lu requested a review from Superjomn January 4, 2018 08:23

def get_scalar(storage, mode, tag, num_records=300):
# if sample amount is negative, 0 or 1, use default
if num_records <= 1:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assert num_records > 1

serious error.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

res.append(data[id])
return [v for v in reversed(res)]
# if sample amount is larger than actual data amount, then choose all data
if num_records > len(data):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

conflicts with line 45

no need to sample again.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

span = float(len(data)) / (num_records - 1)
span_multiply = 0

data_idx = int(span_multiply * span)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

span_multiply -> span_offset ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

span_multiply += 1
data_idx = int(span_multiply * span)

sampled_data.append(data[-1])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sample from data[-1] to data[0], so the latest record can be taken naturally.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

sampled_data = []

while data_idx < len(data):
sampled_data.append(data[len(data) - data_idx - 1])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

len(...) will run every time in the iter
place it outside the whileloop like:

data_size = len(data)

while ...:
    ...
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@Superjomn Superjomn changed the title optimize sampling Jan 4, 2018
Copy link
Contributor

@Superjomn Superjomn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@daming-lu daming-lu merged commit f1db2a5 into develop Jan 4, 2018
@daming-lu daming-lu deleted the optimize_sampling branch January 4, 2018 10:42
jacquesqiao pushed a commit to jacquesqiao/VisualDL that referenced this pull request Jan 12, 2018
* optimize sampling

* fix comments

* len(data) only once
jacquesqiao pushed a commit to jacquesqiao/VisualDL that referenced this pull request Jan 13, 2018
* optimize sampling

* fix comments

* len(data) only once
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants