Skyline's memory profiling is for GPU memory only.
Skyline tracks the memory usage associated with a model's weights and activations. Skyline will also report the peak amount of memory allocated during a training iteration.
Just like the run time report, Skyline also includes the stack trace associated with each activation or weight in the report. Skyline only includes the stack frames associated with files inside your project (i.e. files under your project's root directory).
This table holds the memory used by the model's weights. The
column is the number of bytes used by the weight and
grad_size_bytes is the
number of bytes used by the weight's gradient tensor. The
name column holds
the weight's name, which is assigned by PyTorch.
This table holds the memory used by the model's activations in one training
size_bytes column is the number of bytes used by the
operation_name column is the name of the operation that
generated the activation.
This is a table that stores mappings of Skyline's memory entry types
(activations, weights) to numeric identifiers. Skyline maps weights to an entry
1, and activations to an entry type of
This table maps entries to a
correlation_id, which can be used to look up a
memory entry's relevant stack frames in the
stack_frames table. The
entry_type column contains either
2, which corresponds to the
weights and activations respectively.
For all rows where
entry_type == 1, the
entry_id column will act as a
foreign key for the
id column in the
weight_entries table. Similarly for
all rows where
entry_type == 2, the
entry_id column will act as a foreign
key for the
id column in the
This table holds the stack frames associated with a memory usage entry (both
weights and activations). The
correlation_id column is a foreign key that
correlation_id in the
stack_correlation table. File paths
stored in the
file_path column will be relative to the project's root
directory and line numbers are 1-based.
Skyline does not add an explicit foreign key constraint to the
There may be multiple stack frames associated with any given memory entry (i.e.
ordering column is used to keep track of the
ordering among stack frames that share the same
correlation_id. When sorted
in ascending order by the
ordering column, the stack frames will be ordered
from most-specific (i.e. closest to the weight or operation responsible for
the activation) to least-specific (i.e. farthest from the weight or operation
responsible for the activation).
Connecting to Entries.
To get the stack frames for a given entry, you need to first query the
stack_correlation table to find the
correlation_id associated with your
entry_type combination. Then you can use that
to look up the associated stack frames in this table.
This table holds any miscellaneous memory usage information that is reported by
Skyline. Currently, Skyline only uses this table to report the peak memory
usage during one training iteration. This memory usage is reported using the