PyTorch
来源:互联网 发布:99家居软件 编辑:程序博客网 时间:2024/06/10 00:35
How autograd encodes the history
Each Variable has a .creator
attribute, that points to the function, of which it is an output. This is an entry point to a directed acyclic graph (DAG) consisting of Function
objects as nodes, and references between them being the edges. Every time an operation is performed, a new Function
representing it is instantiated, its forward()
method is called, and its output Variable
s creators are set to it. Then, by following the path from any Variable
to the leaves, it is possible to reconstruct the sequence of operations that has created the data, and automatically compute the gradients.
An important thing to note is that the graph is recreated from scratch at every iteration, and this is exactly what allows for using arbitrary Python control flow statements, that can change the overall shape and size of the graph at every iteration. You don’t have to encode all possible paths before you launch the training - what you run is what you differentiate.
In-place operations on Variables
Supporting in-place operations in autograd is a hard matter, and we discourage their use in most cases. Autograd’s aggressive buffer freeing and reuse makes it very efficient and there are very few occasions when in-place operations actually lower memory usage by any significant amount. Unless you’re operating under heavy memory pressure, you might never need to use them.
There are two main reasons that limit the applicability of in-place operations:
- Overwriting values required to compute gradients. This is why variables don’t support
log_
. Its gradient formula requires the original input, and while it is possible to recreate it by computing the inverse operation, it is numerically unstable, and requires additional work that often defeats the purpose of using these functions. - Every in-place operation actually requires the implementation to rewrite the computational graph. Out-of-place versions simply allocate new objects and keep references to the old graph, while in-place operations, require changing the creator of all inputs to the
Function
representing this operation. This can be tricky, especially if there are many Variables that reference the same storage (e.g. created by indexing or transposing), and in-place functions will actually raise an error if the storage of modified inputs is referenced by any otherVariable
.
In-place correctness checks
Every variable keeps a version counter, that is incremented every time it’s marked dirty in any operation. When a Function saves any tensors for backward, a version counter of their containing Variable is saved as well. Once you access self.saved_tensors
it is checked, and if it’s greater than the saved value an error is raised.
- PyTorch
- PyTorch
- PyTorch
- pytorch
- pytorch
- Pytorch
- Pytorch是什么?关于Pytorch
- pytorch参考
- PyTorch开源
- 安装pytorch
- pytorch可视化
- 扩展PyTorch
- pytorch 使用
- pytorch-tutorial
- pytorch-hook
- pytorch 入门
- pytorch入门
- pytorch-errors
- append
- Java多线程之线程返回值
- 十大Intellij IDEA快捷键
- 蓝桥杯 算法提高 线段和点
- [python]利用BeautifulSoup进行简单图片抓取
- PyTorch
- 王爽 汇编语言 第三版 实验14 访问 CMOS RAM
- 腾讯云数据库团队:SQL Server 数据加密功能解析
- Modsecurity配置限制DoS攻击
- Mysql GROUP_CONCAT 将Decimal类型转化为 blob类型
- tensorflow笔记:流程,概念和简单代码注释
- Problem L-12 Reverse Text
- Python- 解决PIP下载安装速度慢
- 命里有时终须有,命里无时莫强求