Photo by @hasanalmasi

PyTorch Governance and History

PyTorch’s design philosophy and increasing popularity makes it an interesting programming language to better understand

It would be strange for an ‘outsider’ someone who does not frequently use PyTorch to write about governance and history within the programming language. I decided to try to map out parts and bits of the story as best as I can. Understanding this the text you are about to read is an attempt at doing so.

It is one to follow for many reasons.

According to an article from 2019 by Rohit Yadav:

• “According to reports, PyTorch grew 194% in the first half of 2019 alone (Jan-Jun 2018 to Jan-Jun 2019).
• A total of 1,800 papers mentioned TensorFlow, while PyTorch more or less had similar mention in the papers
• Although in the StackOverflow survey 2019, TensorFlow was head and shoulders ahead of PyTorch in popularity, PyTorch was 2nd in the most loved tools, whereas TensorFlow was a distant 5th.”

1. “TorchScript: One of the merits of PyTorch is its usability and readability. On the other hand, developers complain about boilerplate code, thereby, making it complex for new users. Although it has its own advantages, new users and researchers who mostly do not focus on production-level code, embrace PyTorch as coding is very straightforward. Torchscript delivers flexibility in transitioning between eager mode and graph mode for taking advantages of both the processes.
2. Distributed Training: Through asynchronous execution, it helps researchers to parallelise computation for processing a large batch of input data. Researchers often prefer a plethora of data for their projects as their sole focus is to get the most accurate results. However, in production, organisations try to find the balance between quality and usability.
3. Tools & Libraries: The above advantages, over the years, have led to an active community of researchers. Therefore, one can get more insightful support to mitigate their problems. Consequently, aspirants of deep learning technology are adopting PyTorch.”

There are likely far more reasons than mentioned above.

A short history

In October 2016 PyTorch began as an internship project by Adam Paszke.

At the time he was working under Soumith Chintala, a core developer of Torch.

Torch is an open-source machine learning library, a scientific computing framework, and a script language based on the Lua programming language. It had an initial release October 2002.

PyTorch then got two more core developers on board and around 100 alpha testers from different companies and universities.

Original authors were Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan.

I added a job description to the current PyTorch maintainers and they are:

• Adam Paszke (apaszke). Senior Research Scientist at Google.
• Soumith Chintala (soumith). Researcher at Facebook AI.
• Edward Yang (ezyang). Research Engineer, Facebook AI Research.
• Greg Chanan (gchanan). Technical Lead / Software Engineering Manager at Facebook.
• Dmytro Dzhulgakov (dzhulgakov). Technical Lead at Facebook AI.
• (sunsetting) Sam Gross (colesbury). Software Engineer at Facebook AI Research.

PyTorch is an open source machine learning library. It is released under the Modified BSD license.

It is used for applications such as computer vision and natural language processing, and was primarily developed by Facebook’s AI Research lab (FAIR).

A number of pieces of Deep Learning software are built on top of PyTorch, including:

• Tesla Autopilot.
• Uber’s Pyro.
• HuggingFace’s Transformers.
• PyTorch Lightning.
• Catalyst.

In this PyTorch provides two high-level features:

1. Tensor computing (like NumPy) with strong acceleration via graphics processing units (GPU)
2. Deep neural networks built on a tape-based automatic differentiation system.

Facebook operates both PyTorch and Convolutional Architecture for Fast Feature Embedding (Caffe2).

• “The Open Neural Network Exchange (ONNX) project was created by Facebook and Microsoft in September 2017 for converting models between frameworks. Caffe2 was merged into PyTorch at the end of March 2018.”

“PyTorch defines a class called Tensor (torch.Tensor) to store and operate on homogeneous multidimensional rectangular arrays of numbers.”

PyTorch supports various sub-types of Tensors.

In some sense PyTorch Tensors are similar to NumPy Arrays.

(NumPy is a library for the Python programming language, adding support for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays.)

It can be operated on a CUDA-capable Nvidia GPU.

(CUDA (Compute Unified Device Architecture) is a parallel computing platform and application programming interface (API) model created by Nvidia.)

Governance Philosophy and Guiding Tenets

Looking at the PyTorch website their governance is described there.

“PyTorch adopts a governance structure with a small set of maintainers driving the overall project direction with a strong bias towards PyTorch’s design philosophy where design and code contributions are valued.“

There are core maintainers, and a broader set of core developers that directly can merge pull requests and own various parts of the core code base.

Beyond the maintainers and core devs, the community is encouraged to:

• Contribute.
• File issues.
• Make proposals.
• Review pull requests.
• Be present in the community.

Anyone given contributions and willingness can write for PyTorch.

“Based on this governance structure, the project has the following core operating tenets by which decisions are made and overall culture is derived:

1. Code contributions matter much more than corporate sponsorship and independent developers are highly valued.
2. Project influence is gained through contributions (whether PRs, forum answers, code reviews or otherwise)”

To some extent this is limited information about a very interesting language, and I would very much like to learn more.

If you have any sources you would recommend I read about the history or governance of PyTorch feel free to share in a response!

This is #500daysofAI and you are reading article 430. I am writing one new article about or related to artificial intelligence every day for 500 days.