The latest paper by David Patterson & Google TPU team reveals details of the world most efficient and one of the most powerful supercomputers for DNN Acceleration - TPU v3. The one which was used to train BERT.
We recommend that you definitely read the full text, but here are insights and tldr highlights

Key Insight:
The co-design of an ML-specific programming system (TensorFlow), compiler (XLA), architecture (TPU), floating-point arithmetic (Brain float16), interconnect (ICI), and chip (TPUv2/v3) let production ML applications scale at 96%–99% of perfect linear speedup and 10x gains in performance/ Watt over the most efficient general-purpose supercomputers.

More highlights:

🐣🐤🐔 Three generations
There are 3 generations of TPU now released, TPU v1 used fixpoint arithmetic and was used for inference only. TPU v2 and v3 operate in floating-point and used for training. TPU v4 results were presented in MLPerf summer release, but there is no public information available. The TPU architecture differs from CPU with
▪️ Two Dimensional array processing units (instead of 1D vector SIMDs in CPU)
▪️Narrower data (8-16 bits)
▪️ Drop complex CPU features - caches and branch prediction

🐮🤜🐤 Fewer cores per chip (two oxen vs 1024 chickens)
NVidia put thousands of CUDA cores inside their chip. TPU v3 has only 2 TensorCores per chip. It's way easier to generate a program for 2 beefier cores than to swarm of wimpier cores.
Each TensorCore includes the following units:-
▪️ICI(Inter Core Interconnects) - connect core across different chips-
▪️HBM, stacked DRAM on the same interposes substrate-
▪️Core Sequencer - manages instructions and performs scalar operations-
▪️Vector Processing Unit, performs vectors operation for 1D and 2D vectors-
▪️Matrix Multiply Unit (MXU)

🐱🐶❓ From inference to training chip
Key challenges on the way from inference chip V1 to training hardware V2
▪️ Harder parallelization
▪️ More computation
▪️ More memory
▪️ More programmability
▪️ Wider dynamic range of data

✂️🧮✂️ Brain Float
IEEE FP16 and FP32 use (1+8+23) and (1+5+7) bits for the sign, exponent, and mantissa values respectively. In practice, DNN doesn't need mantissa precision of FP32, but the dynamic range of FP16 is not enough. Using of FP16 also requires loss scaling.
The compromised bf16 keeps the same 8 bits for exponent, as FP32, but reduced mantissa - only 7 bits instead of 23.
BF16 delivers reducing space usage and power consumption with no loss scaling in software required.

🍩🧬⚡️ Torus topology and ICI
TPU v1 was an accelerator card for CPU 'based computer. TPUv2 and v3 are building blocks of the supercomputer. Chips connected with ICI interface, each running at ~500Gbits/s. ICU enables direct connection between chips, so no need of any extra interfaces. GPU/CPU based supercomputers have to apply NVLink and PCI-E inside computer chase and InfiniBand network and switches to connect them.
Chips in TPUv2 and v3 clusters are connected in 2D Torus topology (doughnut ) and achieve an unbelievable linear scale of performance growth with increasing of chips number.


🛠⚙️🖥 XLA compiler (to orchestrate them all)
TF programs are graphs of operations, where tensor-arrays are first-class citizens. XLA compiler front-end transforms the TF graph into an intermediate representation, which is then efficiently mapped into selected TPU (or CPU/GPU) architectures. XLA maps TF graph parallelism across hundreds of chips, TensorCores per chip, multiple units per core. XLA provides precise reasoning about memory use at every point in the program.
Young XLA compiler has more opportunities to improve than a more mature CUDA stack.


🌲🐰🦊 Green Power (Forest animals approves)
TPU v3 supercomputer already climbed on the 4th row of TOP500 ranking, but what is remarkable - it demonstrates an overwhelming 146.3 GFLops/Watt performance. The nearest competitor has 10 times and lower number.

Original Paper
A Domain Specific Computer for training DNN

🤔🌳🏃‍♂️ Decision trees accelerating

One would be surprised, but DNNs do not exhaust the list of ML algorithms. In fact, few businesses can find an application of CV or NLP, few have a significant amount of speech or photo data, where DNN shows SOTA results.

But most of them have a huge amount of irregular table data - financial market prices, customer data, base station activity logs, or windmills breakdown statistics.
And that's where decision trees get up on stage. There are 3 major frameworks on the market nowadays, who provides frameworks for training ensembles of decision trees with gradient boosting. It's XGboost, CatBoost and LightGBM. Read here to learn more about them here or here

Due to the specific nature, good match between algorithm and hardware organizations, decision trees can be significantly accelerated on FPGA.
We will cover 2 stories here

🌝🇩🇪🥨 Xelera Decision Tree Acceleration
Germany-based startup Xelera offers FPGA devices as a hardware backend for decision trees inference acceleration. The company claims 700x both speedup and latency improvement. FPGA results were estimated on the cloud AWS F1 FPGA instances and on the Xilinx Alveo U50.
The secret as to why FPGAs perform so well on this class of workloads is their unique memory architecture, which consists of thousands of independent blocks of on-chip memory. This memory is not only highly parallel a key difference to the GPU memory is that it can handle highly parallel, irregular memory accesses very well.

🇺🇸🔬🎳 FPGAs for Particles classification
HLS4ML is an open-source framework from the Cornell University team, who is working in CERN, where FPGAs are used for trigger condition detection or particle classification.
HLS4ML generates an HLS description of the ML algorithm, which you may feed to the HLS synthesis tool (i.e. Vivado HLS) to generate an FPGA configuration file.

The recent paper describes how to use HLS4ML to generate FPGA firmware and host software for decision trees acceleration.

Taking as an example a multiclass classification problem from high energy physics, we show how a state-of-the-art algorithm could bedeployed on an FPGA with a typical inference time of 12 clock cycles (i.e., 60 ns at a clock frequency of 200 MHz)

🤓🤓🤖🤓🤓
How PCIe 5 and its Smart Friends Will Change Solution Acceleration

Nice article by Scott Schweitzer, Xilinx

Keynotes:

🥦PCI-E Gen5 offers not only throughput bandwidth doubling, but also Compute Express Link (CXL) and a Cache Coherent Interconnect for Accelerators (CCIX) promise to create efficient communication between CPUs and accelerators like SmartNIC or co-processors.

🕸 CCIX configurations include direct attached, switched topologies, and hybrid daisy chain.  it can take memory from different devices, each with varying performance characteristics, pool it together, and map it into a single non-uniform memory access (NUMA) architecture. Then it establishes a Virtual Address space, enabling all of the devices in this pool access to the full range of NUMA memory

🥥 SmartSSDs, also known as computational storage, place a computing device, often an FPGA accelerator, alongside the storage controller within a solid-state drive. This enables the computing device in the SmartSSD to operate on data as it enters and exits the drive, potentially redefining both how data is accessed and stored.

👩‍🔬 SmartNICs, are a special class of accelerators that sit at the nexus between the PCIe bus and the external network. While SmartSSDs place computing close to data, SmartNICs place computing close to the network

👩‍👩‍👧‍👧 SmartNICs and DPUs (data processing units) that leverage PCIe 5 and CXL or CCIX will offer us richly interconnected accelerators that will enable the development of complex and highly performant solutions

🇨🇭🇹🇷 Onur Mutlu, world leading researcher of computer architectures, (SAFARI group, ETH) published short thought provoking keynotes
Intelligent Architectures for Intelligent Machines 
Submitted on 13 Aug 2020
 
Highlights:
 💡Data access is still a major bottleneck.

🤹‍♂Current the processor-centric design paradigm is  a dichotomy between processing and memory/storage: data has to be brought from storage and memory units to compute units, which are far away from the memory/storage.  This processor-memory dichotomy leads to large amounts of data movement across the entire system, degrading performance and expending large
amounts of energy.
 
Modern Architectures are poor at:
⚡️Dealing with data: they are designed to mainly store and move data, as opposed to actually compute on the data
⚡️Taking advantage of vast amounts of data and metadata  available to them during online operation and over time.
⚡️Exploiting different properties of application data. They are designed to treat all data as the same
 
Intelligent architecture
Intelligent architecture should handle (i.e., store, access, and process) data well.
 
Key  principles:
🐙Data-centric: minimizing data movement and maximizing the efficiency with which data is handled,
🦐Datadriven: the architecture should make datadriven, self-optimizing decisions in its components
🐠Data-aware":  the architecture should make datacharacteristics-aware decisions in its components and across the entire system
 
Read the full keynote here
🐋

How to Evaluate Deep Neural Network Processors
Vivienne Sze

Why FOPS/W metric is not enough
Common metric for hardware efficieny measurement is FOPS/W (floating-point operations per second per watt) or TOPS/WTerra FOPS/W. However, TOPS/W alone is not enough. It often goes along with, the peak performance in TOPS, which gives the maximum efficiency since it assumes maximum utilization and thus maximum amortization of overhead. However, this does not tell the complete story because processors typically do not operate at their peak TOPS and their efficiency degrades at lower utilization. Following 6 metrics and it's combination must be considered:

🎯1. The accuracy determines if the system can perform the given task. To evaluate it, several benchmark were proposed, among them MLPerf.

⏱2. The latency and throughput determine whether it can run fast enough and in real time. Throughput is the amount of inferences per second and latency is the time period between the input sample arrival and generation of the result. Batching technique improves throughput, but degrades latency. Thus, achieving low latency and high throughput simultaneously can sometimes be at odds depending on the approach, and both metrics should be reported

🏋🏼3. The energy and power consumption primarily dictate the form factor of the device where the processing can operate. Memory read and write operations are still the main consumers of the power, not an arithmetic. 32b DRAM read takes 640 pH and 32b FP multiply 0.9pJ.

🤑4. The cost, which is primarily dictated by the chip area and external memory BW requirements, determines how much one would pay for the solution. Custom DNN processors have a higher design cost (after amortization) than off-the-shelf CPUs and GPUs. We consider anything beyond this, e.g., the economics of the semiconductor business, including how to price platforms, to be outside the scope of this article. Considering the hardware cost of the design is important from both an industry and a research perspective as it dictates whether a system is financially viable

🐍5. The flexibility determines the range of tasks it can support. To maintain efficiency, the hardware should not rely on certain properties of the DNN models to achieve efficiency, as the properties of DNN models are diverse and evolving rapidly. For instance, a DNN processor that can efficiently support the case where the entire DNN model (i.e., all of the weights) fits on chip may perform extremely poorly when the DNN model grows larger, which is likely, given that the size of DNN models continues to increase over time;

🚡6. The scalability determines whether the same design effort can be amortized for deployment in multiple domains (e.g., in the cloud and at the edge) and if the system can efficiently be scaled with DNN model size.

👨‍👩‍👧‍👦7. Interplay Among Different Metrics
All the metrics must be accounted for to fairly evaluate the design tradeoffs

🐭Case 1. Tiny binarized NN architecture, with very low power consumption, high throughput, but unacceptable accuracy.
🐘Case 2. Complete floating point DNN chip, high throughput and moderate chip power consumption. But it's pure arithmetic chip with MACs and all the data read/write/storage is performed outside chip. So total system power consumption will very high.

🔮🧙
Five trends that will shape the future semiconductor technology landscape
Sri Samavedam, senior vice president CMOS technologies at imec

⤴️Trend 1: Moore’s Law will continue, CMOS transistor density scaling will roughly continue to follow Moore’s Law for the next eight to ten years.

⤵️Trend 2: ... but logic performance improvement at fixed power will slow down
Node-to-node performance improvements at fixed power – referred to as Dennard scaling – have slowed down due to the inability to scale supply voltage. Researchers worldwide are looking for ways to compensate for this slow-down and further improve the chip’s performance

🎎Trend 3: More heterogeneous integration, enabled by 3D technologies
We see more and more examples of systems being built through heterogeneous integration leveraging 2.5D or 3D connectivity, like SoC FPGA, HBM, CPU and GPU on the same interposer.

🗜Trend 4: NAND and DRAM being pushed to their limits
Emerging non-volatile memories on the rise. The emerging non-volatile memory market is expected to grow at >50% compound annual growth rate – mainly driven by the demand for embedded magnetic random access memory (MRAM) and standalone phase change memory (PCM).

⌚️Trend 5: Spectacular rise of the edge AI chip industry
With an expected growth of above 100% in the next five years, edge AI is one of the biggest trends in the chip industry. As opposed to cloud-based AI, inference functions are embedded locally on the Internet of Things (IoT) endpoints that reside at the edge of the network, such as cell phones and smart speakers

Read the full text

🎉🥳DataFest2020 will take place this weekend 🥳🎉

The OpenDataScience community presents DataFest2020 , that will happen the following weekend, 19-20th of September.
This year's festival will be fully online.
Please, check the festival landing page and you will find a list full of fascinating sections.
Each section includes several tracks - videos and reading materials - and interactive chat rooms for discussions and Q&A.

@PDP11ML channel and our good friends will present Domain Specific Hardware section.
We will talk about market trends, technical details, share success stories and highlight SOTA solution.
Hardware section content and time schedule will be presented here

Participation is free of charge, all tracks in our sections are in English and you are very welcome to join us.

See you soon!

Why Nvidia wants ARM, by WSJ

🤓 Huang’s Law, Silicon chips that power artificial intelligence more than double in performance every two years.

📟AI Goes from the cloud to the edge (dishwashers, smartphones, watches, hoovers)

🗜 ARM develops ultra-low-power CPUs and ML cores

💡This movement of AI processing from the cloud to the “edge”—that is, on the devices themselves—explains Nvidia’s desire to buy Arm, says Nexar co-founder and CEO Eran Shir.

🇨🇳The pace of improvement in AI-specific hardware will make possible a range of applications both utopian and dystopian,

📲Uses of mobile AI are multiplying, in phones and smart devices ranging from dishwashers to door locks to lightbulbs, as well as the millions of sensors making their way to cities, factories and industrial facilities. And chip designer Arm Holdings—whose patents Apple, among many tech companies large and small, licenses for its iPhone chips—is at the center of this revolution.

🐜Over the last three to five years, machine-learning networks have been increasing by orders of magnitude in efficiency, says Dennis Laudick, vice president of marketing in Arm’s machine-learning group. “Now it’s more about making things work in a smaller and smaller environment,” he adds.

Source: WSJ

​​Intel: IoT Enchanced processor increase Perfomace, AI, Security

🌶Motivation: By 2023, up to 70% of all enterprises will process data at the edge.

🍗AI-inferencing algorithms can run on up to 96 graphic execution units (INT8) or run on the CPU with vector neural network instructions (VNNI) built in. With Intel® Time Coordinated Computing (Intel® TCC Technology) and time-sensitive networking (TSN) technologies, 11th Gen processors enable real-time computing demands

🏋🏽‍♂️Software Tools: Edge Software Hub’s Edge Insights for Industrial and the Intel® Distribution of OpenVINO™ toolkit

🦀Use cases:
Industrial, Retail, Healthcare, Smart City, Transportation

ESP

The ESP Project page, platform for FPGA and ASIC SoC accelerator design, was significantly updated since the last time we mentioned it

Now it includes many more materials, guides and videos:

📡 How to design accelerator in Vivado HLS and Mentor Graphics Catapult HLS

🔬 Describes HLS4ML Flow. Original HLS4ML papers covered only the core design path, but this guide help you to integrate the core into the computer system

📺 Integrating with Nvidia GPUs sthrough NVDLA

📒 List of the related papers

ESP is run by the System-Level Design (SLD) group at Columbia University, led by Professor Luca P. Carloni.

💥🕯🤓A boom in low cost edge AI chips using the RISC-V technology is coming says Facebook’s chief AI scientist Yann LeCun

🐜The move to RISC-V for running neural networks for edge AI applications is accelerated by the proposed takeover of ARM by Nvidia, says Yann LeCun, chief AI scientist at Facebook speaking at the Innovation Day of French research lab CEA-Leti.
“There is a change in the industry and ARM with Nvidia makes people uneasy but the emergence of RISC-V sees chips with a RISC-V core and an NPU (neural processing unit),” he said.

🍕“These are incredibly cheap, less than $10, with many out of China, and these will become ubiquitous,” he said. “I’m wondering if RISC-V will take over the world there.”

📟“Certainly edge AI is a super important topic,” he said. “In the next two to three years, it’s not going to be exotic technologies, it’s about reducing the power consumption as much as possible, pruning the neural net, optimising the weights, shutting down parts of the system that aren’t used," said LeCun.

🤿 "The target is AR devices with chips in the next two to three years with devices in the five years, and that’s coming,” he said.

Presentation by Philip Harris & Jeff Krupa (MIT)
Heterogeneous Computing at the LHC

TL;DR

🎓 FastML Collaboration is group founded by P.Harris and Nhan Tran to adapt DNN to LHC data flow, but already goes far beyond. HLS4ML tools is part of the project.

💫 Proton collisions (events) occurs at 40MHz in the CMS detector, a new collision each 25ns and there 8Mb of data per collision and it gives 320Tb/s. There's no chance to catch them all for now.

🐾 There are 3 triggering levels, that select only "interesting event" for offline-computing at rate 8Gb/s. ML Models (Decision Trees and DNNs) are used for events classification. It creates huge challenges both for throughput, and latency requirements.

☁️ Described system integrates FPGAs and GPUs accelerators in the cloud through the network, to make it available for researches.

🧩 This huge and largescale work includes may famous institutions, among them Fermilab, MIT, CERN, AWS and Microsoft Brainwave project and can be applied not only to HEP, but also Astrophysics and Gravitational Wave Detection

- YouTube video
- Slides Link (Dropbox)