Abstract: Several real-world applications of machine learning (ML) systems such as robotics, autonomous cars, assistive technologies, smart manufacturing, and many other Internet-of-Things (IoT) applications require real-time inference with low energy consumption. The surge in demand for specialized hardware for AI applications has resulted in a rapidly expanding industry for edge AI accelerators. Anticipating this trend, several companies have developed their own specialized accelerators such as the NVIDIA Jetson Nano, Intel NCS2, and Google TPU. While many conventional neural networks can be readily deployed on many of these platforms, the support for deploying more advanced and larger models such as transformers on them has yet to be researched and developed. In this talk, we discuss two of our recent projects in which we utilize optimization mechanism such neural architecture search (NAS) and system-level innovations such as modifying the computational graphs, partitioning, and refactoring the unsupported operations to efficiently deploy ML models on edge accelerators for computer vision and natural language processing tasks.

Bio: Dr. Ramtin Zand is an assistant professor of the Computer Science and Engineering and the principal investigator of the Intelligent Circuits, Architectures, and Systems (iCAS) Lab at the University of South Carolina. The iCAS lab has close collaborations with and is supported by several multinational companies including Intel, AMD, and Juniper Networks, as well as federal agencies such as National Science Foundation (NSF). Dr. Zand has authored more than 50 journal and conference articles and two book chapters and received recognition from ACM/IEEE including the best paper runner-up of ACM GLSVLSI’18, the best poster of ACM GLSVLSI’19, and best paper of IEEE ISVLSI’21, as well as featured paper in IEEE Transactions on Emerging Topics in Computing. He has received the NSF CAREER award in 2024. His research focus is on neuromorphic computing, edge computing, processing-in-memory, and AI/ML hardware acceleration.

Online at: https://us06web.zoom.us/j/8440139296?pwd=b09lRCtJR0FCTWcyeGtCVVlUMDNKQT09&omn=85050122519

Register Here

Get MATLAB here

More courses here

DISSERTATION DEFENSE

Department of Computer Science and Engineering

University of South Carolina

Author : Madushan Abeysinghe
Advisor : Dr. Jason Bakos
Date : March 18, 2024
Time:  10 am – 11: 00 pm

Place : Teams

Meeting ID: 234 681 711 471

Passcode: wiatfJ

Abstract

Digital signal processors (DSP), which are characterized by statically-scheduled Very Long Instruction Word architectures and software-defined scratchpad memory, are currently the go-to processor type for low-power embedded vision systems, as exemplified by the DSP processors integrated into systems-on-chips from NVIDIA, Samsung, Qualcomm, Apple, and Texas Instruments. DSPs achieve performance by statically scheduling workloads, both in terms of data movement and instructions. We developed a method for scheduling buffer transactions across a data flow graph using data-driven performance models, yielding a 25% average reduction in execution time and a reduction of up to 85% DRAM utilization for randomly-generated data flow graphs. We also developed a heuristic instruction scheduler that serves as a performance model to guide the selection of loops from a target data flow graph to be fused. By strategically selecting loops to fuse, performance gains can be achieved by eliminating unnecessary transactions with memory and increasing functional unit utilization. This approach has helped us achieve up to 1.9x speedup on average for sufficiently large data flow graphs used in image processing.

CSE Graduate Student research poster presentations. 

Below is a tentative agenda for the day. If you plant to attend the luncheon, please RSVP by at this link  by Monday, March 11.

Location: 550 Assembly St, Room 2277, Columbia, SC 29201

10:30 am - 12:00 pm: Morning Poster session, 2nd floor hallways

12:00 pm - 1:30 pm: Lunch in room 2277

1:30 pm - 3:00 pm: Afternoon poster session, 2nd floor hallway

3:00 pm - 4:00 pm: Presentations and poster session winners, room 1400 (1st floor)

DISSERTATION DEFENSE

Author : Liang Zhao
Advisor : Dr. Song Wang
Date : March 11, 2024
Time:  8:30am-10:00am
Place : Teams Link

Abstract

Scene texts refer to arbitrary texts presented in an image captured by a camera in the real world. The tasks of scene text detection and recognition from complex images play a crucial role in computer vision, with potential applications in scene understanding, information retrieval, robotics, autonomous driving, etc. Despite the notable progress made by existing deep-learning methods, achieving accurate text detection and recognition remains challenging for robust real-world applications. The challenges in scene text detection and recognition stem from: 1) diverse text shapes, fonts, colors, styles, layouts, etc.; 2) countless combinations of characters with unstable attributes for complete detection, coupled with background interruptions that obscure character strokes and shapes in text recognition; and 3) the need for effective coordination of multiple sub-tasks in end-to-end learning. The fundamental issue lies in the absence of a particularly discriminative representation for the detection task, which involves locating exact complete words with unfixed attributes, and for the recognition task, which entails differentiating similar characters within words. Our research aims to address these challenges and enhance scene text detection and recognition by improving text discriminative representation. In this study, we focus on two interconnected problems: 1) Scene Text Recognition (STR), which involves recognizing text from scene images, and 2) Scene Text Spotting (STS), which entails simultaneously detecting and recognizing multiple texts in scene images.

Addressing the challenges of Scene Text Recognition (STR), the presence of text variations and complex backgrounds remain significant hurdles due to their impact on text feature representation. Numerous existing methods attempt to mitigate these issues by employing attentional regions, bounding boxes, or polygons. Despite these efforts, the text regions identified by such methods often retain undesirable background interference. In response, we propose a Background-Insensitive Network (BINet) that explicitly incorporates text Semantic Segmentation (SSN) to enhance the text representation and reduce the background interruptions. This approach eliminates the need for extensive pixel-level annotations in the STR training data. To maximize the benefits of semantic cues, we introduce novel segmentation refinement and embed?ding modules that refine text masks and strengthen visual features. Experimental results demonstrate that our proposed method significantly improves text recognition in the presence of complex backgrounds, achieving state-of-the-art performance across multiple public datasets.

In tackling the problem of Scene Text Spotting (STS), we introduce two novel developments. Given that the task involves a multi-task model dedicated to locating and recognizing texts in scenes, the coordination of multiple sub-tasks can exert a significant impact on each other and, subsequently, on the overall performance. Current end-to-end text spotters commonly incorporate independent sequential pipelines to conduct different multi-tasks. However, this unidirectional pipeline leads to information loss and error propagation among sub-tasks. In light of these observations, we present CommuSpotter, designed to enhance multi-task communication by explicitly and concurrently exchanging compatible information throughout the scene text spotting process. Experimental results demonstrate that our improved text representation for both sub-tasks enhances performance across public datasets.

Another prominent limitation in multi-tasks coordination in Scene Text Spotting (STS) lies in the capability of extracting and refining text representation of instances for multiple sub-tasks. Existing methods often utilize features from Convolutional Neural Networks (CNNs) and shrink the text regions in representation to perform sequential tasks. Nevertheless, the effectiveness of these methods is primarily constrained by the contextual biases inherent in the representation of CNN backbones. These biases are challenging to filter out, complicating the identification of randomly appearing texts and introducing confusion in discerning the similar characters within text instances. In response to these challenges, we propose a novel approach named Assembling Text Spotter (ATS) to mitigate the problem. ATS initially decouples image contextual information from text structure information through the separation of dual backbones. The disentanglement of image and text information eliminates the need for filtering out one from the other. Subsequently, they are dynamically and purposely aligned to generate discriminative representations for different sub-tasks. Extensive experiments conducted on existing scene text datasets demonstrate competitive performance results across multiple benchmarks for scene text spotting.

DISSERTATION DEFENSE

Author : Mohammed Essa Fawzy Essa

Advisor : Dr. Ramtin Zand

Date : Feb 29, 2024

Time:  4:00 pm – 6:00 pm

Place : Teams

Link: https://teams.microsoft.com/dl/launcher/launcher.html?url=%2F_%23%2Fl%2…

Abstract

Machine learning (ML) has become ubiquitous, integrating into numerous real-life applications. However, meeting the computational demands of ML systems is challenging, as existing computing platforms are constrained by memory bandwidth, and technology scaling no longer yields substantial improvements in system performance. This work introduces novel hardware architectures to accelerate ML workloads, addressing both compute and memory challenges.

In the compute domain, we explore various approximate computing techniques to assess their efficacy in accelerating ML computations. Subsequently, we propose the Approximate Tensor Processing Unit (APTPU), a hardware accelerator that utilizes approximate processing elements to replace direct quantization of inputs and weights in ML models, thereby enhancing performance, power efficiency, and area utilization. The APTPU achieves significant throughput gains while maintaining comparable accuracies. In the memory domain, we present the In-Memory Analog Computing (IMAC) architecture as an effective solution to the data movement issues faced by von Neumann architectures. IMAC employs memristive devices in crossbars and analog activation functions to efficiently execute matrix-vector multiplication (MVM) and non-linear vector operations in ML workloads. Finally, we integrate IMAC with TPU and APTPU architectures to capitalize on their combined strengths in accelerating MVM and matrix-matrix multiplication operations across various ML workloads. This integration leads to noteworthy performance enhancements and reduced memory requirements. This work provides design baselines and automation tools for architects to seamlessly incorporate the proposed compute and memory solutions into their custom systems.

This talk on the intersection of Physics & AI will give a high level overview of physics based neural networks, applications of physics in NLP, healthcare, data generation, autonomous vehicles, data generation etc. The talk will also have a deep dive on "Incorporating Physics into Data-Driven Computer Vision" which will cover approaches like modifications to datasets, network design, loss function, optimization and regularization schemes.
 

Bio: Parth Patwa is a data scientist at Amazon, working on generative AI. He graduated from UCLA. Previously he was working as a ML scientist at MIT Pathcheck. His publication in NLP and CV have over 1000 citations.

Zoom
 

About
Human society of this century is facing several fundamental challenges including global climate change, energy crisis, and public health crisis such as cancers and COVID-19. Common to their solutions are the discovery of novel materials, molecules, proteins, and drugs. Designing these functional atomic structures is challenging due to the astonishing complexity of the interatomic interactions, sophisticated physical/chemical/geometric constraints and patterns to form stable structures, and how the structures relate to their functions. Like most other engineering design activities, currently, the mainstream paradigm of material design is the rational design approach, which emphasizes a causal understanding of the structure-function relationship and depends on heuristic expert knowledge and explicit design rules. However, the traditional material design paradigm is facing increasing challenges in designing extraordinary functional materials that can effectively meet our needs: it usually leads to sub-optimal solutions in the huge chemical design space due to their limited search capability; it is difficult to handle huge amount of implicit knowledge and constraints, and cannot exploit such rules for efficient design space exploration; it needs too many explicit design rules; it is difficult to design highly constrained structures such as the periodic inorganic crystals.

In this talk, I will introduce the transformative shift from rational materials design to the data-driven deep generative material design paradigm, in which known materials data are fed to the deep generative models to learn explicit and implicit knowledge of atomic structures and then exploit them for efficient structure generation. This is inspired by the deep learning based Artificial Intelligence Generated Content (AIGC) technologies that have been accelerating in generating authentic images, videos, texts, music, and human voices. Our work shows that designing images and texts shares many characteristics with the task of designing proteins, molecules, and materials, in which building blocks of different levels are assembled together to form specific stable or meaningful structures that satisfy diverse grammatical, physical, chemical or geometric constraints. While Nature has used the physical apparatus of DNA as the information carrier of synthesis rules for protein synthesis and biochemistry through evolution, deep neural networks can also be exploited similarly to achieve Nature's way of material design by learning the designing rules from known materials or from computational simulations. Just as a female frog can give birth to a frog without knowing how a frog is grown from a zygote through a developmental process, we show that our deep generative materials design works in a similar design-without-understanding process.

-----------------------
Bio: Dr. Jianjun Hu is professor of computer science at the University of South Carolina. He directs the Machine Learning and Evolution Laboratory (http://mleg.cse.sc.edu). Dr. Hu received his Ph.D. of computer science in 2004 from Michigan State University in the areas of machine learning and evolutionary computation and then conducted postdoc studies in bioinformatics at Purdue University and University of Southern California from 2004 to 2007. His current research interests include AI for science, machine learning, deep learning, evolutionary algorithms, and their application in material informatics, bioinformatics, health informatics, and automated design synthesis with a total of more than 200 papers. Dr. Hu is the winner of the National Science Foundation Career Award. His research has been funded by NSF, DOE, NIH. He can be reached at jianjunh@cse.sc.edu

Details: https://www.linkedin.com/events/7162579233444192256/about/

Abstract - The emergence of very large foundation models such as GPT, Stable Diffusion, DALL-E and Midjourney has dramatically altered the trajectory of progress in AI and its applications. The enthusiasm for AI has expanded beyond the realm of AI researchers and has reached the general population; indeed it asserts we are living in an exciting time of scientific proliferation. The present-day capability of AI exhibits promising forms of intelligence with wider usability, but it also possesses unexpected limitations and is susceptible to potential misuse. AI has reached a level where discerning AI-generated content, be it in text, images or videos, has become notably challenging, which we term as "eloquence” characteristics. Conversely, the worrisome rise of hallucinations of AI models raises credibility issues, which we refer to as "adversity" characteristics. Recently, the governments of both the United States and the European Union have put forth their preliminary proposals concerning the regulatory framework for the safety of systems powered by AI. AI systems that adhere to these regulations in the future will be referred to by a recently coined term “Constitutional AI”. The primary objective of regulatory frameworks is to establish safeguards against the misuse of AI systems. In the event of any such misuse, these frameworks aim to impose penalties on the individuals, groups, and/or organizations responsible for such misconduct. "CIVILIZING AI" embodies a nuanced equilibrium between the machine's eloquence and its inclination towards adversarial behavior, with the goal of enforcing constitutional principles. Have a look if you wish to pre-read: https://analyticsindiamag.com/this-usc-professor-from-kolkata-is-on-a-j…

Brief Bio
----------------
Currently, Dr. Das holds the position of Research Associate Professor at The Artificial Intelligence Institute (AIISC), University of South Carolina, USA. Previously, he served as a founding researcher and played a pivotal role in establishing Wipro Labs in Bangalore, India, from its inception. He maintains an association with Wipro Labs in the capacity of an Advisory Scientist. He also holds an adjunct faculty position at IIT Patna. In the past, he also worked for Samsung Research, India. Managing/guiding/collaborating 100+ people across all the aforementioned organizations.

He has held two academic postdoctoral positions, one in Europe and the other in the USA. During his tenure in the USA, he worked at the University of Texas-Austin, while his European postdoc was awarded by the European Research Consortium for Informatics and Mathematics (ERCIM) and hosted at NTNU, Norway. He earned his Ph.D. in Engineering from Jadavpur University, India, with active collaboration with the Tokyo Institute of Technology, Japan. Dr. Das boasts nearly two decades of research experience in NLP, with a substantial publication record of over 120+ research papers spanning a wide array of topics and holds an h-index of 36

Details at https://www.linkedin.com/events/aiiscseminar-civilizingai-exami71606855…

DISSERTATION DEFENSE

Department of Computer Science and Engineering

University of South Carolina

Author : Xiaoguang Li

Advisor : Dr. Song Wang

Date : Feb 7, 2024

Time:  11 am – 12: 30 pm

Place : Teams

Link: https://teams.microsoft.com/l/meetup-join/19%3ameeting_MWJkNGI5OTYtNzk5…

Abstract

Image inpainting is an important challenge in the computer vision field. The primary goal of image inpainting is to fill in the missing parts of an image. This technique has many real-life uses including fixing old photographs and restoring ancient artworks, e.g., the degraded Dunhuang frescoes. Moreover, image inpainting is also helpful in image editing. It has the capability to eliminate unwanted objects from images while maintaining a natural and realistic appearance, e.g., removing watermarks and subtitles. Disregarding the fact that image inpainting expects the restored result to be identical to the original clean one, existing deep generative inpainting methods often treat image inpainting as a pure generative task and emphasize the naturalness or realism of the generated result. Although achieving significant progress, the existing deep generative inpainting methods are far from real-world applications due to the low generalization across different scenes. As a result, the generated images usually contain artifacts or the filled pixels differ greatly from the ground truth. To address this challenge, in this research, we propose two approaches that utilize the predictive filtering technique to improve the image inpainting performance. Furthermore, we harness the predictive filtering technique and inpainting pretraining to tackle the challenge of shadow removal effectively. Specifically, for the first approach, we formulate image inpainting as a mix of two problems, i.e., predictive filtering and deep generation. Predictive filtering is good at preserving local structures and removing artifacts but falls short to complete the large missing regions. The deep generative network can fill the numerous missing pixels based on the understanding of the whole scene but hardly restores the details identical to the original ones. To make use of their respective advantages, we propose the joint predictive filtering and generative network (JPGNet). We validate our first approach on three public datasets, i.e., Dunhuang, Places2, and CelebA, and demonstrate that our method can enhance three state-of-the-art generative methods (i.e., StructFlow, EdgeConnect, and RFRNet) significantly with slightly extra time costs. For the second approach, inspired by this inherent advantage of image-level predictive filtering, we explore the possibility of addressing image inpainting as a filtering task. We first study the advantages and challenges of image-level predictive filtering for inpainting: the method can preserve local structures and avoid artifacts but fails to fill large missing areas. Then, we propose the semantic filtering by conducting filtering on the deep feature level, which fills the missing semantic information but fails to recover the details. To address the issues while adopting the respective advantages, we propose a novel filtering technique, i.e., Multi-level Interactive Siamese Filtering (MISF). The extensive experiments demonstrate that our method surpasses state-of-the-art baselines across four metrics, i.e., L1, PSNR, SSIM, and LPIPS. In the end, we employ the predictive filtering technique and inpainting pretraining to address the shadow removal problem. Specifically, we find that pretraining shadow removal networks on the image inpainting dataset can reduce the shadow remnants significantly: a naive encoder-decoder network gets competitive restoration quality w.r.t. the state-of-the-art methods via only 10% shadow & shadow-free image pairs. After analyzing networks with/without inpainting pretraining via the information stored in the weight (IIW), we find that inpainting pretraining improves restoration quality in non-shadow regions and enhances the generalization ability of networks significantly. Additionally, shadow removal fine-tuning enables networks to fill in the details of shadow regions. Inspired by these observations we formulate shadow removal as an adaptive fusion task and propose the Inpaint4Shadow:Leveraging Inpainting for Single-Image Shadow Removal. The extensive experiments show that our method empowered with predictive filtering and inpainting outperforms all state-of-the-art shadow removal methods.