CSE Graduate Student research poster presentations. 

Below is a tentative agenda for the day. If you plant to attend the luncheon, please RSVP by at this link  by Monday, March 11.

Location: 550 Assembly St, Room 2277, Columbia, SC 29201

10:30 am - 12:00 pm: Morning Poster session, 2nd floor hallways

12:00 pm - 1:30 pm: Lunch in room 2277

1:30 pm - 3:00 pm: Afternoon poster session, 2nd floor hallway

3:00 pm - 4:00 pm: Presentations and poster session winners, room 1400 (1st floor)

DISSERTATION DEFENSE

Author : Liang Zhao
Advisor : Dr. Song Wang
Date : March 11, 2024
Time:  8:30am-10:00am
Place : Teams Link

Abstract

Scene texts refer to arbitrary texts presented in an image captured by a camera in the real world. The tasks of scene text detection and recognition from complex images play a crucial role in computer vision, with potential applications in scene understanding, information retrieval, robotics, autonomous driving, etc. Despite the notable progress made by existing deep-learning methods, achieving accurate text detection and recognition remains challenging for robust real-world applications. The challenges in scene text detection and recognition stem from: 1) diverse text shapes, fonts, colors, styles, layouts, etc.; 2) countless combinations of characters with unstable attributes for complete detection, coupled with background interruptions that obscure character strokes and shapes in text recognition; and 3) the need for effective coordination of multiple sub-tasks in end-to-end learning. The fundamental issue lies in the absence of a particularly discriminative representation for the detection task, which involves locating exact complete words with unfixed attributes, and for the recognition task, which entails differentiating similar characters within words. Our research aims to address these challenges and enhance scene text detection and recognition by improving text discriminative representation. In this study, we focus on two interconnected problems: 1) Scene Text Recognition (STR), which involves recognizing text from scene images, and 2) Scene Text Spotting (STS), which entails simultaneously detecting and recognizing multiple texts in scene images.

Addressing the challenges of Scene Text Recognition (STR), the presence of text variations and complex backgrounds remain significant hurdles due to their impact on text feature representation. Numerous existing methods attempt to mitigate these issues by employing attentional regions, bounding boxes, or polygons. Despite these efforts, the text regions identified by such methods often retain undesirable background interference. In response, we propose a Background-Insensitive Network (BINet) that explicitly incorporates text Semantic Segmentation (SSN) to enhance the text representation and reduce the background interruptions. This approach eliminates the need for extensive pixel-level annotations in the STR training data. To maximize the benefits of semantic cues, we introduce novel segmentation refinement and embed?ding modules that refine text masks and strengthen visual features. Experimental results demonstrate that our proposed method significantly improves text recognition in the presence of complex backgrounds, achieving state-of-the-art performance across multiple public datasets.

In tackling the problem of Scene Text Spotting (STS), we introduce two novel developments. Given that the task involves a multi-task model dedicated to locating and recognizing texts in scenes, the coordination of multiple sub-tasks can exert a significant impact on each other and, subsequently, on the overall performance. Current end-to-end text spotters commonly incorporate independent sequential pipelines to conduct different multi-tasks. However, this unidirectional pipeline leads to information loss and error propagation among sub-tasks. In light of these observations, we present CommuSpotter, designed to enhance multi-task communication by explicitly and concurrently exchanging compatible information throughout the scene text spotting process. Experimental results demonstrate that our improved text representation for both sub-tasks enhances performance across public datasets.

Another prominent limitation in multi-tasks coordination in Scene Text Spotting (STS) lies in the capability of extracting and refining text representation of instances for multiple sub-tasks. Existing methods often utilize features from Convolutional Neural Networks (CNNs) and shrink the text regions in representation to perform sequential tasks. Nevertheless, the effectiveness of these methods is primarily constrained by the contextual biases inherent in the representation of CNN backbones. These biases are challenging to filter out, complicating the identification of randomly appearing texts and introducing confusion in discerning the similar characters within text instances. In response to these challenges, we propose a novel approach named Assembling Text Spotter (ATS) to mitigate the problem. ATS initially decouples image contextual information from text structure information through the separation of dual backbones. The disentanglement of image and text information eliminates the need for filtering out one from the other. Subsequently, they are dynamically and purposely aligned to generate discriminative representations for different sub-tasks. Extensive experiments conducted on existing scene text datasets demonstrate competitive performance results across multiple benchmarks for scene text spotting.

DISSERTATION DEFENSE

Author : Mohammed Essa Fawzy Essa

Advisor : Dr. Ramtin Zand

Date : Feb 29, 2024

Time:  4:00 pm – 6:00 pm

Place : Teams

Link: https://teams.microsoft.com/dl/launcher/launcher.html?url=%2F_%23%2Fl%2…

Abstract

Machine learning (ML) has become ubiquitous, integrating into numerous real-life applications. However, meeting the computational demands of ML systems is challenging, as existing computing platforms are constrained by memory bandwidth, and technology scaling no longer yields substantial improvements in system performance. This work introduces novel hardware architectures to accelerate ML workloads, addressing both compute and memory challenges.

In the compute domain, we explore various approximate computing techniques to assess their efficacy in accelerating ML computations. Subsequently, we propose the Approximate Tensor Processing Unit (APTPU), a hardware accelerator that utilizes approximate processing elements to replace direct quantization of inputs and weights in ML models, thereby enhancing performance, power efficiency, and area utilization. The APTPU achieves significant throughput gains while maintaining comparable accuracies. In the memory domain, we present the In-Memory Analog Computing (IMAC) architecture as an effective solution to the data movement issues faced by von Neumann architectures. IMAC employs memristive devices in crossbars and analog activation functions to efficiently execute matrix-vector multiplication (MVM) and non-linear vector operations in ML workloads. Finally, we integrate IMAC with TPU and APTPU architectures to capitalize on their combined strengths in accelerating MVM and matrix-matrix multiplication operations across various ML workloads. This integration leads to noteworthy performance enhancements and reduced memory requirements. This work provides design baselines and automation tools for architects to seamlessly incorporate the proposed compute and memory solutions into their custom systems.

This talk on the intersection of Physics & AI will give a high level overview of physics based neural networks, applications of physics in NLP, healthcare, data generation, autonomous vehicles, data generation etc. The talk will also have a deep dive on "Incorporating Physics into Data-Driven Computer Vision" which will cover approaches like modifications to datasets, network design, loss function, optimization and regularization schemes.
 

Bio: Parth Patwa is a data scientist at Amazon, working on generative AI. He graduated from UCLA. Previously he was working as a ML scientist at MIT Pathcheck. His publication in NLP and CV have over 1000 citations.

Zoom
 

About
Human society of this century is facing several fundamental challenges including global climate change, energy crisis, and public health crisis such as cancers and COVID-19. Common to their solutions are the discovery of novel materials, molecules, proteins, and drugs. Designing these functional atomic structures is challenging due to the astonishing complexity of the interatomic interactions, sophisticated physical/chemical/geometric constraints and patterns to form stable structures, and how the structures relate to their functions. Like most other engineering design activities, currently, the mainstream paradigm of material design is the rational design approach, which emphasizes a causal understanding of the structure-function relationship and depends on heuristic expert knowledge and explicit design rules. However, the traditional material design paradigm is facing increasing challenges in designing extraordinary functional materials that can effectively meet our needs: it usually leads to sub-optimal solutions in the huge chemical design space due to their limited search capability; it is difficult to handle huge amount of implicit knowledge and constraints, and cannot exploit such rules for efficient design space exploration; it needs too many explicit design rules; it is difficult to design highly constrained structures such as the periodic inorganic crystals.

In this talk, I will introduce the transformative shift from rational materials design to the data-driven deep generative material design paradigm, in which known materials data are fed to the deep generative models to learn explicit and implicit knowledge of atomic structures and then exploit them for efficient structure generation. This is inspired by the deep learning based Artificial Intelligence Generated Content (AIGC) technologies that have been accelerating in generating authentic images, videos, texts, music, and human voices. Our work shows that designing images and texts shares many characteristics with the task of designing proteins, molecules, and materials, in which building blocks of different levels are assembled together to form specific stable or meaningful structures that satisfy diverse grammatical, physical, chemical or geometric constraints. While Nature has used the physical apparatus of DNA as the information carrier of synthesis rules for protein synthesis and biochemistry through evolution, deep neural networks can also be exploited similarly to achieve Nature's way of material design by learning the designing rules from known materials or from computational simulations. Just as a female frog can give birth to a frog without knowing how a frog is grown from a zygote through a developmental process, we show that our deep generative materials design works in a similar design-without-understanding process.

-----------------------
Bio: Dr. Jianjun Hu is professor of computer science at the University of South Carolina. He directs the Machine Learning and Evolution Laboratory (http://mleg.cse.sc.edu). Dr. Hu received his Ph.D. of computer science in 2004 from Michigan State University in the areas of machine learning and evolutionary computation and then conducted postdoc studies in bioinformatics at Purdue University and University of Southern California from 2004 to 2007. His current research interests include AI for science, machine learning, deep learning, evolutionary algorithms, and their application in material informatics, bioinformatics, health informatics, and automated design synthesis with a total of more than 200 papers. Dr. Hu is the winner of the National Science Foundation Career Award. His research has been funded by NSF, DOE, NIH. He can be reached at jianjunh@cse.sc.edu

Details: https://www.linkedin.com/events/7162579233444192256/about/

Abstract - The emergence of very large foundation models such as GPT, Stable Diffusion, DALL-E and Midjourney has dramatically altered the trajectory of progress in AI and its applications. The enthusiasm for AI has expanded beyond the realm of AI researchers and has reached the general population; indeed it asserts we are living in an exciting time of scientific proliferation. The present-day capability of AI exhibits promising forms of intelligence with wider usability, but it also possesses unexpected limitations and is susceptible to potential misuse. AI has reached a level where discerning AI-generated content, be it in text, images or videos, has become notably challenging, which we term as "eloquence” characteristics. Conversely, the worrisome rise of hallucinations of AI models raises credibility issues, which we refer to as "adversity" characteristics. Recently, the governments of both the United States and the European Union have put forth their preliminary proposals concerning the regulatory framework for the safety of systems powered by AI. AI systems that adhere to these regulations in the future will be referred to by a recently coined term “Constitutional AI”. The primary objective of regulatory frameworks is to establish safeguards against the misuse of AI systems. In the event of any such misuse, these frameworks aim to impose penalties on the individuals, groups, and/or organizations responsible for such misconduct. "CIVILIZING AI" embodies a nuanced equilibrium between the machine's eloquence and its inclination towards adversarial behavior, with the goal of enforcing constitutional principles. Have a look if you wish to pre-read: https://analyticsindiamag.com/this-usc-professor-from-kolkata-is-on-a-j…

Brief Bio
----------------
Currently, Dr. Das holds the position of Research Associate Professor at The Artificial Intelligence Institute (AIISC), University of South Carolina, USA. Previously, he served as a founding researcher and played a pivotal role in establishing Wipro Labs in Bangalore, India, from its inception. He maintains an association with Wipro Labs in the capacity of an Advisory Scientist. He also holds an adjunct faculty position at IIT Patna. In the past, he also worked for Samsung Research, India. Managing/guiding/collaborating 100+ people across all the aforementioned organizations.

He has held two academic postdoctoral positions, one in Europe and the other in the USA. During his tenure in the USA, he worked at the University of Texas-Austin, while his European postdoc was awarded by the European Research Consortium for Informatics and Mathematics (ERCIM) and hosted at NTNU, Norway. He earned his Ph.D. in Engineering from Jadavpur University, India, with active collaboration with the Tokyo Institute of Technology, Japan. Dr. Das boasts nearly two decades of research experience in NLP, with a substantial publication record of over 120+ research papers spanning a wide array of topics and holds an h-index of 36

Details at https://www.linkedin.com/events/aiiscseminar-civilizingai-exami71606855…

DISSERTATION DEFENSE

Department of Computer Science and Engineering

University of South Carolina

Author : Xiaoguang Li

Advisor : Dr. Song Wang

Date : Feb 7, 2024

Time:  11 am – 12: 30 pm

Place : Teams

Link: https://teams.microsoft.com/l/meetup-join/19%3ameeting_MWJkNGI5OTYtNzk5…

Abstract

Image inpainting is an important challenge in the computer vision field. The primary goal of image inpainting is to fill in the missing parts of an image. This technique has many real-life uses including fixing old photographs and restoring ancient artworks, e.g., the degraded Dunhuang frescoes. Moreover, image inpainting is also helpful in image editing. It has the capability to eliminate unwanted objects from images while maintaining a natural and realistic appearance, e.g., removing watermarks and subtitles. Disregarding the fact that image inpainting expects the restored result to be identical to the original clean one, existing deep generative inpainting methods often treat image inpainting as a pure generative task and emphasize the naturalness or realism of the generated result. Although achieving significant progress, the existing deep generative inpainting methods are far from real-world applications due to the low generalization across different scenes. As a result, the generated images usually contain artifacts or the filled pixels differ greatly from the ground truth. To address this challenge, in this research, we propose two approaches that utilize the predictive filtering technique to improve the image inpainting performance. Furthermore, we harness the predictive filtering technique and inpainting pretraining to tackle the challenge of shadow removal effectively. Specifically, for the first approach, we formulate image inpainting as a mix of two problems, i.e., predictive filtering and deep generation. Predictive filtering is good at preserving local structures and removing artifacts but falls short to complete the large missing regions. The deep generative network can fill the numerous missing pixels based on the understanding of the whole scene but hardly restores the details identical to the original ones. To make use of their respective advantages, we propose the joint predictive filtering and generative network (JPGNet). We validate our first approach on three public datasets, i.e., Dunhuang, Places2, and CelebA, and demonstrate that our method can enhance three state-of-the-art generative methods (i.e., StructFlow, EdgeConnect, and RFRNet) significantly with slightly extra time costs. For the second approach, inspired by this inherent advantage of image-level predictive filtering, we explore the possibility of addressing image inpainting as a filtering task. We first study the advantages and challenges of image-level predictive filtering for inpainting: the method can preserve local structures and avoid artifacts but fails to fill large missing areas. Then, we propose the semantic filtering by conducting filtering on the deep feature level, which fills the missing semantic information but fails to recover the details. To address the issues while adopting the respective advantages, we propose a novel filtering technique, i.e., Multi-level Interactive Siamese Filtering (MISF). The extensive experiments demonstrate that our method surpasses state-of-the-art baselines across four metrics, i.e., L1, PSNR, SSIM, and LPIPS. In the end, we employ the predictive filtering technique and inpainting pretraining to address the shadow removal problem. Specifically, we find that pretraining shadow removal networks on the image inpainting dataset can reduce the shadow remnants significantly: a naive encoder-decoder network gets competitive restoration quality w.r.t. the state-of-the-art methods via only 10% shadow & shadow-free image pairs. After analyzing networks with/without inpainting pretraining via the information stored in the weight (IIW), we find that inpainting pretraining improves restoration quality in non-shadow regions and enhances the generalization ability of networks significantly. Additionally, shadow removal fine-tuning enables networks to fill in the details of shadow regions. Inspired by these observations we formulate shadow removal as an adaptive fusion task and propose the Inpaint4Shadow:Leveraging Inpainting for Single-Image Shadow Removal. The extensive experiments show that our method empowered with predictive filtering and inpainting outperforms all state-of-the-art shadow removal methods.

Date: 2 February 2024 (Friday)

Time: 2:20pm to 3:20pm

Location: SWGN 2A27 (preferred) or online on Zoom

Historically, Artificial Intelligence has taken a symbolic route for representing and reasoning about objects at a higher-level or a statistical route for learning complex models from large data. To achieve true AI in complex domains such as healthcare, it is necessary to make these different paths meet and enable seamless human interaction. First, I will introduce learning from rich, structured, complex and noisy data. One of the key attractive properties of the learned models is that they use a rich representation for modeling the domain that potentially allows for seam-less human interaction. I will present the recent progress that allows for more reasonable human interaction where the human input is taken as “advice” and the learning algorithm combines this advice with data. I will present these algorithms in the context of several healthcare problems -- learning from electronic health records, clinical studies, and surveys -- and demonstrate the value of involving experts during learning.

Speaker Bio:

Sriraam Natarajan is a Professor and the Director for Center for ML at the Department of Computer Science at University of Texas Dallas, a hessian.AI fellow at TU Darmstadt and a RBDSCAII Distinguished Faculty Fellow at IIT Madras. His research interests lie in the field of Artificial Intelligence, with emphasis on Machine Learning, Statistical Relational Learning and AI, Reinforcement Learning, Graphical Models and Biomedical Applications. He is a AAAI senior member and has received the Young Investigator award from US Army Research Office, Amazon Faculty Research Award, Intel Faculty Award, XEROX Faculty Award, Verisk Faculty Award, ECSS Graduate teaching award from UTD and the IU trustees Teaching Award from Indiana University. He is the program chair of AAAI 2024, the general chair of CoDS-COMAD 2024, AI and society track chair of AAAI 2023 and 2022, senior member track chair of AAAI 2023, demo chair of IJCAI 2022, program co-chair of SDM 2020 and ACM CoDS-COMAD 2020 conferences. He was the specialty chief editor of Frontiers in ML and AI journal, and is an associate editor of JAIR, DAMI and Big Data journals.

SUMMARY: Imagine living in an intuitively interactive space without the need to understand its grammar.
One does not need to interact in a specific way or use voice commands, like Amazon echo, or always wear
something, like a smart band to control the space and the devices therein. Sharing this intelligent space with
others does not also degrade the individual user experience. Interestingly, this vision of seamless smart
spaces is not novel and quite dated; however, we have yet to occupy this kind of space regularly. For this
vision to become an everyday reality, we argue that there is a need for multi-user continuous activity tracking
through passive sensing mechanisms. In this talk, I will discuss how we have developed a pervasive sensing
system for continuous multi-user room-scale tracking of human activities using a single COTS mmWave
radar. We will further highlight how such a sensing system can be utilized for continuous monitoring of
driving activities on the road, to prevent road accidents from driver distraction and promote safe driving. We
will conclude the talk with some potential research challenges where mmWave sensing can help.

BIO: Dr. Sandip Chakraborty is an Associate Professor in the Department of Computer Science and Engineering at
the Indian Institute of Technology (IIT) Kharagpur, and is leading the Ubiquitous Networked Systems Lab (UbiNet). He
obtained his Bachelor’s degree from the Jadavpur University, Kolkata in 2009 and M. Tech and Ph.D. from IIT
Guwahati, in 2011 and 2014, respectively. The primary research interests of Dr. Chakraborty are Pervasive and
Ubiquitous Computing, Computer Systems, and Human-Computer Interactions. He received various awards including
the Google Award for Inclusion Research in Societal Computing (2023), the Indian National Academy of Engineering
(INAE) Young Engineers’ Award 2019, and the Honorable Mention Award in ACM SIGCHI EICS 2020. Dr. Chakraborty
is one of the founding chairs of ACM IMOBILE, the ACM SIGMOBILE Chapter in India. He is working as an Area
Editor of Elsevier Ad Hoc Networks journal and Elsevier Pervasive and Mobile Computing journal. Further details
about his works and publications can be found at https://cse.iitkgp.ac.in/~sandipc/.

Abstract 
Agriculture, vital for global sustenance, is increasingly challenged by a knowledge gap arising from language and literacy barriers, especially in regions like India. This seminar explores the transformative role of Generative AI in overcoming these obstacles, providing a comprehensive overview of its technical and practical applications. The use of Large Language Models (LLMs), enhanced with vernacular voice-based conversational interfaces, has revolutionized the delivery of farming advice in native languages, thereby bridging critical information gaps. 
 

The seminar also emphasizes the significance of robust data curation in developing such AI models. In countries like India, a substantial amount of agricultural data is produced by various institutes in regional languages to assist farmers. However, this data often remains inaccessible for various reasons. Effective data collection, validation, and integration are crucial to ensure the accuracy and contextual relevance of the information provided. 
 

Dhenu 1.0, a pioneering domain-specific model, demonstrates how Generative AI can be customized for bilingual communication in English and Hindi, providing real-time, personalized farming advice. This model's cost-effectiveness and accuracy are particularly beneficial where diverse languages and varying literacy levels among farmers can hinder access to essential agricultural information. 
 

This seminar aims to highlight how innovative solutions like KissanAI and Dhenu 1.0 can transform agricultural practices, making expert knowledge accessible and understandable to all farmers. This approach not only promotes more informed and efficient farming decisions but also showcases the potential of AI in driving sustainable and inclusive agricultural development.

About the speaker: Bio:

Pratik Desai is a seasoned entrepreneur and computer scientist passionate about leveraging knowledge extraction through AI and Semantic Web technologies. Before venturing into AI applications for Agriculture, Dr. Desai honed his skills by co-founding three innovative startups in Silicon Valley. Grounded in his family's agricultural heritage, Dr. Desai established KissanAI. His vision for KissanAI has been to contribute meaningfully to his roots by democratizing the use of AI in agriculture. KissanAI stands out as a vernacular AI CoPilot platform meticulously designed to offer voice-based assistance tailored to the agricultural sector, particularly addressing the challenges faced by farmers in developing nations by overcoming literacy and language barriers. Dr. Desai’s pioneering work in incorporating generative AI in the Agriculture domain has led to the development of Dhenu, the world's first agricultural vertical-specific Large Language Model (LLM). More: https://www.linkedin.com/in/pratikkumardesai/

For those who cannot join in person: Zoom
https://us06web.zoom.us/j/8440139296?pwd=b09lRCtJR0FCTWcyeGtCVVlUMDNKQT…;