THESIS DEFENSE Department of Computer Science and Engineering University of South Carolina Author : Adel D. Rajab Advisor : Dr. Chin-Tser Huang Date : Nov 10th 2017 Time : 2:00 pm Place : Meeting room 2265, Innovation Center Abstract The Optical Bust Switching (OBS) network has become one of the most promising switching technologies for building the next-generation Internet backbone infrastructure. However, OBS networks still face a number of security and Quality of Service (QoS) challenges, particularly from Burst Header Packet (BHP) flooding attacks. First, we proposed and developed a new security model that can be embedded into OBS core switch architecture to prevent BHP flooding attacks. The countermeasure security model allows the OBS core switch to classify the ingress nodes based on their behavior and the amount of reserved resources not being utilized. A malicious node causing a BHP flooding attack will be blocked by the developed model until the risk disappears or the malicious node redeems itself. Using our security model, we can effectively and preemptively prevent a BHP flooding attack regardless of the strength of the attacker. In the second part of this dissertation, we investigated the potential use of machine learning (ML) in countering the risk of the BHP flood attack problem. In particular, we proposed and developed a new series of rules, using the decision tree method to prevent the risk of a BHP flooding attack. The proposed classification rule models were evaluated using different metrics to measure the overall performance of this approach. The experiments showed that using rules derived from the decision trees did indeed counter BHP flooding attacks, and enabled the automatic classification of edge nodes at an early stage. In the third part of this dissertation, we performed a comparative study, evaluating a number of ML techniques in classifying edge nodes, to determine the most suitable ML method to prevent this type of attack. The experimental results from a preprocessed dataset related to BHP flooding attacks showed that rule-based classifiers, in particular decision trees (C4.5), Bagging, and RIDOR, consistently derive classifiers that are more predictive, compared to alternate ML algorithms, including AdaBoost, Logistic Regression, Naive Bayes, SVM-SMO and ANN-MultilayerPerceptron. Moreover, the harmonic mean, recall and precision results of the rule-based and tree classifiers were more competitive than those of the remaining ML algorithms. Lastly, the runtime results in ms showed that decision tree classifiers are not only more predictive, but are also more efficient than other algorithms. Thus, our findings show that decision tree identifier is the most appropriate technique for classifying ingress nodes to combat the BHP flooding attack problem.