reveal.js

IEEE Bash Scripting

### Compiled vs Interpreted languagess
            | Compiler | Interpreter |
            | -------- | ----------- |
            | "Compiles" down a program into binary (machine code) |  Does not save machine langauge |
            | Must first compile a program into machine code before running | Interprets each line and executes each as it reads them |
            | Tries to make use of the cpu it compiles a program for | Usually less finetuned for a CPU |
            | Usually favored in production | Usually favored in development and programming environments |

Compiled vs Interpreted Example

program.c program

#include <stdio.h>
int main() {
  int a = 36;
  int b = 6;
  printf("The answer to life, the universe,\
  and everything: %d\n", a+b);
  return 0;
}

program.py

a = 36;
b = 6;
print("The answer to life, the universe,\
 and everything: {}".format(a+b))

Compiled vs Interpreted Example

program.c program

#include <stdio.h>
int main() {
  int a = 36;
  int b = 6;
  printf("The answer to life, the universe,\
  and everything: %d\n", a+b);
  return 0;
}

$ gcc program.c -o program

program.py

a = 36;
b = 6;
print("The answer to life, the universe,\
 and everything: {}".format(a+b))

Compiled vs Interpreted Example

program.c program

#include <stdio.h>
int main() {
  int a = 36;
  int b = 6;
  printf("The answer to life, the universe,\
  and everything: %d\n", a+b);
  return 0;
}

$ gcc program.c -o program
$

program.py

a = 36;
b = 6;
print("The answer to life, the universe,\
 and everything: {}".format(a+b))

Compiled vs Interpreted Example

program.c program

#include <stdio.h>
int main() {
  int a = 36;
  int b = 6;
  printf("The answer to life, the universe,\
  and everything: %d\n", a+b);
  return 0;
}

$ gcc program.c -o program
$ ./program

program.py

a = 36;
b = 6;
print("The answer to life, the universe,\
 and everything: {}".format(a+b))

Compiled vs Interpreted Example

program.c program

#include <stdio.h>
int main() {
  int a = 36;
  int b = 6;
  printf("The answer to life, the universe,\
  and everything: %d\n", a+b);
  return 0;
}

$ gcc program.c -o program
$ ./program
The answer to life, the universe, and everything is 42
$

program.py

a = 36;
b = 6;
print("The answer to life, the universe,\
 and everything: {}".format(a+b))

Compiled vs Interpreted Example

program.c program

#include <stdio.h>
int main() {
  int a = 36;
  int b = 6;
  printf("The answer to life, the universe,\
  and everything: %d\n", a+b);
  return 0;
}

$ gcc program.c -o program
$ ./program
The answer to life, the universe, and everything is 42
$

program.py

a = 36;
b = 6;
print("The answer to life, the universe,\
 and everything: {}".format(a+b))

$ python program.py

Compiled vs Interpreted Example

program.c program

#include <stdio.h>
int main() {
  int a = 36;
  int b = 6;
  printf("The answer to life, the universe,\
  and everything: %d\n", a+b);
  return 0;
}

$ gcc program.c -o program
$ ./program
The answer to life, the universe, and everything is 42
$

program.py

a = 36;
b = 6;
print("The answer to life, the universe,\
 and everything: {}".format(a+b))

$ python program.py

Compiled vs Interpreted Example

program.c program

#include <stdio.h>
int main() {
  int a = 36;
  int b = 6;
  printf("The answer to life, the universe,\
  and everything: %d\n", a+b);
  return 0;
}

$ gcc program.c -o program
$ ./program
The answer to life, the universe, and everything is 42
$

program.py

a = 36;
b = 6;
print("The answer to life, the universe,\
 and everything: {}".format(a+b))

$ python program.py
The answer to life, the universe, and everything: 42

Compiled vs Interpreted Example

program.c program

#include <stdio.h>
int main() {
  int a = 36;
  int b = 6;
  printf("The answer to life, the universe,\
  and everything: %d\n", a+b);
  return 0;
}

$ gcc program.c -o program
$ ./program
The answer to life, the universe, and everything is 42
$

program.py

a = 36;
b = 6;
print("The answer to life, the universe,\
 and everything: {}".format(a+b))

$ python program.py
The answer to life, the universe, and everything: 42
$

Python Shell

$ python

Python Shell

$ python
>>>

Python Shell

$ python
>>> a = 36

Python Shell

$ python
>>> a = 36
>>>

Python Shell

$ python
>>> a = 36
>>> b = 6

Python Shell

$ python
>>> a = 36
>>> b = 6
>>>

Python Shell

$ python
>>> a = 36
>>> b = 6
>>> print("The answer to life, the universe, and everything: {}".format(a+b))

Python Shell

$ python
>>> a = 36
>>> b = 6
>>> print("The answer to life, the universe, and everything: {}".format(a+b))
The answer to life, the universe, and everything: 42
>>>

Python Shell

$ python
>>> a = 36
>>> b = 6
>>> print("The answer to life, the universe, and everything: {}".format(a+b))
The answer to life, the universe, and everything: 42
>>> a+b

Python Shell

$ python
>>> a = 36
>>> b = 6
>>> print("The answer to life, the universe, and everything: {}".format(a+b))
The answer to life, the universe, and everything: 42
>>> a+b
42
>>>

Shells

MacOS

zsh
bash

Shells

Windows

Command Prompt
Windows PowerShell

Shells

Linux

bash
dash
zsh
ksh

Unix systems can do

cat /etc/shells

POSIX IEEE 1003.1 (2017)

Most POSIX Compliant Shell

Yash

Control Logic

If Statement:

if COMMAND; then
  # Insert more commands here
fi

Control Logic

If Statement:

if COMMAND; then
  # Insert more commands here
fi

if COMMAND; then
  # Insert more commands here
else
  # Do otherwise
fi

Control Logic

If Statement:

if COMMAND; then
  # Insert more commands here
fi

if COMMAND; then
  # Insert more commands here
else
  # Do otherwise
fi

if [[ 2 -eq 2 ]]; then
  # Insert more commands here
else
  # Do otherwise
fi

Control Logic

And and Or:

if [[ "$STRING1" == "$STRING2" && "$STRING1" == "$STRING3" ]]; then
  printf "%s is equal to both strings" "$STRING1"
fi

Control Logic

And and Or:

if [[ "$STRING1" == "$STRING2" || "$STRING1" == "$STRING3" ]]; then
  printf "%s is equal to either string" "$STRING1"
fi

### Piping '|'
              ```bash
              $ ps aux
              ```
              ```bash
              lfreche+ 4129510  0.0  0.0  19528  3924 ?        Ss   Mar11   0:00 -bash
              lfreche+ 4129970  0.0  0.0  13412  3928 ?        S    Mar11   0:00 bash
              key021   4130009  0.0  0.0 1437128 57112 ?       Sl   Feb26   0:39 python3 /home/key021/group/bin/aflowpy -unique ./Binary_Nitrides_POSCARS ./Binary_Nitrides_POSCARS_unique
              lfreche+ 4130115  0.0  0.1 995328 145764 ?       Sl   Mar11   1:08 /home/lfrechette/.vscode-server/bin/1e790d77f81672c49be070e04474901747115651/node --dns-result-order=ipv4first /home/lfrechette/.vscode-server/bin/1e790d77f81672c49b
              lfreche+ 4130152  0.0  0.0 795692 24876 ?        Sl   Mar11   0:02 /home/lfrechette/.vscode-server/bin/1e790d77f81672c49be070e04474901747115651/node /home/lfrechette/.vscode-server/bin/1e790d77f81672c49be070e04474901747115651/out/bo
              aikeliu  4130207  0.2  0.0 5818856 15612 ?       Sl   Feb14 103:12 /home/aikeliu/hyperion-lib/dynamical-sdp/.stack-work/install/x86_64-linux-tinfo6-libc6-pre232/d78446221ff916ba982e2670820af44b68f1282e7cd9e8bfe0ffdc98ca25fc96/9.4.5/
              key021   4136260  0.0  0.0 1840604 75284 ?       Sl   Feb26   0:48 python3 /home/key021/group/bin/aflowpy -unique ./Binary_Nitrides_POSCARS ./Binary_Nitrides_POSCARS_unique
              rus043   4140541  0.0  0.0 202644  3176 ?        Sl   Feb27   0:00 /usr/libexec/dconf-service
              key021   4145103  0.0  0.0 1840604 65448 ?       Sl   Feb26   0:36 python3 /home/key021/group/bin/aflowpy -unique ./Binary_Nitrides_POSCARS ./Binary_Nitrides_POSCARS_unique
              key021   4145681  0.0  0.0 1775068 60696 ?       Sl   Feb26   0:43 python3 /home/key021/group/bin/aflowpy -unique ./Binary_Nitrides_POSCARS ./Binary_Nitrides_POSCARS_unique
              root     4149236  0.0  0.0 136692  9664 ?        Ss   09:22   0:00 sshd: klupo [priv]
              yryang   4149756  0.0  0.0 111100   884 pts/357  S+   09:22   0:00 srun --partition=gpu-shared --pty --account=csb176 --ntasks-per-node=10 --nodes=1 -t 01:00:00 --wait=0 --mem=128G --gpus=1 --export=ALL /bin/bash
              key021   4150972  0.0  0.0 1775068 77812 ?       Sl   Feb26   0:47 python3 /home/key021/group/bin/aflowpy -unique ./Binary_Nitrides_POSCARS ./Binary_Nitrides_POSCARS_unique
              key021   4155614  0.0  0.0 1234364 73640 ?       Sl   Feb26   0:51 python3 /home/key021/group/bin/aflowpy -unique ./Binary_Nitrides_POSCARS ./Binary_Nitrides_POSCARS_unique
              root     4156782  0.0  0.0 141040  4456 ?        Ss   Feb16   0:00 sshd: jcardenas [priv]
              jcarden+ 4156818  0.0  0.0 7994352 9156 ?        Sl   Feb16   7:51 sshd: jcardenas@notty
              jcarden+ 4156854  0.0  0.0  33716  3336 ?        Ss   Feb16   1:11 /usr/libexec/openssh/sftp-server
              root     4162585  0.0  0.0 136692  4824 ?        Ss   Feb23   0:00 sshd: ghafar [priv]
              ghafar   4162616  0.2  0.0 103184 19772 ?        Ss   Feb23  75:19 /usr/lib/systemd/systemd --user
              fgutier+ 2810605  0.0  0.0 136692  5468 ?        S    09:46   0:00 sshd: fgutierrez@pts/324
              fgutier+ 2826236  0.0  0.0  12148  1176 pts/324  S+   09:46   0:00 grep --color=auto fgutierrez
              ghafar   4162618  0.0  0.0 316460  2916 ?        S    Feb23   0:00 (sd-pam)
              ghafar   4162630  0.0  0.0 136692  3240 ?        S    Feb23   0:00 sshd: ghafar@notty
              ghafar   4162631  0.0  0.0  29412  2364 ?        Ss   Feb23   0:00 /usr/libexec/openssh/sftp-server
              klupo    4162807  0.0  0.0 136824  6064 ?        S    09:23   0:00 sshd: klupo@pts/94
              klupo    4163279  0.0  0.0  20896  5636 pts/94   Ss   09:23   0:00 -bash
              ghafar   4163633  0.0  0.0  29412  2364 ?        Ss   Feb23   0:00 /usr/libexec/openssh/sftp-server
              key021   4163964  0.0  0.0 1775068 64288 ?       Sl   Feb26   0:40 python3 /home/key021/group/bin/aflowpy -unique ./Binary_Nitrides_POSCARS ./Binary_Nitrides_POSCARS_unique
              aikeliu  4164435  0.2  0.0 5751268 16456 ?       Sl   Feb14 103:59 /home/aikeliu/hyperion-lib/dynamical-sdp/.stack-work/install/x86_64-linux-tinfo6-libc6-pre232/d78446221ff916ba982e2670820af44b68f1282e7cd9e8bfe0ffdc98ca25fc96/9.4.5/
              root     4165667  0.0  0.0 136692  5136 ?        Ss   Feb21   0:00 sshd: sz550 [priv]
              sz550    4165744  0.0  0.0 136692  3580 ?        S    Feb21   0:00 sshd: sz550@notty
              ```

Writing a Bash Script

$ notepad++ script.sh

#!/bin/bash

Writing a Bash Script

$ notepad++ script.sh

#!/usr/bin/env bash

Writing a Bash Script

$ notepad++ script.sh

#!/usr/bin/env bash
# Comment1: Today is a good day

echo "Hi there"
VAR1="21"
...

Writing a Bash Script

$ notepad++ script.sh
$ bash script.sh
Hi there
...
$

#!/usr/bin/env bash
# Comment1: Today is a good day

echo "Hi there"
VAR1="21"
...

Writing a Bash Script

$ notepad++ script.sh
$ chmod +x script.sh
$ ./script.sh
Hi there
...
$

#!/usr/bin/env bash
# Comment1: Today is a good day

echo "Hi there"
VAR1="21"
...

Scripts

#!/bin/bash --


control='ok'

adddir() {
  [[ -d $1 ]] || mkdir $1;
}

add_class() {

  # make the directories for seperate classes
  adddir Material/$1 && ln -sL ../../Material/$1 Classes/$1/Material
  adddir Homework/$1 && ln -sL ../../Homework/$1 Classes/$1/Homework
  adddir Textbooks/$1 && ln -sL ../../Textbooks/$1 Classes/$1/Textbooks


  if [[ -f Material/$1/syllabus.pdf ]] then
    # a little scuffed but looks for regex assuming only 1 will meet
    ln -sL ../../Material/*yllabus*.pdf Material/$1/syllabus.pdf
  fi
}

add_textbook() {
  book=$(basename $1)
  class=''

  # Check that Classes is not empty
  if [[ ! -d Classes/ ]] || [[ ! $(ls Classes/) ]]; then
    printf 'You are missing some Classes to setup\n'
    printf 'Please add that first\n'
    exit
  fi

  # choose class
  printf 'Choose a class: \n'
  printf '     %s\n' $(ls -d Classes/*/ | cut -d \/ -f2) #, funny this prints like a loop for every element of these
  printf ' : '
  read -r class
  if [[ ! -d Classes/$class ]]; then
    printf 'That is not a class!\n'
    printf 'Exiting...\n'
    exit 1;
  fi


  mv -i $1 $HOME/Documents/School/Textbooks/
  ln -s $HOME/Documents/School/Textbooks/${book} $(pwd -P)/Textbooks/$class/
}

usage() {
  printf '%s:

  Usage: %s [-c] [-h] [-t]

  Options:
    -c    Add a class
    -h    Show help (this message)
    -t    Add a textbook. Takes in a textbook, moves to ~/Documents/School/Textbooks
          and then creates a soft link in the classes Textbook directory.
          %s -t /path/to/textbook 

' $0 $0 $0
  exit 1
}


# check you are in the right place
if [[ $(basename $(dirname $(pwd))) != 'School' ]]; then
  printf 'You are not in the a Sem/Quarter directory!'
  usage
fi

while getopts ":hct:" arg; do
  case "${arg}" in
    c) 
      control=''
      ;;
    t) 
      # give a way to read the next argument
      if [[ -z ${OPTARG} ]]; then
        usage
      fi 
      add_textbook ${OPTARG}
      exit 0;
      ;;
    h) 
      usage
      ;;
    *)
      usage
      ;;
  esac
done

# Need to choose classes
if [[ ! -d Classes/ ]] || [[ $control != 'ok' ]]; then
  # add the directory if not existent
  adddir Classes

  # add classes with a prompt
  printf 'Set a class %s, bro: ' $(whoami)
  read -r control
  adddir Classes/$control

  while [[ $control != 'ok' ]]; do
    printf 'Add another class or say "ok" to finish [ok]: ' 
    # if read then overwrite control
    read -r control
    [[ $control != 'ok' ]] && adddir Classes/$control  
  done

else
  printf 'Classes/ already exists. Perhaps you meant to use [-c] flag.\n'
  usage
fi


# Add the dirs if they are not there yet
adddir Material
adddir Homework
adddir Textbooks

for i in $(ls -d Classes/*/ | cut -d \/ -f2); do

  # check if class hasn't been edited
  if [[ ! $(ls Classes/$i/) ]]; then
    printf 'Adding class %s\n' $i
    add_class $i
  fi

done

printf 'done'

https://devhints.io/bash

- functions, associative arrays, string substition, etc. - Look at a good cheatsheet

Simple Info

#!/bin/bash
KERNEL="$(uname -r)"
MEM="$(free -h | awk '/^Mem:/ {print $2}')"

echo "Hostname: " $HOSTNAME
echo "Username: " $USER
echo "Kernel Version: " $KERNEL
echo "Total Memory: " $MEM

Simple Number Comparison

#!/bin/bash
age=18
if [ "$age" -ge 18 ]; then
    echo "You are an adult."
else
    echo "You are a minor."
fi

Create a zip

#!/bin/bash
backup_dir="/path/to/backup"
source_dir="/path/to/source"
zip -r "$backup_dir/$backup_file" "$source_dir"

Simple Find and Replace

#!/usr/bin/env bash
# Demo 1

BADNAME="myVar"
GOODNAME="my_var"
for i in $(find . -type f); do
  if [[ "$i" != "./script.sh" ]]; then
    if grep "$BADNAME"  $i > /dev/null; then
      echo "Found in the phrase in ${i}";
      sed -i "s/$BADNAME/$GOODNAME/g" $i > /dev/null;
    fi
  fi
done

Http Server

$ ls 
Downloads/  Documents/  Videos/  ...

Http Server

$ cd Downloads/

Http Server

$ ls 
Bash-Workshop.jpg  Me-Playing-The-Saxophone.mp3  some-papers/

Http Server

$ ip a | grep inet
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host proto kernel_lo 
inet 100.80.134.178/20 brd 100.80.143.255 scope global dynamic noprefixroute wlp2s0
inet6 fe80::3977:6f8:fbce:e909/64 scope link noprefixroute

$ ifconfig a | grep inet

Http Server

$ python -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
127.0.0.1 - - [12/Mar/2024 10:28:39] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [12/Mar/2024 10:28:41] "GET /Me-Playing-The-Saxophone.mp3 HTTP/1.1" 200 -

WGet

$ cd Downloads
$ ls
Bash-Workshop.jpg  Me-Playing-The-Saxophone.mp3  some-papers
$

WGet

$ cd Downloads
$ ls
Bash-Workshop.jpg  Me-Playing-The-Saxophone.mp3  some-papers
$ wget https://cseweb.ucsd.edu/classes/fa23/cse120-a/lectures/mem.pdf

WGet

$ cd Downloads
$ ls
Bash-Workshop.jpg  Me-Playing-The-Saxophone.mp3  some-papers
$ wget https://cseweb.ucsd.edu/classes/fa23/cse120-a/lectures/mem.pdf
--2024-03-12 11:27:28--  https://cseweb.ucsd.edu/classes/fa23/cse120-a/lectures/mem.pdf
Resolving cseweb.ucsd.edu (cseweb.ucsd.edu)... 132.239.8.30
Connecting to cseweb.ucsd.edu (cseweb.ucsd.edu)|132.239.8.30|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2497853 (2.4M) [application/pdf]
Saving to: ‘mem.pdf’

mem.pdf                                                   100%[=====================================================================================================================================>]   2.38M  6.58MB/s    in 0.4s    

2024-03-12 11:27:30 (6.58 MB/s) - ‘mem.pdf’ saved [2497853/2497853]
$ clear

WGet

$ ls 
Bash-Workshop.jpg  Me-Playing-The-Saxophone.mp3  mem.pdf  some-papers
$

Get CSE Class Slides

#!/usr/bin/bash -x 
location="cseweb.ucsd.edu/classes/fa23/cse120-a/lectures/"

wget -r -np -R "index.html*" https://$location
class_list=$(ls -rt ./$location)


count=1
for i in $class_list; do

  mv -n ./$location/$i ./${count}_${i}
  count=$((count + 1))

done


rm -rf cseweb.ucsd.edu

wget and http server

https://ieeeucsd.org/
https://supercomputing-club.sdsc.edu/
https://ieeeucsd.org/bash

$ mkdir temp/
$ cd temp
$ wget -k -r -p https://supercomputing-club.sdsc.edu/

wget and http server

https://ieeeucsd.org/
https://supercomputing-club.sdsc.edu/
https://ieeeucsd.org/bash

$ python -m http.server

wget and http server

https://ieeeucsd.org/
https://supercomputing-club.sdsc.edu/
https://ieeeucsd.org/bash

$ python -m http.server

Try visiting localhost:8000 if you completed these steps

Get every Canvas file from your course!

Here's a script to download all files for a given Canvas course. You will need to get an API Token from Canvas for UCSD and the course ID. To get the course ID, take the ID from the course's URL. Go to your browser, open your course page, you will get a URL like this: https://canvas.ucsd.edu/courses/123456 The ID is the number at the end. Place it at the end of the variable COURSE_ID below with quotes. For the API token, click on your profile picture, then click "Settings". After that, create a "New Access Token" under the "Approved Integrations" section. Add a name and expiration date. You will get a token on the dialog after that looks something like this: "12345~..." Just put it down in the variable API_TOKEN below. This uses "jq" to filter the JSON received into a readable format for our script. '.[]' takes every element from the array and lists it line-by-line, while the second part creates a string containing the fields from the request, comma-delimited. Try running the command above on its own and experiment with different "jq" queries. There are cheat sheets online you can find, it's cool! For each line from the above query, we'll get the filename and the URL. Then, for all of them, we will download the file in the directory "files/" and just name it as-is on the page. The line variable looks something like this: "NAME OF FILE,URL TO DOWNLOAD FILE" Awk is a scripting language for text manipulation. You can use to quickly parse comma-delimited values, which is what I do below. Download the file and move it to the files directory. "-s" silences the progress bar that shows up with cURL, you can remove the "-s" to see it.

#!/usr/bin/env bash

COURSE_ID="123456"
API_TOKEN="12345~..."

mkdir -p files/

FILES=$(
	curl \
	-X GET \
	-H "Authorization: Bearer $API_TOKEN" \
		https://canvas.ucsd.edu/api/v1/courses/$COURSE_ID/files | \
		jq -r '.[] | "\(.filename),\(.url)"'
)

echo "$FILES" | while IFS= read -r line; do
	FILENAME=$(echo $line | awk -F',' '{print $1}')
	URL=$(echo $line | awk -F',' '{print $2}')
	echo "Downloading $FILENAME..."
	curl -s "$URL" -o "$FILENAME"
	mv "$FILENAME" "files/$FILENAME"
done

echo "Done! All your files are in the 'files/' directory."

Bash in Supercomputing

Most computers run some Unix derivative

Student Cluster Competition 2022 Tech Stack

SCC Applications Coming Soon!

Our Site

https://supercomputing-club.sdsc.edu/

Thank you

https://ieeeucsd.org/
https://supercomputing-club.sdsc.edu/
https://www.sdsc.edu/